Kuznetsov, who played for the first time since Jan. 19, put the Capitals ahead 1-0 after his shot deflected off Penguins defenseman John Marino on a power play at 5:18 of the first period.
The Penguins answered with two goals in 32 seconds.
Rust tied it 1-1 at 10:29. After taking a backhand pass from Crosby, Mike Matheson moved it to Rust to tip it inside the right post.
Brandon Tanev gave Pittsburgh a 2-1 lead by scoring on a rebound off a shot from Cody Ceci at 11:01 before Vrana tied it 2-2 on a backhand at 3:26 of the second period.
"It's really frustrating. It's a short season and every point matters here," Vrana said. "It feels like we can't let these games slip away here. Good thing we have a practice tomorrow, get to work on some things and clear out some things. Make sure we're going to be more prepared and ready for the next game here."
Rust put the Penguins ahead 3-2 with a wrist shot at 6:44, ending a stretch of 21 straight unsuccessful power plays.
Guentzel made it 4-2, Pittsburgh's first two-goal lead of the season, when he had a shot blocked before following with a wrist shot from the slot at 10:44.
Nicklas Backstrom scored his 250th NHL goal by stealing the puck from Evgeni Malkin for a backhand, cutting it to 4-3 at 17:49.
"I think overall we're trying to play hard and trying to do the right things, but it seems like we're doing the same mistakes over and over again," Backstrom said. "These easy goals against us. It's just an area we have to get better on. Defensive zone, that's where the game is decided."