Teddy Blueger scored shorthanded to give the Penguins a 2-1 lead at 2:40 of the second period. He took a pass from Brandon Tanev and skated around K'Andre Miller to the net for a backhand five-hole.
Kapanen made it 3-1 with his fourth goal in five games at 19:40. Evgeni Malkin passed between Kevin Rooney's legs to Kapanen in the slot for a wrist shot.
Strome cut it to 3-2 with a power-play goal after his shot deflected off Penguins defenseman John Marino's skate at 3:18 of the third period.
"You obviously want to put your best effort out there," Strome said. "I thought the second half of the game, we were much better. Goals at the end of the first and second period kind of killed us a little bit. I thought a much better effort than the night before, but unfortunately, when you're chasing this team and you lose both, it's frustrating. There's not much we can take away from this one in terms of feeling good."
Pavel Buchnevich had two chances to tie it late in the third. After having a wrist shot turned away at the right post at 17:29, Buchnevich raced past Brian Dumoulin to take a pass from Mika Zibanejad in the slot for another wrist shot Jarry denied with a pad save at 17:47.
The Rangers outshot the Penguins 15-1 in the third period.
"[Jarry] was solid in the third," Pittsburgh coach Mike Sullivan said. "I don't think we had our best period in the third, and he made some big saves for us, particularly in the last two and a half minutes of the game ... He was big when we needed him to be. That's part of winning in this league."
Crosby scored an empty-net goal from the red line for the 4-2 final with 33 seconds remaining.
New York has lost five of six games against Pittsburgh this season and has led in each of those losses.
"Four or five times, they've come from behind to beat us in close ones," Rangers forward Chris Kreider said. "They cost us enough the other night. So at the end of the day, we have to find a way to get those two points."
NOTES: Kinkaid, born in Farmingville, New York, is the first goalie born in the New York metropolitan area to start a game for the Rangers. He is the first United States-born goalie to start for them since March 31, 2004 (Mike Dunham against the Buffalo Sabres). Kinkaid made his Rangers debut Sunday, when he made 16 saves on 18 shots in relief of Alexandar Georgiev. ... Penguins forward Jared McCann did not play because of an upper-body injury sustained Sunday. He was placed on injured reserve Tuesday.