It was Hellberg's fifth career game and second this season; he made four saves in relief of Henrik Lundqvist in a 7-6 loss against the Dallas Stars on Jan 17.
WATCH: All Penguins vs. Rangers highlights
"I am a little overwhelmed right now," Hellberg said. "This is something I've been fighting for for five years, so to be able to get this opportunity in the last game of the season is very big for me."
Jimmy Vesey scored the winning goal at 7:23 of the third period. It was Vesey's 16th goal of the season and second since Feb. 19.
New York also got goals from Derek Stepan and Ryan McDonagh, who missed the previous four games because of an undisclosed injury.
In addition, the Rangers had Rick Nash, Mats Zuccarello and Nick Holden in the lineup. They each missed the previous two games. Jesper Fast missed his third straight game.
Kevin Hayes, Chris Kreider and Brendan Smith were healthy scratches. Lundqvist was the backup to Hellberg, who was recalled Sunday and started because Antti Raanta was held out with a knee injury after getting hit by a shot in practice Friday.
Rangers coach Alagn Vigneault said Raanta is day to day but should be available to backup Lundqvist in Game 1.
"We're going into the playoffs as healthy as a team can be at this time of the year," Vigneault said.
The Penguins (50-21-11), who open the playoffs against the Columbus Blue Jackets at PPG Paints Arena, lost their second straight game, but rested nine healthy regulars, including captain Sidney Crosby.