Mika Zibanejad and Jesper Fast each scored a shorthanded goal, and Adam Fox scored a power-play goal for the Rangers (13-9-3), who are 4-0-1 in their past five games and 8-3-2 in their past 13.
"It was a pretty good game for us," Georgiev said. "We got the lead early. We played a pretty simple game. We didn't let them have too much. We had quite a bit of penalties today but at the same time guys played great (on the) PK today."
WATCH: [All Rangers vs. Devils highlights]
Georgiev, who was making his first start in four games, earned the third shutout of his career and first since a 3-0 win (29 saves) at the Philadelphia Flyers on March 31.
"[Georgiev] was big; he stood tall," Rangers coach David Quinn said. "When you look at it, 4-0 and you kill that many penalties, your goalie needs to be good and he was good when he needed to be.
"In the first period we did a lot of good things but unfortunately some turnovers ignited some of their opportunities on the rush, but Georgie had a big part in this win, that's for sure."