Chris Kreider gave New York a 1-0 lead at 1:47 of the first period, scoring off a 2-on-1 with Mika Zibanejad.
Ryan Dzingel scored 2:25 into the third period to tie it 1-1 with a one-timer off a pass from Foligno on a 3-on-1.
Panarin gave Columbus a 2-1 lead with 5:33 left in the third period, splitting two Rangers players before beating Georgiev top shelf.
"No disrespect, [Georgiev] is a great goalie and he's played well for them, but you get that many Grade As or opportunities, you're going to put one or two past him," Foligno said. "We knew it was just a matter of time, so it was nice to get those two in the third."
Bobrovsky saved the only shot he faced in overtime and the three attempts he faced in the shootout, clinching the win and a playoff berth with a save on Rangers defenseman Tony DeAngelo in the third round.
"I'd rather take a 2-1 win," Bobrovsky said. "That's safe plus medicine for the heart, but it is what it is. We get it done in the shootout and we are happy."