Artemi Panarin had a goal and an assist for the Blue Jackets (28-19-3), who have lost four straight. Sergei Bobrovsky made 22 saves.
"It [stinks]," Columbus captain Nick Foligno said. "It [stinks] to lose that way in the last minute. Guys played really hard, played to our identity tonight, and [it] just doesn't sit well with me getting scored on late like that."
Mark Scheifele gave the Jets a 1-0 lead at 4:34 of the first period, scoring with a one-timer off a pass from Blake Wheeler.
Panarin tied it 1-1, beating Hellebuyck with a wrist shot at 8:52 for his 20th goal.
Josh Anderson gave the Blue Jackets a 2-1 lead at 13:30, scoring from the slot off a pass from Panarin.
"It's a hard one for our guys, for sure," Columbus coach John Tortorella said. "They played hard, did a lot better job as far as what we asked for them in the battles. It's a tough one not to come up with anything."
Mason Appleton tied it 2-2, scoring on a backhand that trickled through Bobrovsky's pads at 13:21 of the second period.
The tie lasted 57 seconds. Oliver Bjorkstrand roofed a wrist shot to give the Blue Jackets a 3-2 lead at 14:18.
Columbus forward Cam Atkinson was hit in the face with a puck in warmups and did not play. Mark Letestu took his spot, playing 9:54 in his season debut. Tortorella did not have an update on Atkinson, who leads the Blue Jackets with 28 goals and is second to Panarin with 50 points.
"I thought our guys rallied, played a really good hockey game," Tortorella said of the response to Atkinson's injury.