Casey DeSmith made 10 saves on 12 shots in the first period before being replaced to start the second by Tristan Jarry, who made 15 saves.
Joonas Korpisalo made 26 saves for the Blue Jackets (18-19-1), who defeated the Philadelphia Flyers 2-1 on Thursday.
"No matter how tough it is, you've just got to find a way," Columbus forward Sean Kuraly said. "Five things, almost, happened in that third period, big events, and every single one was a dagger for us."
Dumoulin scored from high in the left circle off a pass from Crosby 32 seconds into the first period to give Pittsburgh a 1-0 lead. Kris Letang had an assist to extend his NHL career-long point streak to 10 games, the longest active streak in the NHL.
Gustav Nyquist made it 1-1 at 10:40 on his NHL-leading fourth shorthanded goal. He broke down the left wing and whiffed on his initial shot, which fooled DeSmith. As Nyquist neared goal line, he shot under the goalie's right arm to the far corner.
Boone Jenner gave Columbus a 2-1 lead at 18:46. He took a carom off the end boards and shot between DeSmith's pads.
Crosby tied it 2-2 with 29 seconds left, redirecting a shot by Bryan Rust.
Pittsburgh has won 15 of 17 since Dec. 4.
"We've shown an ability to play the game a certain way that that gives us the best chance to win, and it starts with an attitude," Penguins coach Mike Sullivan said. "And then it's a commitment to play the game the right way. And then it's attention to detail and knowing what your job is and trusting that your linemate or your teammate is going to do theirs."
Matheson scored with a shot from the blue line that Korpisalo got a piece of, but the puck trickled behind him before Nyquist was able to pull it back across the goal line.