Nico Hischier and Paul Cotter scored for the Devils (24-15-3), who have lost four straight overall and five in a row on the road. Markstrom made 21 saves.
"When we play hard and we play with structure and purpose, we've shown we're a good team," New Jersey coach Sheldon Keefe said. "We've also shown that when we don't, we're not a good team."
Kovalenko put the Sharks in front 1-0 at 17:40 of the first period. William Eklund passed in front for Kovalenko, who went to the net and scored with a low shot that got through Markstrom.
It was Kovalenko's first goal in 11 games with San Jose since he was traded from the Colorado Avalanche on Dec. 9.
"It's [an] amazing feeling," Kovalenko said. "I'm just happy for this moment and I'm going to keep trying to score."
Hischier tied it 1-1 with a power-play goal at 4:19 of the second period. Jack Hughes sent a pass from low in the left circle that went in off Hischier's skate at the right post.
"We're frustrated," said Hischier, the New Jersey captain. "Obviously, we're going through a rough stretch right now. We've got to stick together because when you do, you grow together and come out of the other side."