The Ducks challenged for offside, but the goal stood and Werenski scored on a one-timer from the slot on the ensuing power play for a 3-2 lead at 19:49.
"It can flip like that, it's a momentum game," Ducks defenseman Hampus Lindholm said. "They came at us hard there and we can't feel sorry about ourselves."
Anaheim pressured Columbus late in the third period and forced defenseman Seth Jones into a 3:26 shift, but couldn't tie the score.
"We had them hemmed in there for a while and we had a tremendous amount of pressure on them," Ducks coach Dallas Eakins said. "That was part of the game plan in the third, and it's just too bad we weren't able to get rewarded."
Werenski scored into an empty net with 1:33 remaining to make it 4-2.
With Miller still out of the net for the extra skater, Kase scored on a tip-in front to cut it to 4-3 with 44 seconds left.