Ovechkin passed Brett Hull for fourth in NHL history when he scored his 12th of the season at 10:19 of the first period to make it 1-1.
Ovechkin trails Wayne Gretzky (894), Gordie Howe (801) and Jaromir Jagr (766) in NHL goals.
"It means a lot," Ovechkin said. "When you start playing hockey, when you come to the NHL, you never thought you'd be in that position with those names."
Kuraly scored at 2:33 of the first period to give Columbus a 1-0 lead. The forward from the Columbus suburb of Dublin, Ohio, signed a four-year contract on July 28 after five seasons with the Boston Bruins.
Hathaway scored at 24 seconds and 7:23 of the second period to make it 3-1 before Columbus scored twice in 22 seconds.
"I thought we came on, played really good from the start and got the lead," Capitals coach Peter Laviolette said. "And then we just made a couple of mistakes that we typically haven't made, and all of a sudden they gained some momentum and then they were pressing, and they hit the post a couple of times and the second period got away from us."