Matt Murray made 27 saves for Pittsburgh. Craig Anderson made 39 for Ottawa.
"It's surreal. It doesn't feel like it's actually happening but it is," Anderson said. "We played our hearts out and gave everything we have and we've got nothing to be ashamed of. We laid it out there and we put it on the line. Guys were dog tired out there and battling and it just wasn't in the cards for us."
Senators forward Ryan Dzingel tied it 2-2 with 5:19 left in the third period off a rebound.
Pittsburgh took a 2-1 lead with 8:16 remaining. After Dion Phaneuf was called for interference on Phil Kessel, Justin Schultz shot past a screen by Kunitz and off the right post. Schultz returned Thursday after missing four games because of an upper-body injury
Kunitz gave the Penguins a 1-0 lead at 9:55 of the second period before Mark Stone tied it 1-1 20 seconds later.
Kunitz passed to Conor Sheary, leading to a 2-on-1 facing Erik Karlsson. Sheary passed over Karlsson's stick to Kunitz, who shot inside the left post.
Sheary was in the lineup after being a healthy scratch the past two games. In the second period, Sullivan moved Sheary from the fourth line to the first with Sidney Crosby and Kunitz.
Karlsson set up the tying goal by passing to Stone and allowing him to put a shot over Murray to make it 1-1 at 10:15 of the second. The goal was Stone's first since he scored in the Games 5 and 6 of the second round against the New York Rangers.