The Bayern Munich striker, who had also given his side a 1-0 lead in the first half, notched his second goal of the Group A game when he steered home a cross by captain Philipp Lahm. It was Gomez's 18th goal in his 45th appearance for the national side.
Gomez put Germany ahead just before the break, scoring from close range after Austria's defense failed to clear the ball after a corner taken by Toni Kroos.
Austria equalized in the 50th when Germany defender Arne Friedrich tried to clear a cross by David Alaba before it reached Martin Harnik, but could only deflect the ball past his wrongfooted goalkeeper Manuel Neuer.
''Football is a crazy game, you must always keep going,'' Gomez said. ''I am going to try to play on like this, so the coach will have a lot of options to choose from.''
Germany coach Joachim Loew called it ''a lucky victory'' for his team.
''We did not show our usual class. Otherwise, Austria would have had more troubles,'' Loew said. ''Austria fought hard for its last chance. In the end, the three points are the only thing that matters to us.''
''I have to compliment my team,'' Loew said. ''Many players who were at the World Cup are really near the limits of their strengths now after a heavy season.''
The Euro 2008 finalist was without its main goalscorer Miroslav Klose, who bruised his ribs in Sunday's friendly against Uruguay, and never looked comfortable in front of 47,500 fans at the sold-out Ernst Happel Stadium.
Gomez missed two chances to give Germany an early lead after Austria got the the first scoring opportunity of the match in the fourth minute, when Harnik sent a header just wide from a Christian Fuchs free kick.
Germany quickly responded when Gomez saw his lob from short range saved by Christian Gratzei, and Austria's goalkeeper collected a header by the Bayern Munich striker just minutes later.
Lukas Podolski came closest in a lively opening phase, hitting the bar with a thundering 20-meter effort. It was the winger's only shot on goal in his 88th game for Germany.
Austria tried to ease the pressure on its defense by pressuring Germany in midfield, a strategy that worked for the rest of the first half as Germany failed to gain control the match.
Real Madrid's Sami Khedira, playing his first game since tearing a thigh muscle on April 20, failed to stamp his authority in Germany's midfield and was replaced by Holger Badstuber halfway through the second half.
Austria coach Dietmar Constantini left striker Marc Janko out of the starting lineup in favor over the faster Erwin Hoffer. His had more possession after the break but was let down by its passing in the final third. Hoffer and Harnik came close but saw shots go wide, while the unmarked Stefan Kulovits hit the near post with a volley from a difficult angle.
It was Germany's 80th match and 56th victory in European Championship history. The last time Germany lost an European Championship away game was in 1998, when Turkey won 1-0 in Bursa.
''We played okay but conceded goals at the worst possible moments,'' Constantini said. ''That's bitter, but it's no coincidence. That's the class of Germany.''
Austria has never qualified for a European Championship but took part in the 2008 event as a co-host with Switzerland.
Austria: Christian Gratzei, Ekrem Dag (Zlatko Junuzovic, 66), Emanuel Pogatetz, Florian Klein, Christian Fuchs, Julian Baumgartlinger, Paul Scharner, Stefan Kulovits, David Alaba, Martin Harnik (Daniel Royer, 81), Erwin Hoffer (Marc Janko, 88).
Germany: Manuel Neuer, Philipp Lahm, Arne Friedrich, Mats Hummels, Marcel Schmelzer, Sami Khedira (Holger Badstuber, 69), Mesut Oezil, Thomas Mueller, Toni Kroos (Dennis Aogo, 90), Mario Gomez, Lukas Podolski (Andre Schuerrle, 67).