Mayer played a solid all-around match and did not face a single break point in pulling off the biggest upset of the tournament and defeating the world No. 2 in their first-ever meeting.
But the Spaniard, who earlier had complained about the difficulty of adjusting to the different types of balls used on the Asian swing, appeared out of rhythm on court in his third-round match against the 23rd-ranked Mayer and was unable to assert his dominance in the rallies.
The 10-time Grand Slam champion dropped serve in the seventh game of the second set and once again two games later to bow out of the Shanghai event in the same round as last year, when he lost to Austria's Jurgen Melzer.
"It's a really nice feeling," Mayer said. "He's such a great competitor on court. For sure it is the biggest win in my career."
Nadal, meanwhile, rued his poor play on the return of serve.
"He played really well," Nadal said. "I didn't. I felt that I was not playing bad from the baseline. But my return, when he was serving the first serve, was terrible tonight. I only won four points in whole match (on his) first serve."
"Today is a tough lose, that's the truth. (It) was not my day. I am out of the tournament. I am going to work hard for the next tournament," the world No. 2 said.
The loss was one of the few defeats Nadal has suffered this year against players ranked outside the top 10. The Spaniard has lost several finals of big tournaments - including Wimbledon and the U.S. Open - to Novak Djokovic and ceded his No. 1 ranking to the Serbian this summer.
With his win and fellow Spaniard Nicolas Almagro's loss to American Andy Roddick, the fifth-ranked Ferrer also qualified for the season-ending, eight-man Barclays ATP World Tour Finals in London, joining Nadal, Murray, Djokovic and Switzerland's Roger Federer.
Ferrer had previously qualified for the year-end championship on two other occasions, reaching the final in 2007.
The Spaniard will next take on Roddick, with the winner of that match to face either Mayer or Lopez in Saturday's semifinals.
Murray, the top-ranked player left in the tournament, advanced to the quarterfinals with a 6-4, 3-6, 6-3 win over Switzerland's Stanislas Wawrinka and will next play Australia's Matthew Ebden, who upset France's Gilles Simon 6-2, 2-6, 7-6 (10-8).
The Shanghai Rolex Masters is one of nine ATP World Tour Masters 1000 events, which rank just behind the Grand Slams and the Barclays ATP World Tour Finals in terms of ranking points and prestige.