Nadal raced away with the first set of Wednesday night's match and went up 3-1 in the second set but Dodig then responded with some strong serving and big hitting from the baseline to force a tiebreaker, which he won with some gutsy play.
The Spaniard seemed to have the match under control again when he went up 3-0 in the third set and later served for the match at 5-3, but Dodig refused to go away and took advantage of Nadal's uncharacteristically high unforced error count and his own strong serving to stay alive and force another tiebreaker.
Finally, at the three-hour, eight-minute mark of the contest, the 41st-ranked Dodig fired a blistering crosscourt backhand winner to give himself the improbable victory.
Nadal, who came into the match a tad rusty after not playing a tournament match since his loss in the Wimbledon final in early July, was noticeably erratic with the forehand, ordinarily his best shot.
But his form would have been more than good enough if not for the powerful serving of Dodig - who racked up a total of 19 aces for the contest - and surprising shot-making from the baseline.
"I had to close the match with the 5-3 (lead) in the third set. I had to close that match probably in the second when I was playing very well, 6-1, 3-1, playing very high level of tennis," Nadal said in the post-match interview.
"After that, the tennis is like this. You are winning without many problems, you have one mistake, and after the opponent starts to serve very well, playing very aggressive."
The defeat marked the world No. 2's earliest loss at an ATP World Tour Masters 1000 event since May 2008 (Rome) and lends more urgency to next week's tournament in Cincinnati, the Spaniard's last chance to get in match practice against top-flight competition before the U.S. Open, which gets underway Aug. 29.
Nadal is the defending champion at the year's last Grand Slam event.
In other second-round action Wednesday involving Spanish players, Nicolas Almagro defeated Ukrainian Sergiy Stakhovsky 6-2, 7-6 (7-5), Feliciano Lopez lost to American Mardy Fish 6-2, 6-3 and Fernando Verdasco was ousted by Serbia's Janko Tipsarevic 6-4, 6-7 (5-7), 6-3.
Among the Latin American contingent, Argentina's Juan Martin del Potro was upset in second-round action, falling to another Croat - Marin Cilic - 6-3, 6-4, and Brazil's Thomaz Bellucci lost to Frenchman Richard Gasquet 6-1, 6-4.
The nine ATP World Tour Masters 1000 tournaments rank just behind the four Grand Slam tournaments and the season-ending Barclays ATP World Tour Finals in terms of ranking points and prestige.