Luis Muriel gave Colombia a 1-0 lead in the 56th minute, but Costa Rica came back with goals in the 63rd by John Ruiz and in the 65th by Mynor Escoe to take the lead and threaten the hosts with a surprise elimination.
However Colombia's Pedro Franco equalized with a 79th-minute header before Rodriguez's dramatic late winner.
Colombia has won all four of its matches in the tournament and looks like a genuine contender, thriving on wild support at home and the edge of playing at an altitude of 2,600 meters (8,600 feet) in Bogota.
A tight match was settled just seconds into stoppage time when English referee Mark Clattenburg awarded a penalty after Ariel Soto pulled down Colombia's second-half substitute Duvan Zapata. Rodriguez stepped up and rammed home the penalty for his third goal of the tournament.
The victory came just hours after the head coach of the senior team, Hernan Dario Gomez, resigned after reports he struck a woman in a bar. The incident took place Saturday in Bogota.
''It was a tough day for us, very tough,'' said Colombia U20 coach Eduardo Lara. ''The head of the coaching staff (Gomez) made a decision and this hurt all of the group. This team is a famliy. Maybe because of this, our first half was one of our worst.''
Costa Rica coach Ronald Gonzalez said his team grew quickly during the tournament.
''Finally, we played without fear and without any complexes and showed what we can do,'' Gonzalez said.
Muriel, of Italian club Udinese, gave Colombia the lead when he chased the ball into the area as Costa Rica keeper Aaron Cruz came 10 meters (yards) off his line to meet him. Muriel won the chase and chipped the ball home for his fourth goal of the tournament.
Costa Rica equalized in the 63rd when Ruiz ran on to a long ball and drilled a 15-meter (yard) shot under Colombia keeper Cristian Bonilla. Two minutes later, Mynor Escoe barged through several Colombia defenders and slotted home a shot from 18 meter (yards).
Costa Rica seemed headed to victory before Pedro Franco headed home in a cross from Rodriguez, who maneuvered on the wing before finding an opening to put the ball into the area.
The first half was a standoff with Colombia having more possession but Costa Rica looking more menacing, particularly on the counterattack.
Both teams hit the crossbar once.
Costa Rica's Joel Campbell banged it off the woodwork in the 23rd minute on a 30-meter (yard) free kick. Campbell also threatened on another free kick, as did teammate Juan Golobio.
Colombia's Jose Valencia hit the crossbar in the 35th with a long shot that sailed over the fingertips of Costa Rica keeper Cruz. Michael Ortega also came close with Rodriguez and Muriel being held in check.