Franck Ohandza put Cameroon ahead in the 79th minute in regulation time, but Carlos Orrantia equalized for Mexico in the 81st.
Cameroon missed all three of its penalties in the shootout while Mexico converted its three.
Mexico goalkeeper Jose Rodriguez stopped Franck Ohandza's shot, then Idriss Nguessi struck the crossbar and Christ Mbondi hit the post to end the contest.
Erick Torres, Ulises Davila and Marvin Pinon all found the net with confident shots for Mexico, whose quarterfinal opponent will be host nation Colombia, which converted a stoppage-time penalty to beat Costa Rica 3-2.
''We are happy, but we are not completely satisfied,'' Mexico coach Juan Carlos Chavez said. ''We know there is still a lot to improve. It was a difficult match, with few opportunities, but here we are, it's time to start thinking about Colombia. It will be very special to play against the host.''
Ohandza put Cameroon in front by firing a hard shot into the top of the net from just inside the penalty area after receiving a perfect pass from Mbondi.
But Mexico answered quickly and midfielder Orrantia equalized from close-range after a long pass into the area. He dived feet first near the far post to knock the left-flank cross into the net.
Cameroon coach Martin Ndtoungou Mpille said the defense's lack of attention for a few moments led to the equalizer, and then the pressure got to the players in the shootout.
''There is no other explanation (for the missed penalties),'' he said.
The teams created very few scoring chances during regulation at the Hernan Ramirez Villegas stadium in Pereira, but Mexico pressured near the end and threatened the most, especially with a few long-range shots. Cameroon's only chances came on crosses from set pieces.
In extra time, Mexico almost scored in the 115th, but Nestor Araujo's shot was saved by Cameroon goalkeeper Jean Efala. Torres also had his attempt saved by Efala a few minutes earlier.
Mexico advanced to the knockout stage in second place in Group F, behind Argentina and ahead of England and North Korea. Cameroon was second to Portugal in Group B, which also had New Zealand and Uruguay.