Mazembe were under pressure after being held to a 1-1 draw in the first leg of their last-32 tie at home in Lubumbashi last weekend
The away-goal proved to be crucial for CAPS United who withstood the pressure from Mazembe in the second leg in Harare, earning a 0-0 draw.
Mazembe, won won the African Champions League in 2015, were left frustrated by the Zimbabwean underdogs on Sunday in their search for a goal.
As in the first leg in Lubumbashi, CAPS United's goalkeeper, Edmore Sibanda, stood firm to deny the DR Congo giants a chance to score.
It means Mazembe now drop into the African Confederation Cup play-off round, where they will look to defend the title they won in 2016.
The result is a huge accomplishment for CAPS United of Zimbabwe who have returned to the African Champions League after an 11-year absence.
They will now enter the group phase of the tournament which has been expanded from eight to 16 clubs.
Champions League results (winners progress to group phase, losers to Confederation Cup):
TP Mazembe of the Democratic Republic of Congo suffered a shock exit from this year's African Champions League, losing on the away-goals ruling to Zimbabwe's CAPS United in Harare.