Former heavyweight world champion Haye, 35, dropped Gjergjaj three times before the referee waved the fight off.
Haye's next fight in September will be against American veteran Shannon Briggs, who knocked out Argentina's Emilio Zarate on the undercard.
Haye's ultimate aim is a match with either IBF champion Anthony Joshua or WBA and WBO champion Tyson Fury.
Fury having still not forgiven Haye for pulling out of two scheduled fights against him in 2013, Joshua is the more realistic target.
Manchester's Fury faces a rematch against Wladimir Klitschko on 9 July, while London's Joshua faces unbeaten American Dominic Breazeale on 25 June.
"I felt amazing," said Haye, who now has 28 wins (26 KOs) and two defeats from 30 professional contests.
"The fans have come out to see the future of heavyweight boxing. People thought I was the past, but I am still the future.
"Anthony Joshua is a fight I would relish. But next up for me is Shannon Briggs. He came over to England and talked a big game.
"I told him if he came over and beat someone I would fight him in September. Let's go chump, let's go chump."
Haye was out of the sport for three and a half years after undergoing shoulder surgery in 2013 but returned with a first-round knockout of Mark de Mori in January.
And he floored the unbeaten Gjergjaj with a right hand inside the first 30 seconds at the O2 arena, before dropping him with a left jab and a chopping right in the second.
Haye's Swiss-based opponent made it to his feet but referee Terry O'Connor had seen enough.
The 44-year-old Briggs, a former WBO world champion, lost world title challenges against Vitali Klitschko in 2010 and Britain's Lennox Lewis in 1998.
Briggs, who is now based in London, took four years out of the sport after the Klitschko defeat, during which he suffered from depression and ballooned to 400lb.
But he has won all nine fights since returning, including his latest one-round demolition of Zarate.
Haye has pledged to donate 10% of ticket sales for his latest fight to Nick Blackwell.
The 25-year-old Blackwell will not fight again after sustaining serious head injuries in his middleweight fight against Chris Eubank Jr in March.
Britain's David Haye continued his comeback with a second-round stoppage of Kosovo's Arnold Gjergjaj in London.