Data from more than 35,000 surveys suggests that habitats have declined by more than 50% since the 1970s.
The report's authors believe that over-fishing and disease is mainly to blame.
They say the trend could continue if nothing is done, but with protection the reefs could bounce back.
Carl Gustaf Lundin, director of IUCN's Global Marine and Polar Programme, said the findings were alarming.
"The reefs support a number of different countries and populations," he said.
"Tourism is one of the biggest industries, and the health of the reef is essential to the well-being of many of the people living there. And of course they are immensely beautiful and wonderful places as well."
The report, which was also authored by the Global Coral Reef Monitoring Network, looked at data collected between 1970 and 2012 from 90 reef habitats and is the most comprehensive assessment of corals in this region.
The authors concluded that many reefs in the Caribbean were undergoing a dramatic transformation - changing from colourful realms, bursting with life, to barren habitats, covered in slimy algae.
They said the damage had been driven by the loss of some of the reefs' key inhabitants.
In the 1980s, a disease, thought to have been brought in from the Panama Canal, wiped out a large number of sea urchins. While fishing on the reefs has also caused parrotfish to plummet.
Both of these creatures graze on reefs, and without them, algal species take over.
Dr Lundin said: "We saw that reefs with no grazers ended up getting smothered by algae. And after a period of time they see a significant or even complete collapse of the reef area."
He said that if no action was taken, the outlook for Caribbean corals would be bleak.
"If we do nothing, I'm afraid the most likely scenario is that we will continue the slippery slope to slime," he said.
"We'll lose a tremendous amount of coral cover and we'll end up with algal reefs with a much lower diversity of species."
However, the report concluded that there was evidence that damaged reefs could be revived if they were protected.
Dr Lundin said: "We have been able to document a number of cases where we've seen recovery of degraded reefs, so that is very positive - it means we shouldn't give up hope."
The Caribbean is not the only area that has seen losses to coral reefs. Around the world, these habitats are doing badly.
Conservationists say that warming ocean temperatures are also driving some of the losses.
As temperatures rise, corals lose the tiny algae that live in their tissues, causing them to turn white - a process known as coral bleaching.
Follow Rebecca on Twitter
Many of the Caribbean's coral reefs could vanish in the next 20 years, according to a report published by the International Union for the Conservation of Nature (IUCN).