The Pomeranian-cross called Milo was spotted by a member of the public in water near Appley Bridge, Wigan, on Tuesday evening.
The male dog, aged about five, was wearing a blue harness to which a 1.25kg exercise weight had been tied.
The RSPCA said the animal - who had been in the canal for two weeks - was micro-chipped, but it was out-of-date.
Inspector Johnny Brooks said: "We have not been able to establish how he died.
"We are very keen to find out who owned Milo after 29 April, so we can find out how he came to be in the canal."
A terrier-type dog was found in similar circumstances in Bootle, Liverpool in May, however the RSPCA do not believe that the two incidents are related.
A dog with an exercise weight attached to his harness has been found dead in a Greater Manchester canal.