It would be "shameful" to  spend huge sums of money on the celebrations when "our people are dying of cholera", he said, state television reported.
Cholera has killed about 60 people in Tanzania in the last three months.
This will be the first time in 54 years Tanzania will not hold celebrations to mark independence from the UK.
Mr Magufuli's party, Chama Cha Mapinduzi (CCM), has governed since colonial rule ended on 9 December 1961, winning a new mandate in elections last month.
Many people were caught by surprise by Mr Magufuli's announcement but have welcomed the move, the BBC's Sammy Awami reports from the main city, Dar es Salaam.
They feel it shows his commitment to ending lavish spending and tackling the cholera outbreak which has caused widespread concern, he says.
But it remains to be seen whether people will heed his call to spend independence day cleaning up their residential areas and work-places, our reporter adds.
Independence day celebrations are usually marked with a presidential address, a military parade and performances by music groups at the National Stadium in Dar es Salaam.
Mr Magufuli's spokesman Gerson Msigwa did not say how much would be saved by cancelling the event but said the money would be spent on hospitals, and the fight against cholera - a major problem in poor areas where there is a lack of proper toilets.
Mr Magufuli has announced a range of cost-cutting measures since he took office, including a ban on unnecessary foreign travel by government officials.
Last week, he ordered the cost of a party to inaugurate the new parliament to be slashed from $100,000 (Â£66,000) to $7,000.
He also sacked the head of the main state hospital after finding patients sleeping on the floor during a surprise visit to the facility.
Mr Magufuli beat opposition candidate Edward Lowassa in last month's election to become Tanzania's fifth president since independence.
Nicknamed "The Bulldozer", Mr Magufuli is reputed to be a no-nonsense, results-driven politician.
He was the works minister, before being elected president.
Tanzania's newly elected President John Magufuli has cancelled independence day celebrations, and has ordered a clean-up campaign instead.