Anders Ygeman said charter aircraft would be used to deport the migrants but it would take several years.
Some 163,000 migrants applied for asylum in Sweden in 2015, the highest per capita number in Europe.
The numbers have fallen significantly since Sweden imposed tighter border controls this year.
Along with Germany, the Scandinavian country is a prime destination for refugees and other migrants entering the EU illegally.
Of the approximately 58,800 asylum cases processed in Sweden last year, 55% were accepted.
Of those facing expulsion, Mr Ygeman was quoted in Swedish media as saying: "We are talking about 60,000 people but the number could climb to 80,000."
But he later tweeted to say he had not taken a position on how many migrants had grounds for asylum, it being a matter for the authorities and the courts.
Sweden earlier this week became the latest of a number of European nations to see tensions over migrants heightened by violence. A 15-year-old asylum seeker was arrested in Molndal, near Gothenburg, after a 22-year-old asylum centre employee was stabbed to death.
Mikael Ribbenvik, head of operations at the Swedish Migration Agency, told the BBC that assessing all the asylum applications would be "an enormous feat to accomplish" and would require more resources from the government.
"A lot of people leave voluntarily and a lot of people abscond. And then we have a few people that are staying on that are impossible to remove because of identification purposes," he said.
More than one million refugees and migrants travelled to Europe last year, most fleeing conflicts in Syria, Iraq and Afghanistan.
The UN says another 46,000 people have arrived in Greece so far this year, with more than 170 killed making the dangerous crossing from Turkey.
In the latest such accident, at least 11 migrants drowned - mostly children - after their boat capsized off the island of Samos, the Greek coast guard says. Several are still missing.
On Wednesday, a draft European Commission report said Greece "seriously neglected" its obligations to control the external frontier of Europe's passport-free Schengen zone.
Greek government spokeswoman Olga Gerovasili accused the Commission of "blame games" and said it had failed to act on a programme agreed last year to relocate tens of thousands of migrants and refugees stranded in Greece.
Denmark has also faced criticism this week after approving legislation to seize the valuables of refugees in the hope of limiting the influx of migrants.
Some have likened the Danish proposals to the confiscation of gold and other valuables from Jews by the Nazis during the Holocaust.
Sweden may reject the asylum applications of up to 80,000 migrants and should prepare to deport them, the interior minister says.