Australia Day is celebrated on 26 January, marking the arrival of the first British convict fleet in 1788.
Some argue that marked the start of the dispossession of Indigenous people, so it is not a day to be celebrated.
"Invasion Day" protests are being held in several Australian cities.
Mr Joyce told Sydney radio station 2GB that he was "just sick of these people who... want to make us feel guilty about it".
"They don't like Christmas, they don't like Australia Day, they're just miserable gutted people, and I wish they'd crawl under a rock and hide for a little bit."
He added that the debate was "political correctness gone mad" and said: "This is Australia Day, and if you don't like it, I don't know mate, go to work. Do something else."
Prime Minister Malcolm Turnbull does not support a date change, and told reporters at a barbecue that he thought most Australians felt the same.
"It is a day where we celebrate the rich diversity of all of our cultures - from our First Australians... to the new citizens, migrants who come from such [a] diverse range of countries."
Australians are marking the national day with community events, barbecues and firework displays.
But protests and marches have been held in several cities including Sydney, Melbourne and Perth, by people saying the day is insensitive to Aboriginal Australians and Torres Strait Islanders, who died in large numbers - from disease, conflict and land loss - following British colonisation.
In New South Wales, Governor David Hurley, who represents the Queen, addressed Indigenous Australians, recognising that "the coming of the First Fleet wreaked a terrible impact on your people, your families, your culture".
The West Australian city of Fremantle this year cancelled its Australia Day plans, and will be holding a "culturally inclusive" alternative celebration on 28 January.
"What we're doing is coming up with something that is actually more Australian," Fremantle Mayor Brad Pettit told the Australian Broadcasting Corporation.
West Australian Premier Colin Barnett, however, has called on the city to "pull your head in" and honour the day. The move was also expected to attract far right protests.
Last year popular radio station Triple J was the subject of a lobbying campaign to release their annual list of top 2016 songs, known as Hottest 100, on an alternative date rather than Australian Day as is the station's tradition.
The station said it would stick to 26 January in 2017, but added that the release date for future years was "under review" as it would consult indigenous communities.
One alternative date suggested for Australia Day has been 1 March.
Former cabinet minister Ian Macfarlane says this marks the date the federal government took control of many functions of the original Australian colonies in 1901, following their independence from Britain.
Australian comedian Jordan Raskopoulous has also suggested "May 8" as an appropriate day, because it sounds like "mate".
Australia's Deputy Prime Minister Barnaby Joyce has criticised people calling for the date of the national day to be changed, saying they should "crawl under a rock and hide".