Several students were injured and opposition MPs said four were killed, but the government denied any deaths.
The students were marching from their campus in the capital, Port Moresby, towards parliament where Mr O'Neill faces a possible no confidence motion.
They are demanding that he stand down and answer corruption allegations. Mr O'Neill has denied wrongdoing.
In a statement later on Wednesday, Mr O'Neill said "a small group of students were violent, threw rocks at police and provoked a response that came in the form of tear gas and warning shots".
"The factors that led to students being injured are yet to be ascertained," he said, condemning "agitators responsible for instigating a violent confrontation".
The government said an inquiry would be launched to determine what caused the clashes.
In a separate development, parliament was adjourned on Wednesday until August.
Footage obtained by the BBC appeared to show a large crowd of students at the campus running away as shots and tear gas were fired.
Political tension in the country has been rising for weeks, with thousands of students at the University of Papua New Guinea boycotting classes demanding the resignation of PM O'Neill. Classes were officially suspended last month.
In May, Mr O'Neill responded to a petition from students saying that he would not be resigning. He said the corruption allegations against him were of "questionable political intent", as reported by ABC.
Papua New Guinea was ranked one of the most corrupt countries in the world in 2012 by Transparency International.
In 2014, Mr O'Neill himself was accused of fraud by the national anti-corruption watchdog, which issued a warrant for his arrest. The warrant has not been carried out so far.
According to the World Bank, 70% of the country, the most linguistically diverse in the world, lives in poverty.
Medical officials in Port Moresby told Reuters that at least 10 students had been admitted "in a difficult situation".
Images circulating on social media also showed injured students being carried away.
There have been conflicting reports of casualties. Opposition figures told parliament that four people had died, while other reports said one person was dead.
Reuters quoted an unnamed aid agency as saying it had heard reports four students died at a clinic on the campus in the Waigani suburb of Port Moresby.
The main hospital in the capital told local news site EMTV that it had seen no fatalities.
Papua New Guinea Police Commissioner Gari Baki said no deaths had been reported, according to the Associated Press news agency.
One eyewitness, David Rupa, told the BBC he was on his way to work when he became caught in the protest at about 08:50 local time (23:50 GMT Tuesday).
He said he could see tear gas and people running for cover and heard shots fired.
"I saw policemen hit and kick girls who couldn't run [fast enough] and were bashed up. I cried and was told I will be shot if I was going to take photos or video."
He also said he had seen smoke coming from a dormitory at the university, and that parents were coming down to the campus to make sure their children were not hurt.
Gary Juffa, an MP and vocal critic of the prime minister, said on Twitter he had been told the shooting began with an argument between one student and a police superintendent.
Reports suggest the students had refused to hand over their protest leader to police.
EMTV said the university had suspended classes to deal with the student boycott, but they were supposed to resume this week.
Police in Papua New Guinea have opened fire on students protesting against Prime Minister Peter O'Neill.