UK Independence Party and French National Front both performed strongly. The three big centrist blocs all lost seats, though still hold the majority.
The outcome means a greater say for those who want to cut back the EU's powers, or abolish it completely.
UK PM David Cameron said the public was "disillusioned" with the EU.
Mr Cameron said their message was "received and understood".
LIVE: Full coverage of results
French President Francois Hollande has called an urgent meeting of his cabinet, as Prime Minister Manuel Valls promised tax cuts a day after the results which he described as "a shock, an earthquake".
Chancellor Angela Merkel - whose party topped the poll in Germany - described the far right victories as "remarkable and regrettable" and said the best response was to boost economic growth and jobs.
Jose Manuel Barroso, outgoing president of the European Commission, stressed that the pro-EU blocs still had "a very solid and workable majority".
He said a "truly democratic debate" was needed to address the concerns of those who did not vote, or "voted in protest".
(Based on exit polls/provisional results)
European election results: At a glance
EU supporters will be pleased that election turnout was slightly higher, at 43.1%, according to provisional European Parliament figures.
By Gavin HewittEurope editor
That would be the first time turnout had not fallen since the previous election - but would only be an improvement of 0.1%.
"The people have spoken loud and clear," a triumphant Marine Le Pen told cheering supporters at National Front (FN) party headquarters in Paris.
"They no longer want to be led by those outside our borders, by EU commissioners and technocrats who are unelected. They want to be protected from globalisation and take back the reins of their destiny."
Provisional results suggested the FN could win 25 European Parliament seats - a stunning increase on its three in 2009.
Despite the huge gains for anti-EU parties and those that wish to reduce the power of the EU, it is  unlikely that the European Parliament will adopt a fundamentally different approach.
The main centre-right and left groupings still hold the majority of seats between them. Add in the Liberals and the Greens and parliament is overwhelmingly pro-EU. In that sense if the main groups in the parliament choose to ignore those who didn't vote for them last night, they can. National politicians however, can't afford to ignore them.  That's where this election is likely to really shake things up.
This will have an impact not just on domestic politics, but on the national leaders when they meet in Brussels at the European Council - where their job is to shape the direction of the EU.
Protest parties will force EU rethink
Across the board, the centre-right European People's Party was set to win 214 out of the 751 seats, with 28.5% across the bloc, according to estimated results issued by the European Parliament. That would make it the biggest group - but with more than 60 seats fewer than before.
That puts it ahead of the Socialist group with 189 seats (25.17%), Liberals with 66 (8.79%) and Greens 52 (6.92%).
EPP leader Jean-Claude Juncker insisted the majority of people had voted to be a part of Europe.
"The extreme right, contrary to what some of the media has said, did not win this election," he said.
"We will have a clear pro-European majority in this house," added the man who is the frontrunner to be the next president of the European Commission.
The Eurosceptic Europe of Freedom and Democracy group appeared to have around 38 seats, including the 24 confirmed seats for the UK Independence Party (UKIP).
But the number of non-attached right-wing MEPs is set to rise, boosting the Eurosceptic camp.
UKIP leader Nigel Farage, commenting on his party's success in Britain, said: "The inevitability of European integration ends tonight."
The anti-bailout hard-left group in parliament was set to make big gains, largely thanks to Syriza in Greece and United Left in Spain, gaining about 12 seats.
Martin Schulz, the former Socialist president of the European Parliament - said of the FN victory: "It's a bad day for the European Union, when a party with a racist, xenophobic and anti-Semitic programme gets 25% of the vote."
The election is the biggest exercise in multi-national democracy in the world. The vote will affect the lives of the EU's 500 million citizens.
The parliament's powers have expanded since the last election in 2009, and it is hoping to have a decisive say in who gets the EU's top job, president of the European Commission.
You can follow full coverage with all the latest updates at bbc.co.uk/vote2014.
Eurosceptic and far-right parties have seized ground in elections to the European parliament, in what France's PM called a "political earthquake".