Belgian music royalty collecting firm SABAM wanted the social network Netlog to stop users infringing copyright.
But the court said the filtering required would contravene rights to freedom of business, personal data and freedom of information.
The judgement could have consequences for similar cases across the EU.
Netlog, a social network developed by Ghent-based Massive Media NV, says it has more than 95 million members throughout Europe.
Michael Gardner, head of the intellectual property practice at law firm Wedlake Bell, said: "The European Court appears to have ruled out the idea that operators of social network sites and ISPs can be forced - at their own expense - to impose blanket monitoring and filtering aimed at stopping infringements."
However he added: "The ruling doesn't stop rights owners seeking more limited injunctions against social networking sites or ISPs, but they will have to be more 'proportionate' in scope and effect."
The ECJ's decision will be used by courts across the EU where this aspect of European law is in question.
SABAM had asked a Belgian court to require that Netlog cease unlawfully making available works from its repertoire and face 1,000-euro (Â£828) fines for every day it failed to comply.
Netlog said SABAM was, in effect, requiring it to filter its content, indiscriminately monitoring all of its users - something they argued was contrary to the EU's E-Commerce Directive.
The case was referred to the European Court of Justice to decide the legality of such a filtering system
Inits judgement the ECJ ruledthat a system would be a "serious infringement" of Netlog's freedom to conduct its business "since it would require Netlog to install a complicated, costly, permanent computer system at its own expense".
But it also said there was a risk of infringing rights to the protection of personal data, as a filtering system would require "the identification, systematic analysis and processing of information connected with the profiles created on the social network".
Additionally it might also restrict freedoms to send and receive information, as the system "might not distinguish adequately between unlawful content and lawful content, with the result that its introduction could lead to the blocking of lawful communications".
This is the second judgement the court has made in this area involving SABAM. An earlierCourt of Justice ruling in November sided with an ISPwhen it decided against what it said was, in effect, a request that it monitor traffic for illegally copied material.
The European Internet Providers Association said this latest ruling was a positive step: "We therefore welcome the greater legal certainties and protection granted by the court which are necessary to guarantee the openness of the internet."
The ruling comes amid fierce debate over anti-piracy measures contained in the Anti Counterfeiting Trade Agreement (Acta) - an international treaty intended to help protect intellectual property rights.
The Open Rights Group, an organisation which opposes Acta, said in a statement: "It's good to see courts promoting our rights by swatting away plans to snoop on people's use of social networks.
"It is especially timely because, as seen in agreements like Acta, policymakers everywhere find it much harder to respect our rights when making intellectual property policy."
However Acta's supporters argue it is necessary to prevent the widespread infringement of intellectual property rights.
A social network cannot be required to install an anti-piracy filtering system, the European Court of Justice (ECJ) has ruled.