We have a windows 2003 AD domain users use IE6 clients.
Browser caching is turned off as a domain policy.
Question is: Is it possible to override this for a group of users for a specific web site?
Group Policy objects linked to Active Directory Sites (assuming you're talking about an AD Site) are processed second, right after Local Group Policy. This means that any domain-level (or lower, OU-level) GPOs will "override" them by default. You could tick the "No Override" setting on the site-level GPO such that it "overrides" the domain-level GPO that has the unwanted setting.
If your computers aren't arranged into AD Sites and you're using the term "site" somewhat more generically then you'll probably be stuck creating a security group that contains the affected computers and either denying those computers the right to apply the GPO with the unwanted settings or creating a GPO that "undoes" the unwanted setting and allowing only those computers the rights to apply the new GPO.
Thanks for the edit.
There is no method to make IE maintain a cache for only specific web sites. The cache settings aren't tied to the "security zones", etc.