The best prevention is to:
- use your own trusted computer (malware that could capture form data),
- install your OS/programs from trusted source,
- do not install browser extensions/user scripts (especially from untrusted sources) or at the very least use private browsing (with all extensions disabled) when you go to these sites,
- use a modern up-to-date browser,
My original answer mostly was concerned with your title
How do i protect myself from "Form grabbing" attack? versus instructing you on how to write security software. Otherwise I agree with CodesInChaos remark: "It's an unwinnable battle".
There are two distinct places form-grabbing could occur: either from the browser (e.g., an XSS attack loading malicious JS, or a malicious extension installed in the browser) OR monitoring API calls via hooking.
There are a variety of ways of hooking, but if a user/malware has sufficient privileges to insert API hooks, they probably also have enough power to make the output of your hooking detection tests untrustable.
For example, if someone was mounting this style of attack to log all http/https requests you could detect try to detect if the shared library (aka DLL) has changed (granted every benign upgrade to that library will trigger false positives) or contains a certain malware signature.
Now let's say the altered shared library is saved to disk in its altered form. Now your anti-hooking program computes a cryptographic hashsum to check that the library has not been altered to compare against the known function of the unmodified library. But what prevents the attacker from just hooking into those calls to calculate the checksum and altering the output to be the pre-modified checksum whenever the malicious checksum comes up? Nothing, unless of course you are analyzing a hard disk from a safe operating system (e.g., mounted the drive read-only from another computer not infected with the malware).
Granted, due to threats of runtime modification of libraries, you really need to check the instructions in the memory of the process that is running versus the static files. In this sort of situation, you really need to have the potential malware running to capture the API-hooking attack, though all your safe utilities cannot be trusted as the system has already been compromised. (You could still scan the disk from a safe setup for whatever program is eventually altering the memory to insert the API-hooks, but its a much harder task where you only find the types of attacks you know to look for).