Calculating CRCs (or, better, sha1sums) on both files requires reading every byte anyway. If you do a byte-by-byte comparison, you can quit as soon as you see a mismatch -- and you don't have to worry about two different files that happen to have the same checksum (though that's vanishingly unlikely for sha1sum). So if you're doing the comparison locally, a byte-by-byte comparison will be at least as fast as a checksum comparison (unless you've already computed the checksums anyway).
On the other hand, checksum comparisons are useful when you're comparing files that aren't on the same machine; the checksums can be computed locally, and you don't have to transfer the entire content over the network.
Hybrid approaches are also possible. For example, you might compute and compare checksums for the two files a chunk at a time, which can avoid reading the whole files (if they differ) while also avoiding transmitting the while file across the network. The rsync protocol does something like this.
Note that using a simple CRC gives you a fair chance of a collision, as Dave Rager mentioned in his answer. Use at least sha1sum, or even something more recent. (Don't try to invent your own hashing algorithm; the folks who developed sha1sum know far more about this stuff than either of us.)
As for the likelihood of collision, if you use a decent hash like sha1sum you pretty much don't have to worry about it, unless someone is deliberately and expensively constructing files whose sha1sums collide (generating such collisions is not currently feasible, though it may become so in the foreseeable future). Quoting Scott Chacon's "Pro Git", section 6.1:
Here’s an example to give you an idea of what it would take to get a
SHA-1 collision. If all 6.5 billion humans on Earth were programming,
and every second, each one was producing code that was the equivalent
of the entire Linux kernel history (1 million Git objects) and pushing
it into one enormous Git repository, it would take 5 years until that
repository contained enough objects to have a 50% probability of a
single SHA-1 object collision. A higher probability exists that every
member of your programming team will be attacked and killed by wolves
in unrelated incidents on the same night.
Byte-by-byte comparison is good for local comparisons. sha1sum is good for remote comparison, and presents no significant chance of false positives.