Since DNS usually runs over UDP, response packets can be readily spoofed. UDP packets are identified by the combination of source and destination IP address and source and destination port numbers.
The classic DNS poisoning attack is to send a DNS server a query which you think will cause the server to do a recursive lookup, and then blast away at the server with spoofed response packets for the secondary request you expect the server to make. If you get the right IP/port combinations and transaction ID and your answer arrives first, your faked answer could be accepted and inserted into the cache.
Well, we know the source and destination IP address ahead of time, and we know that DNS runs on port 53, so that's one of the port numbers. Assuming DNS queries get sent FROM port 53 as well as being sent TO port 53, then we've got all 4. Then all we have to worry about is the transaction ID, which is a 16-bit number. That means each packet we send has a 1 in 65K chance of striking gold. We could easily spam out several thousand packets in just a second or so, which gives us a decent chance of poisoning the server's cache. If we push out 10K response packets before the "real" response comes in, that gives us about a 1 in 6 chance overall of success. And if it fails, we just wait for expiry and try again.
But, if the server uses a random outbound port number on recursive lookup queries, then that adds another 16 bits worth of material that we have to guess right for our response to be accepted. So that's a total of 32-bits, or about 4 billion possibilities. That means that our odds for each packet go from 1 in 65,536 to 1 in 4,294,967,296, which becomes a bit prohibitive.
So while you can't be 100% certain that a response isn't spoofed, the odds are weighted much more in the favor of the defender if he randomizes his source ports for his secondary lookups; enough so to dissuade most attackers from investing the resources to try the attack.