Elsewhere in the Pathfinder SRD, under Combat, you can find this quote:
If penalties reduce the damage result to less than 1, a hit still deals 1 point of nonlethal damage.
So your unbuffed cats and rats will always cause 1 point of nonlethal damage on hit. The rat has more HP, the cat has more attacks, and it's down to a numbers game to see who knocks the other unconscious first, after which the victor can quietly eviscerate the loser and feast on his lovely, lovely XP.
Addendum: In case you're wondering how critical hits work in this scenario, it's quite simple. Two paragraphs later we have this:
Sometimes you multiply damage by some factor, such as on a critical hit. Roll the damage (with all modifiers) multiple times and total the results.
So you don't have to mess around with totaling rolls and applying penalties twice. You just calculate damage twice and then sum it up. So a critical hit for our cat or rat will cause simply 2 points of nonlethal damage. The cat, with its three natural attacks, could theoretically pull of 6 points of damage in one round and and one-shot the heck out of the rat. Rat-boy is stuck with a max of 2 points of damage, and will take at the very least two rounds to whack-a-cat.