Most of the time, if you are downvoted for a legitimate reason, that reason should either be readily apparent or will become so as more answers/comments arrive. At least, I've found that to be the case. I can then delete my answer, or make it CW if I think for some reason it is still useful despite being wrong.
Sometimes you aren't downvoted for a legitimate reason, and you can usually tell as these are one-off votes that aren't echoed by others. So what? It's 2 rep. If you're lucky, someone else will see that you didn't deserve the downvote and give you a pity upvote you wouldn't have otherwise gotten. +8 for you. IMHO, it's not a big enough deal enough to warrant sacrificing the integrity of the anonymous system.
I understand that often people who propose this say that it isn't about the rep. While you might prefer to know who downvoted you, or for what reasons, the community is invested in the anonymous voting process. I think that you'd have to show that the system is badly broken in order to make this fundamental change.