If you ever find yourself using that line, thinking it excuses you from accusations of racism, stop. Just stop. Think about what you are doing: you are taking a diverse group of people, categorizing them by the color of their skin, and are about to make a sweeping generalization about all of them.
You know what that is? It’s racism.