There are lots of different kinds and degrees of color blindness, not just one set kind.
So, for different people, different colors are bad.
So, I guess it depends on who you are talking to.
I'm not colorblind, but I've known one or two people that are.
A trick that I would use (not sure if its the best, but it makes sense to me) is to convert a copy of your image to greyscale. If you have any problems seeing anything with it in black and white, chances are that some people might have trouble seeing it in the colored version.
If you are doing text or colors, try to make it stand out not only by color, but also by brightness and saturation. that makes it easier to see.