The answer to this question is complicated, and one way to see that it can't be simple is that the sky isn't a single color. It's bluer near the zenith and whiter near the horizon. It's gray on a hazy day. It's red near the setting sun.
The figure below (Bohren, Atmospheric Optics, http://homepages.wmich.edu/%7Ekorista/atmospheric_optics.pdf ) below shows the result of multiplying the solar spectrum by the $1/f^4$ of Rayliegh scattering. This gives us a spectrum that is strongly peaked in the violet, but that still has quite a bit of light throughouth the rest of the spectrum.
This is all assuming single scattering. Multiple scattering also occurs, and is most common near the horizon. Multiple scattering brightens the light and equalizes the intensities of the different wavelengths.
We also have haze and possibly other forms of particulate matter such as smog. This produces Mie scattering. Mie scattering on a cloud of particles with a range of sizes produces more uniform scattering as a function of wavelength than Rayleigh scattering.
The net result of multiple scattering and Mie scattering is to whiten the color of the sky considerably.
You can have particles in the atmosphere that are actually colored. This is not a major effect on earth, but it is the main factor responsible for the color of the Martian sky.
Next we hit the cone cells of the eye, which contain pigments that act as filters for three overlapping bands. Philip Gibbs' answer shows the curves for these filters. There is also the Purkinje effect here, which shifts the spectral response of the eye depending on the level of light, which will affect the color of the sky at dusk compared to its color at noon.
Finally we get to an additional layer of processing by the nervous system, as described by the opponent processing theory of Hering, Hurvich, and Jameson. This probably has a significant effect on the color of the sky. On these graphs, you can see that (1) the red-green function has a surprising rise near the short-wavelength end of the spectrum (which I don't think can be explained just based on the tiny secondary bump in the red cells' filtering function), and (2) we perceive a color to be blue when the red-green function is near zero. Examining the spectrum on the graph by Bohren, and then considering the whitening that would result from Mie scattering and multiple scattering, it looks like the final step of passing it through the red-green function would probably produce a net negative result, which would make the sky tend to appear blue or cyan -- as we observe. For the sky to appear violet, it would have to have a net positive value of the red-green function, and that seems impossible with this spectrum.
A good online article with links to several other, more detailed PDFs is here: http://www.orionsarm.com/xcms.php?r=oa-page&page=gen_skyonalienworlds