I've never fired an M4, but I know from simple deduction that it's not a terribly accurate weapon.
Here is how: The M16 is not very accurate (it's not necessarily supposed to be, accuracy is not a goal in modern infantry rifle design). Shorten the barrel and stock and you have an M4. Less barrel = less accurate (in general); Less stock = Less control = less accurate.
The M4 is also a generally poor weapon. The effect of shortening the highly refined M16A2 was to create a gas capture system that is too short and therefore overheats. They shortened the barrel without changing the twist rate and it therefore has a low muzzle velocity and therefore is inordinately loud. It's designed to have all its parts easily swappable and they therefore are not securely fastened and the weapon has a tendency to "rattle". Also the magazine is often incorrectly loaded resulting in it having feeder jams.
Although troops are fairly happy with it because if you load the mag correctly, there is the perception that it won't jam up no matter how dirty you get it.
The next model is going to incorporate design aspects of the HK416 (which is a German derivation of the AR15, which is what the M16 and M4 are both based on).