The specs page for the Nikon V1 says the following about what parameters can be recorded into its H264 quicktime files:
HD movies 1,920 x 1,080/60i (59.94 fields/s *)
1,920 x 1,080/30p (29.97 fps)
1,280 x 720/60p (59.94 fps)
Slow-motion movies 640 x 240/400 fps (plays at 30p/29.97 fps)
320 x 120/1,200 fps (plays at 30p/29.97 fps)
Movies recorded in still image mode 1,072 x 720/60p (59.94 fps)
Motion Snapshot 1,920 x 1,080/60p (59.94 fps) (plays at 24p/23.976 fps)
* Sensor output is about 60 fps.
So what parameters are your files recorded with?
Have you tried transcoding the files to a better intermediate codec (like ProRes, Animation or PNG codecs)?