Scientists found that young men who suffered disturbed sleep had sperm counts that were more than a quarter lower than those who slept more.
Those who suffered disturbed sleep with less than six hours a night also had lower sperm quality and smaller testicles.
It is thought that poor sleep interferes with the body's ability to produce sperm.
Tina Kold Jensen, who led the study at the University of Southern Denmark, said: "Men with a high level of sleep disturbance had a 29% lower adjusted sperm concentration.
"This appears to be the first study to find associations between sleep disturbances and semen quality."
The researchers, whose work is published in the American Journal of Epidemiology, added that they hope to examine whether restoring normal sleep patterns helped to improve fertility in men.
They studied 953 young men from Denmark aged around 18-years-old and asked them to fill in questionnaire to assess their sleep.
They then compared their responses to sperm samples taken from the men.
Those who had suffered poor sleep in the past four weeks had lower sperm counts and less mobile sperm. They were also more likely to have deformed sperm.
Sleep loss has in the past been linked with increased risk of obesity, breast cancer and heart disease.
Experts believe the ideal amount of rest is between seven and eight hours a night.
Dr Remy Slama, a fertility expert at Joseph Fourier University in Grenoble France, said the study added another aspect of modern life that may explain falling fertility rates among men.
He said: "Intervention studies would also be helpful to identify whether sleep disturbances, or improvement of sleep quality, can lead to short-term variations in semen parameters.
"This study adds another suspect to the list of factors possibly influencing male fecundity potential, which also includes (being) overweight, exposure to tobacco smoke, exposure to pollutants.
"Even if each of these factors has a weak impact at the individual level, the large number of factors and relatively high prevalence of exposure in the general population make it likely that at the population level, lifestyle and environmental factors put a high burden on male fecundity potential."