I think that another reason (though it's been touched on in a number of responses here) is the fact that the human population no longer depends on just the "fittest" as it might in populations of other animals where natural survival ability dictates far more which individuals will live to pass on their genes.
In his essay Darwin's Untimely Burial
(1976), Stephen Jay Gould specifically addresses what Darwin meant by "fitness", and notes: "criteria of fitness independent of survival can be applied to nature and have been used consistently evolutionists." And concludes: "Darwin's independent criterion of fitness is, indeed, 'improved design,' but not 'improved' in the cosmic sense that contemporary Britain favored. To Darwin, 'improved' meant only 'better designed for an immediate, local environment.'"