Much that is published in scientific journals is of astonishingly poor quality.
We have become accustomed to a steady barrage of reports of hazards lurking in our environment that MAY pose a threat our health and that of our children.
These include, among others, low-level radiation exposure from nuclear power plants and nuclear waste; possible water contamination from hydraulic fracturing; and exposure to a wide range of chemicals, including pesticides and industrial pollutants, in food, water, air, and consumer products.
Potential hazards like these need to be studied and will be studied, but the public needs to realize that much that is published in scientific journals, and even in reputable journals, can be of astonishingly poor quality and is of absolutely no relevance to non-researchers. But, far from these papers being ignored, they often get seized on by the media and high-lighted as if they provided serious, actionable evidence of a hazard.
How can this happen? How do mediocre or plain-wrong studies get accepted for publication in scientific journals? In essence the explanation is simple. Scientists and scientists-in-training need to find questions to work on and need to publish their results in order to put themselves on the map and to advance in their careers. Journals want to publish articles on topics that appear to be important and that will engage readers. And, scientists who review the work, being human, may be influenced by the general buzz in the media concerning certain hazards.
All that stands between the publication of a poor piece of work and the public are journal editors and the peer reviewers who agree to donate their time to evaluate a paper for publication. For the most part, the system works, and there is a relatively high quality of publications, particularly in the more reputable journals. Nevertheless, quality is highly variable across different journals and different content areas – for example, the standards for validation of results in genetic epidemiology studies have been greatly strengthened in the past few years.
Furthermore, as is true of any system dependent on human beings, peer review is imperfect. It is only as good as the critical acumen that reviewers bring to the task, and there are inevitable failures, i.e., papers that get published but should never have been published.
The statistical methodologist John Ioannidis of Stanford has argued that “most research findings are false” And the unfortunate truth is that virtually anything – no matter how bad – can get published somewhere.
A recent example of a failure of the system is the publication of a paper in the American Journal of Epidemiology, one of the two top journals in the field of epidemiology. The paper entitled “Urinary Bisphenol A and Obesity in US Children” came out earlier this year.
Publication of this paper in the venerable journal put out by the Johns Hopkins School of Public Health is, I think, significant, first because it shows how the peer review system can fail.
Furthermore, it may alert us to the danger when a paper deals with a topic that has received enormous publicity and caused public alarm, as is the case with BPA. This may have lowered the threshold for publication. In other words, to the extent that the reviewers and editor were aware that the paper was weak and its results dubious, they may have over-ridden these reservations on the grounds that the paper was on a topic of great interest. Finally, publication of this paper may be symptomatic of a lower level of scientific rigor and overall quality prevailing in the area of environmental health, an area where there is great public and media interest, as opposed to other research areas that do not evoke so much interest.
Let’s take a look at the paper.
The authors used data from the National Health and Nutrition Survey (NHANES) to examine the association between urinary BPA and obesity in children aged 6-18 years. They report a positive association between BPA and obesity: that is, compared to children in the lowest quartile of BPA, children in the highest quartile were 2 and ½ times as likely to be obese. This after adjusting for such factors as age, sex, ethnicity, etc.
Point 1. This is a cross-sectional study, meaning that all the information was collected at one point in time. This is the weakest type of design and it does not tell us anything about the temporal relationship between BPA exposure and obesity, i.e., did the former precede the latter?
Point 2. Since BPA was measured at only one point in time, we know nothing about how reliable this single measure is to characterize a child’s habitual level of the compound.
Point 3. What does a urinary BPA measurement tell us about exposure? Careful studies in animals and adult humans show that BPA is efficiently metabolized and >99% is excreted in the urine, even in newborns. In a study in which 20 adult volunteers ate meals from cans for 1 day, resulting in high BPA exposure compared to the general population, BPA in blood was largely undetectable and was 42 times lower than in urine, suggesting little or no BPA accumulation.
Point 4. Up until now, one might say that the paper was worth publishing even in spite of its limitations. However, when the authors present their results broken down by sex and ethnicity, one is brought up short (see Table 6 below). The association between BPA and obesity is present only in ONE out the four groups, i.e., White boys. There is not a glimmer of a relationship among Non-White girls or boys nor among White girls. Furthermore, there is not really a convincing trend in White boys: 15.56, 8.47, and 18.89. So, even though the trend statistic may be highly significant (<0.0001), the trend is anything but clear.
Finally, one notices that the numbers of subjects are much greater among Non-whites than among Whites – more than twice as great. Therefore, it is interesting that among Non-Whites, where there is much greater statistical power to detect an association, there is no evidence of one.
If there were a real relationship, one would expect to see it in girls as well as boys and in Non-Whites as well as Whites. As it stands, this looks like a chance finding that the authors latched onto and converted into a paper that simulates a scientific paper.
Perhaps most objectionable, the authors use the trick of combining these very different groups, and reporting an OVERALL statistically significant association for the total study group. But one is left to wonder what a result means that is observable only in White boys, but not in the other three groups.
If this paper had appeared in one of the many less rigorous and less respected journals, I would not have written this column. But it is noteworthy when a blue-chip and highly-respected journal fails to live up to its standards and publishes something of such questionable value. This alerts us to the need to be vigilant and critical and to not give credence and currency to work that does nothing to advance our understanding on the question of BPA and its possible health effects.
It also alerts us to the reality that the ability – and willingness — to be critical of scientific papers in order to reduce contamination of the literature by meaningless results are commodities that are in short supply.
Geoffrey Kabat is a cancer epidemiologist at the Albert Einstein College of Medicine and author of Hyping Health Risks: Environmental Hazards in Everyday Life and the Science of Epidemiology.