A study published recently in the Annals of Internal Medicine sought to probe whether published randomized controlled trials (RCTs) cited relevant prior work. After all, if we all stand on the shoulders of giants, we should at least check our footing before attempting another leap. This is especially important in medical research, where patients are put at risk and where there has been documented evidence of citation invention and diversion.
The authors plumbed meta-analyses to compile lists of all the possible papers on a topic, figuring that meta-analysts have an inclusive approach to the literature (an approach that trialists might not share). They looked back 40 years through the literature to arrive at their dataset. Doing so, they found that not only were the trialists only citing 21% of the trials they possibly could have, they were claiming to be the first trial on a topic. Trials citing the fewest percentage of possible trials did this more often, but even in the top quintile, it still occurred.
Now, there may be multiple reasons why reference lists might be on the lean side. One that immediately occurred to me was space considerations. Because this study spanned four decades of trials, much of what was published was generated before online supplements and so forth. Print constraints, and in the older studies, true difficulties assessing the entire scope of the literature (-40 years = 1970), might explain some of the findings. But the authors dismiss this — without measuring it. In fact, how they dismiss it is almost suspiciously dismissive:
The possibility that journal space limitations are causing this lack of citation seems unlikely. We find it implausible that authors are being forced to limit themselves to 2 or fewer of their most critical citations by page or reference list limitations.
Since this isn’t my first rodeo, I won’t dismiss that explanation too quickly. If a researcher is writing up a trial, one that starts from Point A and ends at Point B, he or she might reasonably do some hand-waving at Point A’s best studies, then move right along, citing the studies that matter on the journey to Point B. As the authors themselves concede, trialists’ motivations for citation are not informed by exhaustive citation.
In fact, there are hints that the reference list problem might be a style problem and not a substance issue. Here are a few quotes from the study to that effect:
The surprising constancy of the number of cited trials with increasing numbers of citable trials meant that the 2 proportional citation measures . . . must be interpreted with care.
A small improvement (P > 0.001) was seen in trials published after 2000 . . .
However, [differences in citation patterns between trialists and meta-analysts] does not account for the constant average number of citations as the number of citable trials increased.
Studies not cited in a published RCT may have been cited in a funding proposal or institutional review board application.
My interpretation isn’t nearly as nefarious as the authors’. In my mind, the expectation is that the report of a trial tips its hat to some of its predecessors. If it’s easy to find more (which it became around 2000), there will be an extra tip of the hat or two. But since research reports are almost as stylized as Kabuki theater, a lot of formal structures will be mimicked, tending to the mean. Also, authors might have gone into exhaustive detail in their funding requests or during institutional reviews, but trimmed reference lists back for publication.
To bolster the speculation that form is the culprit here, it’s worth noting that the Annals of Internal Medicine itself places a cap of “75 or fewer bibliographic references” on authors in its instructions for authors. Other major medical journals — where most RCTs are published — also trim references routinely in order to meet page budgets.
Of course, the researchers could have asked. Surely many of the trialists being scrutinized via citation lists and databases like the Web of Science are still alive and kicking. Why not call them up and ask why they didn’t do a more extensive job of citing the literature? Were there good reasons? Were they purposely duplicitous? Were they inadvertently ignorant? Did they feel the reference list was relatively unimportant in the grand scheme of things?
Ultimately, it’s hard to know whether this trial demonstrates a weakness in research, flaws in research presentation, problems with editorial policies, or limitations around information availability — or all of the above.
It’s clear we can do better, but this study does little to illuminate the path forward.