Two clinicians who “utilize detoxification methods in their clinical practices” together with a third author have published an epic review in Translational Psychiatry claiming to evaluate the evidence for the involvement of environmental contaminants in autism. While we obviously want to limit contaminant exposure, autism doesn’t emerge here as the reason for doing so. Something that does emerge, however, looks a lot like bias and not much like systematic evaluation.
Given that the review is so wide-ranging (it’s really a laundry list of studies), I’ve focused here on a category that’s likely of greatest interest: ‘autism spectrum disorder (ASD) and toxicant exposures’. (The rest of the review features the same general problems I characterize below.)
Some potential candidates in this category have been investigated to the point of redundancy, including mercury (per the review: “the largest prospective study reported no significant association between methylmercury and ASD”). Others, like “air pollution,” are vague and variable, at best, involving studies with layers of mathematical contortions and conflicting conclusions that seem to cancel each other out. That doesn’t even include the gaping holes in the underlying assumptions, which I’ve addressed elsewhere.
As the authors of this review duly note but never quantify in a clear way (the reader must dig through thousands of words to pin down numbers), most of these investigations of chemical environmental factors and autism suffer limitations that include their retrospective nature, their reliance on parent memory (as opposed to data collected in real-time), and the lack of confirmation of ASD diagnoses. For example, for pesticides (which in some cases already have been banned or are being phased out; some of these studies are almost 40 years old), three prospective reports relied on parent-completed behavior checklists that aren’t autism specific. One of these studies identified a “trend” to a relationship between checklist scores and pesticide exposure, but using another developmental index, found improved scores with pesticide exposure. The air pollution findings are similarly contradictory between studies (e.g., one finds an ozone correlation; another does not; ditto for particulates smaller than 2.5 microns).
The review authors claim that “a majority of the studies examining a potential association between ASD and estimated environmental toxicant exposures reported a significant relationship.” They looked at 36 or 37 studies, for this category, depending on how you count them. Since they didn’t officially add it up for the reader, I will: Five were prospective, the type that is expected to yield the best evidence.
Of these five prospective studies, one identified conflicting relationships; the data generally didn’t involve confirmed ASD diagnoses; two are characterized as having a “positive” finding but showed only a “trend,” according to the authors; and one (and the largest and longest-running) found no association between the “toxicant” in question (methylmercury) and ASD. In other words, based on the five studies with the best (prospective) design, the evidence for an association of autism with toxicants is on life support.
The authors nevertheless take their “majority” claim a bit further and state that 92% of the studies “found a significant association between estimated environmental toxicant exposures.” This assertion gives each of these studies equal representation even though they are by no means created equal and in some cases, their results directly contradict one another. The authors repeat this 92% value four times in their paper, seemingly aware of what a compelling sound bite it makes. This tendency (trend?) to make all studies equivalent in relevance appears to be a habit of theirs, as they self-cite another review of theirs from 2012 (without noting their authorship) and claim that it shows 89% of the 190 cited papers “implicating (sic) an association (of toxicants) with ASD.”
A good systematic analysis with well-supported conclusions weights studies based on design (case, case-control, retrospective, prospective) and validity of evidence (based on factors such as population size, data collection, etc.), rather than simply calculating the percent allegedly reporting the result the authors want to see. In addition, strict inclusion criteria for systematic reviews are critical for such papers because, as the Cochrane Reviews website notes, without them:
… an individual or company might actively seek to discuss and combine only the research which supports their opinions, prejudices or commercial interests. In contrast, a Cochrane Review circumvents this by using a predefined, rigorous and explicit methodology.
Based on the methods that the current review authors describe, they applied no such systematic method for relevance or weighting of studies or for evaluating and presenting the collected data. Instead, they applied only general criteria, such as exclusion of animal studies, and don’t seem to have limited the coverage years–hence the presence of studies dating to the 1970s. It’s also strange that they found space to include a huge table of values for heavy metal content in hair and urine (which have been called into question), among other tissues, but didn’t tabulate the studies cited in this section of the review and clearly identify each by study type and relevance in a readily accessible way.
Furthermore, a trend is not significance, and a look at some of the odds or risk ratios of studies they say found a “positive” result shows less-than-impressive numbers. A closer look at many of the retrospective studies they cite—and I’ve taken that closer look at several, as have others—makes their limitations abundantly clear.
That didn’t stop the authors from concluding (again) that
Despite these limitations, the majority of the reviewed studies implicated multiple toxicants in ASD risk.
As a close examination of the presented studies indicates, that assertion seems an overstatement, at best, and given the limitations, self-contradictions within and among the cited reports, and overall mixed results, the conclusion of ‘implication’ doesn’t seem to follow. So, I took a look at the study authors, wondering what biases might have led to such a skewed presentation and conclusions and this insistence on “92%” as a valid measure.
Here’s what I found:
The review article funding came from the Autism Research Institute, begun by the late Bernard Rimland (vaccines-cause-autism proponent) and the origin of the controversial and now-defunct Defeat Autism Now! conferences and clinician registry.
The first and third authors of this review, Daniel Rossignol and Richard Frye, are scientific advisors to Jenny McCarthy’s Generation Rescue, a group committed to the idea that vaccines and mercury cause autism. Rossignol and second author Stephen Genuis declared the conflict of interest regarding using “detoxification methods” in their practices (those would be applied, presumably, for “metal toxicity” etc.). Each author has other interests in the conclusions from the review (including related to oft-mentioned metabolic disorders, among others).
Rossignol was a Defeat Autism Now! Doctor who tells parents on his clinic Website that children with an autism diagnosis “in many cases” will need to have a large number of tests ordered. These tests include analyses for lead and “increased heavy metal or pesticide levels in the kidney … markers of the metal burden in the body.” And, of course, at the top of his “Articles” page, you will find a link to this latest open-access review with its conclusions that such toxicants are implicated in autism. In addition to his “detox in clinical practice” disclosure, he’s a big fan of HBOT, which the FDA has warned us about.
We all bring bias to what we do. Avoiding this bias in science requires vigilance. Scientists have guidelines to follow for systematic reviews that aid in that vigilance, guidelines that get a mention in the Rossignol review but largely go unused, except for inclusion of a PRISMA flowchart. There’s a reason the Institute of Medicine, in its standards for systematic reviews for therapeutic medical and surgical interventions, states that these standards for authorship on a systematic review
require each team member to disclose potential COI and professional or intellectual bias; exclude individuals with a clear financial conflict; and exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users.
Not heeding such advice combined with an unscientific focus on a simplistic measure like “92%” does no favors for autism research. I’m sure that won’t stop the “92%” soundbite from making the rounds in certain circles and feeding the alt-med cottage industry around autism.