This is perhaps the first real crack in the wall for the almost-universal use of the null hypothesis significance testing procedure (NHSTP). The journal, Basic and Applied Social Psychology (BASP), has banned the use of NHSTP and related statistical procedures from their journal. They previously had stated that use of these statistical methods was no longer required but can be optional included. Now they have proceeded to a full ban.
The type of analysis being banned is often called a frequentist analysis, and we have been highly critical in the pages of SBM of overreliance on such methods. This is the iconic p-value where <0.05 is generally considered to be statistically significant.
The process of hypothesis testing and rigorous statistical methods for doing so were worked out in the 1920s. Ronald Fisher developed the statistical methods, while Jerzy Neyman and Egon Pearson developed the process of hypothesis testing. They certainly deserve a great deal of credit for their role in crafting modern scientific procedures and making them far more quantitative and rigorous.
However, the p-value was never meant to be the sole measure of whether or not a particular hypothesis is true. Rather it was meant only as a measure of whether or not the data should be taken seriously. Further, the p-value is widely misunderstood. The precise definition is:
The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true.
Perhaps one of the greatest threats to the enterprise of Science-Based Medicine is research fraud and misconduct. Rigorous research methods can be used to minimize the effects of bias, but when those methods themselves are the problem there is no easy fix. Related to this is the need for transparency. When fraud or misconduct is uncovered it erodes confidence in the system because it provokes speculation about how much fraud and misconduct has not been uncovered.
A recent study published in JAMA looks at one aspect of this issue – reporting of misconduct uncovered by the FDA. The good news here is that FDA trials, those that will be used to apply to the FDA for approval of a drug, are carefully monitored and inspected by the FDA. This is an important quality control measure. When the FDA uncovers misconduct it takes steps to correct it. If the misconduct is severe enough then any data that is associated with the poor research practices will be excluded from the trial so as not to taint the results. Even an entire study can be disqualified if necessary.
The problem highlighted by the study is that there is no systematic way for the FDA to communicate its findings through the peer-reviewed literature. Tainted studies, or ones that require a correction or retraction (because the violations were discovered after publication) may therefore persist in the peer-reviewed literature without any indication of the uncovered misconduct.
The standard features of quackery are all there. Proponents of this particular therapy claim that a normal condition is a disease. They make false claims about the cause of this disease. They then charge thousands of dollars for their fake treatment to cure the fake disease, and claim success rates that are not backed by any statistics.
In this case the fake disease is homosexuality, for which there is now a solid consensus that it is a normal variation of human sexuality. The fake treatment is conversion therapy. Recently a New Jersey judge ruled that conversion therapists cannot claim that homosexuality is a disease or disorder. The Southern Poverty Law Center reports:
Superior Court Judge Peter F. Barsio Jr. found that it “is a misrepresentation in violation of [New Jersey’s Consumer Fraud Act], in advertising or selling conversion therapy services, to describe homosexuality, not as being a normal variation of human sexuality, but as being a mental illness, disease, disorder, or equivalent thereof.”
The judge also ruled that [New Jersey conversion therapy provider Jews Offering New Alternatives for Healing (JONAH)] is in violation of the Consumer Fraud Act if it offers specific success statistics for its services when “client outcomes are not tracked and no records of client outcomes are maintained” because “there is no factual basis for calculating such statistics.”
Image credit: Wellcome Images, Wellcome Library, London, via Wikimedia Commons.
Last week I gave a quick overview of standard treatment options for migraine, a severe form of recurrent headaches. As promised, this week I will address some common treatments for migraine that I don’t think are supported by the evidence.
Acupuncture is the CAM modality that, it seems to me, has infiltrated the furthest into mainstream medicine, including for the treatment of migraine. In fact the The American Headache Society includes acupuncture on its list of recommended treatments. The reason for this is that acupuncture proponents have been able to change the rules of clinical research so that essentially negative or worthless studies of acupuncture are presented as positive.
I reviewed the evidence for acupuncture and migraine previously, demonstrating the multiple problems with the acupuncture literature in general, and specifically acupuncture in migraines. Most studies suffer from at least one fatal flaw: they are not properly blinded, they do not include a control, they mix acupuncture with non-acupuncture variables (mostly including electrical stimulation in the treatment group), comparison groups are not adequately treated, they make multiple comparisons to maximize chance outcomes, or they are simply too small making them susceptible to all the usual problems of bias in research.
What we don’t see is a consistent and clinically-relevant effect in properly-controlled double-blind trials where the variables of acupuncture are isolated.
I am a headache specialist and so I receive many questions, through SBM, NeuroLogica or listeners of the Skeptic’s Guide to the Universe, about how to best treat headaches, or about a specific, often unusual, treatment. Migraines and severe headaches are very common. According to the latest statistics:
14.2% of US adults 18 or older reported having migraine or severe headache in the previous 3 months in the 2012 NHIS. The overall age-adjusted 3-month prevalence of migraine in females was 19.1% and in males 9.0%, but varied substantially depending on age. The prevalence of migraine was highest in females 18-44, where the 3-month prevalence of migraine or severe headache was 23.5%.
That means about 28 million Americans suffer from migraines. Percentages do vary from continent to continent, but not dramatically. Migraine, therefore, is a huge burden. Headaches can be debilitating when severe, and so also are a major source of lost productivity.
This will be a two-part series reviewing some of the options for treating migraines, focusing on science-based treatments in part I, and non-science-based treatments in part II. None of this is intended to give specific medical advice for any individual. If you have severe headaches you should consult your physician. I will simply be reviewing the evidence for various options, focusing on migraine specifically.
Caffeine, a common trigger for migraines and headaches
As we search for a logo for SBM or the SfSBM, Mark Crislip has been a strong advocate of using an image of Sisyphus, endlessly pushing a boulder up a hill only to have it roll back down again. It’s a bit too self-defeating to be enthusiastic about that suggestion, but it does reflect a common feeling among all of us here at SBM – promoting science can be a frustrating endeavor.
Our frustration reflects a broader phenomenon, that it is difficult to persuade people with facts and logic alone. People tend to prefer narrative, ideology, and emotion to facts. The high degree of scientific illiteracy in the culture presents another barrier.
In recent years psychologists have demonstrated experimentally what we have come to understand through personal experience, that people engage in a host of cognitive defense mechanisms to protect their beliefs from the facts. We jealously guard our world view and are endlessly creative in shielding it from refutation.
A recent series of experiments published by Friesen, Campbell, and Kay in the Journal of Personality and Social Psychology demonstrates that one strategy commonly used to protect our beliefs is to render them unfalsifiable, or at least incorporate unfalsifiable elements. (more…)
SBM frequently receives questions from readers asking for more information or even challenging our position on various topics. We make extensive efforts to answer such questions, since engaging with the public is one of the primary purposes of this blog. In fact, I specifically chose the blog format because of its interactive nature and the ability to rapidly respond to items in the news or being discussed publicly.
Sometimes it’s helpful to provide answers to questions in the form of its own post. I do this when the questions are common or explore some new or interesting angle of a topic. I am also more likely to engage when the questions are polite and genuine.
We recently received the following e-mail which meets all these criteria, so here is my response. I will reprint the e-mail in sections as I address each question.
I have the utmost respect for the scientific method, and we subscribe to the Skeptical Inquirer. I respect much of what your organization does, and I do not believe that Reiki or Therapeutic Touch is effective, unless the person receiving these therapies believe they work. However, your organization seems to go out of its way to disprove things like the benefit of organic produce which has less pesticides than conventional produce. You claim that natural pesticides could be just as harmful. Here are some examples of these natural pesticides: apply 1 tablespoon of canola oil and a few drops of ivory soap to the leaves of plants and vegetables to repel insects. Also, apply 2 TBSPS of hot pepper sauce with a few drops of ivory soap to leaves, use baking soda and water or pureed onions to repel insects. How can you claim that these innocuous substances are as harmful as conventional pesticides?
A newly-published review of neuroscience research looking at the predictive value of functional and anatomical imaging raises interesting questions about the role of such studies in learning, psychiatric treatment, and even the treatment of criminals. “Prediction as a Humanitarian and Pragmatic Contribution from Human Cognitive Neuroscience” by Gabrieli, Ghosh, and Whitfield-Gabrieli and published in Neuron, does a thorough job of explaining the current state of the research and pointing to where future research is needed.
The basic idea is to use noninvasive imaging to look at the structure or function of the brain as a way of predicting future behavior, and then using those predictions to help guide treatment and education interventions, and perhaps decisions regarding parole or further treatment of criminal behavior. This concept raises many issues, including the technology being used, the state of the research, the ultimate potential for this line of research, and ethical considerations.
The major question underlying this entire endeavor is, to what extent is brain anatomy and function destiny? (more…)
3D model of the molecular structure of glyphosate.
There is an ideological subculture that is motivated to blame all the perceived ills of the world on environmental factors and corporate/government malfeasance. Often this serves a deeper ideological drive, which can be anti-vaccine, extreme environmentalism, or anti-GMO. The latest environmental bogeyman making the rounds is glyphosate, which is being blamed for (you guessed it) autism.
Glyphosate is the active ingredient in the herbicide Roundup. It has been widely used for about 40 years, and with the introduction of GM crops that are Roundup resistant, its use has increased significantly in the last 20 years. It has therefore become a popular target for anti-GMO fearmongering.
Glyphosate is one of the least toxic herbicides used. It inhibits the enzyme 5-enolpyruvylshikimic acid-3-phosphate synthase which interferes with the shikimic pathway in plants, resulting in the accumulation of shikimic acid in plant tissues and ultimately plant death. The enzyme and pathway do not exist in animals, which is why toxicity is so low. Still, chemicals can have multiple effects and so toxicity needs to be directly measured and its epidemiology studied. (more…)
It’s always preferable to have objective empirical evidence to inform an opinion, rather than just subjective impressions. Confirmation bias will make it seem as if the facts support your opinion, even when they don’t. Of course, when objective evidence (such as published studies) does seem to support your position, you still have to keep your critical shields up. Confirmation bias can still kick in, resulting in cherry-picking favorable evidence, finding fault with studies whose conclusions you don’t like, and too-easily accepting those that confirm your position.
I therefore had to be careful in evaluating the following study from the BMJ, because it nicely confirms what I and many others here at SBM have been saying for years – recommendations made by TV doctors, particularly Dr. Oz, are unreliable and insufficiently based on evidence.
This was a prospective study that:
…randomly selected 40 episodes of each of The Dr Oz Show and The Doctors from early 2013 and identified and evaluated all recommendations made on each program.