Hype over science: Does acupuncture really improve the chances of success for in vitro fertilization?
There it was on Friday greeting me on the ABC News website: “Study: Acupuncture May Boost Pregnancy” in bold blue letters, with the title of the webpage being “Needles Help You Become Pregnant.” The story began:
It sounds far-fetched sticking needles in women to help them become pregnant but a scientific review suggests that acupuncture might improve the odds of conceiving if done right before or after embryos are placed in the womb.
The surprising finding is far from proven, and there are only theories for how and why acupuncture might work. However, some fertility specialists say they are hopeful that this relatively inexpensive and simple treatment might ultimately prove to be a useful add-on to traditional methods.
By the end of the day, the story was all over the media, including radio, TV, news websites, the blogosphere, and various other outlets, all trumpeting the message that a scientific study says that acupuncture can help infertile couples conceive. Nary a skeptical word seemed to be found. Knowing very well just how far parents will go to conceive, I was curious: Did this study actually say what the media says it said? What was so new and radical about this study that it rated a press release and a lot of promotion? Do we here at SBM (particularly Steve) need to rethink our extreme skepticism about acupuncture, given the poor quality evidence and lack of even a glimmer of a convincing physiologic mechanism to explain its supposed activities?
First, let’s see what the ABC News story said about it:
The analysis was led by Eric Manheimer, a researcher at the University of Maryland School of Medicine, and paid for by a federal agency, the National Center for Complementary and Alternative Medicine. Results were published Friday in the British medical journal, BMJ.
Acupuncture involves placing very thin needles at specific points on the body to try to control pain and reduce stress. In fertility treatment, it is thought to increase blood flow to the uterus, relax the cervix and inhibit “fight or flight” stress hormones that can make it tougher for an embryo to implant, Manheimer said.
The analysis pools results from seven studies on 1,366 women in the United States, Germany, Australia and Denmark who are having in vitro fertilization, or IVF. It involves mixing sperm and eggs in a lab dish to create embryos that are placed in the womb.
Right away, this tells us that the study is not an original study at all, but rather an meta-analysis of existing studies. Moreover, its appearance on the very same day that my co-blogger Dr. Kimball Atwood IV explained how evidence-based medicine (EBM) was never designed to deal with so-called complementary and alternative medicine (CAM) because it does not take into account scientific prior probability is a fantastic coincidence. This study seems almost custom-made to illustrate his point. Before I examine the study itself, I’ll tell you what I mean.
Consider the use of acupuncture in the situation of infertility. Why would anyone think that it would do anything at all to aid acupuncture? Let’s see what the study authors themselves say:
Acupuncture has been used in China for centuries to regulate the female reproductive system. Three potential mechanisms for its effects on fertility have been postulated. Firstly, acupuncture may mediate the release of neurotransmitters, which may in turn stimulate secretion of gonadotrophin releasing hormone, thereby influencing the menstrual cycle, ovulation, and fertility. Secondly, acupuncture may stimulate blood flow to the uterus by inhibiting uterine central sympathetic nerve activity. Thirdly, acupuncture may stimulate the production of endogenous opioids, which may inhibit the central nervous system outflow and the biological stress response.
Note first the appeal to ancient knowledge, which is in itself a fallacy. After all, bloodletting and purging with toxic metals were also used for centuries as remedies; that does not mean that they actually did anything therapeutic. The rest is mostly speculation. For instance, the reference supporting the claim that acupuncture may stimulate bloodflow to the uterus was not looking at acupuncture but electroacupuncture, which is an entirely different beast. Moreover it was a small study (ten women), and it was not randomized and had no control group. Even so, given that electric current was passed through the lumbosacral area, this may well have stimulated nerves leading to the uterus. In other words, this study does not show that acupuncture itself does anything to uterine blood flow. If correct, it shows that passing an electric current through L4-S3 levels may stimulate uterine blood flow. (Indeed, it states in the paper itself that acupuncture points were chosen for innervation of the uterus.) Moreover, the reference used to justify the claim that endogenous opiods may be the mechanism behind an acupuncture effect is not even an acupuncture study. The bottom line, is that no plausible physiologic mechanism has been shown or proposed. As we now know, when clinical studies are done about a hypothesis with a very low prior plausibility, noise predominates, producing a disturbingly high proportion of seemingly “positive” studies.
Doing a meta-analysis of such studies can be a very effective means of amplifying that noise.
Let’s look at the meta-analysis itself. Perhaps the most unusual thing about studies of acupuncture for IVF is that, unlike most conditions for which acupuncture is claimed to work, the outcomes are not subjective. There’s not much in the way of interpretation. Outcomes include pregnancy, continued pregnancy (more than 12 weeks), and live births. Not present are the usual sorts of subjective measures that make the placebo effect such a concern in most studies requiring double-blinding. Basically, the authors scanned multiple databases for articles, including Medline, Embase, Cochrane Central, and the Chinese Biomedical Database. They also searched the proceedings of three major annual conferences on assisted reproduction for abstracts. Selection criteria were fairly stringent. A total of 108 initial studies were identified and then ultimately whittled down to only 7, with a total of 1,366 patients. Because the authors were looking at acupuncture as an adjuvant to embryo transfer, only trials in which acupuncture was administered within one day of the procedure were included. Outcomes on clinical pregnancy, ongoing pregnancy, or live birth had to be extractable. Unfortunately, no data regarding a number of factors that affect the chance of successful IVF were included, such as age, duration of infertility, number of previous attempts at IVF). So what did the authors find? Overall, three studies reported a benefit, three reported a statistically not significant trend towards a benefit, and one showed no benefit. In essence, the authors determined that, according to these studies, women undergoing acupuncture before IVF had a “relative risk” of pregnancy of 1.65 (95% confidence interval 1.24 to 2.14) compared to those who did not, 1.87 (95% confidence interval 1.40 to 2.49), and 1.91 (95% confidence interval 1.39 to 2.64).
Looking at the actual seven studies themselves, I note that three of them were not blinded. In other words, no sham acupuncture was administered to controls. The authors go through a lot of contortions and arguments to justify including studies that were not blinded, much less double-blinded on the basis that the placebo effect is much less likely to affect objective outcomes. I do not believe that this is a valid argument for two reasons. First, if nervousness during implantation of the embryo has any role in inhibiting successful implantation, a woman who believes she is getting an effective adjuvant may be more relaxed during the procedure and thus more likely to conceive. Indeed, there is suggestive evidence, albeit weak, that acupuncture may indeed produce relaxation, and there’s even an anecdote from the ABC News article:
Dr. Ann Trevino, a 37-year-old family physician who recently moved to Houston, is pregnant, and a believer. She had three unsuccessful pregnancy attempts with intrauterine insemination before trying acupuncture with IVF at a fertility clinic in San Antonio where she used to live.
“I had been reading about acupuncture, probably like every other patient on the Internet. I was just willing to do anything possible to improve our chances,” she said. With acupuncture, “I just felt very warm and relaxed” when the embryos were placed.
If it’s the case that either relaxation promotes implantation or nervousness inhibits it, then anything that results in relaxation during embryo placement would result in higher fertility rates. No needles would be needed. Second, and more importantly, the obstetrician implanting the embryos knows who is and is not getting acupuncture. There’s no way of knowing whether that might have affected how the procedure was done or the decision regarding the number of embryos to be implanted. The authors try to sidestep this important issue thusly:
Blinding of physicians performing the embryo transfer is another potential source of bias (performance bias), and three of the seven included trials did not blind the physicians.w1 w6 w7 Considering the cost of embryo transfer and the importance of successful transfers to maintaining high pregnancy rates at clinics, we think that physicians would be motivated primarily to perform a successful procedure for all patients, rather than to show that acupuncture, a non-proprietary treatment, is an effective adjuvant procedure. In subfertility trials in general, where outcomes are entirely objective, blinding of either patients or physicians is “infrequently attempted,”17 and such blinding components are not always considered as critical elements related to the evaluation of risk of bias.35
It is correct that reference 17 states that blinding is infrequently attempted. However, this is also what was said:
Blinding is important because trial personnel are naturally susceptible to hunches about the effectiveness of one or both trial treatments and only if they are blinded can anyone be confident that decisions and assessments are not affected by such intuitive influences. Double-blinding in assisted reproductive technology trials is infrequently attempted, and most gonadotrophin trials are either not blinded, or the outcome assessors are blinded to allocation as a reasonable compromise. It is logical to assume that double-blinding would bring about an increased confidence in clinical trial results and in theory this is simply a matter of making equivalent preparations for each drug. However, in reality the investigational drug would need to have indistinguishable primary packaging material compared to the approved comparator. This is very difficult to arrange and most likely would require new qualification studies. True double-blinding would be optimal, but in practice it remains very difficult.
Funny, but in oncology we manage to produce placebos that have packaging indistinguishable from that of the drug being studied. I wonder why this is so much more difficult in IVF. In any case, this reference does not support the contention that blinding of the treatment team to experimental group is not important in IVF. Moreover, reference 35, although its title states that it’s about design and statistics, in reality seems to be mainly about statistics and reporting rather than overall trial design. Moreover, this is what it says about blinding (or “concealment of allocation”):
Concealment of allocation has received increased attention since an empirical study by Schulz et al. (1995) found it to be one of the quality criteria for randomized trials most strongly associated with bias. Their study of 250 trials in the field of pregnancy and childbirth found 32% to be adequately concealed and 18% stating a specific method of randomization. Our figures of 34 and 64% respectively suggest an improvement in reporting of randomization over recent years, but demonstrate that there is still a widespread lack of appreciation of the importance of reporting allocation concealment.
This reference does not support the authors’ claim that double blinding is unimportant in IVF trials either. In fact, what it does suggest is that far too many studies of IVF, not just those looking at IVF and acupuncture, are poorly designed. Remember, it’s not an issue of whether or not the physicians are sufficiently motivated to want to do their best for their infertile patients. It’s about unconscious changes in procedure or clinical decision-making that can result from a failure to be blinded to the intervention.
One of the most telling conclusions of the study comes from the subgroup analyses. The authors found that out of nine subgroups studied, only one grouping mattered. When the analysis was restricted to three studies in which a higher rate of pregnancy was noted in the control group than the other studies, the “effect” of acupuncture virtually evaporated, and the relative risk fell to 1.24 (95% confidence range 0.86 to 1.77). This could suggest one of two things: either the “effect” (if effect there is) is greater in groups with lower success rates, or it could mean that noise and/or standardization in these groups is greater or more prone to bias.
Finally, we haven’t yet considered the issue of publication bias, which was recently discussed in the context of trials of antidepressant drugs on this very blog. One of the biggest problems in both conventional medicine, and probably even more so in “CAM,” is that positive studies that show a result are more likely to be published. They’re more interesting, and investigators (not to mention reviewers) are more excited about them. Negative studies are less likely to be published or, if published, are more likely to show up in lesser journals. The authors of the study under discussion even admit this, although they try to discount it.
I suppose it is possible that acupunture might increase IVF success rates, although there is no physiologically plausible reason for it to do so other than possibly the ability of acupuncturists to soothe and relax their patients. Even so, this study does not provide particularly compelling evidence to support such a conclusion. Certainly, it does not provide evidence worthy of the hype that it got after its release. A combination of failure to blind in most of the studies included coupled with the likelihood of publication bias serve to cast extreme doubt on the conclusion of this study. It also shows the peril of applying meta-analysis to CAM modalities. Here we have researchers going to huge effort to apply the scientific method to a hypothesis for which there is no plausible mechanism, which is based on a concept that has never been detected or measured (meridians through which life force, or qi, flows), and studies that are most likely dominated by noise and false positives of the type Dr. John Ioannidis warned us about. In other words, it’s a really long run for a really short slide.
Manheimer E, Zhang G, Udoff L, Haramati A, Langenberg P, Berman BM, Bouter LM. Effects of acupuncture on rates of pregnancy and live birth among women undergoing in vitro fertilisation: systematic review and meta-analysis. BMJ 2008;0(2008):bmj.39471.430451.BEv1 (7 February), doi:10.1136/bmj.39471.430451.BE.