Oops, they did it again.
You think the media would learn after the last time, but no….
There it was on Friday greeting me on the ABC News website: “Study: Acupuncture May Boost Pregnancy” in bold blue letters, with the title of the webpage being “Needles Help You Become Pregnant.” Wow, what a claim! Naturally, I had to know more. Fortunately (or unfortunately, depending on your point of view) The story began:
It sounds far-fetched sticking needles in women to help them become pregnant but a scientific review suggests that acupuncture might improve the odds of conceiving if done right before or after embryos are placed in the womb.
The surprising finding is far from proven, and there are only theories for how and why acupuncture might work. However, some fertility specialists say they are hopeful that this relatively inexpensive and simple treatment might ultimately prove to be a useful add-on to traditional methods.
It wasn’t long before multiple media outlets, blogs, and websites were trumpeting the message that a real, honest-to-goodness scientific study has shown that acupuncture really can help infertile couples conceive. This was big, and nary a skeptical word seemed to be found. Knowing all too well just how far parents will go to conceive, I had to know: Did this study actually say what the media says it said? What was so new and radical about it that it rated so much promotion on a Friday?? Should I start to rethink my extreme skepticism about whether acupuncture is anything more than a placebo, given the poor quality evidence generally used to support its supposed efficacy in all sorts of pathphysiologically related unrelated conditions and the lack of even a glimmer of a convincing physiologic mechanism to explain its supposed activities?
Here’s what ABC said about it:
The analysis was led by Eric Manheimer, a researcher at the University of Maryland School of Medicine, and paid for by a federal agency, the National Center for Complementary and Alternative Medicine. Results were published Friday in the British medical journal, BMJ.
Acupuncture involves placing very thin needles at specific points on the body to try to control pain and reduce stress. In fertility treatment, it is thought to increase blood flow to the uterus, relax the cervix and inhibit “fight or flight” stress hormones that can make it tougher for an embryo to implant, Manheimer said.
The analysis pools results from seven studies on 1,366 women in the United States, Germany, Australia and Denmark who are having in vitro fertilization, or IVF. It involves mixing sperm and eggs in a lab dish to create embryos that are placed in the womb.
Right away, I knew that the study is not an original study at all. Instead, it’s a meta-analysis of existing studies. Dr. Kimball Atwood IV has made a strong case that evidence-based medicine (EBM) was never designed to deal with so-called complementary and alternative medicine (CAM) because EBM< does not take into account scientific prior probability. Consequently, it may never have occurred to the originators of the concept of the "hierarchy of evidence" that basic science evidence should rate higher than level 5, one of the lower levels. Sadly, this meta-analysis seems almsot custom-made to illustrate his point. Before I examine the study itself, I'll tell you what I mean. Consider the use of acupuncture in the situation of infertility. Why would anyone think that it would do anything at all to aid acupuncture? Let's see what the study authors themselves say:
Acupuncture has been used in China for centuries to regulate the female reproductive system. Three potential mechanisms for its effects on fertility have been postulated. Firstly, acupuncture may mediate the release of neurotransmitters, which may in turn stimulate secretion of gonadotrophin releasing hormone, thereby influencing the menstrual cycle, ovulation, and fertility. Secondly, acupuncture may stimulate blood flow to the uterus by inhibiting uterine central sympathetic nerve activity. Thirdly, acupuncture may stimulate the production of endogenous opioids, which may inhibit the central nervous system outflow and the biological stress response.
I really hate it when people claiming to be scientists appeal ancient knowledge in this manner. It’s a particularly annoying fallacy. Yes, it might mean that there’s something to acupuncture, but it’s even more likely to be meaningless. After all, bloodletting and purging with toxic metals were also used for centuries as remedies; that does not mean that they actually did anything therapeutic. The rest is mostly speculation. For instance, the reference supporting the claim that acupuncture may stimulate bloodflow to the uterus was not looking at acupuncture but electroacupuncture, which is most definitely not acupuncture. They didn’t have electricity (at least as far as we know) in ancient China. Including electricity, particularly in acupuncture studies looking at pain relief, merely mimics transcutaneous electrical stimulation (TENS), which is not an “alternative” therapy at all, but an accepted “conventional” therapy supported by EBM. Moreover the reference was a small study (ten women). Worse, it was not randomized and had no control group. Even so, given that electric current was passed through the lumbosacral area, it may well have stimulated nerves leading to the uterus. In other words, this study does not show that acupuncture itself does anything to uterine blood flow. If its results are correct, it shows that passing an electric current through L4-S3 levels may stimulate uterine blood flow. (Indeed, it states in the paper itself that acupuncture points were chosen for innervation of the uterus.) Moreover, the reference used to justify the claim that endogenous opiods may be the mechanism behind an acupuncture effect is not even an acupuncture study. The bottom line, is that no plausible physiologic mechanism has been shown or proposed. As we now know, when clinical studies are done about a hypothesis with a very low prior plausibility, noise predominates, producing a disturbingly high proportion of seemingly “positive” studies.
Doing a meta-analysis of such studies can be a a spectacular method of amplifying that noise. So what about the meta-analysis itself? Does it deserve a dose of Respectful Insolence™? I thinkl you know the answer to that one.
Perhaps the most unusual thing about studies of acupuncture for IVF is that, unlike most conditions for which acupuncture is claimed to work, the outcomes are not subjective. They are not shades of gray, but yes-no outcomes. These include pregnancy, continued pregnancy (more than 12 weeks), and live births. Not present are the usual sorts of subjective measures of pain or other measures that make the placebo effect such a concern in most acupunctures studies. That at least makes data comparison between studies easier. Basically, the authors scanned multiple databases for articles, including Medline, Embase, Cochrane Central, and the Chinese Biomedical Database. They also searched the proceedings of three major annual conferences on assisted reproduction for abstracts. Selection criteria were fairly stringent. A total of 108 initial studies were identified and then ultimately whittled down to only 7, with a total of 1,366 patients. Because the authors were looking at acupuncture as an adjuvant to embryo transfer, only trials in which acupuncture was administered within one day of the procedure. Outcomes on clinical pregnancy, ongoing pregnancy, or live birth had to be extractable. Unfortunately, no data regarding a number of factors that affect the chance of successful IVF were included, such as age, duration of infertility, number of previous attempts at IVF). So what did the authors find? Overall, three studies reported a benefit, three reported a statistically not significant trend towards a benefit, and one showed no benefit. In essence, the authors determined that, according to these studies, women undergoing acupuncture before IVF had a “relative risk” of pregnancy of 1.65 (95% confidence interval 1.24 to 2.14) compared to those who did not, 1.87 (95% confidence interval 1.40 to 2.49), and 1.91 (95% confidence interval 1.39 to 2.64).
Looking at the actual seven studies themselves, I can’t help but note that three of them were not blinded. In other words, no sham acupuncture was administered to the control group. The authors twist themselves into pretzels in the discussion, going through multiple contortions and excuses to justify including studies that were not blinded, much less double-blinded on the basis that the placebo effect is much less likely to affect objective outcomes. I do not accept this as a valid argument for two reasons. First, if nervousness during implantation of the embryo has any role in inhibiting successful implantation, a woman who believes she is getting an effective adjuvant may be more relaxed during the procedure and thus more likely to conceive. Indeed, there is some suggestive evidence, albeit weak, that acupuncture may indeed produce relaxation, and there’s even an anecdote from the ABC News article:
Dr. Ann Trevino, a 37-year-old family physician who recently moved to Houston, is pregnant, and a believer. She had three unsuccessful pregnancy attempts with intrauterine insemination before trying acupuncture with IVF at a fertility clinic in San Antonio where she used to live.
“I had been reading about acupuncture, probably like every other patient on the Internet. I was just willing to do anything possible to improve our chances,” she said. With acupuncture, “I just felt very warm and relaxed” when the embryos were placed.
As an aside, there is one thing really annoying about this anecdote that makes me fear for the critical thinking skills among my fellow physicians, and that’s the fact that Dr. Trevino is comparing apples and oranges, namely intrauterine insemination (which is basically the same thing as artificial insemination) with IVF. If she has something wrong that makes conception via sperm and egg meeting in the Fallopian tube impossible or very unlikely, then of course IVF would be more likely to work. In either case, the two are different procedures, and often the reason fertility specialists proceed to IVF is because artificial insemination has been tried multiple times and has failed. Be that as it may, if it is true that either relaxation promotes implantation or nervousness inhibits it, then anything that results in relaxation during embryo placement would result in higher fertility rates. No needles would be required, just an effective strategy to induce relaxation in the woman before she undergoes embryo transfer.
What’s worse about the inclusion of studies that were not double-blinded, however, is that in such studies not only does the patient know whether she got acupuncture or not but the OB/GYN implanting the embryos also knows. He or she knows who is and is not getting acupuncture beforehand. There’s no way of knowing whether such knowledge might affect how the procedure is done or the decision regarding the numbers of embryos to be implanted in each woman. The authors try unsuccessfully to sidestep this crucially important issue thusly:
Blinding of physicians performing the embryo transfer is another potential source of bias (performance bias), and three of the seven included trials did not blind the physicians.w1 w6 w7 Considering the cost of embryo transfer and the importance of successful transfers to maintaining high pregnancy rates at clinics, we think that physicians would be motivated primarily to perform a successful procedure for all patients, rather than to show that acupuncture, a non-proprietary treatment, is an effective adjuvant procedure. In subfertility trials in general, where outcomes are entirely objective, blinding of either patients or physicians is “infrequently attempted,”17 and such blinding components are not always considered as critical elements related to the evaluation of risk of bias.35
It is correct that reference 17 states that blinding is infrequently attempted. However, this is also what was said in the very same article the authors cite as evidence that blinding of the health care team is not that important:
Blinding is important because trial personnel are naturally susceptible to hunches about the effectiveness of one or both trial treatments and only if they are blinded can anyone be confident that decisions and assessments are not affected by such intuitive influences. Double-blinding in assisted reproductive technology trials is infrequently attempted, and most gonadotrophin trials are either not blinded, or the outcome assessors are blinded to allocation as a reasonable compromise. It is logical to assume that double-blinding would bring about an increased confidence in clinical trial results and in theory this is simply a matter of making equivalent preparations for each drug. However, in reality the investigational drug would need to have indistinguishable primary packaging material compared to the approved comparator. This is very difficult to arrange and most likely would require new qualification studies. True double-blinding would be optimal, but in practice it remains very difficult.
Funny, but in oncology somehow we manage to produce placebos that have packaging indistinguishable from that of the drug being studied. True, it’s not always easy to accomplish, but still we manage it pretty well. I wonder why this is so much more of a problem in studies of IVF. In any case, this reference does not support the contention that blinding of the treatment team to experimental group is not important in IVF> Moreover, reference 35, although its title states that it’s about design and statistics, in reality seems to be mainly about statistics and reporting rather than overall trial design. It, too, mentioned double-blinding, but under the term “concealment of allocation”:
Concealment of allocation has received increased attention since an empirical study by Schulz et al. (1995) found it to be one of the quality criteria for randomized trials most strongly associated with bias. Their study of 250 trials in the field of pregnancy and childbirth found 32% to be adequately concealed and 18% stating a specific method of randomization. Our figures of 34 and 64% respectively suggest an improvement in reporting of randomization over recent years, but demonstrate that there is still a widespread lack of appreciation of the importance of reporting allocation concealment.
You guessed it. This reference does not support the authors’ claim that double blinding is unimportant in IVF trials either. In fact, what it does suggest is that far too many studies of IVF, not just those looking at IVF and acupuncture, are poorly designed and/or poorly reported. Remember, when it comes to double blinding it is not an issue of whether or not the physicians are sufficiently motivated to want to do their best for their infertile patients. It’s about unconscious changes in procedure or clinical decision-making that can result from a failure to be blinded to the intervention.
One of the most telling conclusions of the study comes from the subgroup analyses. The authors found that out of nine predefined subgroups examined, only one grouping mattered. When the analysis was restricted to three studies in which a higher rate of pregnancy was noted in the control group than the other studies, the “effect” of acupuncture virtually evaporated, and the relative risk fell to 1.24 (95% confidence range 0.86 to 1.77). This could suggest one of at least two things: either the “effect” (if effect there is) is greater in groups with lower success rates, or it could mean that noise and/or standardization in these groups is greater or more prone to bias.
Finally, we haven’t even considered the issue of publication bias, which was recently discussed in the context of trials of antidepressant drugs. One of the biggest problems in both conventional medicine, and probably even more so in “CAM,” is that positive studies that show a result are more likely to be published. They’re more interesting, and investigators (not to mention reviewers) are more excited about them. Negative studies are less likely to be submitted for publication or published. Moreover, if published, they tend to show up in lesser journals than they might otherwise have if positive. The authors of the study under discussion even admit this as a possibility, although they try very hard poo-poo it away.
I suppose it is possible that acupunture might increase IVF success rates, although there is no physiologically plausible reason for it to do so other than possibly the ability of acupuncturists to soothe and relax their patients prior to and during the procedure. Given its shortcomings, this study does not provide particularly compelling evidence to support the use of acupuncture as an adjunct to embryo transfer during IVF to increase success rates. Certainly, it does not provide evidence worthy of the hype that it received shortly after its release. A combination of failure to blind in most of the studies included coupled with the likelihood of publication bias serve to cast extreme doubt on the conclusion of this study.
Another disturbing aspect of this study is that it also shows the peril of applying meta-analysis to CAM modalities. Here we have researchers going to huge effort to apply the scientific method and rigorous statistical analysis to a hypothesis for which there is no plausible mechanism, which is based on a concept that has never been detected or measured (meridians through which life force, or qi, flows), and studies that are most likely dominated by noise and false positives of the type Dr. John Ioannidis warned us about. In other words, trying to do a meta-analysis on woo is most definitely a long run for a really short slide.
