Believe it or not, there was one area of so-called “alternative” medicine that I used to be a lot less skeptical about than I am now. Homeopathy, I always realized to be a load of pseudoscientific magical thinking. Ditto reiki, therapeutic touch, and other forms of “energy healing.” It didn’t take an extensive review of the literature to figure that out, although I did ultimately end up doing fairly extensive literature reviews anyway. Then, the more I looked into the hodge-podge of “healing” modalities whose basis is not science but rather prescientific and often mystical thought, the less impressed I was.
Even so, there was always one modality that I gave a bit of a pass to. There was one modality that, or so I thought, might actually have something to it. There was one modality that seemed to have a bit of suggestive evidence that it might do something more than a placebo. I’m referring to acupuncture. No, I never bought all the mystical mumbo-jumbo about how sticking needles into “meridians” somehow alters or “unblocks” the flow of a mysterious “life force” known as qi that is undetectable by science. I did wonder if perhaps it worked as a counterirritant or by releasing endorphins.
Then I actually started paying attention to the scientific literature regarding acupuncture, including literature like this and this. The more I read, the more I realized something. I realized that there was far less to acupuncture than I had previously thought, and, even with my previous openness to it, I hadn’t thought all that much about it anyway. What I had thought about it was that it might have a very mild beneficial effect. What I know now is that acupuncture is almost certainly no more than an elaborate placebo. What I know now is that virtually every study of acupuncture claiming to show a positive effect has serious methodological flaws and that the better-designed the study the less likely there is to be an effect. What I now know is that any study without a true “sham” acupuncture arm is worthless, and that well-designed studies show “sham” acupuncture to be no different than “real” acupuncture; i.e., no different than placebo.
And now comes yet another in a long line of studies that is consistent with just that, and, worse, it’s billed as (and probably is) the “largest randomized study of acupuncture ever done.” Too bad it depends on what you mean by “randomized.” Too bad the press coverage misses the point:
For the current study, published in the journal Cephalalgia, German researchers followed more than 15,000 adults with chronic headaches; all had been suffering from either migraine or tension-type headaches at least twice a month for 1 year or more.
Of these patients, nearly 3,200 agreed to be randomly assigned to either have acupuncture added to their regular therapy or to stay with their usual care alone. The rest of the patients began on acupuncture treatment.
All of the acupuncture patients received up to 15 sessions over 3 months, and all study patients were reassessed after 6 months.
In the end, the study found, acupuncture patients reported greater pain improvements than those who stayed with their usual care only. At the outset, they reported an average of 8.4 headache days over 3 months; that dropped to 4.7 by the study’s end.
Take a minute here. If you’re a regular reader of this blog, I’m betting that you can pick out the huge methodological flaw in this study from just the press report alone. Do you have it yet? Don’t worry, I’ll get to it very soon. However, I don’t like to rely on just the news coverage of such a study. Whenever possible, I always like to go to the original study, and, as I usually do, I did just that. Here’s the abstract:
We aimed to investigate the effectiveness of acupuncture in addition to routine care in patients with primary headache (> 12 months, two or more headaches/month) compared with treatment with routine care alone and whether the effects of acupuncture differ in randomized and non-randomized patients. In a randomized controlled trial plus non-randomized cohort, patients with headache were allocated to receive up to 15 acupuncture sessions over 3 months or to a control group receiving no acupuncture during the first 3 months. Patients who did not consent to randomization received acupuncture treatment immediately. All subjects were allowed usual medical care in addition to study treatment. Number of days with headache, intensity of pain and health-related quality of life (SF-36) were assessed at baseline, and after 3 and 6 months using standardized questionnaires. Of 15 056 headache patients (mean age 44.1 ± 12.8 years, 77% female), 1613 were randomized to acupuncture and 1569 to control, and 11 874 included in the non-randomized acupuncture group. At 3 months, the number of days with headache decreased from 8.4 ± 7.2 (estimated mean ±s.e.) to 4.7 ± 5.6 in the acupuncture group and from 8.1 ± 6.8 to 7.5 ± 6.3 in the control group (P < 0.001). Similarly, intensity of pain and quality of life improvements were more pronounced in the acupuncture vs. control group (P < 0.001). Treatment success was maintained through 6 months. The outcome changes in non-randomized patients were similar to those in randomized patients. Acupuncture plus routine care in patients with headache was associated with marked clinical improvements compared with routine care alone.
Got it yet? Here’s the study design graphically shown:
Got it yet?
I’m sure that most of you do; so I’ll just move on. This study has three design flaws so glaring that I almost don’t even care what it shows because the flaws are so significant that they scuttled the study before it even started. Here they are:
- The study is not only not double-blind, it’s not even blinded in any way. Both the patients and the health care practitioners know who is receiving what therapy. That alone makes its result entirely explainable by placebo effects.
- There isn’t even an attempt at a sham acupuncture group. Remember my previous posts on the importance of sham acupuncture and how sham acupuncture is indistinguishable from “real” acupuncture.
- The “randomization” isn’t even really a randomization. Of 15,056 patients with a complaint of headache, only 3,404 accepted randomization to control or acupuncture. Normally a clinical investigator, when faced with this situation, studies only the patients who agreed to be randomized. Not these intrepid woo-mavens! They included the remaining 11,652 patients anyway. Actually, they included 11,874 nonrandomized patients.
There are a number of other flaws, but, really, they pale in significance to the three above and are hardly worth mentioning except in passing. For example, there were significant differences between the randomized and nonrandomized groups, including higher pain intensity and a shorter duration of chronic headaches, making them prime candidates to be prone to regression to the mean. The study is also suspect because it lumps together all headaches, rather than separating out the migraine headaches, which have a different physiological mechanism behind them than run-of-the-mill headaches. Another problem was that the authors relied on questionnaires, rather than a pain diary. Because a questionnaire relies on patient memory, rather than the patient writing down an incident as it happens, it’s prone to recall bias. There are also numerous other nits to pick, but none of them even come close to the three flaws listed above.
Regarding the failure to use a sham acupuncture group or to blind, it makes me wonder if all the studies coming out showing that sham acupuncture and “real” acupuncture are indistinguishable are starting to get to acupuncture advocates to the point where they really aren’t even trying anymore. After all, an unblinded study is almost guaranteed to produce an effect, but an investigator has no way of knowing whether that effect is greater than placebo if there is no valid placebo group. True, the authors did all sorts of fancy statistics and handwaving to try to take the reader’s mind off of this fundamental fatal flaw, but none of that changes anything.
Reading the discussion is very instructive, as it demonstrates very well the torturing of language and logic that is used by advocates of “alternative” medicine. While they admit flat out that this was an unblinded study and that , the excuse used was that this was a “pragmatic” study designed “chosen to reflect general medical practice.” Yes, that’s a great excuse not to do the necessary placebo/sham acupuncture control that would make the results of this study interpretable. It’s also a lovely excuse to allow patients in essence to self-select for acupuncture by refusing randomization, thus making the likelihood of a placebo effect even greater–except that there’s no control that allow us to know if it’s just a placebo effect or not. Here’s part of what they argue:
Although differences with respect to both baseline characteristics and treatment outcomes between randomized and non-randomized patients were small in absolute numbers, our findings indicate that randomization was associated with some selection effects. Therefore, the use of study designs that also include non-randomized patients appears to be desirable.
It is of note, however, that treatment benefits were similar in the randomized and nonrandomized acupuncture groups after adjusting for baseline differences. This suggests that the results of randomized trials can be representative of routine medical care situations, at least in large pragmatic studies.
No, it suggests that the placebo effect was operative in both the randomized patients and the patients who refused to be randomized. The use of study designs that include non-randomized patients is only “desirable” if you want to maximize the chances of a seemingly positive result. Bravo, though, to the study authors for having the chutzpah to try to change this study’s most glaring weakness into a strength. It was a nice try, but it won’t fly.
What’s truly depressing is the editorial by Dr. H-C Diener of the University Hospital Essen in Germany, where he actually makes this argument:
Despite the fact that I have major design issues with the study, my view is that studies like this have to be published in high ranked journals to promote discussion on trial design in non-drug trials
Dr. Diener actually lists two of the same flaws that I did, namely the unblinded nature of the study and the lack of adequate “sham” acupuncture controls. In spite of this, he still argues that this article should be published in high ranked journals “to promote discussion”? Funny, but I always thought that high ranked journals are high ranked because they post the most scientifically sound and medically interesting articles. Think New England Journal of Medicine. I always thought the reason for such journals to publish an article is because it is scientifically sound and studies a clinically important and/or interesting question. As a reason to publish such an article, “to promote discussion” is about as far down on the list of reasons as “because the author has nice hair.”
The bottom line is that this study is yet another of a long line of studies of “complementary and alternative” medicine that are entirely consistent with the placebo effect. Worse, it didn’t even really try to distinguish between a treatment effect and placebo effect. Maybe that’s the point. Whatever the point was, what I do know is that if I were a German citizen, I’d be mightily pissed off that so much money was wasted on this study. I’d also wonder why it was a consortium of insurance companies who funded the study. Maybe I was wrong about insurance companies funding woo. Maybe it is cheaper in the long run for them to pay for CAM than actual scientific medicine that’s more than just a highly elaborate placebo.
REFERENCES:
1. Jena, S., Witt, C., Brinkhaus, B., Wegscheider, K., Willich, S. (2008). Acupuncture in patients with headache. Cephalalgia, 28(9), 969-979. DOI: 10.1111/j.1468-2982.2008.01640.x
2. (2008). Acupuncture for the treatment of headaches: more than sticking needles into humans?. Cephalalgia, 28(9), 911-913. DOI: 10.1111/j.1468-2982.2008.01650.x
