It’s always disappointing to see a good journal fall for bad medicine, particularly when it’s in your field. For example, the Journal of Clinical Oncology (affectionately referred to by its abbreviation JCO) is the official journal of the American Society of Clinical Oncology (ASCO) and probably the most read clinical journal by those involved in the clinical care of cancer patients. Just as most oncologists, surgeons, and radiation oncologists who specialize in the care of cancer patients belong to ASCO, most of them also at least peruse JCO on a regular basis because major results of large cooperative group trials are often published there. Basically, everyone wants to publish his oncology clinical trial in JCO. Getting into Clinical Cancer Research is good, but getting in JCO is great.
So when I see a study that some of you have been e-mailing and Tweeting at me in JCO, I can’t help but wonder what the hell happened to peer review at the journal. I also realize that, being in the biz myself, by criticizing the peer review at JCO I might endanger my own chances of ever publishing in there given that my meatspace identity isn’t exactly a particularly well-kept secret (nor is it meant to be anymore). Oh, well. It needs to be said. Earlier this week there appeared in the Early Release Articles a study by Jun J. Mao and colleagues at the University of Pennsylvania entitled Electroacupuncture Versus Gabapentin for Hot Flashes Among Breast Cancer Survivors: A Randomized Placebo-Controlled Trial. Sounds neat, right? Sounds like a head-to-head battle between every acupuncturist’s version of transcutaneous electrical nerve stimulation (TENS), electroacupuncture (EA), versus gabapentin (a.k.a. Neurontin) in a battle royale to see which can alleviate hot flashes the best.
And make no mistake, hot flashes are an incredibly important problem in oncology, particularly for breast cancer patients. Patients who have breast cancer that is estrogen receptor positive, which means that the cells are responsive to estrogen, are frequently treated with drugs that block estrogen action. If a woman is postmenopausal, the usual drug used is a member of a class of drugs called aromatase inhibitors. If the woman is premenopausal, the usual drug used is Tamoxifen. In either situation, the most common—and frequently the most troubling—side effect of the drug is menopausal symptoms, mainly hot flashes. What a lot of people who aren’t cancer doctors or patients don’t understand is that these hot flashes can be horrible. Believe it or not, I learned this lesson with the very first patient I treated as an attending surgeon, lo these many years ago. This woman had such horrendous hot flashes from her Tamoxifen, as well as psychological symptoms, that she and her oncologist, after attempts at dose reductions and other strategies to ameliorate her symptoms, jointly decided that she couldn’t take it any more. Unfortunately, for patients with hormone-responsive breast cancers, the most bang for the buck in terms of preventing recurrence comes from the hormonal therapy, at least equal to that of any chemotherapy. So the degradation of quality of life and sometimes even symptoms necessitating cessation of hormonal therapy are a major problem in oncology. Unfortunately, our existing treatments are not the greatest, which opens the door to woo such as acupuncture. Where there’s woo, there’s quackademic medicine to study the woo, and that’s just what this study is.
Basically, this study is a bait-and-switch. I realize that I just discussed another bait-and-switch acupuncture study less than a week ago, but sometimes a doc’s gotta do what a doc’s gotta do.
So where’s the bait and switch? In this case, it’s on more levels than you think. In fact, before I dive into the story itself, let’s look at the spin being put on the results. I like to call this how not to report on an acupuncture study:
Needles beat pills for treating hot flashes in breast cancer survivors, according to a new trial that compared acupuncture, “sham” acupuncture, the medication gabapentin and a placebo pill.
Interestingly, sham acupuncture came in second place for effectiveness, the researchers said.
Furthermore, the effects of acupuncture were “significant and enduring for hot flashes while gabapentin’s effect only happened when a patient was taking the medication,” said study first author Dr. Jun Mao, an associate professor of family medicine and community health at the University of Pennsylvania in Philadelphia.
The study was published Aug. 24 in the Journal of Clinical Oncology.
As usual, the reporting that “acupuncture beat pills for hot flashes” is a completely inaccurate assessment of what the study actually showed. The study showed nothing of the sort. Barring an incredibly strong effect, it couldn’t possibly have (more on that later). Sadly, dissecting the BS attached to acupuncture studies is a rather specialized skill set; so perhaps I’m being too harsh. There were others who fell for this. When this study was presented as an abstract at the San Antonio Breast Cancer Symposium in December, similar stories were written about it. Of course, given the depth and breadth of science presented at the SABCS, it never ceases to amaze me how the press always seems to zero in on studies like this one, which tell us very little but give the appearance that acupuncture works.
Acupuncture is, of course, a theatrical placebo. Its effects are nonspecific, which is why it is such an adaptable placebo. As I joked sarcastically last time, if you believe its practitioners and adherents, acupuncture can treat almost any disease or health problem to which human beings are prone, which is why last week I could be blogging about a bait and switch study of acupuncture to treat hypertension and this week I can be writing about a bait and switch study of acupuncture to treat hot flashes caused by hormonal treatment in breast cancer patients. That’s why several years ago I was writing about worthless acupuncture studies examining acupuncture for hot flashes.
This study, in fact, shares a major flaw with the one I blogged about in 2008. Specifically, it has no control group that received no treatment at all. For a study that goes on and on about placebo effects and how acupuncture provides more of a placebo effect than pills, that’s a rather glaring omission. So let’s look at the study design. Basically it’s a four arm study, two arms of which look at electroacupuncture (EA) versus acupuncture and the other two at gabapentin versus a placebo. Interestingly in this study compared to most electroacupuncture studies, the EA is considered the “real” acupuncture and the standard acupuncture (SA) is described as the “sham,” which, I have to admit, is not inappropriate, given that acupuncture is a sham and electroacupuncture is basically transcutaneous electrical nerve stimulation rebranded as the sham that is acupuncture. (We all know that those ancient Chinese must have had access to electricity, right?) It is funny, though, because the SA group did use retractable needles and really were sham acupuncture. Consequently, the investigators weren’t really comparing EA versus the same acupuncture without the current but rather EA to at true sham.
So basically, the groups were as follows: electroacupuncture (EA); standard acupuncture (SA), gabapentin (GP), and placebo pill (PP). There were 120 patients with stage I to III breast cancer treated for cure at the Abramson Cancer Center of the Hospital of the University of Pennsylvania in Philadelphia and with no evaluable disease (i.e., no detectable disease left after treatment) and at least two hot flashes per day for at least a month. They excluded patients with stage IV disease, active breast cancer treatment, initiation or change in hormonal adjuvant therapy within the past 4 weeks, plans to initiate or change hormonal treatment in the coming 14 weeks, pregnancy or breast feeding, bleeding disorders or use of warfarin/heparin, an allergy to or previous use of GP for hot flashes, current use of an anticonvulsant, or documented renal failure in the past 12 months. They were then randomly assigned to one of the four groups according to this schema (click to embiggen):
In any case, the EA and SA groups received their “acupuncture” treatment for 30 minute sessions, twice a week for two weeks, then once a week for six more weeks. Meanwhile, in the GP versus PP groups, the GP group received 900 mg per day of gabapentin or placebo for eight weeks. Outcomes were measured once a week using the average hot flash composite score (HFCS) as measured by the Daily Hot Flash Diary, a commonly used instrument to measure hot flash severity and frequency. Adverse events were also monitored. Patient outcomes were followed for 24 weeks.
Of course, in any acupuncture study, blinding is of critical importance; so whenever you see an acupuncture study, look at the blinding. In this case, monitors blinded to treatment group tallied adverse events (AEs). That’s good. The patients were blinded to treatment group between EA and SA and between GP and PP, but were not blinded to whether they fell into one of the acupuncture groups or the medication groups. So was the principal investigators, study investigators, study staff, and the statistician. The acupuncturists, however, were not blinded to whether the patients received EA or SA, which is another huge flaw, almost as bad as the lack of a no treatment control group. In fact, it’s bad that they didn’t even blind patients between the acupuncture and pill groups. All that would have required would have been to give all the patients in the two acupuncture groups a placebo pill and all the patients receiving pills sham acupuncture and then blind the acupuncturists to experimental group by using a needle that retracts in such a way that the acupuncturist can’t tell if it really went into the skin or not (such needles do exist) and then having every patient hooked up to the current generator, which could be set to light up and have its gauge show that it’s producing current whether it actually is or not. That way, everyone can be blinded. Then add a wait list/no treatment control. But that’s not what they did. I realize my way would be more difficult and expensive, but it is the only way likely to generate anything resembling useful data.
Here’s another aspect:
Because placebo response to a pill has been previously reported to be approximately 20%,20 assuming baseline HFCS had a mean of 16 with a standard deviation of 4, we needed 26 participants in each placebo group to detect a 20% difference (3.2 HFCS/day) with 80% power using a two-sided significance level of .05. Assuming a 15% dropout rate, we needed to recruit 30 subjects per arm, for a total of 120 subjects. Our trial was not designed to evaluate the efficacy of EA or GP because sample size requirements would be much larger on the basis of previous literature. Evaluations of the short- and long-term effects between EA and other groups were secondary aims.
In other words, this study shows nothing about whether EA is actually efficacious, because it wasn’t designed to. It’s too small. So right away, any results must be interpreted in light of this knowledge, which is why all the discussion of placebo effects. First, however, here is the singularly unimpressive result of the study in the form of a single graph:
First, notice that all groups appeared to be improving right up until the end of the eight week treatment phase and are bunched up. How the authors interpret this result is rather interesting:
By week 8 (end of the intervention), acupuncture produced a significantly greater placebo effect than did pills, with the SA group having a significantly lower HFCS than the PP group (−2.39; 95% CI, −4.60 to −0.17; P = .035). Compared with PP, EA also had improvement in hot flashes (−4.1; 95% CI, −7.0 to −1.3; P = .005), whereas GP showed nonsignificant improvement (−1.8; 95% CI, −3.9 to 0.2; P = .085). We observed differences among treatment groups in the HFCS score (P < .001 for time and treatment interaction), with −7.4 for EA, −5.9 for SA, −5.2 for GP, and −3.4 for PP (Table 2 and Fig 2). Participants in the active treatment groups (EA and GP) experienced 47.8% and 39.4% improvement in hot flashes, respectively, compared with baseline. Placebo participants in the SA and PP groups experienced 45.0% and 22.3% improvement, respectively.
So, in other words, everyone improved (which is usually what happens on clinical trials of symptoms with a large subjective component), and somehow the authors see this as evidence that “sham acupuncture” produces a stronger placebo effect than the placebo pills. They also state that the SA group had less nocebo effect (equated with AEs, which I don’t consider appropriate given that some of the AEs included bruising In actuality, this might be true, but we really don’t know if that’s true. For one thing, there is no control group receiving no treatment. How do we know that a no treatment group wouldn’t also show a roughly 20% decline in HFCS composite scores just due to the subjects being on a clinical trial? We don’t. In fairness, the authors acknowledge this limitation. They also acknowledge that their study wasn’t powered to detect efficacy, which puzzled me to no end. Why bother to do the study if it can’t demonstrate efficacy? And why say things like this at the end:
In conclusion, acupuncture is associated with enhanced placebo effects and lower nocebo effects when compared with taking pills for hot flashes among survivors of breast cancer. Eight weeks of EA produced promising short- and long-term treatment outcomes for hot flashes compared with other comparators and had fewer adverse effects than GP. These preliminary findings need to be confirmed in larger studies with long-term follow-up. For survivors of breast cancer experiencing hot flashes, acupuncture could be preferable to GP because of sustained benefit after treatment and fewer adverse effects, whereas patients who dislike needles or do not have the time required for acupuncture treatments may prefer GP.
The results with the sham acupuncture, which bested gabapentin, suggest that “there is more than a placebo effect with the sham acupuncture,” said Dr. Gary Deng, interim chief of the integrative medicine service at Memorial Sloan Kettering Cancer Center in New York City. “There is a component of behavior of doing a sham procedure, so it psychologically may trigger a different kind of reaction from patients versus taking the placebo pill.”
Deng pointed out that clinicians have come to realize that the placebo effect is very important in treatment. “In fact, in clinical practice, every doctor uses it all of the time,” he said. “The so-called bedside manner or communication with patients — all of these enhance the effect of the patients feeling they’re getting something.”
No one is quite sure why placebos work for some people and not for others, said Deng. “It’s like psychotherapy,” he added. “Why does it work for some people and not others?” He suggested that differences in anatomy and genetics might be possible explanations, but said “there is a fertile field for further research.”
Because the most rigorous studies of acupuncture (and other “complementary and alternative medicine” or “integrative medicine” therapies) are almost always negative and the totality of the literature on acupuncture is consistent with its being a “theatrical placebo,” acupuncture advocates, like CAM advocates in general, are now selling it as the “power of placebo.” Of course, we know that more invasive interventions produce a stronger placebo effect. For instance, injections tend to produce stronger placebo effects than pills and surgical interventions produce the strongest placebo effects of all. So it would not be surprising if as theatrical a procedure as acupuncture produced a stronger placebo effect than a pill. If you accept that what was measured were placebo effects (and I do not without a no treatment control), then all this study basically showed was that SA is a better placebo than a pill, a result we could already have surmised from the literature, meaning that there was really no need to do this study.
In any event, I’ve never argued that placebo effects can’t be important. However, there is an inherent problem with using them, and that’s that you have to lie to the patient, attempts to demonstrate the existence of “placebo effects without deception” that show nothing of the sort notwithstanding. Worse, when a fantasy treatment based on a prescientific understanding of human physiology and disease retconned by Chairman Mao into a seemingly effective therapy is used as the placebo, the message is not so much that we could get the same effects using ethical means of increasing placebo effects, such as good bedside manner, encouraging words, and the like. It’s that magic works through placebo effects, which is exactly how this study will be sold.
In the end there are two bait and switches in this study: electroacupuncture sold as acupuncture and acupuncture sold as a powerful placebo that has clinical utility. Same as it ever was.
