Here we go again.
Tuesday night and yesterday, you probably saw it, plastered all over the media, in the newspapers, on ABC, on the radio, in press releases, and around the blogosphere. Yes, it was another bit of science by press release, with news outlets practically falling all over themselves to hype the results of an acupuncture study reported earlier this week at the annual meeting of the American Society for Therapeutic Radiology and Oncology (ASTRO). Leading the pack was ABC News:
A new medical study finds that acupuncture, an ancient form of healing that has been around for thousands of years, is as good as, or better than modern medicine in helping ease the side effects of breast cancer treatment.
The findings, which were presented today at the American Society for Therapeutic Radiology and Oncology’s (ASTRO) annual meeting in Boston, suggest that this ancient therapy can give cancer patients a wide range of benefits above modern medicine.
This study suggests nothing of the sort. In fact, like most acupuncture studies, it’s riddled with methodological shortcomings that make its results, in essence, meaningless. I’ll explain why shortly. But first, let’s check out the credulous coverage:
Most women in the study said they saw the same dramatic effect from the acupuncture treatment as Azar did.
“Acupuncture is equal to drug therapy in decreasing hot flashes,” said Dr. Eleanor Walker at the Henry Ford Hospital and lead author of the study. And even better, she said, it has no side effects.
Yes, this study originates in my hometown, and Dr. Walker has been all over the news serving up such bon mots such as this:
Doctors have been prescribing the antidepressant venlafaxine — sold under the name brand Effexor — for off-label use in alleviating hot flashes, Walter said. But oral medications are difficult for some breast cancer patients to keep down because of the nausea associated with chemotherapy, and Effexor has a long list of side effects, including constipation.
“My patients were complaining,” Walter said. “They were sick of taking pills. Wasn’t there something I could do?”
Don’t you just love the false dichotomy? If patients are sick of taking pills or don’t like the side effects of the pills, then the answer must be woo, of course! It’s so obvious! It also never fails to puzzle me that patients who hate taking a couple some pills are more than willing to inconvenience themselves to go to a clinic three times a week to have needles stuck into them for a half hour or an hour. I never could figure that out. Be that as it may, into the picture comes a familiar face, whom I’ve discussed before, reappearing to pontificate piously:
Barrie Cassileth, chief of integrative medicine services at Memorial Sloan-Kettering, said acupuncture can benefit most patients with “an open mind.”
“I think many more patients should try acupuncture,” Cassileth said. “It is easy. It’s pleasant. It’s inexpensive. There are no risks involved.”
Dr. Cassileth, as you may recall, was the principal investigator for a trial testing whether acupuncture can alleviate hot flashes in women with breast cancer taking anti-estrogen therapy that I discussed in detail in December. In fact, it was about as well-designed and scientifically rigorous a trial as could be carried out for acupuncture in practical terms. Its main shortcoming was that the practitioners were not blinded and that it used only patient reporting rather than a combination of patient reporting of hot flashes and objective measurements. Because it was a pretty well-designed study, not surprisingly, it was also a completely negative study, failing to find any significant effect for acupuncture beyond that of a placebo. You’d think That Dr. Cassileth would have been deterred or less enthusiastic, but apparently not. Here she is saying that acupuncture works if you have an “open mind.” Does that mean it doesn’t work if you don’t have an open mind? No wonder we nasty, close-minded skeptics think acupuncture doesn’t do anything more than an elaborate placebo! But I digress. In fact, she is very much still a booster of acupuncture, to the point of praising a study that is markedly inferior to hers.
How inferior?
Well, let’s look at the design. I had a heck of a time finding the abstract, which is all there was to go on. Granted, there was enough information in some of the news reports and press releases to tell me why this study is worthless, but I always like to go to the source whenever I can. Fortunately, despite its credulous blogging about the study, Pharmalot provided the actual abstract, entitledAcupuncture for the Treatment of Vasomotor Symptoms in Breast Cancer Patients Receiving Hormone Suppression Treatment. Basically, the design was simple. Dr. Walker compared acupuncture treatments versus venlafaxine (Effexor), the latter of which is an antidepressant sometimes used as an alternative to hormone replacement therapy (HRT) to relieve menopausal symptoms in women in whom HRT isn’t a good idea, such as women taking estrogen blocking drugs for breast cancer. Forty-seven women with stage 0 to III breast cancer being treated with either Tamoxifen or Arimidex were randomized to receive either acupuncture or venlafaxine for twelve weeks. Patients logged the severity and frequency of their hot flashes daily for one week prior to the study and during the study, as well as at regular intervals for a year after treatment. Other outcome measures that were assessed before, during, and after treatment included menopause-specific quality of life, general health status (SF-12), Beck Depression Inventory, and the frequency and kinds of side effects.
The results were reported as follows:
Both acupuncture and venlafaxine groups exhibited significant decreases in hot flashes and other menopausal, quality of life symptoms, as well as decreases in depressive symptoms. These changes were similar in the 2 groups, indicating that acupuncture is at least as effective as venlafaxine in reducing vasomotor and other symptoms associated with anti-estrogen hormonal treatment of breast cancer. Additionally, numerous patients treated with venlafaxine reported negative side effects including nausea, dry mouth, headache, difficulty sleeping, dizziness, double vision, increased blood pressure, constipation, fatigue, anxiety, feeling ”spaced out,” and body jerking during the night. Patients treated with acupuncture experienced no negative side effects. They reported increased energy, clarity of thought, sexual desire, and overall sense of well-being (compared to pretreatment).
So what’s the problem? Have you spotted it yet? If you’re a regular reader here, I bet you have. Yes, this study is completely unblinded. Patients knew what group they were in. But is that a problem? After all, it’s represented that venlafaxine is routinely used for the treatment of hot flashes; so if there was no difference between the two groups then acupuncture must have been doing as well as the drug in relieving these patients’ hot flashes, right?
Not so fast, there, pardner.
First, one needs to realize that at best venlafaxine is mildly effective when it comes to menopausal symptoms. It’s nowhere near as effective as estrogen, and its relatively weak relief comes at the price of some significant side effects that lead a fair number of women to stop taking the drug because they consider the side effects worse than the hot flashes (which the drug doesn’t reduce all that much anyway). Indeed, a meta-analysis from 2006 published in JAMA failed to find a significant effect on menopausal symptoms due to venlafaxine, although the meta-analysis has been criticized and there have been more recent randomized trials suggesting a benefit. The consensus of the studies is that venlafaxine appears to decrease patient self-reported hot flashes but not necessarily physiologically documented hot flashes. That’s not such a big deal, given that it’s the patient perception of hot flashes that’s so troublesome. What’s more of a confounding factor in any study looking at therapy-induced menopause is that hot flashes virtually always tend to become less frequent and troublesome regardless of treatment (or lack of treatment), and there tends to be a significant placebo effect in all such studies.
What’s a bigger deal is that the abstract did not report the dose of venlafaxine used. Perusing the literature, one thing that became apparent to me is that low dose venlafaxine (37.5 mg per day) appears to be less likely to be effective than high dose venlafaxine (75 mg per day). The problem, of course, is that higher doses are more likely to produce side effects. If this acupuncture study used 37.5 mg a day in the venlafaxine group, it’s quite possible that there was little or no effect from the drug. Even at 75 mg, it’s not clear that the benefit would be high enough to overcome the potential confounding factors inherent in the design of this study. One study suggested that the drug did little for menopausal symptoms but, because it’s an antidepressant, helped women cope with them in their everyday activities, which is a plausible possibility. The bottom line is that, given the small size of the acupuncture study, it’s entirely plausible–even likely–for investigators to have found no difference between the venlafaxine group and the acupuncture group, even if there really were a difference favoring venlafaxine just on statistical power considerations alone. Indeed, for such a small trial, the effect of venlaxafine is probably not strong enough or reliable enough to justify only a head-to-head trial comparing it with another modality when the object is to test whether this other modality is effective.
Let me describe for you a much better design for this study. Because the interventions involved are so different, the best way to produce a true double blinding of the study subjects and the investigators would be to randomize the patients into at least three groups:
- Placebo acupuncture plus placebo venlaxafine (the true placebo group)
- Placebo acupuncture plus venlaxafine (the venlaxafine group)
- Acupuncture plus placebo venlaxafine (the acupuncture group)
The sham acupuncture, to be most rigorous, would have to consist of the special retractable needles that blind both the patients and practitioners to the experimental group. The above design would produce real blinding, because the patients would all think they are receiving both treatments and have no idea which of the two is the placebo. If the investigators wanted to get fancy, they could add a no treatment group to gauge the true size of the placebo effect (which in many of studies of treatments for menopausal symptoms is greater than 30%) or an acupuncture plus venlaxafine group, although the latter of the two is probably not necessary unless the investigators have a real interest in testing for additive or synergistic effects of the two modalities. Other designs are possible, but the one chosen for the study as presented at ASTRO is about the worst design possible, other than a nonrandomized design. Indeed, I find it very disappointing that the Susan J. Komen Foundation actually wasted its donors’ dollars on this study, given its methodological flaws. Even with its methodological flaws, I’m surprised ASTRO accepted it for a scientific talk (as opposed to a poster). I can only guess the organizers were interested in presenting “provocative” results, which meeting organizers sometimes are.
Finally, over at Science-Based Medicine, Steve Novella makes an excellent point about this study. Normally studies progress from small pilot studies, which may be unblinded or even unrandomized, which, if promising, form the basis for larger, more rigorous clinical studies. Acupuncture study methodology has evolved to the point where there is no real need to do these sorts of pilot studies anymore. There are good placebo and sham acupuncture techniques. It may be difficult to blind practitioners, but there is really no reason why an acupuncture study in which the patients are not blinded should ever be done anymore. There’s no point anymore (excuse the pun). Doing small unblinded studies of this sort represents a regression in the usual arc of small pilot studies to larger, better controlled, randomized trials. It’s OK to go back to do pilot studies if new observations suggest new applications for acupuncture, but, as demonstrated by the Cassileth study from last year, CAM advocates have been studying acupuncture for hot flashes for years, and acupuncture for hot flashes due to antiestrogen therapy has been studied for quite a while now. Indeed, a recent randomized trial studying this very question concluded that their “results suggest either that there is a strong placebo effect or that both traditional and sham acupuncture significantly reduce hot flash frequency.” In other words, the question of whether acupuncture can relieve menopausal symptoms is not a “new” question that requires pilot studies anymore. It’s been studied more than enough to be beyond that question; yet such studies persist.
As a breast cancer surgeon, I appreciate just how bad hot flashes can be for breast cancer patients. Not only do they significantly impact quality of life, but they can jeopardize treatment. Indeed, my first breast cancer patient that I treated as an attending found them so unbearable that she ultimately decided she couldn’t stand it anymore and decided to stop taking her Tamoxifen. Granted, in her case, the additional benefit of the Tamoxifen was probably relatively small, but there are patients for whom it is much larger. Failure to relieve menopausal symptoms induced by therapy can in some cases increase a patient’s risk of recurrence. Consequently, because it is not safe to administer hormones to women with estrogen-responsive breast cancers, it is very important to find better, non-hormonal methods of treating these symptoms. It is thus understandable that even unscientific methods would be tempting, especially because current treatments are only mildly effective and have side effects.
The problem is that practitioners have taken advantage of this hole, as they have for other syndromes and problems for which conventional medicine is not as efficacious as we would like. Worse, they have insinuated their way into academia so that academic physicians who really should know better decide that, no matter how implausible the treatment (in this case, acupuncture) is on a scientific and physiological basis, it “deserves” study simply because it’s apparently popular, never mind scientific plausibility and the very high level of improbability bordering on the impossible. This leads to a cycle of pilot studies showing an apparent benefit of this woo or other, followed by larger and better designed studies that fail to confirm that benefit. At that point, rather than doing what scientists do with most “conventional” or drug therapies and abandoning them, they keep at it. Instead of seeing the “pruning” of ineffective or less effective treatments, which are then supplanted by effective or more effective treatments, we see a “regression” back to pilot studies, which, given their methodological shortcomings, then show the same apparent “benefit” again. Meanwhile the media aids and abets this cycle by credulously trumpeting any nonsense that comes from these poorly powered studies. It’s also rather interesting, don’t you think, that negative studies of acupuncture never find themselves being reported all over the media, as this one did. It’s not just studies of acupuncture, either, but virtually any “alternative” medicine.
Once complete for a question, the cycle then starts all over again or moves on to a different set of nonscientific and utterly implausible treatments, while patients are left not knowing what to think, or, worse, thinking that woo will help them, with the media reporting and promoting the few apparently “positive” trials and ignoring the vast mass of negative evidence. And on and on and on the cycle goes.
