Categories
Bad science Clinical trials Integrative medicine Medicine Quackery

PLOS ONE publishes a howler of a bad acupuncture network meta-analysis

Meta-analyses can sometimes suffer from the “GIGO problem” (garbage in, garbage out). The publication of a “crappy” acupuncture “network meta-analysis” for acupuncture and chronic constipation illustrates the GIGO problem on steroids a.nd reveals a problem with peer review at PLOS ONE, where it was published

I write about acupuncture a lot because it’s the form of quackery that is arguably the most “respectable” and accepted among academic medical institution and “conventional” doctors. Indeed, a decade or more ago I used to be (almost) among the doctors who thought that there might be something to acupuncture. Obviously, I never believed in any of the mysticism underlying acupuncture meridians and “unblocking” the flow of qi with needles. However, because needles actually penetrate skin in acupuncture, I thought that maybe there was a mechanism by which acupuncture could “do something” therapeutic. Of course, the more I read the medical literature and actual studies purporting to show a therapeutic effect due to acupuncture, the more I realized how poor the evidence was and how the highest quality evidence out there showed that acupuncture is basically nothing more than a theatrical placebo. Basically, it doesn’t matter where you stick the needles or even if you stick the needles in. The effect is the same. Unfortunately, many journals are complicit in spreading pseudoscience. Such were my thoughts as I was made aware by a reader of a new acupuncture network meta-analysis published late last month in PLOS ONE out of the Shenzhen Longhua District Central Hospital in China entitled “Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis.” It is truly one of the worst acupuncture meta-analyses that I’ve ever seen and reveals a real problem with peer review at PLOS ONE.

A meta-analysis is a type of study that seeks to aggregate data from high quality studies in order to come to a conclusion about an intervention or therapy. When done rigorously and without bias, meta-analyses can be a very useful method for assessing the state of the evidence with regard to a particular treatment. However, meta-analyses are subject to the rule of GIGO (garbage in, garbage out). Basically, aggregating a bunch of poor-quality studies does not result in reliable conclusions.

This, however, is something I hadn’t encountered much before, mainly a network meta-analysis. So I had to look it up:

Systematic reviews use explicit, pre-specified methods to identify, appraise, and synthesize all available evidence related to a clinical question. When appropriate, systematic reviews may include a meta-analysis, that is, the statistical combination of results from two or more separate studies. Some systematic reviews compare only two interventions, in which a conventional pair-wise meta-analysis may be conducted, while others examine the comparative effectiveness of many or all available interventions for a given condition. When the comparative effectiveness of a range of interventions is of interest, appropriate statistical methodology must be used for analysis.

Also called mixed treatments comparison or multiple treatments comparison meta-analysis, network meta-analysis expands the scope of a conventional pair-wise meta-analysis by analyzing simultaneously both direct comparisons of interventions within randomized controlled trials (RCTs) and indirect comparisons across trials based on a common comparator (e.g., placebo or some standard treatment) [1, 2, 3, 4, 5]. In the simplest case, one may be interested in comparing two interventions A and C. Indirect evidence can be obtained from RCTs of either A or C versus a common comparator B (Figure 1), keeping intact the randomized comparisons within the RCTs [1, 2, 3, 4, 5]. When both direct and indirect evidence are available, the two sources of information can be combined as a weighted average when appropriate.

Right off the bat, network meta-analysis sounds very suspect as a methodology. Certainly, if regular meta-analyses are subject to GIGO and cherry picking studies, this sort of meta-analysis is even more so. In any case, the reason the authors chose a network meta-analysis is obvious. There are no studies directly comparing acupuncture to the various laxatives and drugs to which the authors wish to compare them. I knew that the study would be bad right from the introduction, where the authors opine:

The methods of treating chronic constipation currently include drugs and acupuncture. Common drugs include lactulose, polyethylene glycol, linaclotide, lubiprostone, bisacodyl, prucalopride, tegaserod, etc. But most of drugs have some side effects which couldn’t be tolerated by patients, complementary or alternative treating methods with less side effect [sic] are needed in our practice. An increasing number of studies have shown that acupuncture may have certain effects on chronic constipation. However, there have been no studies comparing acupuncture and commonly used oral drugs. The comparison between drugs and acupuncture will bring us a clear understanding of those interventions. Furthermore, various acupoints have been targeted to treat chronic constipation. Therefore, this study was designed to solve these unanswered questions.

This study was a systematic review of the literature and collected data from published randomized controlled trials (RCTs) of acupuncture and common oral drugs in the treatment of chronic constipation up to December 2017. This study aimed to answer the following questions: (1) what are the curative effects and side effects of acupuncture and drugs to improve chronic constipation?; (2) what is the curative effect of sham acupuncture as a placebo?; and (3) what are the acupoints to cure constipation?

See what I mean? The authors state as a fact that acupuncture is a method of treating chronic constipation. While that is true on a trivial level (obviously there are quacks out there treating constipation with acupuncture), the implication behind the statement is that it is an accepted method of treatment. Also notice the false dichotomy: Drugs used to treat constipation are framed as having unacceptable side effects, while acupuncture is framed as somehow more desirable. In any case, the authors searched PubMed, the Cochrane Library, Embase and 4 Chinese databases [China National Knowledge Infrastructure (CNKI), Wanfang Database, VIP Database and Chinese Biomedical Database (CBM)] for randomized clinical trials, using as their criteria:

  • Clinical trials of adult patients
  • Clinical trials of single drugs
  • Clinical trials with treatment duration greater than two weeks
  • Publications that were not comments or commentaries
  • Studies that excluded pregnant or lactating women, patients with peptic ulcers, rectal disease, or liver or other systemic disease, and patients with previous history of gastroduodenal surgery, neurologic diseases or neurologic surgery

The authors identified a total of 1,516 articles were obtained from their literature search, of which 36 duplicates were deleted and 1,438 articles were excluded, leaving a total of 40 trials including 11,032 patients with chronic constipation. The studies encompassed a total of 10 treatment modalities including A: prucalopride; B: tegaserod; C: lubiprostone; D linaclotide E: bisacodyl; F: PEG; G: lactulose; H: acupuncture; I: sham acupuncture; and J: placebo (sucrose, etc.). So right there is a problem, namely that there are ten interventions and only forty studies. Given that there are 12 studies involving acupuncture (7 of acupuncture versus lactulose, 4 involving acupuncture versus sham, and one comparing two different forms of acupuncture), for some interventions, there are only two studies (range: 1-8). None of this stopped the authors from paddling bravely up the river of bad science by comparing all the studies and comparing acupuncture with each of the other interventions and concluding:

By including more than 40 articles for network meta-analysis (NMA), acupuncture was found to be more effective than drugs in treating chronic constipation with nearly no side effects. The mechanism of action of acupuncture for the treatment of chronic constipation remains unclear, but previous studies have also shown its possible role in the treatment of chronic constipation. An important cause of chronic constipation is the decreased motility of the colon, resulting in a decreased urge to defecate, bloating, and abdominal pain or discomfort. [51] Existing studies indicate that acupuncture at heterotopic acupoints facilitates distal colonic motor ability by activating M3 receptors and somatic afferent C-fibres in normal, constipated, or diarrhoeic rats. Acupuncture can also increase distal colonic motility in normal and abnormal mice with a possible role of M3 receptors [52]. Previous studies have suggested that the ST36 acupoint can change the motor function of the colon. [53]

That’s right. The authors concluded that acupuncture was more effective than any drug commonly used to treat chronic constipation and that it had “nearly no” side effects. That’s rather convenient, given that only one of the acupuncture studies included in the meta-analysis even bothered to report side effects.

That’s not all, either. There’s the way that the authors assessed effect size. Basically, they measured and compared the proportion of patients “with improvement in chronic constipation.” No measure is made of how much constipation improved, whatever tool various investigators might have chosen to use to assess severity of constipation in each study. Think of it this way by considering two hypothetical studies. One study shows that 50% of patients saw their constipation improve based on one treatment, and those 50% averaged a 90% improvement. Now consider another study that shows that 50% of patients improved, but by only an average of 20%. Those two studies would count the same because both report that 50% of their patients got better, but obviously they do not show the same result. Obviously, you’d want the treatment with an average improvement of 90% rather than 20%.

But that’s still not all. Let’s take a look at the Forest plot of the conventional paired meta-analysis that the authors carried out before doing their network thing:

Forest plot of acupuncture and medications
Forest plot of acupuncture and medications

The authors report that, compared with placebo (J), acupuncture significantly improved the symptoms of chronic constipation, as did most of the other interventions. Note also, though, that sham acupuncture also improved chronic constipation symptoms with an odds ratio of 3.1 (95% confidence interval 1.5-7.2). Just look at the lower panel. That’s as good as many of the drugs. There’s a red flag right there, as far as I’m concerned. There’s a reason for this, as described by Ross Pomeroy:

The vast majority of the laxative studies were double-blind, randomized, placebo-controlled trials published in peer reviewed journals. On the other hand, all of the acupuncture studies were unblinded, four were not peer-reviewed, five were published in questionable journals like the Shanghai Journal of Acupuncture and Moxibustion, and all were based in China, which is notoriously friendly to acupuncture and alternative medicine. Thus, it’s no surprise that acupuncture showed stronger results in those studies.

So basically what we have here is a perfect example of what acupuncturists do to try to “prove” that their quackery works for a wide variety of conditions. It is, however, a new wrinkle in bad acupuncture meta-analyses. Instead of just GIGO, it compares garbage (the results of most acupuncture studies) with the results of better quality studies (the studies of medications) in order to conclude that the garbage is even better than medicine. Never mind that ten different acupuncture points were used, that sham acupuncture “worked” almost as well as acupuncture, and that, well, their study is the best meta-analysis so far:

However, a previous meta-analysis showed no significant curative benefits of acupuncture compared with sham acupuncture in the treatment of chronic constipation. Only some studies from China have shown that acupuncture is superior to drugs. [54] A previous study from China showed that acupuncture and sham acupuncture had similar effects on functional constipation, but moxibustion was used with ST23 and ST27 acupoints in this study. Moreover, this study was based on a small sample size [55], and therefore, we are unable to ensure the accuracy of these findings. In addition, the curative effect of sham acupuncture remains to be confirmed.

Our study was unique in that the included studies of acupuncture used the deep pricking method instead of the ordinary acupuncture method. The depth of the deep pricking method is deeper than that of ordinary acupuncture method. Further studies are needed to confirm whether the deep pricking method has a better curative effect on chronic constipation than ordinary acupuncture. In the present study, many studies conducted after 2012 were included and the studies using the same acupuncture points were included to ensure consistency. RCTs using consistent acupuncture points will yield more definitive results.

I note that the outcome evaluated in the meta-analysis mentioned above was the change in the number of weekly spontaneous bowel movements.

This is the sort of thing acupuncturists do, though. Here’s another example, but not a meta-analysis. Rather, it’s a recently published randomized controlled acupuncture trial to treat chronic constipation that compares He acupuncture, Shu‐mu acupuncture, He‐shu‐mu acupuncture, and orally administered mosapride. Notice something? There’s no placebo control, no sham acupuncture control, and, you’ll notice if you read further, no blinding. Believe it or not, the mosapride group functioned as the control group. Surprise! Surprise! All groups improved, which is exactly what one would expect in such an unblinded trial. Basically, this is as worthless a clinical study as I’ve ever seen, and Neurogastroenterology & Motility should be ashamed for having published it.

Back to the meta-analysis, though. Not surprisingly, physicians and scientists were puzzled at how PLOS ONE could publish dreck as bad as this and took Twitter:

Steve Salzberg even resigned from the editorial board over this article:

PLOS ONE even responded:

We’ll see what happens. After all, as James Coyne has noted, it took PLOS ONE a long time to retract a more obviously dubious study:

Here’s the retraction notice:

Following publication of this article [1], concerns were raised about the scientific validity of the study as well as a potential competing interest that was not declared. The PLOS ONE Editors discussed the concerns with the authors and consulted external experts. In light of our editorial assessment and advice received in the expert consultations, we are retracting this article due to concerns about the scientific validity of the research question, study design, and conclusions.

Specifically, we are concerned about the overall design of the study, which aims to detect effects of a reagent diluted to such a degree that the solution is not expected to contain biochemically relevant levels of antibody. The consulted experts also raised concerns about the validity and rigor of the immunoassay system used in the study. The enzyme-linked immunosorbent assay (ELISA) used was adjusted to give barely detectable signals, which renders the assay particularly susceptible to interference. In light of these issues, we consider that the article does not present sufficient or reliable evidence to support the conclusions.

Stealth homeopathy paper indeed. I note that this paper was first published in 2014. PLOS ONE also notes undisclosed conflicts of interest.

Unfortunately, it turns out that this is not the first time PLOS ONE has fallen prey to acupuncture pseudoscience. For instance, PLOS ONE published one of Helene Langevin’s dubious studies trying to “map” acupuncture meridians using a combination of electrical impedance measurements and ultrasound. In addition, it also published a bad fMRI study of “laser acupuncture” purporting to show effects on neural activity in the brain. And let’s not forget that the followup to Andrew Vickers’ original highly dubious acupuncture meta-analysis was published by PLOS ONE. Indeed, acupuncture pseudoscience seems to have found a home on PLOS ONE. Just for yucks, I did a search on the journal’s website, yesterday, and what did I find? Page after page of search results of studies published about acupuncture. I might very well have to look into some of them, because they look bad. Indeed, although I can’t tell you which paper I reviewed, I can tell you that, the one time I peer reviewed an acupuncture manuscript for PLOS ONE, the pressure was to publish, as I recall its being two reviewers against me. Yes, I might very well have to revisit this topic.

Scientific and clinical studies of pseudoscience like acupuncture and homeopathy pose a significant challenge to journals, including PLOS ONE. The number of people who both know enough about the modalities being tested to evaluate them and are not advocates of those modalities is small. This results in reliance on peer reviewers who are supporters or practitioners of the quackeries being evaluated. It also allows an acupuncture believer associated with Deepak Chopra like Neil Theise to spin highly preliminary findings having nothing to do with acupuncture as evidence supporting acupuncture. Still, I have to wonder what went wrong at PLOS ONE in this particular case. I can sort of see how the stealth homeopathy paper could fool a peer reviewer not familiar with homeopathy, but I have a hard time seeing how this network meta-analysis paper could pass muster. Whatever the case, PLOS appears to have a problem, and acupuncture still doesn’t work—for constipation or anything else.

By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

19 replies on “PLOS ONE publishes a howler of a bad acupuncture network meta-analysis”

No doubt this meta-analysis will metastasize and become part of the lore that says “studies” show acupuncture works.

There was an acupuncture-helping-addicts-in-recovery story in the Columbus Dispatch yesterday, 100% in the tank for this modality (specifically, auricular acupuncture as performed on addicts by a chiropractor), citing in part a judge who sends addicts to get this treatment.

“It came about because Pickaway County Common Pleas Judge P. Randall Knece had earlier this year heard about auricular acupuncture (utilizing only external ear points) from a recovery home where he sometimes sends defendants from his court. He asked Laux (the chiro) if it was true that this practice could help those overcoming addictions. Laux said yes. He is certified by the National Acupuncture Detoxification Association and uses its method…
Knece has pledged almost $13,000 to help cover the $55 cost per treatment for those in recovery who don’t have Medicaid or private insurance. The money will come out of $434,000 he has received from the state to build innovative justice programs to help keep low-level, nonviolent offenders out of prison…

Barry Bennett, executive director of PARS…said he has read the research that shows people in recovery who undergo acupuncture stay clean at a higher rate than those who don’t. Some drugs courts, for example, report that the rates of offenders who used again dropped by as much as 10 percent when acupuncture was a part of that treatment. That was good enough for him.”

Never mind the comprehensive reviews by Cochrane, Rand et al showing no demonstrable effect of acupuncture on addiction of various types. Or the excellent article in Science-Based Medicine on the subject of auricular acupuncture for addiction recovery.

I contacted the editor of the newspaper to fill him in on all the evidence the reporter, um, overlooked, so no doubt I’ll quickly get an apologetic response from the Dispatch on how it dropped the ball on this story. 😉

As a form of psychosocial support, it’s probably better than the 12-step cult, but tastes vary. (Yes, I’ve known a junkie.)

Tell the Dispatch they should be looking at Judge Moses in Hocking County. He runs the drug courts there, and just started a Vivitrol drug court that is seperate from the suboxone drug courts. He spoke to the Ohio Association of Advanced Practice Nurses last Fall and I was very impressed with what he’s doing. He’s passionate about drug treatment that works over criminalization.

You know if you start sticking a bunch of needles in me, I would probably become unblocked fairly quickly. That is probably the mechanism by which acupuncture works, it scares the sh!t out of you.

I want to be sure about this network thing. They compared medical interventions to acupuncture except that the primary literature used did not include direct comparisons between medical interventions and acupuncture save one. Very creative, not to mention their deeming of studies included as having low publication bias.

A bit off topic – AARP has endorsed “dry needling” as a “fix” for pain in the May AARP Bulletin. As a person who suffers chronic pain, I’ve decided to pass, even though they categorize it as a “fix.”

The authors identified a total of 1,516 articles were obtained from their literature search, of which 36 duplicates were deleted and 1,438 articles were excluded

Is a 97% false positive rate for this kind of literature search normal? It makes sense to exclude the duplicates, but why were the other 1438 excluded articles excluded? If the reasons for exclusion are not clearly stated, that gives the appearance of skullduggery.

I am not sure about that high a “rejection” rate but very high rates are not unusual.

My understanding is that when one does that kind of key word search can turn up a lot of articles that just are not specific to the question being asked or may not report required statistics, or the statistics are not in a usable form.

Often, much of the material may turn out to be editorials, letters to the editor, literature reviews, and heaven only knows what else.

Good point Eric. They explain inclusion/exclusion criteria for the search but don’t have the hits listed and why they were excluded; they have only included studies listed. Cochrane lists all studies from their search and why they were excluded along with bias rankings of the included studies. I so wish Dana Ullman would show up and explain why this is such an awesome study.

I can speak for that one, being the third author:

<

blockquote>Of over 7000 articles retrieved, 58 (19 PET and 39 fMRI) satisfied all inclusion criteria and were included in the analysis (Table 1). Several studies reported activity from multiple task and control conditions. For our analysis, only conditions incorporating either no overt task or a simple task used to maintain attention were considered. To maintain consistency among the control conditions, only task contrasts with a low-level baseline (silence, tone, or noise) were included. For some studies, more than one contrast satisfied our criteria and all were included in the analysis. This procedure was employed to maximize the sensitivity of the analysis, but could potentially bias the results toward samples for which more than one contrast was included.

Of the top of my memory, we had 10 to 12 criteria to work with, the primary and most important one being that the analysis for the publication included in our meta-analysis had to be done on the whole brain, not any region of interest (as much as possible but Dr Samson had the final say on that along with Dr Zeffiro). Still, the number are 58 studies out of over 7000 (0.83% met inclusion criteria).

I haven’t a clue about the current study here discussed by Orac (of which I’ll probably read the post during next weekend) but yeah, that can happen.

Alain

Unfortunately, woo peddlers will spin this as abundant proof that acupuncture works and is science or evidence based. Usually, they are thrilled with meta- analysis as multiple studies are involved as if that means that they are also meaningful. Mostly they have a sole outlier or two to discuss. Quantity doesn’t equal quality.
PLOS is often mentioned because they can sometimes find abysmal studies that fit their world view.

Yeah, unfortunately, AARP often recommends some of the softer-types of woo, all the while strongly encouraging its members to be vigilant of financial scams and medical fraud. Seems they don’t know where to draw the line.

So basicly they took a bunch of biased papers and came up with a biased result. Why am I not surprised?

Prediction: The next thing Acupuncturists will claim is why their woo works is this newly discovered subcutaneous organ, which I’m not sure even exists.They will of make the claim that their needles somehow stimulate this heretofore unknown organ, causing it to heal what ails you.

Comments are closed.