PLOS ONE publishes a howler of a bad acupuncture network meta-analysis

I write about acupuncture a lot because it’s the form of quackery that is arguably the most “respectable” and accepted among academic medical institution and “conventional” doctors. Indeed, a decade or more ago I used to be (almost) among the doctors who thought that there might be something to acupuncture. Obviously, I never believed in any of the mysticism underlying acupuncture meridians and “unblocking” the flow of qi with needles. However, because needles actually penetrate skin in acupuncture, I thought that maybe there was a mechanism by which acupuncture could “do something” therapeutic. Of course, the more I read the medical literature and actual studies purporting to show a therapeutic effect due to acupuncture, the more I realized how poor the evidence was and how the highest quality evidence out there showed that acupuncture is basically nothing more than a theatrical placebo. Basically, it doesn’t matter where you stick the needles or even if you stick the needles in. The effect is the same. Unfortunately, many journals are complicit in spreading pseudoscience. Such were my thoughts as I was made aware by a reader of a new acupuncture network meta-analysis published late last month in PLOS ONE out of the Shenzhen Longhua District Central Hospital in China entitled “Comparison of acupuncture and other drugs for chronic constipation: A network meta-analysis.” It is truly one of the worst acupuncture meta-analyses that I’ve ever seen and reveals a real problem with peer review at PLOS ONE.

A meta-analysis is a type of study that seeks to aggregate data from high quality studies in order to come to a conclusion about an intervention or therapy. When done rigorously and without bias, meta-analyses can be a very useful method for assessing the state of the evidence with regard to a particular treatment. However, meta-analyses are subject to the rule of GIGO (garbage in, garbage out). Basically, aggregating a bunch of poor-quality studies does not result in reliable conclusions.

This, however, is something I hadn’t encountered much before, mainly a network meta-analysis. So I had to look it up:

Systematic reviews use explicit, pre-specified methods to identify, appraise, and synthesize all available evidence related to a clinical question. When appropriate, systematic reviews may include a meta-analysis, that is, the statistical combination of results from two or more separate studies. Some systematic reviews compare only two interventions, in which a conventional pair-wise meta-analysis may be conducted, while others examine the comparative effectiveness of many or all available interventions for a given condition. When the comparative effectiveness of a range of interventions is of interest, appropriate statistical methodology must be used for analysis.

Also called mixed treatments comparison or multiple treatments comparison meta-analysis, network meta-analysis expands the scope of a conventional pair-wise meta-analysis by analyzing simultaneously both direct comparisons of interventions within randomized controlled trials (RCTs) and indirect comparisons across trials based on a common comparator (e.g., placebo or some standard treatment) [1, 2, 3, 4, 5]. In the simplest case, one may be interested in comparing two interventions A and C. Indirect evidence can be obtained from RCTs of either A or C versus a common comparator B (Figure 1), keeping intact the randomized comparisons within the RCTs [1, 2, 3, 4, 5]. When both direct and indirect evidence are available, the two sources of information can be combined as a weighted average when appropriate.

Right off the bat, network meta-analysis sounds very suspect as a methodology. Certainly, if regular meta-analyses are subject to GIGO and cherry picking studies, this sort of meta-analysis is even more so. In any case, the reason the authors chose a network meta-analysis is obvious. There are no studies directly comparing acupuncture to the various laxatives and drugs to which the authors wish to compare them. I knew that the study would be bad right from the introduction, where the authors opine:

The methods of treating chronic constipation currently include drugs and acupuncture. Common drugs include lactulose, polyethylene glycol, linaclotide, lubiprostone, bisacodyl, prucalopride, tegaserod, etc. But most of drugs have some side effects which couldn’t be tolerated by patients, complementary or alternative treating methods with less side effect [sic] are needed in our practice. An increasing number of studies have shown that acupuncture may have certain effects on chronic constipation. However, there have been no studies comparing acupuncture and commonly used oral drugs. The comparison between drugs and acupuncture will bring us a clear understanding of those interventions. Furthermore, various acupoints have been targeted to treat chronic constipation. Therefore, this study was designed to solve these unanswered questions.

This study was a systematic review of the literature and collected data from published randomized controlled trials (RCTs) of acupuncture and common oral drugs in the treatment of chronic constipation up to December 2017. This study aimed to answer the following questions: (1) what are the curative effects and side effects of acupuncture and drugs to improve chronic constipation?; (2) what is the curative effect of sham acupuncture as a placebo?; and (3) what are the acupoints to cure constipation?

See what I mean? The authors state as a fact that acupuncture is a method of treating chronic constipation. While that is true on a trivial level (obviously there are quacks out there treating constipation with acupuncture), the implication behind the statement is that it is an accepted method of treatment. Also notice the false dichotomy: Drugs used to treat constipation are framed as having unacceptable side effects, while acupuncture is framed as somehow more desirable. In any case, the authors searched PubMed, the Cochrane Library, Embase and 4 Chinese databases [China National Knowledge Infrastructure (CNKI), Wanfang Database, VIP Database and Chinese Biomedical Database (CBM)] for randomized clinical trials, using as their criteria:

• Clinical trials of adult patients
• Clinical trials of single drugs
• Clinical trials with treatment duration greater than two weeks
• Publications that were not comments or commentaries
• Studies that excluded pregnant or lactating women, patients with peptic ulcers, rectal disease, or liver or other systemic disease, and patients with previous history of gastroduodenal surgery, neurologic diseases or neurologic surgery

The authors identified a total of 1,516 articles were obtained from their literature search, of which 36 duplicates were deleted and 1,438 articles were excluded, leaving a total of 40 trials including 11,032 patients with chronic constipation. The studies encompassed a total of 10 treatment modalities including A: prucalopride; B: tegaserod; C: lubiprostone; D linaclotide E: bisacodyl; F: PEG; G: lactulose; H: acupuncture; I: sham acupuncture; and J: placebo (sucrose, etc.). So right there is a problem, namely that there are ten interventions and only forty studies. Given that there are 12 studies involving acupuncture (7 of acupuncture versus lactulose, 4 involving acupuncture versus sham, and one comparing two different forms of acupuncture), for some interventions, there are only two studies (range: 1-8). None of this stopped the authors from paddling bravely up the river of bad science by comparing all the studies and comparing acupuncture with each of the other interventions and concluding:

By including more than 40 articles for network meta-analysis (NMA), acupuncture was found to be more effective than drugs in treating chronic constipation with nearly no side effects. The mechanism of action of acupuncture for the treatment of chronic constipation remains unclear, but previous studies have also shown its possible role in the treatment of chronic constipation. An important cause of chronic constipation is the decreased motility of the colon, resulting in a decreased urge to defecate, bloating, and abdominal pain or discomfort. [51] Existing studies indicate that acupuncture at heterotopic acupoints facilitates distal colonic motor ability by activating M3 receptors and somatic afferent C-fibres in normal, constipated, or diarrhoeic rats. Acupuncture can also increase distal colonic motility in normal and abnormal mice with a possible role of M3 receptors [52]. Previous studies have suggested that the ST36 acupoint can change the motor function of the colon. [53]

That’s right. The authors concluded that acupuncture was more effective than any drug commonly used to treat chronic constipation and that it had “nearly no” side effects. That’s rather convenient, given that only one of the acupuncture studies included in the meta-analysis even bothered to report side effects.

That’s not all, either. There’s the way that the authors assessed effect size. Basically, they measured and compared the proportion of patients “with improvement in chronic constipation.” No measure is made of how much constipation improved, whatever tool various investigators might have chosen to use to assess severity of constipation in each study. Think of it this way by considering two hypothetical studies. One study shows that 50% of patients saw their constipation improve based on one treatment, and those 50% averaged a 90% improvement. Now consider another study that shows that 50% of patients improved, but by only an average of 20%. Those two studies would count the same because both report that 50% of their patients got better, but obviously they do not show the same result. Obviously, you’d want the treatment with an average improvement of 90% rather than 20%.

But that’s still not all. Let’s take a look at the Forest plot of the conventional paired meta-analysis that the authors carried out before doing their network thing:

The authors report that, compared with placebo (J), acupuncture significantly improved the symptoms of chronic constipation, as did most of the other interventions. Note also, though, that sham acupuncture also improved chronic constipation symptoms with an odds ratio of 3.1 (95% confidence interval 1.5-7.2). Just look at the lower panel. That’s as good as many of the drugs. There’s a red flag right there, as far as I’m concerned. There’s a reason for this, as described by Ross Pomeroy:

The vast majority of the laxative studies were double-blind, randomized, placebo-controlled trials published in peer reviewed journals. On the other hand, all of the acupuncture studies were unblinded, four were not peer-reviewed, five were published in questionable journals like the Shanghai Journal of Acupuncture and Moxibustion, and all were based in China, which is notoriously friendly to acupuncture and alternative medicine. Thus, it’s no surprise that acupuncture showed stronger results in those studies.

So basically what we have here is a perfect example of what acupuncturists do to try to “prove” that their quackery works for a wide variety of conditions. It is, however, a new wrinkle in bad acupuncture meta-analyses. Instead of just GIGO, it compares garbage (the results of most acupuncture studies) with the results of better quality studies (the studies of medications) in order to conclude that the garbage is even better than medicine. Never mind that ten different acupuncture points were used, that sham acupuncture “worked” almost as well as acupuncture, and that, well, their study is the best meta-analysis so far:

However, a previous meta-analysis showed no significant curative benefits of acupuncture compared with sham acupuncture in the treatment of chronic constipation. Only some studies from China have shown that acupuncture is superior to drugs. [54] A previous study from China showed that acupuncture and sham acupuncture had similar effects on functional constipation, but moxibustion was used with ST23 and ST27 acupoints in this study. Moreover, this study was based on a small sample size [55], and therefore, we are unable to ensure the accuracy of these findings. In addition, the curative effect of sham acupuncture remains to be confirmed.

Our study was unique in that the included studies of acupuncture used the deep pricking method instead of the ordinary acupuncture method. The depth of the deep pricking method is deeper than that of ordinary acupuncture method. Further studies are needed to confirm whether the deep pricking method has a better curative effect on chronic constipation than ordinary acupuncture. In the present study, many studies conducted after 2012 were included and the studies using the same acupuncture points were included to ensure consistency. RCTs using consistent acupuncture points will yield more definitive results.

I note that the outcome evaluated in the meta-analysis mentioned above was the change in the number of weekly spontaneous bowel movements.

This is the sort of thing acupuncturists do, though. Here’s another example, but not a meta-analysis. Rather, it’s a recently published randomized controlled acupuncture trial to treat chronic constipation that compares He acupuncture, Shu‐mu acupuncture, He‐shu‐mu acupuncture, and orally administered mosapride. Notice something? There’s no placebo control, no sham acupuncture control, and, you’ll notice if you read further, no blinding. Believe it or not, the mosapride group functioned as the control group. Surprise! Surprise! All groups improved, which is exactly what one would expect in such an unblinded trial. Basically, this is as worthless a clinical study as I’ve ever seen, and Neurogastroenterology & Motility should be ashamed for having published it.

Back to the meta-analysis, though. Not surprisingly, physicians and scientists were puzzled at how PLOS ONE could publish dreck as bad as this and took Twitter:

There’s bad studies, but this #acupuncture nonsense is on a whole other level, shame on @PLOSONE https://t.co/L2u127tZwG by @RCScience (via @ACSHorg) cc @CaulfieldTim @gorskon @McGillOSS @ScienceBasedMed

— Philippe Chouinard (@DrPChouinard) April 29, 2018

Steve Salzberg even resigned from the editorial board over this article:

I agree, this is appalling bad pseudoscience that never should have been published. I just wrote to @PLOSONE and officially resigned from their Editorial Board. https://t.co/9bbBQtvlY0

— Steven Salzberg (@StevenSalzberg1) April 29, 2018

PLOS ONE even responded:

Thank you very much for bringing these concerns to our attention. We are looking into this matter and will also be in touch with you separately to discuss further.

— PLOS ONE (@PLOSONE) April 30, 2018

We’ll see what happens. After all, as James Coyne has noted, it took PLOS ONE a long time to retract a more obviously dubious study:

Please to see @PLOSONE today retracted stealth #homeopathy article, but process should not have taken years, even if authors objected. https://t.co/WhJXU3X8rI #retraction https://t.co/QlPxm75UhQ

— James C.Coyne (@CoyneoftheRealm) May 3, 2018

Here’s the retraction notice:

Following publication of this article [1], concerns were raised about the scientific validity of the study as well as a potential competing interest that was not declared. The PLOS ONE Editors discussed the concerns with the authors and consulted external experts. In light of our editorial assessment and advice received in the expert consultations, we are retracting this article due to concerns about the scientific validity of the research question, study design, and conclusions.

Specifically, we are concerned about the overall design of the study, which aims to detect effects of a reagent diluted to such a degree that the solution is not expected to contain biochemically relevant levels of antibody. The consulted experts also raised concerns about the validity and rigor of the immunoassay system used in the study. The enzyme-linked immunosorbent assay (ELISA) used was adjusted to give barely detectable signals, which renders the assay particularly susceptible to interference. In light of these issues, we consider that the article does not present sufficient or reliable evidence to support the conclusions.

Stealth homeopathy paper indeed. I note that this paper was first published in 2014. PLOS ONE also notes undisclosed conflicts of interest.

Unfortunately, it turns out that this is not the first time PLOS ONE has fallen prey to acupuncture pseudoscience. For instance, PLOS ONE published one of Helene Langevin’s dubious studies trying to “map” acupuncture meridians using a combination of electrical impedance measurements and ultrasound. In addition, it also published a bad fMRI study of “laser acupuncture” purporting to show effects on neural activity in the brain. And let’s not forget that the followup to Andrew Vickers’ original highly dubious acupuncture meta-analysis was published by PLOS ONE. Indeed, acupuncture pseudoscience seems to have found a home on PLOS ONE. Just for yucks, I did a search on the journal’s website, yesterday, and what did I find? Page after page of search results of studies published about acupuncture. I might very well have to look into some of them, because they look bad. Indeed, although I can’t tell you which paper I reviewed, I can tell you that, the one time I peer reviewed an acupuncture manuscript for PLOS ONE, the pressure was to publish, as I recall its being two reviewers against me. Yes, I might very well have to revisit this topic.

Scientific and clinical studies of pseudoscience like acupuncture and homeopathy pose a significant challenge to journals, including PLOS ONE. The number of people who both know enough about the modalities being tested to evaluate them and are not advocates of those modalities is small. This results in reliance on peer reviewers who are supporters or practitioners of the quackeries being evaluated. It also allows an acupuncture believer associated with Deepak Chopra like Neil Theise to spin highly preliminary findings having nothing to do with acupuncture as evidence supporting acupuncture. Still, I have to wonder what went wrong at PLOS ONE in this particular case. I can sort of see how the stealth homeopathy paper could fool a peer reviewer not familiar with homeopathy, but I have a hard time seeing how this network meta-analysis paper could pass muster. Whatever the case, PLOS appears to have a problem, and acupuncture still doesn’t work—for constipation or anything else.