Clinical trials Complementary and alternative medicine Medicine

NCCAM and “pragmatic trials.” Again.

I’m not alone in pointing this out, but if there’s one thing about research and clinical trials into “complementary and alternative medicine” (CAM) that has become very apparent to me over the years, it’s that the more rigorous the study the less likely it is to show an effect. In normal research, the usual progression in clinical research goes from small pilot and observational studies to small to medium-sized randomized studies, ultimately culminating in large randomized, double-blind studies that are tightly controlled to eliminate as many potential biases and confounders as possible, as well as to minimize placebo effects. Of course, when you do that for studies of most CAM interventions, what you will almost always find is that they have no effect. In science-based treatments with real effects, the effect will persist as the trials become more rigorous. True, the effect size might decrease, as biases are eliminated with increasingly rigorous trials, but the effect will persist. Not so for an ineffective treatment with homeopathy. You might see effects in preliminary studies, which tend to be uncontrolled, unblinded, and prone to bias and placebo effects, but decreasing effect size as the rigor of clinical trials increases is pretty much the sine qua non of CAM therapies.

All of which is why CAMsters like what are known as “pragmatic trials.” Pragmatic trials are a type of clinical trial meant to mimic the “real world” in order to study how treatments work outside of the cozy, controlled confines of randomized clinical trials. Such trials are said to measure “effectiveness,” while randomized clinical trials measure “efficacy.” Now, pragmatic trials have their place. For a treatment that has been shown to be efficacious in randomized clinical trials, it can be useful to test it under less-controlled situations to see if it’s still effective. The reason is that, once a drug or treatment is approved and reaches wider clinical usage, it is suddenly used on a much wider variety of patients, patients who don’t fit the strict inclusion and exclusion criteria of a good clinical trial. Often that means that the effectiveness is lower than the efficacy. However, there is an exception, and that’s when a treatment has little or no efficacy. In that case, often the effectiveness will appear to be greater than the efficacy, thanks to placebo effects, confirmation bias, expectancy effects, and the like.

So which sorts of studies do you think that the National Center for Complementary and Alternative Medicine (NCCAM) is interested in? Just check out this recent post on the NCCAM blog by NCCAM director Josephine Briggs entitled Let’s Get “Pragmatic” About Pain:

The “real world” question is an active topic across the National Institutes of Health (NIH), reflected in growing interest in what are being called effectiveness studies or pragmatic trials. Of course, NIH clinical studies happen in the real world, but usually under conditions that are tightly specified and controlled. In the typical interventional study—whether it is of a new drug, a procedure, or a new behavioral approach—simultaneous use of other therapies is limited, patient eligibility criteria are closely specified, only highly experienced practitioners are engaged, nurse coordinators closely monitor compliance, and so on. Careful control of all aspects helps ensure study results can be replicated. It helps to create “internal validity.” And, by reducing the sources of variability, we reduce the need for large numbers of participants. But, there is a tradeoff in this approach to trial design. At least sometimes, when the results are implemented in real-world conditions, the intervention does not work as expected. Hence, the concept is gaining acceptance that we need both explanatory studies to test the efficacy of therapies and pragmatic studies to examine real-world effectiveness.

Yes, pragmatic studies are becoming more accepted these days, but not unless it is already first known that the treatment being subjected to the pragmatic study actually shows efficacy in well-controlled randomized trials. At least, they shouldn’t be. CAM practitioners and quackademics publishing dubious CAM studies tend to short circuit the whole efficacy part and head straight for the apparent effectiveness part. NCCAM, it would appear, is no different, as Briggs then goes on to write:

There are many practical questions emerging from our pain portfolio that seem ready for a more pragmatic approach. We have growing evidence, reflected in systematic reviews and practice guidelines, that a number of the mind-body therapies can have real benefit in pain management. But these results raise many new questions. Do these approaches improve patient well-being when they are integrated into primary care settings? What works well in pain clinics that see referral patients? Do the mind-body approaches reduce opioid abuse? What patient populations are best targeted? How can providers effectively encourage models of self-management?

In other words, let’s do pragmatic studies on “mind-body” therapies (whatever that means), so that we are more likely to find the appearance of effectiveness. Then let’s use that appearance of effectiveness as an “evidence base” to justify “integrating” quackery with science-based medicine. No, I don’t think that Dr. Briggs is consciously advocating this. I think she’s let herself be sucked into the culture of NCCAM and the mindset that rules there. She started out as a respectable scientist with a sincere desire to promote and maintain scientific rigor at NCCAM. Unfortunately, it is an impossible task. When you are placed in a situation where your mission is to study treatments that do not have a preclinical scientific basis, such as acupuncture, reiki, homeopathy, and the like, the only way you can do it is to bypass the normal process of translational science that progresses from observation to basic science to clinical trials and go straight to clinical trials.

NCCAM was the product of a woo-loving Senator (Tom Harkin) throwing his influence around. The NIH didn’t want it. There was no scientific need for it, and scientists didn’t ask for it. Science-based physicians certainly didn’t want it either. Yet we got it. It was forced upon us by a politician who thought that bee pollen had cured his allergies. Unfortunately, it also appears to be here to stay, and to continue to promote pseudoscience through the use of “pragmatic” trials.

By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

16 replies on “NCCAM and “pragmatic trials.” Again.”

Richard Feynman noted that one of the most important things to do in the course of your scientific research is to make sure you are not fooling yourself. He was talking about physics, but it applies equally to medical research. Self-deception is remarkably easy. That’s why controlled trials are so important: they are a check on self-deception. Pragmatic trials are not. They are still useful for ensuring that what works in principle will work in practice, a point which comes up in other scientific fields as well (e.g., a number of mathematical techniques that physicists learn work analytically but fail spectacularly when you try to use them on computers with finite precision). But using pragmatic trials alone puts CAM firmly into the realm of what Feynman called “cargo cult science.” The forms are there, but the planeloads of cargo (in the form of scientifically valid results) do not arrive.

Yes, what is mind-body? I wish I knew!

Studies like these are exactly the type that will be announced loudly at places like PRN and Natural News.
NCCAM provides grist for their mills- and they are mills, grinding out woo ceaselessly.

One large hospital I know of has a mind-body program that seems to focus largely on stress reduction and the effects of stress on the body, but I’ve also seen hints that it may also involve things like reiki and such.

As some jackbrain once said to Randi after a failed experiment, “this is why we don’t do double-blind tests–they never work”. If good science doesn’t support your claim, try Bad Science!

And look at all of the cranks who flock in to join Briggs’ pragmatic movement… “yeah, I am a reflexologist and this sounds like a great idea!” Such is the company kept by she who creeps away from integrity.

@Dr. Coleslaw Burpinsky:
It may have been said to Randi as well, but the version of it that I heard was said to psychologist Ray Hyman with regards to ‘Applied Kinesiology’, a weird form of diagnostics that involves assuming the ideomotor effect is actually unlocking truths.

When these results were announced, the head chiropractor turned to me and said, “You see, that is why we never do double-blind testing anymore. It never works!” At first I thought he was joking. It turned it out he was quite serious. Since he “knew” that applied kinesiology works, and the best scientific method shows that it does not work, then — in his mind — there must be something wrong with the scientific method.

@ Todd W.:

Stress Reduction is an area where woo can creep in: first you’re doing meditation and deep breathing and the next thing you know, they’re talking about past lives!

More seriously, it seems that t’ai chi and yoga as exercise/ recreation can be useful to many people BUT their roots in the Mysterious East predestine their utility to woo meisters.
( In the 1990s, I actually tried some forms out as research for clients who had issues with stress and serious illness).

Note to self:
why is it also the East that beckons the woo-entranced?
Seriously, reading through the folklore and magic of Europe – via Frazer- I am reminded how some of our ancestors, even a few hundred years ago or less, believed in superstitions that rival those of Asia.

And -btw- north, south, east or west- all of Europe has magical thinking percolating beneath the surface.

Perhaps woo-meisters might feature ancient European woo. Consider the possibilities.

“I am reminded how some of our ancestors, even a few hundred years ago or less, believed in superstitions that rival those of Asia.”

I worked with two people who literally believe in witchcraft, one of whom also thinks mental illness is demons.

I think one of my current co-workers thinks vampires are real and the Serbs were talking about the famous one currently haunting some village back home…


Is it my imagination, or has Age of Autism become significantly more fringe this year? I swear their earlier articles used to begin mild mannered and superficially logical, but now it’s straight into CTs and links to sites that are just this side of

why is it also the East that beckons the woo-entranced?

My money’s on the Theosophical diaspora.

I don’t know whom I hate more, dedicated quacks or fair-minded bureaucrats like Dr. Briggs.

@ Nick:

It was unblinded and for a condition highly susceptible to placebo effects. So no, no real merit to it. Hypothesis-generating, but not sufficient to demonstrate any actual effect.

Comments are closed.