The director and deputy director of NCCAM pontificate about “scientific plausibility”

One of the overarching issues, if not the overarching issue that makes so-called “complementary and alternative medicine” (CAM)—or, as it’s now more commonly called, “integrative medicine”—so problematic is prior plausibility. It’s also one of the most difficult to explain to the lay public, because to someone not trained in science it can sound like not being open-minded. I like to joke about this whole concept by saying that it’s good to be open-minded but not so open-minded that your brains fall out. In other words, the main difference between science-based medicine (SBM) and evidence-based medicine (EBM) is how the two deal with this very issue. In EBM, clinical trial evidence, particularly from randomized clinical trials, are the be-all and end-all. I exaggerate, but only a little. Sure, other forms of evidence are present in the EBM hierarchy of evidence, but basic science considerations are relegated to the very lowest rung on the ladder of evidence. In contrast, in SBM, considerations of prior plausibility are much more important, and what determines prior plausibility is the evidence base supporting a therapy, in particular basic science considerations.

One common misconception about SBM is that it requires that we know the mechanism by which a drug or treatment works before we can consider it. In other words, when we reject something like homeopathy as being about as close to completely implausible as it is possible to be, homeopaths will claim that we are saying that, because we don’t know how homeopathy works, we’re dismissing it prematurely. They might have a point if homeopathy actually did, in fact, work, but it doesn’t. Even so, it’s a straw man. When supporters of SBM say that a treatment is implausible on considerations of prior plausibility, we mean something much more specific. Basically, we mean that basic science considerations alone are enough to tell us that the treatment is physically impossible. Take the example of homeopathy again. It’s my favorite example because it’s so easy to use to illustrate my point. Basically, for homeopathy to work, science would have to not just wrong, but spectacularly wrong, about several well-established physical laws and theories in chemistry, physics, and biology. This is far different than saying that something can’t work because we don’t know its mechanism. The same is true of reiki. Or pretty much any form of “energy healing.” Such therapies are so implausible on basic science considerations alone that, absent compelling evidence that they actually work better than placebo (evidence so compelling that it makes us doubt our understandings of physics), it is a waste of resources to do clinical trials.

This brings us to the problem of the National Center for Complementary and Alternative Medicine (NCCAM). It is a highly problematic institution because it is dedicated to studying, in essence, a large number of treatments whose scientific plausibility is highly questionable, while at the same time it also studies therapies that have scientific plausibility, such as diet and exercise, but that have been “rebranded” as somehow “alternative” when in fact they are nothing more than SBM—or should be. The reason I’ve started with this rambling introduction designed for newbie readers of this blog who might not be familiar with these concepts derives from two recent posts on the NCCAM research blog. I tell ya. I ought to send Dr. Josephine Briggs, the director of NCCAM, a thank you note for the amount of blogging material she’s been providing me lately through that blog. This time around, though, it’s both Dr. Briggs and her deputy director, John Killen, Jr., MD providing the blog fodder, Dr. Briggs through a post entitled Our Framework for Research Priorities, which is echoed by her loyal deputy in a post from yesterday entitled On Scientific Plausibility. Together, these two posts suggest that maybe, just maybe, the criticisms that those of us supporting SBM have leveled against CAM in general and NCCAM in particular might be having an effect.

Dr. Briggs cites the latest NCCAM five year strategic plan, a document that I covered extensively when it was published, sarcastically referring to it as saying, “Let’s do some real science for a change!” You think I joke, but I wasn’t. That really is basically what the strategic plan said, admitting that NCCAM funded some crap science in the past (such as studies of homeopathy and distance healing) and, basically promising to do better. The problem is that, after all these years and millions of dollars, NCCAM still wants to have it both ways. It wants to claim to do rigorous science without fully embracing what, exactly, doing rigorous science entails. For example, Dr. Briggs states her post that these are the sorts of questions that NCCAM asks about research grant applications it considers for funding:

These factors, and some examples of questions NCCAM might ask related to each factor, are as follows:

1. Scientific Promise.
   How strong is the body of evidence supporting the concept?
2. Amenability to Rigorous Scientific Inquiry.
   Are there reliable and reproducible methods—e.g., for diagnostics, outcome measures, biological effects, quality control?
3. Potential to Change Health Practices.
   Is it reasonably likely that the results will make a difference to consumers, providers, or policymakers?
4. Relationship to Use and Practice.
   Do the methods and approaches actually address the most important questions about use or practice in the real world?

It’s very hard not to note that, in terms of criterion #1, very few therapies that fall under the rubric of CAM fare very well. When it comes to CAM like homeopathy and “energy healing,” not only is the clinical evidence incredibly unimpressive, but such therapies are completely implausible from basic science considerations alone. Even more “plausible” CAM therapies, such as herbal remedies and acupuncture rarely have an evidence basis that any reasonable scientist could term “robust”; that is, unless he’s either drunk or messing with you. Indeed, the very term CAM is the name for a double standard. If CAM therapies met criterion #1, they wouldn’t need to be treated as a separate category. As the old cliche goes: What do you call “alternative medicine” that has been shown to work by science? Medicine. I would put a twist on that question and ask: What do you call “alternative medicine” that’s plausible enough from a basic science standpoint to warrant a clinical trial? Experimental medicine. See what I mean? When CAM goes up against real science, it’s like golf being played between a pro and me. Without a massive handicap, I would have no prayer. CAM is basically the duffer of scientific medicine. And what’s adding to CAM’s handicap? Big money institutes like the Samueli Institute and the Bravewell Collaborative and, of course, NCCAM and its $125 million yearly budget.

Criterion #2 might explain why NCCAM has during Briggs’ tenure moving away from the more “out there” CAM therapies, like homeopathy and distance healing. Not only are these sorts of interventions not easily amenable to rigorous scientific inquiry by the sub-criteria listed by Briggs, but in fact, as I’ve written about time and time again on this very blog, CAM practitioners frequently say that they want “rigorous” scientific investigations of their magic nostrums, but when the rubber hits the road they are not nearly as eager as they let on to have their treatments subjected to truly rigorous scientific investigation and randomized clinical trials; hence the preference for “pragmatic” trials among, for example, acupuncturists. These trials are listed as being “real world” and studying “real world efficacy,” but in reality they are putting the cart before the horse. Prove efficacy first in well-designed randomized clinical trials. Then—and only then—look at “real world” effectiveness.

Criteria #3 and #4 are where the true uselessness of Dr. Briggs’ criteria reveal themselves. They are perfectly reasonable criteria for SBM. However, when applied to CAM, not so much. For example, CAM practitioners almost never change their treatments or methods, no matter how much evidence accumulates against them. I mean, really. There are still homeopaths out there! There are still reiki practitioners, who believe that they can channel healing energy from the “universal source” (whatever you do, don’t call it god) by holding their hands in special positions over their clients (whatever you do, don’t call it the laying on of hands). Does Briggs really think that acupuncturists, reiki practitioners, herbalists, practitioners of traditional Chinese medicine, and the like will change their practice if research funded by NCCAM shows that some of their nostrums don’t work? Has CAM ever abandoned a treatment that’s been shown not to work? Heck, there are still quacks out there hawking Laetrile, high dose vitamin C for cancer, and the Gonzalez therapy for cancer, even though these have all been shown quite conclusively to be ineffective.

Dr. Briggs’ post was short, sweet, and to the point, even if it was misguided. Dr. Killen’s post is a bit longer, which means that it gives him ore time to get himself into trouble. It doesn’t take very long at all to admit that a lot of CAM is nonsense but nonetheless argue that was should study it anyway:

Scientific plausibility permeates discussions and debates about research on complementary, alternative, or integrative health approaches. This is no surprise; many interventions that fall under this rubric are ensconced in belief systems about illness and health—some ancient and some modern—that lack foundations in modern science. In addition, those who support research on these approaches often fail to articulate a scientifically grounded rationale or approach to research. Thus, it is common to see criticism based on scientific plausibility in the scientific literature, news stories, and blogs.

No doubt. Count this as one of the blogs and myself as one of the bloggers who have repeatedly leveled this very charge against NCCAM, CAM research, and the very concept of “integrative” medicine (or, as I like to refer to it, “integrating” quackery with SBM). Of course, Dr. Killen appears to be admitting that, yes, a lot of CAM is unscientific and a lot of CAM practitioners and advocates of CAM research can’t defend their woo based on science. I can’t argue with that. I’ve said the very same thing many times. The difference is, of course, that I wasn’t making such statements as a prelude to arguing that, even though CAM is utter pseudoscientific nonsense, we should spend millions of dollars to study it anyway. Dr. Killen is constrained by no such qualms:

This criticism often suggests that the existence of research implies either belief in scientifically implausible explanations or ignorance of basic scientific principles and concepts. So how do we justify investment of public resources in research on complementary interventions that are associated with pre-scientific or unscientific explanations? Simply, it is both possible and necessary to disconnect scientific interest from unscientific “trappings.” For example, an objective look at the body of accumulated evidence (from patient reports, clinical observations by many good clinicians, and clinical studies) suggests that some people with chronic low-back pain are deriving meaningful benefit from acupuncture, yoga, or procedures involving spinal manipulation. It is entirely possible to be scientifically curious about that body of evidence and investigate it further, while not in any way embracing scientifically unfounded explanations for those practices. For instance, it is not necessary to believe in meridians or qi to study the effects of the procedure of acupuncture on pain, or to explore the hypothesis that acupuncture mediates pain by conditioning or expectancy effects produced by a convincing ritual combined with a counter-irritant.

This is a bit of a straw man. We don’t imply by such criticisms (at least I don’t imply by such criticisms) that the researchers studying, for instance, acupuncture believe in qi flowing down meridians, but it’s certain that many, if not most, practitioners “on the ground” do. I’ve encountered them. Let’s put it this way, using Dr. Killen’s own example. If you’re going to study “counter-irritation,” which is not an entirely implausible mechanism by which sticking needles into the skin might lead to a perceived reduction in pain (although, from my perspective, it’s not particularly plausible, either, from a physiologic standpoint), then study counter-irritation, rather than acupuncture. The very structure of acupuncture trials buys into the mystical concepts that underlie it: “controls” that are defined by not sticking the needles in the “right places” along meridians, for instance. If a study of acupuncture were truly studying counter-irritation, it wouldn’t need to be dressed up in the mystical, magical, religious trappings of acupuncture. No, Dr. Killen, these studies are of acupuncture, not counter-irritation, and, worse, rather than asking if acupuncture works, they tend to proceed from the premise that acupuncture does work!

NCCAM can’t do that, of course, because it is those very trappings, the qi, the meridians, and various other prescientific, mystical, and religious concepts that underlie so much CAM that distinguish CAM from SBM. Without them, NCCAM has no reason to exist. There is nothing about “counter-irritation” that requires a special Center in the NIH to oversee studies of it. Without qi and meridians, there’s nothing in acupuncture that can’t be studied by the National Institute of Neurological Disorders and Stroke, which funds a lot of the non-woo chronic pain research supported by the NIH. Similarly, strip away the whole mystical aspect of herbalism, and all that’s left is pharmacognosy, the study of medicines derived from plants and other natural sources. If you take the “innate intelligence” and “subluxations” out of chiropractic, all you’re left with is manipulation of the sort that is better performed by physical therapists. Woo is what makes NCCAM. If you strip the woo from all the CAM therapies whose study NCCAM funds, there’s no reason at all that NCCAM couldn’t be dismantled and its components distributed to the appropriate centers and institutes of the NIH. It’s not for nothing that I half-jokingly have invoked the Internet meme that we should take off and nuke NCCAM from orbit. It’s the only way to be sure.

What amazed me more than anything, though, is an admission that Dr. Killen makes at the end. First, he repeats rather emphatically his straw man that “it is a mistake to assume that scientific inquiry is equivalent to acceptance of unscientific mechanistic thinking,” using this as a prelude to admit:

NCCAM’s first decade entailed a relatively broad and investigator-initiated approach to funding. This was appropriate to the time and the state of the available scientific evidence. The four factors we now consider evolved out of lessons learned during those years. So with the benefit of hindsight, it is pertinent to note that a number of studies funded during that timeframe would probably not be funded today because they could not pass our current hurdles regarding plausibility. In fact, the portfolio of research NCCAM has actually funded over the past several years demonstrates clearly that both the peer review process and NCCAM are now using these factors to shape our investments in research.

I can’t believe that Dr. Killen actually wrote that! Did you see what he just admitted? Basically, he just admitted that many of the studies funded by NCCAM back in the day were, from a scientific perpective, utter crap, just as I’ve been saying all along. He even goes so far as to point out that the studies funded a few years ago, during NCCAM’s first decade of existence as a center, wouldn’t pass muster now. Actually, on the basis of the horrifically tight pay lines all over the NIH alone, I’d bet that a lot of those studies wouldn’t have been funded now even if NCCAM hadn’t made a single change to its strategic focus, just as the first NIH grant I ever got back in 2005 would not have been funded today because the paylines have shrunk so much since then that it wouldn’t have passed muster if I had submitted it today. Be that as it may, Dr. Killen has just confirmed more than I would ever have dreamed possible that I was right on when I described NCCAM’s new strategic plan as “Let’s do some real science for a change!”

Unfortunately for Dr. Killen, his pleas to CAM researchers “to carefully parse rationale from “trappings” and give due recognition to the validity of concerns about scientific plausibility” notwithstanding, he’s in a no-win situation. These trappings are inherent to the mixture of various quackeries that make up CAM. They cannot be so easily “extracted,” to leave plausible scientific hypotheses behind. Moreover, the few CAM modalities that have scientific plausibility, such as nutrition and exercise, could just as easily and more appropriately be studied by other institutes and centers at the NIH.

NCCAM is a completely superfluous waste of taxpayer money.