I’ve become known as an advocate for evidence-based medicine (EBM) in the three years since I started this little bit of ego gratification known as Respectful Insolenceâ¢. One thing this exercise has taught me that I might never have learned before (and that I’ve only reluctantly begun to accept as true) is one huge problem with EBM. Not surprisingly, it has to do with how EBM is used to evaluate so-called “complementary and alternative medicine” (CAM) therapies, many of which are highly implausible on a scientific basis, to put it exceedingly generously.
Consider homeopathy, which on a strictly physical basis is about as implausible as it gets based on very well-established science. There’s almost no doubt on a scientific basis that the concepts behind homeopathy are nothing more than magical thinking writ large and then justified with all sorts of pseudoscientific mumbo jumbo invoking everything from the “memory of water” to ridiculous torturings of quantum mechanics that make me wonder just what these people are smoking. However, it’s not hard to find seemingly positive studies suggesting that homeopathy actually does something more than provide a bit of dihydrogen monoxide to the body. True, the better the study, the more likely it is to be negative, with no efficacy shown greater than placebo, but there are some seemingly well-designed studies that purport to show an effect. Granted, the effect is always small and I’ve yet to see any scientifically convincing reports of homeopathy curing cancer or other non-self-limited diseases, but that doesn’t stop the homeopaths.
Or the other purveyors of antiscientific woo, for that matter.
John Ioannidis has, to a great extent, explained why there are so many positive trials of CAM therapies of exceedingly low scientific plausibility. Basically, it has to to with prior probability. Basically, the lower the prior probability of a hypothesis to be tested in a clinical trial to be true, the more likely there is to be false positive trials, far more than the expected 5% false positives that would be expected under ideal conditions using a p-value of 0.05 as the cutoff for statistical significance. Even being aware of this problem, we as advocates of science- and evidence-based medicine have a hard time swatting down individual studies. To the layperson, saying that we must evaluate the totality of the literature is an unsatisfying response to the seemingly positive studies that homeopaths, for example, routinely like to cherry pick.
As I’ve come to realize, the elephant in the room when it comes to EBM is that it relegates basic science and estimates of prior probability based on that science to one of the lowest forms of evidence, to be totally trumped by clinical evidence. This may be appropriate when the clinical evidence is very compelling and shows a very large effect; in such cases we may legitimately question whether the basic science is wrong. But such is not the case for homeopathy, where the basic science evidence is exceedingly strong against it and the clinical evidence, even from the “positive” studies, generally shows small effects. EBM, however, tells us that this weak clinical evidence must trump the very strong basic science, the problem most likely being that the originators of the EBM movement never saw CAM coming and simply assumed that supporters of EBM wouldn’t waste their time investigating therapeutic modalities with an infinitesimally small prior probability of working. But CAM did infiltrate academic medicine, and investigators do investigate such highly unlikely claims. So what to do?
Dr. Kimball Atwood IV explains one proposed solution: the adoption of Bayesian inference for evaluating clinical trial data over the “frequentist” statistical evaluations of clinical trials that have been dominant throughout the careers of every physician now alive. Money quote:
If the prior probability of a hypothesis is small, it will require a large amount of credible, confirming data to convince us to take it seriously. If the prior probability is exceedingly small, it will require a massive influx of confirming data to convince us to take it seriously (yes, extraordinary claims really do require extraordinary evidence).
Which is what I’ve been saying all along with respect to homeopathy, reiki, distance healing, and many other CAM therapies that, on a physical, scientific basis, are exceedingly unlikely to work.
I encourage skeptics and CAM advocates alike to read Dr. Kimball’s post in its entirety and comment.
