How not to do a study on the efficacy of “alternative” medicine

ResearchBlogging.orgIf there’s one thing I’ve learned over the last four years of examining the various forms of woo out there, it’s to be very, very skeptical whenever an advocate of a highly dubious-sounding “therapy” points to a study as “proof” that the therapy, whatever it is, works. Usually, what I find is a small pilot study with inadequate controls or even a poorly designed study. For example, the acupuncture literature is rife with these sorts of studies. It’s also rife with larger studies for which the control was inadequate–or for which there was no real control at all. This phenomenon is generalizable to many, if not most, studies of so-called “complementary and alternative medicine” (CAM), as is another feature, namely that the larger and better designed the study, the less likely it is to find a treatment effect greater than placebo due to the treatment. Another principle is that at a statistical significance level of 95%, at least 5% of studies will appear to find a treatment effect through random chance alone. Guess which studies will be cherry picked and held up as “proof” while the preponderance of studies showing no effect are ignored?

There is at least one other form of studies pointed to by CAM advocates to “prove” that their woo “works.” Indeed, this form is perhaps their favorite crutch to fall back on. It’s what I like to call the “non sequitur” study. In other words, it’s a study that is, at best, only tangentially related to the question at hand, or, as I like to put it, a study that is related to the therapy being argued for only by coincidence. This sort of study is a favorite of homeopathy. Just think of studies about the molecular bonds of water homeopaths like to point to as “evidence” for the “memory of water.” It is this latter form of study that I’m going to deal with here.

Remember about three weeks ago, when I had a bit of fun with one of the most hilariously ludicrous bits of woo that I’ve ever seen, Tong Ren? If you’re really new to the blog and didn’t happen to read my post, I encourage you to go back and do so now. If you do, you’ll see, besides my own inimitably insolent prose, YouTube videos of a man named Tom Tam leading a bunch of people tapping on acupuncture dolls with small hammers and concentrating their “intent” to “heal” a person. I’ve seen a lot of woo before. A lot of woo. But Tong Ren was about the most ridiculous things I’ve ever seen. These people really believe that by taking what looks in essence to be a voodoo doll and tapping on it at the correct acupuncture points, they can direct some vague “energy” undetectable by science to cure cancer and all manner of other diseases. One thing you’ll also see is a news report that mentions a study of Tong Ren being done at the Dana Farber Cancer Institute, a very prestigious institution indeed. Naturally, at every opportunity, the connection to Harvard University is played up, so desperate is Tom Tam to wrap himself in the mantle of seeming legitimacy that the attention of Harvard University suggests. When I poked around various Tong Ren websites, I couldn’t find out anything about the study other than that it appeared to be some sort of survey and that the manuscript had been submitted.

It appears that the manuscript has been accepted and was published recently in a journal I’ve never heard of, namely Complementary Health Practice Review. The article, entitled The Tong Ren Healing Method: A Survey Study, by Amy M Sullivan, EdD (Virginia Commonwealth University School of Medicine), Susan Bauer-Wu, PhD, RN (Emory University), and Michael Miovic, MD (Dana Farber Cancer Institute). Of course, that this study appeared in a journal called Complementary Health Practice Review does not bode well for its quality, and this is no exception. More importantly, the hypothesis examined by this study is related only by coincidence to the hypothesis that Tong Ren “heals” anything. Check out the abstract:

Tong Ren (TR) is an untested energy healing modality with anecdotally-reported effectiveness for a variety of disorders. Study objective: To describe participant reports of effectiveness and safety. Design: Cross-sectional, anonymous survey. Setting: Weekly group sessions in the Northeast US. Participants: Adults attending group sessions. Measures: Changes in conditions attributed to TR. Results: Response rate 89% (n = 265). Cancer (30.6%), endocrine/autoimmune (17.5%) and musculoskeletal disorders (17.2%) were the most commonly reported conditions. Among respondents who had attended more than one session (n = 216), 30% used superlatives (e.g., “amazing”) to describe TR’s impact, and one-third noted improved quality of life. No adverse effects were described. Anxiety, depression, cancer, and autoimmune disorders appeared to have the greatest treatment responses, with 63.8%, 61.0%, 60.3%, and 58.1% of participants with these conditions reporting substantial improvements. Conclusion: This first study documenting self-reported effects of TR shows subjective benefits and no adverse effects. Further research on this approach is warranted.

In other words, what Sullivan et al did was to collect anecdotes. That’s it. Other than that, she characterized the characteristics of the people who sought out Tom Tam’s particular brand of woo. The article begins:

The National Institutes of Health National Center for Complementary and Alternative Medicine (IH NCCAM) has described energy medicine as a domain in complementary and alternative medicine (CAM) in need of scientific research (National Institutes of Health [NIH], 2007). Tong Ren (TR) Healing is a relatively new modality of energy medicine developed in 2001 by Tom Tam, a practitioner of acupuncture and other forms of Traditional Chinese Medicine (TCM;

Ugh. Right from the beginning I can tell I’m dealing with believers, or at least people who are so open-mined that their brains fell out. Don’t believe me? Then get a load of this next section:

The TR method blends TCM with Western biomedical knowledge of neurophysiology, endocrinology, and neuroanatomy. Since 2001, TR has been used to treat symptoms of a variety of diseases and conditions, including cancer, diabetes, AIDS, arthritis, autoimmune diseases, anxiety, and depression, with ongoing classes currently held in 13 countries and 24 states in the United States ( TR practitioners report that thousands of people have sought treatment and reported benefits from this method. This approach to healing, however, has not yet been subject to systematic empirical study.


TR is distinguished from other energy healing methods in two ways. First, it posits that the qi follows actual physiological pathways of the endocrine, circulatory, and central and peripheral nervous systems, in contrast (>or some cases in addition) to the hypothesized energy pathways represented by TCM acupuncture meridians (Tam, 2004; Tom Tam Healing System, 2007). This approach thus aims to facilitate healing by identifying and removing blockages in specific physiological systems that relate to the particular organ or disease. For example, treatment for breast cancer would target not only the tumor itself but, more importantly, the supposed cause(s) of the tumor, which for breast cancer would include blockage at the level of the spinal nerves (particularly T4; Tam, 2004) that emerge from the thoracic vertebrae and cross in front of the internal mammary artery and pectoral muscles. Second, although many energy healing modalities involve one-on-one treatments in which the practitioner is thought to use her or his own energy to promote healing in the recipient, TR group sessions are thought to facilitate the release of specific blockages by the collective intention and attention of the participants in the group (Tam, 2008).

The purpose of this study is to obtain information from users regarding their experience with TR Healing in improving symptoms from disease, disease course, or treatment side effects through an anonymous cross-sectional survey of participants of TR Healing group sessions. Prior to the study, we determined that if 50% or more of participants reported improvements with symptoms of their disease or relief from side effects of treatment, we would consider this sufficient preliminary evidence to support future, more rigorous investigation of this healing method. For this report, we address the following specific research questions: (a) What are the conditions for which participants are seeking healing through TR? (b) For participants who have attended more than one TR session, what is the self-reported effectiveness of the treatment? (c) Does self-reported treatment effectiveness vary by group leader, time-in-treatment, or specific condition? (d) What are the self-reported adverse effects, if any, of this treatment modality?

First off, reading this credulous prose cues me in right away that I am not dealing with anyone interested in actual science. The whole routine about qi flowing along actual physiological pathways instead of (or in addition to) meridians is a nice touch, but it’s nothing but the proverbial putting lipstick on a pig. (Oh, wait, should I have used that metaphor? Our recent election shows that particular saying can have unintended consequences.) Of course, Sullivan et al don’t bother to cite any actual evidence from the peer-reviewed literature for Tong Ren. Oh, no. That’s because there isn’t any. Rather, they cite a bunch of URLs from Tom Tam’s various websites and books by him. Unfortunately, peer review for this particular journal isn’t what one would hope.

Moving on, I suppose there are some circumstances where such a study might be useful. It might give information about who uses woo such as Tong Ren. It might be useful as a psychological study of what draws such people to incredibly implausible “treatments” such as Tong Ren. It might even be useful as a measure of user perception of Tong Ren. What it’s not in the least bit useful for is as evidence for any sort of efficacy due to Tong Ren. Indeed, it is nothing more than the formalized collection of anecdotes, and, when it comes to determining the efficacy of a therapy, the pleural of “anecdote” is not “data.”

The results of the study are essentially as described in the abstract. However, the manner in which the results are reported and discussed tells us nothing. To get an idea, I quote a typical paragraph:

To obtain a qualitative measure of which diseases or conditions appeared to have the strongest response to treatment, we sorted the comments coded as “superlatives” (n = 65 cases) by ICD-9 classification and counted the number of superlative ratings in each disease category. Cancer (n = 20) and endocrine, metabolic, and immune disorders (n = 16) received the highest number of superlative ratings, followed by musculoskeletal disorders (n = 11; Table 5). Table 5 shows three examples of individuals within each of these disease categories. For example, one person with non-Hodgkin lymphoma described TR as “revolutionary” and said that the difference noted from TR was “no trace of cancer” and that “remission” was the most important result of TR treatment. Other disease categories that were associated with comments using superlative terms were digestive system disorders (n = 5), nervous system disorders (n = 4), mental health conditions (n = 4), and respiratory disorders (n = 1). Some improvements that were not coded as superlative were still notable, for example, one participant who was in attendance to treat “bone loss in jaws, potentially loss of at least 7 teeth,” described TR as “energizing, revitalizing,” and after 11/2 years of weekly TR treatments reported that “y teeth are no longer mobile, none have had to be removed, am able to use my teeth without problems.”

Note that these are entirely subjective impressions. Not one whit of clinical correlation is provided to determine whether there was anything to these testimonials other than what the patients said. If there’s one bit of wisdom I’ve tried to impart over the last four years, it’s that the vast majority of testimonials are not what they appear. It’s also that the people who make these testimonials are not stupid, by and large. Credulous, yes. Not trained in the scientific method. Almost always. Unable to realize that correlation does not equal causation. Definitely. Attributing effects that are due either to regression to the mean or treatment effect from “conventional” therapy to the woo they’re undergoing. Almost certainly. Generally, they fall prey to the cognitive quirks that all humans share that lead us to make incorrect conclusions regarding causation when faced with small sample sizes (namely us), or, as physicians, our own “clinical experience” (as Dr. Jay Gordon is fond of invoking). It is these cognitive quirks and shortcomings that make the scientific method necessary. Too bad Sullivan et al seem not to realize that, as they write in their Conclusion section:

Equally intriguing were the write-in descriptions citing “amazing” and “remarkable” improvements, with 30% of respondents who had attended more than one session using superlatives such as these to describe the impact of TR. Many of these descriptions did indeed appear remarkable, including reports of cure or remission from cancer, elimination of pain from arthritis, and reversal of bone loss.

None of which were verified by objective measures or correlated with Tong Ren therapy. Yes, the authors do concede that in the Discussion and (weakly) admit that this study does not provide concrete evidence for the efficacy of Tong Ren, but that admission does not keep them from taking all the “superlative” adjectives used by Tong Ren users, adding some dubious scientific studies, and using them to conclude that Tong Ren (a.k.a. the tapping of little voodoo dolls with little hammers to cure disease) is the greatest thing since sliced bread and should be studied further:

Some prior well-designed research does support the possibility of these effects; research and reviews of research on Ki and Qigong energy healing has documented evidence of inhibition of cancer cell growth in both in vitro studies with human carcinoma cells and in vivo animal studies (Chen, 2004; Ohnishi, Ohnishi, & Nishino, 2006; Ohnishi, Ohnishi, Nishino, Tsurusaki, & Yamaguchi, 2005). A systematic review of Qigong for pain reported evidence for its effectiveness as “encouraging” and warranting further study,(Lee, Pittler, & Ernst, 2007) and in vitro research on Ki energy healing has shown it to stimulate osteoblast and inhibit osteoclast cell activity (Ohnishi, Nishino, Uchiyama, Ohnishi, & Yamaguchi, 2007). If, as proponents have suggested (Kimura et al., 2005), balancing and unblocking the flow of qi energy enhances immune, endocrine, and nervous system function, some of these descriptions of healing from TR are plausible and may reflect measurable improvements in these physiological functions. The reports of strongly positive results indicate that future controlled studies of TR are warranted to test hypotheses about the physiological mechanisms that may be involved.

I’ve discussed these sorts of studies before. Suffice it to say, I’m not nearly as impressed as Sullivan et al apparently are, any more than I was impressed by a teenager who claimed that Tong Ren inhibited the growth of cancer cells.

Reading studies like this, I can’t help but alternately marvel and despair at what a waste of time, effort, and money they are. I note from the Acknowledgments section that the work was paid for by an “unrestricted grant from a private donor to the Department of Social and Behavioral Health, Virginia Commonwealth University School of Medicine.” That means it wasn’t funded by a grant, either from the government or a private foundation, won through peer review and competition with other applicants. Most likely, some woo-friendly wealthy contributor who had heard of Tong Ren paid to have this survey done. Nice work if you can get it. I wish I could find a wealthy patron to fund my research without all that nasty groveling (metaphorically speaking) in front of study sections and pounding my head against the wall to demonstrate that my results are worth funding and publishing.

The bottom line is that this study is yet another example of the phenomenon of “quackademic medicine” that has led so many medical schools to embrace pseudoscience. In the “good old days,” oh, say 20 years ago, no self-respecting medical school would have wanted anything to do with such a survey. A medical school might have tolerated it if funded by an outside source, but it would have been an embarrassment. No more. Now, no quackery is beyond the purview of quackademic medicine, and medical schools now seek it out as a means of enhancing their revenue stream, both through grants from NCCAM, donations from wealthy supporters of woo, and the creation of a new billable set of services that are all cash on the barrelhead without all that nasty mucking about with third party payors. Now, even the most elite medical schools have no sense of shame and–even more depressingly–no sense of science.


A. M Sullivan, S. Bauer-Wu, M. Miovic (2008). The Tong Ren Healing Method: A Survey Study Complementary Health Practice Review DOI: 10.1177/1533210108329265