It often comes as a surprise to proponents of alternative medicine and critics of big pharma that I’m a big fan of John Ioannidis. Evidence of this can easily be found right here on this very blog just by entering Ioannidis’ name into the search box. Indeed, my first post about an Ioannidis paper is nearly a decade old now. These posts were about one of Ioannidis’ most famous papers, this one published in JAMA and entitled Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. It was a study that suggested that at least 1/3 of highly cited clinical trials may either be incorrect or or show a much higher effect than subsequent studies, which settle down to a “real” effect. Of course, this was nothing more than the quantification of what clinical researchers instinctively suspected or knew: That the first published trials of a treatment not infrequently report better results than subsequent trials.
In other contexts, this phenomenon has also been called the “decline effect.” Possible reasons for discrepancies between initial results and later trials may include publication bias (positive studies are more likely to see publication in high-impact journals than negative studies) or time-lag bias (which favors the rapid publication of interesting or important “positive” results). Also, high impact journals like JAMA and NEJM are always on the lookout for “sexy” findings, findings likely to have a strong impact on medical practice or that challenge present paradigms, which may sometimes lead them to overlook flaws in some studies or publish pilot studies with small numbers. In any case, I was not kind to a certain blogger who misinterpreted Ioannidis’s study as meaning that doctors are “lousy scientists.” Of course, lots of people misinterpret Ioannidis’ work, particularly alt med cranks, as they did with his most famous (thus far) paper, entitled Why Most Published Research Findings Are False.
Why do alt med advocates find it surprising that I’m a huge fan of John Ioannidis? The reason, of course, is that Ioannidis has dedicated his life to quantifying where science goes wrong, particularly sciences related to medicine, such as clinical trials, biomarkers, nutrition, and epidemiology. He pulls no punches. Alt med aficionados often labor under the misconception that proponents of science-based medicine are somehow afraid to examine the flaws in how science operates, that we circle the wagons whenever “brave mavericks” like them criticize science. Of course, the reason they criticize science is because it doesn’t show what they think it shows. They also assume that proponents of SBM will react to criticism of their discipline the same way, for instance, homeopaths react to criticism of homeopathy. In general, we don’t. That’s because science embraces questioning. Such questioning is baked into the very DNA of science. Oh, sure, scientists are human too and sometimes react badly to criticism, but we usually manage to shake it off and then seriously look at critiques such as what Ioannidis provides.
That’s why I was immediately drawn to a recent interview with John Ioannidis by Julie Belluz, John Ioannidis has dedicated his life to quantifying how science is broken, a title that has the great virtue of being true. It’s a fascinating read and provides insight into the mind of perhaps the greatest analyst of the scientific method as it’s actually practiced currently publishing. Part of it also shows how prescient Ioannidis was a decade ago when he published the article describing how most published research findings are false:
Julia Belluz: The paper was a theoretical model. How does it now match with the empirical evidence we have on how science is broken?
John Ioannidis: There are now tons of empirical studies on this. One field that probably attracted a lot of attention is preclinical research on drug targets, for example, research done in academic labs on cell cultures, trying to propose a mechanism of action for drugs that can be developed. There are papers showing that, if you look at a large number of these studies, only about 10 to 25 percent of them could be reproduced by other investigators. Animal research has also attracted a lot of attention and has had a number of empirical evaluations, many of them showing that almost everything that gets published is claimed to be “significant”. Nevertheless, there are big problems in the designs of these studies, and there’s very little reproducibility of results. Most of these studies don’t pan out when you try to move forward to human experimentation.
Even for randomized controlled trials [considered the gold standard of evidence in medicine and beyond] we have empirical evidence about their modest replication. We have data suggesting only about half of the trials registered [on public databases so people know they were done] are published in journals. Among those published, only about half of the outcomes the researchers set out to study are actually reported. Then half — or more — of the results that are published are interpreted inappropriately, with spin favoring preconceptions of sponsors’ agendas. If you multiply these levels of loss or distortion, even for randomized trials, it’s only a modest fraction of the evidence that is going to be credible.
This latter problem is the sort of problem that initiatives like the US Food and Drug Administration Amendments Act of 2007 (in the US) and Alltrials.net (primarily in Europe) are intended to correct. The FDAAA mandated that enrollment and outcomes data from trials of drugs, biologics, and devices must appear in an open repository associated with the trial’s registration, generally within a year of the trial’s completion, whether or not the results have been published. Of course, the FDAAA hasn’t been enforced as it should be, with a lot of studies not being published in a timely fashion, but it’s a start. In any case, since Ioannidis first made his big splash, there’s been a lot of research validating his model and showing that a lot of what is published has flaws. The question is: How to do better.
One thing Ioannidis suggests is what’s already happening: More post-publication review. He notes that there are two main places where review can happen: Pre-publication (peer review) and post-publication. How to improve both is a big question in science, as Ioannidis emphasizes in response to the question of how to guard against bad science:
We need scientists to very specifically be able to filter [bad] studies. We need better peer review at multiple levels. Currently we have peer review done by a couple of people who get the paper and maybe they spend a couple of hours on it. Usually they cannot analyze the data because the data are not available – well, even if they were, they would not have time to do that. We need to find ways to improve the peer review process and think about new ways of peer review.
Recently there’s increasing emphasis on trying to have post-publication review. Once a paper is published, you can comment on it, raise questions or concerns. But most of these efforts don’t have an incentive structure in place that would help them take off. There’s also no incentive for scientists or other stakeholders to make a very thorough and critical review of a study, to try to reproduce it, or to probe systematically and spend real effort on re-analysis. We need to find ways people would be rewarded for this type of reproducibility or bias checks.
I’ve often wondered how we can improve peer review. Most people who aren’t scientists don’t understand how peer review is done. Usually an editor tries to get scientists with expertise in the relevant science to agree to review a manuscript, in general two or three reviewers per manuscript. They will find time to squeeze in the review. Sometimes senior scientists who agree to review a paper will ask their postdoc to review it for them. It’s not an ideal system, but the scientists who do peer review are all unpaid. They basically volunteer or are asked and agree to review papers without recompense. Often the academic editors (as opposed to the permanent editorial staff responsible for getting post-review manuscripts ready for publication) are also unpaid. Basically, we scientists tend to do peer review out of a sense of obligation, because it’s considered part of our duty to science, as part of our job. All in all, it’s a very ad hoc system. Given these issues, the system actually works pretty well most of the time.
However, as Ioannidis notes, there are flaws, in particular our inability to analyze the primary data. In fact, I wouldn’t want to have to analyze the primary data of all the manuscripts I review over the course of a year. It would be way too much work, and I’d have to stop reviewing manuscripts. In fact, I was quite annoyed at the last manuscript I reviewed which had 13 (!) dense figures worth of data in its supplemental data system in addition to its 6 figures for the paper. Going through supplemental data sections, which journals now encourage scientists to load up all that data that they don’t want to include in the manuscript, all to be dumped online, has become downright onerous. As much as I agree with Ioannidis that it would be very good for peer review if we could find a way to reward scientists to do such activities, I really have a really hard time envisioning how, under the current financial model of science, this could ever happen.
I do agree with this:
We need empirical data. We need research on research. Such empirical data has started accruing. We have a large number of scientists who want to perform research on research, and they are generating very important insights on how research is applied or misapplied. Then we need more meta-research on interventions, how to change things. If something is not working very well, it doesn’t mean that if we adopt something different that will certainly make things better.
Thus far, Ioannidis has been very good at identifying problems in science. What hasn’t (yet) followed are evidence- and science-based strategies for what to do about it, how to improve. What I do have a bit of a problem with is Ioannidis’ other suggestion:
So if you think about where should we intervene, maybe it should be in designing and choosing study questions and designs, and the ways that these research questions should be addressed, maybe even guiding research — promoting team science, large collaborative studies rather than single investigators with independent studies — all the way to the post-publication peer review.
“Team science” has become a mantra repeated over and over and over again as though it will ward off all the evil humors that have infected the methods and practice of science, and there’s no doubt that for certain large projects team science is essential. Team science is, however, quite difficult. Whether it adds value to scholarship, training, and public health remains to be seen, and it tends to foster an environment in which the rich get richer, leading to a few leaders with a lot of followers. It’s labor-intensive and often conflict-prone. Team science also poses a significant challenge (and risk) to young investigators trying to establish a distinct professional identity. Moreover, it is becoming clear that investments in team science are not as cost-effective as advertised and, indeed, might even be the opposite. Hopefully, Ioannidis’s group is working on methods of evaluating when team science is useful and when it is not.
I do like the idea of post-publication peer review, as well, but that’s an even more difficult sell than peer review. After all, I get credit when I fill out my yearly activity report for my peer review activity, both for scientific manuscripts and grant review. I don’t get credit for post-publication review, where I read an already published paper and comment on it. Heck, blogging for me often represents an exercise in post-publication peer review, wherein I eviscerate bad papers and comment on good papers that catch my interest. Fortunately, there are now more journals permitting comments on their websites, and there is now a websites like PubPeer, which exists to facilitate post-publication peer review, although this is not without peril.
Science is not perfect. It has problems. The limitations and problems inherent in how it is currently practiced point to strategies to improve it. Ironically, it will be the rigorous application of the scientific method to science itself that will likely lead to such improvements. In the meantime, if you want me to illustrate the difference between science-based medicine and “complementary and alternative medicine” (CAM), I can do it. Just ask yourself this: Who is the John Ioannidis of CAM? Arguably, it was Edzard Ernst. Now ask yourself this: How did the CAM community react to him compared to how the community of medical scientists reacts to John Ioannidis? I’ll spell it out. Ernst was attacked, rejected, and castigated. Ioannidis is, for the most part, embraced.
That should tell you all you need to know about the difference between CAM and science-based medicine.