John Oliver teaches us how to interpret medical and scientific studies

It’s no secret that I’m a big fan of John Oliver. When my aging body allows me to stay awake late enough on Sunday nights and there’s a new episode on, I’ll almost always be watching. Since starting his own show Last Week Tonight With John Oliver on HBO, Oliver’s become quite the expert at using humor to explain and bring attention to injustices that are widely ignored and to skewer politicians and blowhards.

This Sunday, he turned his attention on science reporting:

I certainly would have written about this for yesterday, except that at 11:30 PM on Sunday night there’s only one place for me to go, and that’s to bed to try to rest up for the week to come. So you get this on Tuesday. Of course, I was at an evening meeting last night; so this is a perfect topic for today given that it doesn’t need extensive discussion. Of course, for Orac a “brief” post is usually at least 1,000 words. On the other hand, if you watch the whole video above (and you really, really should, as it’s hilarious), you’ll spend more time than reading even one of my longer posts. It’ll be worth it though.

The key quote from the segment above is one that bloggers all over the place have been citing, and this is one of the rare times that I won’t be contrarian, because I agree. Oliver was particularly harsh on the TODAY SHOW. For example, noting a segment in which co-hosts Natalie Morales and Tamron Hall were arguing about the health benefits of whole milk based on various scientific studies in the news regarding its pros and cons. Al Roker then interrupted them and said something amazingly stupid, specifically, “You find the study that sounds best to you. And go with that..”

To which Oliver bitingly replied:

No! No, no, no, no, no, no, no, no! In science, you don’t just get to cherry-pick the parts that justify what you were going to do anyway. That’s religion. You’re thinking of religion.”

Indeed. Not surprisingly, cranks did not like this statement at all. Not at all. Before that, near the beginning of the segment, Oliver noted the plethora of studies about coffee, some showing that it might reverse liver damage, prevent colon cancer, decrease the risk of endometrial cancer, and increase the risk of miscarriage, and used another religion reference:

Coffee today is like God in the Old Testament: It will either save you or kill you depending on how much you believe in its magic powers. After a certain point, all that ridiculous information can make you wonder, is science bullshit? To which the answer is clearly no, but there’s a lot of bullshit masquerading as science.

Indeed. This blog and my not-so-super-secret other blog wouldn’t exist if that weren’t the case. The problem for science communicators, of course, is to distinguish what is good science and what is BS in the context of the avalanche of studies that show up in the news and, more recently, get passed around in social media. If I were to try to analyze and refute all the bad science that shows up in my Facebook feed every day, I’d need to become a full time blogger, and even then I would be able to put much of a dent in the flood of misinformation and questionable science.

Fortunately, for those who don’t read science and medical blogs, Oliver provides a useful primer in a 19 minute segment. First, Oliver points out the problems that result in what a Brit might call dodgy studies showing up in the scientific literature and issues that scientists know about when interpreting scientific studies but that lay people are largely unaware of:

  1. Not all studies are created equal, nor are all journals. There are lots of journals out there that publish weak science because they don’t have the reputation of top tier journals like, say, Cell, Science, or the New England Journal of Medicine.
  2. The pressure to “publish or perish” is intense. I recently went up for promotion (yes, I made it, and my promotion will become official in August). The experience taught me that publishing matters—a lot. Worse, quantity seems to matter more than quality. I have a lot of quality publications, but I finished the process with the distinct sense that I would have been better off come promotion time publishing more articles, even if they were smaller findings in less prestigious journals.
  3. Because of this pressure, there is pressure to come up with eye-catching findings, which can lead scientists, consciously or unconsciously, to tweak their studies. Oliver even brought up p-hacking, the process of collecting lots of data and doing lots of comparisons until a statistically significant comparison is found. I’ve discussed this issue a lot in the context of studies that do many comparisons and don’t control for multiple comparisons. Remember, if the p-value is set at 0.05, even if an experiment is done perfectly 5% of the time any “positive” value will be a false positive. Do a lot of comparisons, and the chance of a false positive approaches 100%.
  4. Replication is important, but happens way less than it should. Of course, that’s because no one wants to fund them or do them. As Oliver put it, there’s no Nobel Prize for fact checking. Of course, I would quibble somewhat about how rare replication studies are. It’s often necessary to replicate another investigator’s results as a prelude to doing experiments that go beyond them, to make sure that the experiments are working on one’s laboratory. Still, Oliver’s point is valid.
  5. Scientists know not to put too much weight on any single study, but that’s not how many of these studies, particularly preliminary studies, are presented to the public. Oliver provided a particularly egregious example that, thanks to a press release, ended up being spun as saying that chocolate as beneficial for pregnancy.
  6. It’s partially our fault. We love studies with interesting conclusions and news directors know it.
  7. Animal studies are often inappropriately extrapolated to humans, sometimes without the fact that a study was carried out in rodents being downplayed in news stories.
  8. Scientists themselves oversimplify and overhype. Oliver even took a swipe at TED Talks, thus endearing himself to me forever.
  9. Then there’s commercial bias. Oliver picked a hilarious example of a study claiming that dehydration degraded driving ability as much as drinking, which was funded by a company that sells rehydration products.

Oliver finished up with a fantastic example, of “everything causes cancer,” complete with a graph. It looks like a figure from John Ioannidis’ famous “cookbook study,” where he examined ingredients in a cookbook and reviewed the literature for studies linking them to cancer. Basically, he found that everything causes cancer and everything prevents cancer.

Oliver is quite correct, though, that Roker’s attitude that you “find the study that sounds best to you and go with that” is the sort of attitude that underlies so much science denialism. It lets antivaccine activists claim that vaccines cause autism when the overwhelming scientific consensus is that they don’t, and it fuels denial that human activity is causing major global climate change. Note that Oliver very clearly says that vaccines do not cause autism and that anthropogenic global climate change is real, both of which are overwhelming scientific consensuses.

I love how Oliver says near the end that if a talk show is going to say “a study says,” it should also have to provide sourcing and context or not mention it at all. Of course, that will never happen. That’s why medicine and science bloggers like me will never lack for topics.