Not surprisingly, in response to my article on the health risks of secondhand smoke yesterday, the “skeptics’ came out in force, although I must admit that even I hadn’t expected quite as large an influx as what appeared. Perhaps I’ll prepare a general response in the near future (and, no, I didn’t take the Surgeon General’s report as the be-all and end-all, but it did make a compelling case for SHS causing increasing the risk of lung cancer and cardiovascular disease at least, and it also served as a convenient aggregator of the many, many studies out there). In the meantime one commenter in particular piqued my interest, as he or she basically reposted a list of quotes by various scientists claiming that relative risks less than 2.0 are basically rubbish and not to be trusted. Not coincidentally, the relative risks of heart disease and lung cancer due to SHS estimated by epidemiological studies range between 1.2 and 1.3 in most studies and meta-analyses, including the aggregation of these studies in the Surgeon General’s report. By implication, this list of quotes, rather than actually finding substantive problems with the actual epidemiological studies or how they were aggregated, are clearly meant to appeal to authority to convince you that any relative risk under 2.0 should be ignored as completely unreliable.
Not surprisingly, my skeptical antennae started twitching when I read this list, as it smacked of crank talking points based on quote mining similar to the anti-Darwin quotes that are the favorites of “intelligent design” creationists. The liberal use of ellipses in the quote by Sir Richard Doll in particular made me suspicious. Unfortunately, that was the one quote whose context I can’t find online. (I will, however, most definitely look in my medical school’s library for the book the next time I’m there.) Consequently, I decided to look up as many of these quotes as I could find online. It didn’t take long for me to find a website where this list of quotes was published virtually verbatim, at Forces.org, among other places.
Forces.org is an amazing website, a virtual repository of smoking crankery beyond what I’ve ever seen before. It not only denies any dangers from secondhand smoke, but denies that smoking itself causes cancer. For example, it claims that medical radiation is a necessary and more important cofactor in causing lung cancer, and even goes so far at one point as to say explicitly, “There is no proof that smoking causes cancer.” Wow. Not even the SHS “skeptics” make the claim that smoking itself doesn’t cause cancer, but Forces.org apparently does, or at the very least strongly implies that smoking isn’t such a big deal in causing lung cancer. I can’t say that I’ve ever seen a website about smoking that’s quite so…well, cranky. I have to thank rrgabe23 for pointing me to this site, however inadvertently and indirectly. There’s enough crankery there to keep both me and fellow SB’er Mark busy for quite a while, should either of us ever be so inclined to wade into the muck there.
But I digress. Back to the quotes.
I’ll start with the easiest one to find, because it was taken from the Journal of the American Medical Association. It’s a big mistake to include one from a journal that’s easily accessible online, because I can pull up the context behind it, and I did. So let’s look at two quotes as published in the list:
FDA – “Relative risks of 2 have a history of unreliability” – Robert Temple, M.D. Food and Drug Administration Journal of the American Medical Association (JAMA), Letters, September 8, 1999
FDA – “My basic rule is if the relative risk isn’t at least 3 or 4, forget it.” – Robert Temple, director of drug evaluation at the Food and Drug Administration.
First off, the second quote comes from a news article published in Science in 1995 entitled Epidemiology Faces Its Limits. Here’s the context:
Robert Temple, director of drug evaluation at the Food and Drug Administration, puts it bluntly: “My basic rule is if the relative risk isn’t at least three or four, forget it.” But as John Bailar, an epidemiologist at McGill University and former statistical consultant for the NEJM, points out, there is no reliable way of identifying the dividing line. “If you see a 10-fold relative risk and it’s replicated and it’s a good study with biological backup, like we have with cigarettes and lung cancer, you can draw a strong inference,” he says. “If it’s a 1.5 relative risk, and it’s only one study and even a very good one, you scratch your chin and say maybe.”
Some epidemiologists say that an association with an increased risk of tens of percent might be believed if it shows up consistently in many different studies. That’s the rationale for meta-analysis-a technique for combining many ambiguous studies to see whether they tend in the same direction (Science, 3 August 1990, p. 476).
In other words, epidemiologists tend to disagree over whether a relative risk of less than 2 is significant in single studies or small numbers of studies, but there is a fairly broad agreement that if a relative risk less than 2 is found in multiple studies done in different places with different methodologies, that’s reasonable evidence that there is more likely than not a real correlation. However, what was really interesting to me was when I looked into the source of the other quote in the letter to JAMA. The letter to JAMA was in response to a letter by Douglas Weed, M.D., Ph.D. of the National Cancer Institute criticizing an article by Dr. Temple entitled Meta-analysis and Epidemiologic Studies in Drug Development and Postmarketing Surveillance. In this article, Dr. Temple suggested a blanket policy under which any epidemiological study reporting a relative risk of less than 2.0 should not be published until it is replicated. He suggested this not because he thinks that studies with relative risks of less than 2.0 are not real, but because he feels that single studies are too prone to problems. In other words, he’s not saying that he believes relative risks under 2.0 should be automatically discounted, but rather that detecting such low relative risks is “problematic”:
The relative risk is a far more important determinant of how and whether adverse events can be detected than whether the events themselves are rare or common. Changes in the rates of relatively common events are often of greatest concern–a 30% increase in myocardial infarction rates, after all, would be more damaging than a 10-fold or even 100-fold increase in the rate of a 1 per million event–but methods to detect these changes other than through controlled trials are problematic.
How ironic. 30% is a relative risk of 1.3, which is right around the value that has become pretty well established for the additional risk of cardiovascular events due to chronic exposure to SHS. One can’t help but wonder if that was the example that Dr. Temple had in mind when he made his statement. He goes on:
A 2- to 3-fold relative risk of a myocardial infarction or death is not a “small” increase in risk in the usual sense; it is far larger, for example, than the benefit of such effective treatments as postinfarction aspirin, β-blockade, angiotensin-converting enzyme inhibition, or thrombolysis. Nonetheless, Taubes found that a sizable group of epidemiologists did not consider findings of relative risks of this magnitude in epidemiologic studies reliable. Some suggested that replication of such a finding in different environments with different methods might be more persuasive than a single study.
Moreover, here’s what Dr. Temple writes in the letter to JAMA including the quote cited above in context:
Fourth, as indicated in my article, relative risks of 2 are not problematic because they are unimportant; it would be very desirable to detect them. Many of the most powerful interventions we have (eg, thrombolysis, use of postinfarction β-blockers) do not create effects as large as 2-fold. The problem is that such risks, when observed in epidemiologic studies, have a history of unreliability, not because of obvious errors or methodological inadequacy, but because selection and other biases cannot be fully controlled in these studies. It is also true, as Weed notes, that failure to see a small effect in these settings also would be unreliable.
Well, well, well, well, that certainly sounds different than the way the quote was presented, doesn’t it? Dr. Temple even states that it would be “very desirable” to detect smaller relative risks! It should also be remembered that his quote comes in the context of his arguing not that we shouldn’t believe relative risks less than 2, but rather that at least two studies should confirm such a low relative risk before we take it seriously. So we have at least one case (and possibly two) of deceptive quote-mining in the list of quotes. That’ll teach ’em to include an easily checked quote in a list like that. (Look for Dr. Temple’s quotes to disappear from future iterations of the list, to be replaced by another, less easily tracked down set of quote-mined quotes.) Unfortunately, the creators of this list weren’t quite so careless with the other quotes, which were either hard or impossible to track down, at least online. Even so, let’s see what else we can find. Next up was the quote, “In epidemiologic research, relative risks of less than 2 are considered small and usually difficult to interpret. Such increases may be due to chance, statistical bias or effects of confounding factors that are sometimes not evident.” – National Cancer Institute, “Abortion and possible risk for breast cancer: analysis and inconsistencies,” October 26, 1994. I did a lot of Google searches looking for the context of this one. All that came up were websites downplaying the risk of SHS and, as in the case of Forces.org. It turns out that, as far as I can tell, this statement came from a press release by the NCI with that date. Unfortunately (and conveniently), it’s not on the Cancer.gov website, meaning that I can’t examine the full context. The NCI news page only appears to go back to 1998. However, I can piece together part of the context.
What I could figure out is that the above press release was apparently in response to a famous (or infamous, depending on your point of view) 1994 study published in the Journal of the National Cancer Institute that reported a 50% elevation in breast cancer risk in women who had had abortions (or a relative risk of 1.5). Of course, it’s not exactly rocket science to express skepticism of a single epidemiological study with a relative risk of less than 2, especially if there is no plausible biological mechanism to explain the result. That’s a whole lot different than examining the results of dozens of studies. If I can find a complete copy of the NIH press release from which that quote is drawn, I may post again on this topic to provide full context.
Finally, we have the quote from a publication by the World Health Organization:”Relative risks of less than 2.0 may readily reflect some unperceived bias or confounding factor, those over 5.0 are unlikely to do so.” – Breslow and Day, 1980, Statistical methods in cancer research, Vol. 1, The analysis of case control studies. I’m half-tempted to order a copy of this publication, just to see what the context is for that quote. Given that it’s just one sentence, I’m very curious.
In any case, for single studies, it is indeed wise to interpret a relative risk of less than 2 with great caution. When many studies all point to a relative risk between 1 and 2, it is reasonable to start to conclude that the findings are probably real. Moreover, as Dr. Temple, one of the scientists quoted above, said, relative risks below 2 may well be real and have real practical and clinical significance, making it desirable to detect them. The problem is, because they are relatively small in non-controlled prospective studies, a lot of confounding factors can indeed interfere with individual studies. Again, that’s why multiple studies are needed. We have that for the dangers of SHS. Moreover, it’s not true that we don’t act based on relative risks less than 2. A couple of prime examples come from just the past couple of years. For example, a few months ago, a pooled meta-analysis of studies looking at cardiovascular risk from Avandia revealed a relative risk of 1.43 for myocardial infarction and 1.64 for death from cardiovascular events. This is just a single study with a relative risk of less than 2.0; yet it led to a widespread change in doctors’ prescribing habits, such that Avandia prescriptions have plummeted precipitously. And, of course, I can’t resist pointing out that Dr. Temple himself cited a number of examples where relative risks of less than 2 are widely accepted as real.
But who’s kidding whom? The real purpose of the list above is nothing more than an appeal to authority to suggest that any relative risk below 2 is bullshit and to be ignored, when such is not the case. What the real message is from epidemiologists is that relative risks below 2 should be viewed with skepticism, particularly if they’re single studies and there is no biologically plausible mechanism to explain the noted association. This is not the case with SHS. Not only is there a biologically plausible mechanism, given that we know that smoking causes cancer and cardiovascular disease, but there are many studies that find a relative risk in the same range.
Such quote collections are a favorite tactic of cranks. Usually, they consist of mined quotes taken out of context to support the crank’s position cherry picked from sources that are hard to track down. Such lists of quotes then propagate far and wide across the Internet by e-mail and on blogs and websites. Sometimes they mutate along the way, with the addition of more quotes or the editing of the quotes that are there.The above list qualifies, because most of the quotes are very difficult to track down to find the context in which they were made and they have the whiff of being cherry-picked to form just such a crank list. In this case, the mistake made was that two of the quotes were very easy to track down online by anyone with access to institutional subscriptions to Science and JAMA. It took me less than 10 minutes to show that the quotes by Dr. Temple, particularly the one about the unreliability of relative risks under 2, were grossly taken out of context and even downright deceptive. It wouldn’t surprise me if the others are, too, although i can’t yet prove it. Even if they aren’t, they remain misleading because they discuss the problems with interpreting relative risks under 2 in single studies, not the more relevant case of when many studies suggest a relative risk of under 2, as is the case with studies on the health risks of SHS.
If anyone can provide the full text and context of any of the remaining quotes to me so that I can examine them in detail, I’d be grateful. In particular, those of you who post and repost those quotes, please send me the context if you have it. (I’m guessing that none of you probably do.) In the meantime, I will keep looking. I hope my library has the book by Sir Richard Doll from which one of the quotes is purported to come.
