In which I disagree with Brian Deer on the issue of how to deal with scientific fraud

I admire Brian Deer. I really do. He’s put up with incredible amounts of abuse and gone to amazing lengths to unmask the vaccine quack Andrew Wakefield, the man whose fraudulent case series published in The Lancet thirteen years ago launched a thousand quack autism remedies and, worst of all, contributed to a scare over the MMR vaccine that is only now beginning to abate. Yes, Andrew Wakefield produced a paper that implied (although Wakefield was very careful not to say explicitly) that the MMR vaccine caused an entity that later became known as “autistic enterocolitis” and later implied that the MMR vaccine causes autism itself. Aided and abetted by the credulous and sensationalistic British press, Wakefield then played the myth he helped create for all it was worth. Were it not for Brian Deer and his dogged investigation into Wakefield’s perfidy, the fraud at the heart of the myth that the MMR vaccine causes autism might never have been discovered and Wakefield might still have his medical license, rather than having been struck off the U.K. medical register.

All of which is why it pains me to have to disagree with Deer now.

Of course, this is not the first time I’ve had a problem with something that Deer’s written, and I’m guessing that it won’t be the last. No one expects that we’ll agree on everything, nor, I hope, would anyone expect that I’d hold my fire when even a usual ally like Deer makes a misstep. My admiration for his having exposed Wakefield is enormous and buys Deer a lot of credit in my estimation when it comes to giving him the benefit of the doubt about what he writes, but that credit and benefit of the doubt only go so far. Unfortunately, in the wake of his vindication with respect to Wakefield’s fraud, Deer seems to have developed a bug up his posterior over science and scientists. Now, I can sort of understand why that might be true, given the seven or eight years of relentless abuse he’s suffered at the hands of the anti-vaccine movement and the chilly reception he’s received from some scientists. I can even kind of understand why Deer has lashed out at Paul Offit, Ben Goldacre, and Michael Fitzpatrick, although I think he made a big mistake in doing so. I also think he’s gone a bit overboard in his latest article, Scientific fraud in the UK: The time has come for regulation. He begins in a very derogatory, ad hominem fashion:

Fellows of the Royal Society aren’t supposed to shriek. But that’s what one did at a public meeting recently when I leapt onto my hobbyhorse: fraud in science. The establishment don’t want to know. An FRS in the audience – a professor of structural biology – practically vaulted across the room in full cry. What got this guy’s goat was my suggestion that scientists are no more trustworthy than restaurant managers or athletes.

Now, obviously I wasn’t at this public meeting, its being in the U.K. and all, but I highly doubt that this particular Fellow of the Royal Society literally “shrieked” at Deer. I really do. Even granting a bit of artistic license, his characterizing it that way is clearly meant to paint a picture of someone who disagrees with him as being shrill and unreasonable, “shrieking” and “vaulting across the room at full cry” at him. Not “strenuously disagreed” with the proposal Deer is arguing for. Not “criticized it harshly.” “Shrieked” and “vaulted across the room at full cry.” Now, maybe the FRS described really did shriek, or maybe scientists did raise their voices to be heard. Who knows? Even if they did, this is not an auspicious start to Deer’s argument. Deer is usually much, much better than that; he usually doesn’t usually take cheap shots like this. Sadly, he takes an even cheaper shot at scientists later, as you will see. But first he has to dump on scientists a bit more:

Restaurant kitchens are checked because some of them are dirty. Athletes are drug-tested because some of them cheat. Old people’s homes, hospitals and centres for the disabled are subjected to random inspections. But oh-so-lofty scientists plough on unperturbed by the darker suspicions of our time.

Is Deer actually proposing surprise inspections of labs? Probably not, but if he’s suggesting through his analogy that this would be likely to catch fraud, he’s going to be sorely disappointed. Such inspections would be even less likely to detect overt fraud than the peer review system. So what is Deer proposing? I’m really not sure, and, rereading Deer’s article, I’m not entirely sure he knows what he’s proposing, either. In any case, the analogy is really, really bad. Unsanitary conditions and practices in kitchens can usually be easily detected by surprise inspections. Ditto hospitals. Random drug tests don’t work quite as well, but they do certainly weed out those who aren’t clever enough to evade them. Often such inspections distort the very thing being regulated. For instance, we just went through our JCAHO inspection a few weeks ago, and I’m not sure that the months of preparation that went into getting ready for the inspection actually made us better as a hospital. All that preparation did get us ready to pass the test. True, inspections of the laboratory used by Wakefield to do PCR looking for vaccine strain measles virus might have turned up the rank incompetence there, but in the majority of cases I highly doubt that such inspections would find evidence of data falsification.

In any case, Deer airily dismisses scientists pointing out that there already exist checks and balances in science, referring to appeals to the scientific method as a method “which separates true from false, like a sheep gate minded by angels.” Damn. I’ll give Deer credit for being a really good writer, but the contempt dripping from his prose is palpable as he dismisses arguments he doesn’t agree with even while admitting he has little evidence to support his assertions:

They heard, of course, that there’s no evidence of a problem: no proof of much fraud in science. Publishing behemoth Reed Elsevier, for example, observed that of 260,000 articles it pumps out in a year, it will typically retract just 70. And for nearly all of these the reason was that the stuff was “plain wrong”, not because it was shown to be dishonest.

This sounds like the old Vatican line about priests and child abuse. Or Scotland Yard and tabloid phone-hacking. And, although I know that the plural of “anecdote” isn’t “data”, the anecdotes of science fraud are stacking up.

Comparing scientists to pedophile priests is the cheapest of cheap shots, and, quite frankly, I resent it. If he were to use that sort of simile at a hearing that I attended, I might be tempted to “shriek” at him too.

As for whether anecdotes of science fraud are “stacking up,” maybe I’m just blinded by being a–you know–actual scientist, but quite frankly, I just don’t see it. To me, data talks, and, quite frankly, Deer doesn’t have much in the way of data. Actually, he doesn’t have any (or at least he doesn’t present any), and he’s actually right about one thing: The plural of “anecdote” isn’t “data.” Yet all he presents are two anecdotes. He has Wakefield, and he has “Woo-Suk Hwang’s fabricated claims in Science about cloning embryonic stem cells.” But he has no real hard data on how common scientific fraud and misconduct are. If anecdotes are what Deer is dealing with, then what does he make of my anecdote? In my 20+ years in science I have never witnessed or had personal knowledge of anyone where I’ve worked falsifying data. In fact, when The Lancet editor Richard Horton is quoted as saying that “flagrant” scientific fraud is “not uncommon,” I have to wonder what, exactly, he referring to. If it were that common, presumably my colleagues and I would have seen some. Don’t get me wrong. I’m not holding up my experience as necessarily being representative. Maybe I am blind. Who knows? What I am doing is trying to point out how relying an anecdotes can easily lead to a distorted picture. The anti-vaccine movement taught us that.

Also, let me just repeat yet another time that I detest scientific fraud, and have written about it on multiple occasions. But fraud is a continuum. Far more common than outright data falsification à la Wakefield is the reporting of half-baked research, the use of inappropriate analyses or selective data reporting to squeeze positive-appearing results out of what are really negative results, or exaggerating the strength and/or importance of their work.

As an example, let’s do a little thought experiment and imagine a situation, for the moment, in which Andrew Wakefield’s Lancet paper was not fabricated in the least, as Deer showed that it was. Pretend, for the moment, that it was a perfectly well-executed case study, with clinical histories accurately reported. Even if that were the case, his paper was merely the result of a measly twelve patient case study. At best, it could generate hypotheses. Making any sort of firm conclusions from such results is profoundly irresponsible, as was promoting such results so publicly. In any case, science actually did eventually sort it out; other studies were done and failed to find any association between the MMR vaccine and autism or enterocolitis in autistic children. The problem was that no one seemed to be listening. Science was self-correcting in Wakefield’s. The damage caused by Wakefield’s fraud was not so much to science, but rather to the public opinion of the safety of the U.K. vaccination program. It is thus the public perception of that science became the problem, not science itself. Look at it this way. Even if there were no fraud and Wakefield’s initial paper had been meticulously carried out, the damage would still have been done because it was how Wakefield reported his results to the press and how the British press credulously lapped up his line of BS, coupled with the irresponsibility (well documented by, yes, Brian Deer) of The Lancet editors and the leadership at the Royal Free Hospital that caused the MMR scare.

Deer continues to report on the House of Commons science and technology committee proceedings, including the report it recently issued, quoting and heartily endorsing its conclusion:

Finally, we found that the integrity of the peer-review process can only ever be as robust as the integrity of the people involved. Ethical and scientific misconduct–such as in the Wakefield case–damages peer review and science as a whole. Although it is not the role of peer review to police research integrity and identify fraud or misconduct, it does, on occasion, identify suspicious cases. While there is guidance in place for journal editors when ethical misconduct is suspected, we found the general oversight of research integrity in the UK to be unsatisfactory. We note that the UK Research Integrity Futures Working Group report recently made sensible recommendations about the way forward for research integrity in the UK, which have not been adopted. We recommend that the Government revisit the recommendation that the UK should have an oversight body for research integrity that provides “advice and support to research employers and assurance to research funders”, across all disciplines. Furthermore, while employers must take responsibility for the integrity of their employees’ research, we recommend that there be an external regulator overseeing research integrity. We also recommend that all UK research institutions have a specific member of staff leading on research integrity.

Deer concludes with a tweak:

The fellows of the Royal Society, I’m sure, won’t like it.

Of course, the question is: Why won’t they like it? It might not be, as Deer seems to think, because they are reflexively resistant to any oversight (although that might be true). Might it not also equally be because this proposal is ill thought out and in general a bad idea? It might.

For example, consider these questions:

  1. What, exactly, would each specific member of each staff of UK research institutions charged with “leading on research integrity” actually do? Seriously. Think about it. What would such a peson do? Would he pop into investigator’s labs? Would he inspect lab notebooks? Would he peruse computer hard drives? Watch students, postdocs, and technicians do experiments? Reanalyze random data? And if he doesn’t do those things, then what, exactly, would he do to stop fraud in his own institution that would have any hope of actually making a difference?
  2. What, exactly, would a regulatory body do? David Colquhoun is spot on in the comments when he asks, “Would it reanalyse each of my single ion channel records to make sure I’d done it right? Would it then check all my algebra to make sure there was no misteke? Even if you could find anyone to do it, that would take as long as doing the work in the first place. I fear that Deer’s suggestion, though made from the best of motives, shows that he hasn’t much idea of how either experiments work or how regulatory bodies (don’t) work.” And if it were just a passive surveillance system that only acts after a complaint is filed, how is that any better than the situation in the US?

The bottom line is that we really don’t know how common fraudulent research, specifically examples of blatant fraud like Wakefield’s, really is. There is evidence that perhaps 2% of researchers admit to having manipulated data, but perhaps 33% admit to having at least once engaged in “questionable” research practices. Of course, such results depend upon how you define “questionable.” Whatever the true incidence of scientific misconduct is, we do know that peer review isn’t very good at catching outright fraud and never has been. So what to do? Another consideration is that any regulatory body could be hijacked by ideologues. I don’t know how much of a problem this is in the U.K., but imagine, for example, someone like global warming denialist Senator James Inohofe taking over a panel on science integrity and what he could do with climate scientists.

Even though peer review isn’t particularly good at finding fraud, it is still one major tool to be used to do so. What, however, should be added to it? The problem with many ideas, such as oversight panels or institutional science integrity officers is in the details. They sound like good ideas on the surface, but when you start to think a bit about the details and what, exactly, such mechanisms would mean and how they would be set up, suddenly it’s not so easy at all. Let’s go back to my thought experiment in which Wakefield’s work was not fraudulent. Now switch back to reality, where Wakefield did commit fraud, but imagine that the ideas advocated by Deer were policy at the time he was signing up patients for his case series. Would Deer’s and the committee’s proposals have stopped Wakefield or exposed him earlier. Would a science integrity officer at the Royal Free Clinic or a science oversight board have made a difference? Maybe, but I highly doubt it. At what point in Wakefield’s fraud would either such mechanism have been able to intervene? Afterward, what, specifically, would have triggered an investigation by either the institution or the regulatory body proposed? After all, it took even Brian Deer himself years to dig up the evidence that started to suggest fraud.

Still, it’s not all bad. Deer is right about one thing, and that’s that the “R” word (responsibility) has to be injected into the system. Here in the U.S., government granting agencies, such as the NIH, investigate allegations of scientific fraud in research funded by the U.S. government, the penalty for fraud being banned from receiving federal grants for a period of time and possibly even criminal charges for defrauding the government. The problem is that the NIH Office of Research Integrity is underfunded and overwhelmed.

More important than any external regulatory force is changing the culture of science so that it is considered acceptable to report suspected scientific misconduct. Science is a means to an end: to find out how nature works, to plumb its mysteries and discover rules by which it works and that we can use to make predictions. Science, to its credit, also tends to work more or less on the honor system, which is why Deer is probably correct that scientists tend not to consider perhaps as much as we should the possibility that an investigator is lying or falsifying data when anomalous results such as Wakefield’s are reported. On the other hand, he appears not to understand that the vast majority of anomalous results are not due to fraud but rather differences in experimental design, analysis, bias (often subtle, but sometimes not), publication bias, and many other factors that can lead scientists astray. Some results are just wrong through sheer random chance. Yet Deer seems to assume based on his experiences with Wakefield but without strong empirical evidence that much of these problems are due to fraud, rather than just bad science, biased science, or random flukes that produce .

Even if Deer were correct that scientific fraud is a massive problem, it doesn’t mean that his blanket condemnation of science in the U.K. or that his likening the culture to the Roman Catholic Church shielding pedophile priests wasn’t way over the top. Unfortunately, burned by his experience pursuing Wakefield, I fear he’s become a bit cynical. It’s also clear that, for all his amazing skill as an investigative journalist, he hasn’t really developed a firm grasp of the nitty gritty of how science actually works and how research is actually done. In the end, science usually does correct itself regardless of the source of error, be it fraud or, well, error. Results in issues that matter will eventually be corrected because other investigators interested enough to expand on a line of investigation need to start by replicating the results that interested them. In the case of fraud, they won’t be able to. Meanwhile, results that other investigators don’t bother to try to replicate as a prelude to moving beyond them usually don’t matter much and have little or no effect on science. It’s a slow and messy process, sometimes maddeningly so, but over time it does work. Science does correct itself.

The problem is, many of the cures to accelerate the process of discovering fraud are potentially worse than the disease. Then there’s another question to consider: How does one determine whether highly novel or potentially groundbreaking work is correct or a mistake or a dead end when there’s nothing to measure it against? One can’t; other scientists can by examining the results and trying to reproduce them so that they can move on.