Categories
Antivaccine nonsense Autism Medicine Quackery

British science accused in the wake of the MMR scare

I was originally going to blog this yesterday, but Dr. Oz’s offenses against science and medicine on his show that aired on Tuesday kind of pushed it out of the way. It’s not that I didn’t think the third part of Brian Deer’s expose of Andrew Wakefield’s fraud worthy of my attention. Rather, the Oz thing really got me peeved, peeved enough to push aside (temporarily, at least) Brian Deer’s deconstruction of how the editors of The Lancet scrambled to cover their proverbial asses, which they proceeded to do with alacrity, as the title of Deer’s article implies: The Lancet‘s two days to bury bad news.

I’ve been critical of The Lancet before for publishing Andrew Wakefield’s case series. Indeed, to this day, I hadn’t been able to figure out how Richard Horton, the editor of The Lancet, kept his job in the wake of l’affaire Wakefield, particularly after Wakefield’s massive conflicts of interest had been revealed in 2004, as well as the possibility that Wakefield had subjected vulnerable children to invasive procedures that were not medically indicated. The reason is not so much that I would expect peer review to have uncovered what we now know to have been blatant fraud. Peer review has a hard time doing that because the default assumption in science tends to be one of honesty, namely that what authors report in manuscripts submitted to journals does not represent falsified data. Oh, sure, editors and reviewers are fully aware that scientists will try to present their data in such a manner to put the best possible spin on their experiments, to make their story as persuasive as possible. That’s expected. What is not expected is outright dishonesty, falsification of data, the offense by Wakefield for which Deer presented strong evidence in his previous two articles.

No, what should have gotten Horton fired, from my perspective, is that he accepted such a poor manuscript. Even taken at face value and even if there had been no scientific fraud, Wakefield’s manuscript, a case series of only 12 children that, even viewed in as favorable a light as possible, didn’t support the hypothesis that MMR vaccination was somehow associated with a new syndrome consisting of regressive autism and enterocolitis. It was the thinnest of thin gruel, which, again, even if viewed in the best possible light, didn’t belong in the pages of The Lancet. Maybe in the pages of some bottom-feeding medical journal or another, but not in The Lancet. This is particularly true given the explosive nature of the implications of Wakefield’s paper, which should have resulted in more caution on the editors’ part to make sure that all the i’s were dotted and t’s crossed.

Of course, in most cases, the consequences of publishing such an article would be a temporary blot on the reputation of a journal, soon forgotten. Not in this case. In this case, the publication of Wakefield’s publication in The Lancet, coupled with Wakefield’s media whoring and the eager acquiescence of the British media, ignited a scare about the MMR vaccine that led parents to abandon the MMR in droves, driving the MMR uptake rate to levels below that necessary to maintain herd immunity. As a result, measles came roaring back in the U.K. such that it is now endemic again. Can Horton be directly blamed for this scare? Not exactly. But his irresponsibility did play a significant role in igniting the scare and at the same time permanently tarnishing the reputation of the journal for which he was responsible. In other words, he screwed up. He had no way of definitely knowing that his screwup would have such far-reaching consequences, although he should have had an inkling, but them’s the breaks. His screwup, instead of being quickly forgotten, leaving him to continue his career with minimal consequence, resulted in consequences that could not be ignored.

If I were in charge of The Lancet, I would have canned Horton’s ass in 2004 after Deer’s first set of revelations, or, if that were not possible (it’s not easy to fire a journal editor), I would have declined to renew his contract when it expired, assuming he has a contract.

Before I go on, though, I must point out one thing. After reading his installment, it is even more clear to me than it was a week ago when I blogged about it that Deer’s attack on Drs. Offit, Goldacre, and Fitzpatrick for supposedly being part of the “old boys network” of physicians downplaying Wakefield’s offenses and his investigative journalism was wildly off key in relationship to this most recent article, which criticizes British medical authorities and the Lancet for letting the Royal Free Hospital and the UCL investigate themselves and laments how medical authorities often don’t take fraud seriously enough. Neither of these are positions that Drs. Offit, Goldacre, or Fitzpatrick would be likely to disagree with in the least, and none of them that I’m aware of, other than Fitzpatrick, ever even criticized the bringing of charges against Wakefield before the GMC.

Deer is completely on key, however, when he describes what happened when the Wakefield situation blew up in 2004 after Brian Deer’s first set of revelations was published. In retrospect, it is not surprising (actually, it shouldn’t have been surprising at the time) that Horton had every motivation to try to minimize the damage to his journal and its reputation, and apparently that’s just what he did, as Deer points out as he describes what happened after his meeting with Horton and the editors of The Lancet in 2004, in which he laid out the evidence for Wakefield’s misconduct:

I had assumed that when I finished Horton would say that an investigation was needed to untangle these complex matters. There were at least three strands: possible research fraud, unethical treatment of vulnerable children, and Wakefield’s conflict of interest through the lawyer. But within 48 hours, and working with the paper’s three senior authors, the journal was to publish a 5000 word avalanche of denials, in statements, unretracted to this day.5 6 7 8 9

It can’t be emphasized enough that Wakefield took advantage of a vulnerable population (autistic children) in order to enrich himself. I mention it here again because it is a very important point.

During the GMC hearings, Horton claimed this:

“In this particular case,” he [Horton] told the GMC tribunal of three doctors and two lay members, seated to his right at the hearing, “we went to the vice-dean of the Royal Free, laid out the nature of the problem, and asked him to investigate and come back to us, as best he could, with his own judgment of the veracity or not of the allegations. In addition to that, we would look at the documentation as best we could and try and form our view as to whether those allegations were true.”

But what really happened was this:

But documents, emails, and replies obtained under the Freedom of Information Act reveal no formal investigation. What emerges is merely a scramble to discredit my claims during the 48 hours after I disclosed the information. They show the journal’s editor, the paper’s senior authors, and the Royal Free medical school, frantically mobilising against me. Were it not for the GMC case, which cost a rumoured £6m (€7m; $9m), the fraud by which Wakefield concocted fear of MMR would forever have been denied and covered up.

Deer then proceeds to describe exactly how this was done. Basically, there was no independent investigation. Right after Deer left Horton, Horton met with the three principal authors of The Lancet paper and devised a strategy wherein all impropriety was denied except for Wakefield’s conflict of interest in having accepted money from lawyers to develop evidence to support lawsuits against vaccine manufacturers, which they did their best to minimize as an offense as a failure on Wakefield’s part to disclose. Fortunately, that part appears to have backfired, as Horton’s concession that Wakefield had had a conflict of interest ignited a media firestorm:

His actions sparked a media firestorm. Although denying all of the most serious of my findings (now proved), he conceded that Wakefield had a conflict of interest–and that weekend was what journalists call “slow.”

The BBC was on the story within half an hour of the statements’ release. Independent Television News called Harris. And all of Fleet Street knew the thrust of an impending splash in the Sunday Times, the UK’s market leading Sunday broadsheet.

The furore blazed from Friday until Wednesday, and beyond.

Science is an enterprise that, because it is a human enterprise, can be corrupted when its practitioners do not possess the personal integrity to be honest in reporting their results. Andrew Wakefield is clearly just such a dishonest man, whose ambition and greed led him not only to accept money from lawyers to develop scientific “evidence” they could use in court while suing vaccine manufacturers, but then to falsify evidence for his case series, misrepresenting the clinical histories of several of the children, and then making plans to produce products based on his unsupported hypothesis that the MMR vaccine can cause autism associated with GI complaints, products that, according to Wakefield’s hopes, could make him many millions of dollars. Unfortunately, a combination of factors can provide motivation for such misconduct, including self-aggrandizement, becoming too enamored of one’s own ideas and insufficiently willing to give them up when they are not supported by the evidence, and just plain greed. While it’s obvious that Wakefield suffered from just plain greed, his ego almost certainly played a role.

It is also truly disheartening that, instead of taking the charges of fraud seriously, and trying to get at the heart of the matter, Richard Horton instead appears to have done his best at damage control. In this, he unfortunately follows a pattern that is all too common in science. An accompanying editorial by Douglas J. Opel, Douglas S. Diekema, and Edgar K. Marcuse, entitled Assuring research integrity in the wake of Wakefield addresses the question of how this can happen, be Wakefield and his misdeeds or other scientific fraud. They correctly point out that the problems that allow people like Wakefield to get away with what Wakefield got away with tends to be a systemic problem and that instances of fraud like Wakefield’s should be treated as adverse events. While this is a useful model as far as it goes, more useful is the concept that the culture of science needs to change:

We must transcend traditional hierarchies and authority gradients to empower everyone in the research enterprise–especially those on the front lines, such as research assistants, data analysts, and project managers–to raise questions and “stop the line.”12 We must train our research leaders–such as department chairs and medical school deans–to manage such inquiries. We must not allow it to be “customary” for journal editors “to discuss and take the word of those against whom the allegations are made.”3 Lastly, when allegations of research misconduct or unethical research are brought to the attention of research leadership, these leaders must recognise that they often have a conflict of interest in managing these allegations.

Most depressingly, it is very telling that it took Brian Deer’s investigation to bring the full extent of Wakefield’s fraud and scientific misconduct. In this case, scientists couldn’t keep an eye on their own house; it took an outsider to uncover the fraud. Science needs to do better. The consequences of not doing better are stark. Even though scientific research quickly refuted Andrew Wakefield’s claims with additional studies, science alone wasn’t enough to bury his fraudulent conclusions once and for all. I’m not even sure that Brian Deer’s revelations will be enough.

By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

176 replies on “British science accused in the wake of the MMR scare”

Peer review has a hard time doing that because the default assumption in science tends to be one of honesty, namely that what authors report in manuscripts submitted to journals does not represent falsified data. Oh, sure, editors and reviewers are fully aware that scientists will try to present their data in such a manner to put the best possible spin on their experiments, to make their story as persuasive as possible. That’s expected. What is not expected is outright dishonesty, falsification of data, the offense by Wakefield for which Deer presented strong evidence in his previous two articles.

Maybe it would be better if they, you know, actually looked at the data instead of the conclusions and the pedigree of the researchers.

“What is not expected is outright dishonesty, falsification of data, …..”

Why isn’t it? Because it’s assumed researchers abide by some kind of 19th century gentleman’s code? With the range of scandals in science that are building up, if scientists want to keep their reputations they’d better stop being so lazy. Publishing crap that you haven’t really reviewed is just as dishonest as producing it.

This kind of stuff is what legitimate science is undermined. Probably the most important science being done today, climate change science, has been damaged by association with this kind of stuff.

Maybe it would be better if they, you know, actually looked at the data instead of the conclusions and the pedigree of the researchers.

Clearly, you’ve never peer reviewed a scientific paper before.

Peer reviewers do look at the data. The problem is that falsified data can be very difficult to detect. In the case of a paper like Wakefield’s, without having access to the primary data, including all the subjects’ medical records, it’s basically impossible. Peer reviewers can’t spend that much time on a single paper, particularly since peer reviewing is one of those unpaid activities that academics are expected to do as part of their academic activities.

It took Brian Deer years of digging to uncover all of Wakefield’s offenses.

As for your crack about climate change science, it’s clear that you don’t know what you’re talking about there, either.

Orac, are you familiar with the Hauser scandal of last year?

In one case, according to an article in The Boston Globe on Tuesday, Gordon G. Gallup Jr. of the State University of New York at Albany asked Dr. Hauser for videotapes of an experiment in which cotton-topped tamarins were said to recognize themselves in a mirror. When he received the videotapes, Dr. Gallup could see no evidence that this was the case. Dr. Gallup did not return a call or respond to e-mail on Wednesday.

Dr. Hauser’s 2002 article in Cognition was published with two co-authors, but he has accepted responsibility for the error. One co-author, Gary Marcus of New York University, said he saw the summary of Dr. Hauser’s experiments but not the raw data. He was informed that there was a problem with the data, but has not seen the result of the investigation.

http://www.nytimes.com/2010/08/12/education/12harvard.html

Note: that was one of the co-authors!

Colleagues of Hauser’s at Harvard and other universities have been aware for some time that questions had been raised about some of his research, and they say they are troubled by the investigation and forthcoming retraction in Cognition.

“This retraction creates a quandary for those of us in the field about whether other results are to be trusted as well, especially since there are other papers currently being reconsidered by other journals as well,’’ Michael Tomasello, co-director of the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany, said in an e-mail. “If scientists can’t trust published papers, the whole process breaks down.’’

http://www.boston.com/news/local/massachusetts/articles/2010/08/10/author_on_leave_after_harvard_inquiry/

When that scandal was breaking I asked several researchers I know who review papers that are published in professional journals if they look at the data and none of them said they normally do. That practice is guaranteed to let through stuff that isn’t, based on the conclusions of the researchers, who may or may not let their interests influence those conclusions on the basis of professional status.

If, as you assert, the normal standards of peer review let through fraud and incompetence, then either there’s something wrong with peer review or with the concept that science is in the business of producing reliable information. I’d like science to be reliable a lot more than that a lot of stuff gets published. Your usual standard that something is “woo” if you don’t happen to like it doesn’t seem to be sufficient to guarantee reliability. Me, I’d like to at least know that the reviewers had looked at what they’re passing through.

Science scandals are as liable to damage the reputation of science as scandals in any other profession.

Since you’re excusing the present system, I’m surprised you’re in such high dudgeon at the results.

Lets suppose Wakefield’s paper was meticulously accurate in terms of its data (OK – it wasn’t, but just for the sake of argument…)

He would have been looking at a possible new phenomenon, which rightly would require explaining, and to generate a possibly plausible hypothesis for this is not necessarily a bad thing to do. In his eyes MMR seemed to be the likely culprit, and therefore this would merit further exploration and investigation. As scientists we should welcome challenges to the prevailing consensus or opportunities to explain new phenomena. So asking the Lancet to publish his paper was probably an OK thing to do.

Whether it should have been published is another matter – until now, I was of the opinion that the Lancet had published the paper in good faith and only later, to spare its blushes once the shit hit the fan, tried to explain matters away. But it seems that before the paper was published serious concerns about it were directly raised with them.

So how should Horton have acted at that point? I believe he should have deferred publication and asked for both an independent investigation into the Deer allegations, and also asked Wakefield to look more closely at his scientific evidence (Wakefield inserted an addendum at the end of the paper to say he had seen another 40 children with the syndrome) and come back with a more thorough analysis.

Of course we now know that of the other cases Wakefield’s unit had seen, precious few of them had parents who blamed the MMR vaccine (so they were useless for his purposes).

The raw data of papers should be, ideally, made publicly available, always — within reason (i.e. anonymized when appropriate.) I don’t see any reason why things shouldn’t work like this, with all the technology that’s available at the moment.

There should be journals that only accept papers if the authors make the raw data and source code available to the public. These journals would and should be considered more credible than the rest.

Beyond increased transparency, raw data can be re-analyzed by other data scientists, who can come up with new findings, and superior methods of data analysis. The data could be used in data mining/analysis competitions, and so forth. It would accelerate the scientific process.

Peer review has a hard time doing that because the default assumption in science tends to be one of honesty

Not just for fraud, as Orac states, but for other kinds of scientific misconduct such as plagiarism. Peer reviewers don’t have the time or the resources to reproduce the results ab initio; the system depends on the reviewer being able to assume (and most of the time this assumption is valid) that the authors of the manuscript performed the described experiment (or simulation or theoretical derivation, in some fields) and are reporting in good faith and in their own words the outcome of that experiment. Where fraud has been committed, it is usually detected after the fact, either by chance (as in the Jan Hendrik Schön case, where a junior scientist happened to notice that two figures purporting to show results from rather different experiments were in fact identical) or after a thorough investigation (as in the Wakefield case).

That’s not to say that peer review as actually practiced is perfect. I’ve seen too many instances of pro forma reviews and cases where the referee is flat-out wrong (both on my own papers and by the other referee on papers I have reviewed). I’ve also read too many papers where my immediate reaction was, “How the %&#@ did that get past the referees?” But these are weaknesses of implementation, not of peer review per se. A more serious flaw that I don’t have an immediate answer for is the perverse incentive effect: those of us who do our refereeing assignments on time and thoroughly tend to be rewarded with more refereeing assignments, while those who do a bad job suffer the penalty of fewer refereeing assignments (i.e., there are incentives for doing the job badly).

I recall that Mr. Deer has made statements in the past regarding problems with the Lancet paper arose with peer-reviewers prior to publication. Of course memory can be faulty so I hope Mr. Deer may be so kind as to correct me or reiterate.

A little OT, but in the business section of Tuesday’s NYTimes there is an article on the AAPS oppositon to the new health plan. It covered some of the stuff they have published, and one phrase that made me happy was a matter-of-fact comment that among their published studies is one “linking child vaccinations to autism, a discredited theory”.
The author of the article also asked the editor of their journal about peer review. ” When asked, he said he could not say that what percentage of those reviewers were members of his own organization.” Link below
http://www.nytimes.com/2011/01/19/business/19physicians.html

Most depressingly, it is very telling that it took Brian Deer’s investigation to bring the full extent of Wakefield’s fraud and scientific misconduct. In this case, scientists couldn’t keep an eye on their own house; it took an outsider to uncover the fraud.

But as you say, peer-review isn’t well equipped to detect this sort of thing. It isn’t really designed to – presumably this is one reason for the importance of independent replication. So perhaps it is unsurprising that it was turned up by a journalist rather than a scientist.

A little OT, but in the business section of Tuesday’s NYTimes there is an article on the AAPS oppositon to the new health plan. It covered some of the stuff they have published, and one phrase that made me happy was a matter-of-fact comment that among their published studies is one “linking child vaccinations to autism, a discredited theory”.
The author of the article also asked the editor of their journal about peer review. ” When asked, he said he could not say that what percentage of those reviewers were members of his own organization.” Link below
http://www.nytimes.com/2011/01/19/business/19physicians.html

Perhaps we can understand why the public will throw up their hands collectively and say, ” You can’t trust research!” And woo-meisters will jump at the chance to dismiss scientific journals entirely.

I once (informally) counselled someone- who had a son with an SMI- who opined, ” But science has been wrong before! Maybe they are wrong about his illness!” While I’m sure that it might be emotionally reassuring to think this way, I don’t think that it’s reasonable to assume that the general outlines of what we know will be suddenly overturned and that overnight the prognosis will be changed to “favorable”.

But *l’affaire* Wakefield reminds that progress often runs a crooked path** – it’s like looking at the history of the Dow- it’s gone *up* -despite many twists and turns and recessions along the way. People generally don’t have that broad historical perspective- be it history of the DJIA or of theories about mental illness or autism. A job for us.

** in his case, very crooked.

While certainly going off base in his reply, I think there is a grain of insight in Anthony McCarthy’s post at #1. During my post-grad research time my good friend and lab director devised this exercise where he had the undergrad research monkeys… err… assistants… read new studies that were released without knowing who the authors were or what journal it was published in. Removing all the credentials associated with a study made them much less forgiving in their analyses of the study and the veracity of the data. Suddenly papers in lower impact journals seemed to be pretty reasonable and papers in high impact journals with big name authors were seen as crap. With the pressure to “publish or die” many studies are being pushed through on prior credibility when they lack in data and insight.

Changing the culture of “publish or die” and throwing in an arm of the peer review in which reviewers are unaware of the author or the journal in question could help decrease the overall number of publishings a year and simultaneously increase the quality of our body of published data.

Assume that uncorroborated case reports or case series are manifestations of spurious correlation, delusion, misunderstanding, error, motivated reasoning, or fraud. This will save everyone a lot of detective work and expense, which is better directed toward more science.

During my post-grad research time my good friend and lab director devised this exercise where he had the undergrad research monkeys… err… assistants… read new studies that were released without knowing who the authors were or what journal it was published in. Removing all the credentials associated with a study made them much less forgiving in their analyses of the study and the veracity of the data. Suddenly papers in lower impact journals seemed to be pretty reasonable and papers in high impact journals with big name authors were seen as crap.

I’ve actually discussed this issue before; I don’t have time to look up the link right now. If memory serves, I discussed the issue of double blind reviewing, where neither the authors nor the reviewers know who is who. There are problems with this concept, but I did like the idea of stripping manuscripts of author names. One problem with this concept is that a lot of areas are relatively small fields, and anyone who knows a lot about a topic will often find it pretty easy to figure out who the authors are based on what they are studying, the references they cite, and the hypotheses their work supports.

As for Mr. McCarthy, he seems to think that I’ve never written about scientific fraud before, when in fact I have written about it on several occasions. I don’t think that anyone can fairly describe me as “defending” the current system, at least not in the way McCarthy insinuated. (His mention of climate change science was particularly off-base, if he thinks that was an example of scientific fraud. It wasn’t.)

I was merely pointing out that peer review can’t examine all the raw data for most studies because it is impractical, given the massive time commitment that it would require. In his discussion of the Hauser case, he seems to be willfully conflating the concept of co-authorship (where not being aware of the raw data, at least all of the raw data related to your part of the study, is indeed not acceptable) and scientific peer review (where it is usually impractical to look over every blot, every bit of raw data, etc.).

My acid test for someone criticizing the peer review system is to ask them what concrete proposals they have to improve it. Double blind peer review is one such proposal, and it is a serious one. Some go to the other extreme and propose completely open peer review, in which the peer reviewers sign the manuscript and their names are listed somewhere in the acknowledgments or in a separate line after the authors. The idea behind this latter proposal is to force peer reviewers to put their names and reputations on the line as well. Similarly, proposals to make all raw data available are not unreasonable, but might well have unintended consequences, such as patients being reluctant to enroll in clinical trials because they believe that they might somehow be identified. (I know anonymization of the data is part and parcel of such proposals, but patients, particularly underserved populations who could most benefit from increased access to clinical trials, are suspicious enough that doctors are doing “Tuskegee” trials already, and such assurances do not usually convince them.) Even so, this might be the best option.

We’ll see if Mr. McCarthy has any concrete suggestions for how to improve the situation that are actually workable.

I’m (sort of) with @titmouse. I think ultimately the solution is not to make peer review better able to detect fraud, because that is very difficult to impossible, but to try to somehow get the message out that one small study of ANYTHING is really meaningless except as a possible kicking off point for more studies. It doesn’t really matter from a scientific perspective that Wakefield was fraudulent rather than just wrong. It was a study with twelve subjects and the results were never replicated. End of story.

@ Elizabeth Reid

In the light of your remarks, if I thought you were a research scientist in a significant area of public interest, I would pull all your papers.

Why would somebody shrug off research fraud? Why would somebody say that Wakefield’s ability to place fraudulent findings on a matter of child health into the Lancet, and keep it there for six years in the face of all the public and professional scrutiny that it received, “doesn’t really matter from a scientific perspective”?

I find your attitude just beggars belief. If you think so lightly of research integrity, believing that armchair reading is sufficient to protect the public, then I would really have to wonder what goes on in your professional world. I really would.

@Brian Deer — this is pretty much what got your knickers in a twist over Offit, as well, isn’t it?

Try to understand — as was pointed out on the other thread, from a scientific perspective, someone might have committed fraud, but might still be correct about his hypothesis. It is unlikely, but it is remotely possible. In order to dismiss the researcher himself from any position of future responsibility, it is necessary to prove fraud. In order to dismiss the hypothesis from any further scientific consideration, it isn’t necesarilly that one has to prove that the researcher committed fraud, because as I said, it is remotely possible that despite the fraud the idea itself could be correct; to ditch the idea from further consideration in science, the hypothesis has to be demonstrated to be wrong.

What part of this perspective, exactly, do you have such a problem accepting? Why exactly?

Maybe in the pages of some bottom-feeding medical journal or another, but not in The Lancet. This is particularly true given the explosive nature of the implications of Wakefield’s paper, which should have resulted in more caution on the editors’ part to make sure that all the i’s were dotted and t’s crossed.

It’s part of an interesting discussion on Sandwalk I referred to some time ago that it is in fact exactly the case that, paradoxically, it can be those papers making the most spectacular (short of outrageous) claims that may receive the least critical examination.

Consider if you are an editor of a journal and reviewers of a paper. The paper is going to make a real splash, earning publicity for the editor and his journal, as well as generating publicity and excitement (research funding?) in the reviewers’ field of endeavor. These are, it seems to me, incentives to get this thing published and to see nagging doubts as minor.

Now consider the editor and reviewers of an extremely well-researched paper confirming prior work in the field. No such positive motivation; in fact the paper, no matter how well-researched, may not even get published.

I think (I hope!) that you have profoundly misunderstood me, but maybe not. Luckily for me I am not a research scientist so I don’t have to be worried that you’re calling my institution right now. (Seriously? You’d be pulling all of my papers because of a blog comment? I’m glad you’re so dedicated – and I don’t mean that one bit sarcastically, your work is awesome and I really admire it – but that seems extreme.)

Of course it matters that Wakefield is fraudulent. I just don’t think that the integrity of the peer review and publication system can ever rest on fraud detection as a major pillar, because it’s just not set up to do that. NO ONE should be making major decisions based on one study of twelve subjects, even if the researcher is assumed to be honest and the journal is The Lancet and so on. It just isn’t remotely enough evidence to do that, and I really wish that that could get communicated to the general public. That way we would be better protected both from deliberate fraud such as Wakefield’s, honest mistakes, and initial results that don’t hold up, all at the same time.

I think what I’m saying is that I make a distinction between, paraphrasing your words, “being able to place a fraudulent paper in the first place” and “being able to keep it there in the face of evidence that it was fraudulent”. Obviously, the factors that let him keep it there despite growing evidence that it was fraudulent need serious scrutiny and (figurative) heads on blocks, but I think the system that let him place it there in the first place worked as well as one could expect. It was a study of twelve subjects. No one could replicate it. That should be that.

Oops, sorry, in my last comment the italics tag was only supposed to apply to “The Lancet”.

@ Luna the cat,

So, leaving aside your snobby “try to understand” advice, is it that it “doesn’t really matter” then that somebody fabricates research which shows vaccines to be safe and effective because, well, they are?

From a scientific point of view, of course.

Hah! I have been called a “snob” by Brian Deer, I’m sure that qualifies me for some sort of distinction! ;-D

@Brian Deer,

Please don’t misunderstand, I’m not trying to downplay your work uncovering the fraud; as with many people here, I admire it immensely and often refer people to the results. But in response to your question: if someone fabricated data that purported to prove that vaccines were safe and effective, what that would mean is that someone else better go and do a *real* test of vaccine safety and efficacy, toot sweet. And the replicated experiments would be what either validated or disproved the idea — the fact that the first study was fraud means that THAT study can’t be taken seriously — it doesn’t mean that the idea behind it is wrong, it just doesn’t mean the idea can’t be assumed in any way to be right. It is the further scientific work which makes that determination.

When I asked my question, what part of this do you have such trouble with, and why, I meant that seriously. It was a question, not a snark; I’d be delighted if you would go into it a bit more, since I’m having trouble understanding your stance at the moment.

I haven’t read anywhere near all of what McCarthy has to say about this, but right away I was highly disturbed that he thinks that problem of fraud can be ameliorated by getting rid of the default presumption of honesty in data. Sounds nice, but just exactly how do you propose going about doing that? The reason data falsification is so hard to detect is not just because scientists usually assume it isn’t happening; it’s that most of the time there is no direct way to even check on it — and even when there is, it’s insanely cost prohibitive to expect that peer reviewers would check on this for every single paper.

As a trivial example, if a researcher is using a piece of equipment to measure something, how do you know that what she wrote down as the measurement is actually what the readout on the screen said? Should there be a videotape of every measurement made, and then the peer reviewers watch all of the tape? Please.

I realize that’s an unrealistic example, that data falsification doesn’t in practice work like that, but I think it illustrates my point. The data is the record. You can’t have a record of the record — or, I suppose you could, but that could be falsified too, all the way back to, as I said, videoing every action the researchers take. Yeah right.

As I understand it, most falsification of data is detected because the falsifier screws up — either he presents data that is just too “neat” to be believe, or in one high profile case I can’t recall, uses the same cooked data set for two different papers saying it came from separate experiments… stuff like that. There would be virtually no way to have detected that kind of thing, because the thing that was being fabricated was the record that you would check to see if there was fabrication going on.

The default presumption of honesty (at least in presenting the data) is unavoidable. Which is not to say that nothing can be done about fraud, but pretending it’s as easy as just checking the data is silly.

There is a huge difference between mis-interpretation of data & lying about data. Wakefield lied – he made the data fit the conclusion that he had already reached (because he was paid a hell of a lot of money get the results the lawyers needed).

I don’t know how Wakefield managed to skate along for so long – perhaps because the whole topic became political/social almost immediately.

We do need to continue to find better ways to check work & replicate results before studies are accepted as “fact.”

Ironically, I’m spoken to researchers in the Pharma industry that see drugs and treatments abandoned even very late in the process, after millions of dollars had been spent, because the results weren’t what was expected. They didn’t fudge the numbers – they accepted the fact that they were wrong & moved on. Wakefield could have done that, but instead, he made the results fit his conclusions – and that’s fraud.

I am aware that you have blogged on the topic before (and quite well, I may add) and that you have offered some concrete ideas for revision and betterment of peer review. I also agree that the commentary about AGW science was completely off base as well. I wanted to take the glimmer of a point made and offer a real world example of a (at least partial) solution to the issue.

I understand and agree that in many fields the pool of experts is narrow enough to be able to deduce who is authoring the article under review. The system can never be “perfect” but it seems to me these are concrete ways to improve it. Your idea about having the reviewers be identifiable and thus accountable for their reviews can further enhance the intellectual honesty. I think that both ideas can be implemented together – and perhaps a system wherein the reviewers’ identities can be held secret (but ultimately available in certain circumstances) should the paper be denied and left open for those accepted (to uphold the level of accountability) can be implemented.

As for Mr. Deer, I personally (and I think most here as well) think very highly of your work and I see little reason to harbor doubt as to your integrity and skill as a journalist. Your work in thoroughly discrediting Wakefield as fraudulent I believe was very important for a number of reasons (which have been outlined by Orac and others in the comments as well). It seems to me an idealized notion that the science should settle it out and Wakefield’s fraud is non-important – and I think that even Offit or Goldacre don’t really think that. They merely make the point that the study itself was so bad it never should have held the weight that it did. Exposing the fraud was a necessary step to demonstrate to the general populace that he and his work should be ignored, discarded, and blighted but for scientific circles it was superfluous. In fact, I had a friend of mine who is a lay person with interest in this send me an email of your work before I had found it myself (I was on holiday and not keeping up). When she mentioned it and asked what I thought my first response was “Oh, well his stuff has been discredited for ages and there is no credulity to anything he has said. Nothing is new.” As I’ve come to realize (and Orac as well) there is something new – from a popular opinion standpoint and from a self-regulation standpoint. As I see it, from a scientific perspective, you have brought to us evidence that we actually need to change and improve our process, have better self-regulation, and more intellectual honesty. Your work has done nothing to change what we have thought about the ramifications of Wakefield’s work from a [I]scientific[I] standpoint – but much very important work from a public opinion standpoint.

Changing the culture of “publish or die” and throwing in an arm of the peer review in which reviewers are unaware of the author or the journal in question could help decrease the overall number of publishings a year and simultaneously increase the quality of our body of published data.

Actually the standard today is that peer review is “blind.” We do know the journal — you have to, because you have to know whether the paper is suitable for the journal — but not the authors. So that’s not really the issue. (Yeah, sometimes you can guess.)

Your work has done nothing to change what we have thought about the ramifications of Wakefield’s work from a scientific standpoint

Oh, I dunno – I think there is an argument to be made that even speaking in terms of “pure” science, whatever that is, there is a distinction between garbage work and fraudulent work.

The thought is interesting to me that Wakefield’s “study” may have been as small as it was (and would have been unimpressive but for the subject matter) because something larger could have been subjected to statistical analysis that may have indicated fraud, not just bad work, since they would have different statistical “signatures” (e.g., as Deer found through investigation, all the errors pointed in one direction).

@cervantes — not all reviewing has author names removed. It depends entirely on the journal.

Brian: You’re conflating two issues in arguing with the commenters (and Offit et al.).

The fact that the study consisted of a handpicked case-study group with no randomization or replication, and subsequent studies found no evidence of an MMR-autism link, is enough to disregard it scientifically even if it wasn’t fraudulent. This is the point Offit and Elizabeth are making.

That it was also fraudulent is why Wakefield should be barred from any kind of medical or scientific practice and no one should trust anything he says. It’s not irrelevant, because Wakefield is still around and (somehow) carries credibility among non-scientific people, but it has little bearing on a paper that was discredited already.

dt @ 4:

I believe that Brian Deer’s investigations into the matter began some years after the publication of Wakefield’s paper (which happened in 1998).

Not to say there weren’t questions raised by reviewers when the paper was being reviewed, of course. Only that Horton & Deer’s discussions would have occured after publication.

If I am incorrect, I trust I will be put right.

For small fields, blinding of authors could be problematic. For example, the big international symposium for my specialization will attract maybe 50 or 60 people from only a few labs. You can pretty much guess which lab produced a manuscript by the study subject and, in ecological studies, the geographic location.

@jud#29: What I am talking about (and I believe the distinction being drawn by the likes of Orac, Offit, et al) is that we in the medical sciences have long ago realized that the science behind the study, and thus the hypothesis posited, were invalid. The distinction becomes important when you think about subsequent studies and clinical recommendations – regardless of the fraud there was no change in clinical recommendations, no change in scientific consensus regarding MMR. We did not need to know of Wakefield’s fraud to feel good about the fact that we did not (as a whole) buy into and change our application of clinical medicine. From the scientific standpoint it was like finding out an inmate on death row was actually guilty of 6 murders instead of 5. Yes, from a political and personal standpoint that data is relevant. But the person is already on death row, convicted, and over with – the legal system did not need that extra bit of information to justify the action.

Wakefield’s study should have held no merit or sway whatsoever. Period. The science was easy enough to know that. Deer’s discovery and proof of fraud relates ONLY to the political and social fallout and the flaws in the process and media coverage of the “study.” It does not relate to or change the impact of the science behind the study. That was crap from the get go.

Elizabeth Reid (#17) comments:

“It doesn’t really matter from a scientific perspective that Wakefield was fraudulent rather than just wrong.”

to which Brian Deer (#18) retorts:

“Why would somebody shrug off research fraud? Why would somebody say that Wakefield’s ability to place fraudulent findings on a matter of child health into the Lancet, and keep it there for six years in the face of all the public and professional scrutiny that it received, “doesn’t really matter from a scientific perspective”?”

As I see it, there are two “non-overlapping magisteria” here: one is the realm of scientific validity or “truth”, if you like; the other is the realm of ethics.

In terms of validity or “truth”, Wakefield’s 1998 Lancet paper was known to be invalid (i.e. attempts to replicate his findings failed to show what he claimed) long before his data were shown to be fraudulent. His pathetic attempts to “vindicate” his research by claiming conspiracy and corruption on the part of his critics were laughed off by the “scientific community” because that’s exactly the wrong way to deal with scientific criticism. To many of us, his claims of persecution only showed that he had no faith in his results and was trying to “save face”.

As Mr. Deer pointed out in his articles, Dr. Wakefield was offered the time and resources to replicate his findings in a larger group of subjects, but he declined to do so. Had this been more widely known, it would have immediately raised suspicion of fraud in the “scientific community”. He was refusing to do what he should do to defend his findings and – as we only later found out – was not able to plead a lack of time or resources to do it. This would have been (and is) highly suggestive of conscious fraud.

In the ethical realm, I believe that any scientist would be appalled that Dr. Wakefield (or any scientist) deliberately falsified data. I can’t believe there would be a single exception. The only reason that science “works” is because we implicitly trust that researchers won’t commit fraud. We expect them to be wrong on occasion, but we expect that the errors and misinterpretations of data will be honest mistakes, not deliberate attempts to deceive. Being wrong doesn’t end a scientific career (unless you’re usually wrong), but a single act of fraud will bring a scientific career – even a long and productive one – to an immediate end.

Although I cannot speak for anyone but myself, I think that it’s probable that most scientists – at least, those who knew anything about Andy Wakefield prior to last week – considered Wakefield’s research a “dead issue” (and dead wrong) long ago. Thus, the recent revelations of fraud only serve to explain why his conclusions were wrong.

From a purely scientific perspective – from the perspective of increasing human knowledge about how the Universe works – Andy Wakefield’s work was dead long before we knew it was fraudulent. It isn’t any less scientifically relevant today than it was last year, since the “relevance scale” only goes down to zero.

Prometheus

Prometheus – you have said what I attempted to much more eloquently. Thank you kindly. I always find your comments well worded. I have been trying to participate in this conversation through a sinus infection and bronchitis whilst packing to catch my flight to Australia tonight.

Thanks, Midnight Rambler (31) and Prometheus (35), you have explained what I was trying to get at much more eloquently than I could. Of course I’m appalled that Wakefield falsified data (and honestly far more appalled that he performed invasive medical procedures on children for no reason, I would be far less bothered by the situation if he’d been falsifying data about the Higgs bosun). But the whole system of peer review and publication is set up explicitly not to be about the credibility of the researcher but about the soundness of the work. It works imperfectly, but it did work in this case, by showing these results were meaningless well before they were shown to be fraudulent. Unfortunately, by then the idea had gotten into the public imagination.

Brian [email protected]:

Please consider that there are multiple valid concerns at issue in L’Affaire Wakefield. An analogy might help:

I am an emergency medic. My friend is a police officer. Leaving a football game, we come upon a scene where a fan of one team is beating the stuffing from a fan of the opposing team [1].

My friend’s priority is apprehending the assailant. Mine is stabilizing the victim.

I can understand your prioritizing the need to deal with the assailant. Please do us the courtesy of respecting our attention to the victim.

[1] This may be a uniquely American thing. Please try to stretch your imagination.

@D.C. — your analogy is extremely British as well. The only thing that differs is what we call “football”. 🙂

I like your analogy, btw.

“No, what should have gotten Horton fired, from my perspective, is that he accepted such a poor manuscript. Even taken at face value and even if there had been no scientific fraud, Wakefield’s manuscript, a case series of only 12 children that, even viewed in as favorable a light as possible, didn’t support the hypothesis that MMR vaccination was somehow associated with a new syndrome consisting of regressive autism and enterocolitis.

Any good scientist would agree, again, a study with more authors than subjects should have been a letter to the editor and not a major journal article. The Lancet did their job poorly and now BMJ’s hit man is crying “fraud.” I doubt that Wakefield committed fraud and certainly don’t believe the biased reportage of a writer out to get Wakefield. They both have exaggerated and we now get to choose which of the two has good intentions and which has a vendetta. I choose . . . Dr. W. for the former and Deer for the latter. Anyone here agree?

Best,

Jay

@Dr. Jay

It’s a good thing that Deer provided lots of citations and evidence to support his accusations, then. Perhaps you might try, y’know, reading them? But that might make you question your hero worship respect for Wakefield, and we can’t have that, no no. After all, if Jenny support him, you have to, too, no?

@ Dr. J

Umm, no. Fraud was unquestionably committed by Wankers. Whether Deer has a vendetta or not, I could not care less. He has uncovered fraud. Perhaps Deer is a bit miffed that it took so damn long for anyone of the responsible authorities (Lancet, medical community, RFH…) to do anything about it.

No.

Wakefield’s “intention” was pretty patently to line his own pockets from the start. And Deer did outstanding work at uncovering a fraud which was most certainly of great public interest.

Personally, I was convinced that Wakefield had committed deliberate fraud before I knew that he had misrepresented patient’s medical data — I figured that since I read the testimony about the Unigenetics lab PCR, and what was done. But Deer uncovered a lot more, and it is in the public interest that he did so.

You are in serious denial, Jay. It takes a big man to admit that he has made a mistake — and patently, you aren’t that big.

Consider if you are an editor of a journal and reviewers of a paper. The paper is going to make a real splash, earning publicity for the editor and his journal, as well as generating publicity and excitement (research funding?) in the reviewers’ field of endeavor. These are, it seems to me, incentives to get this thing published and to see nagging doubts as minor.

I agree, and a corollary of this statement is that journals higher up the food chain are more likely to encounter such papers. The GlamourMags are notorious for publishing splashy papers which ultimately do not stand the test of time. Schön, whom I mentioned in my earlier post, was especially adept at writing papers for the GlamourMags, and the majority of his retracted papers were published there. Being outside the field of medicine, I’m not completely sure of the pecking order, but my understanding is that The Lancet is (or at least was, prior to the Wakefield affair) one of the top medical journals, part of the tier just below the GlamourMags, and therefore susceptible to this phenomenon. I certainly have seen this phenomenon in play at Physical Review Letters, which has a comparable stature in my field: PRL has published some papers which, as subsequent work showed, turned out to be cases where the authors were fooling themselves (though, oddly, none of Schön’s retracted papers were published in PRL).

There is a difference between fooling yourself and committing fraud. One is honest error, and the other is not. But they have some properties in common, among which is that subsequent researchers will fail to duplicate the results. One would have to investigate, as Mr. Deer has done here, to distinguish the two cases.

I doubt that Wakefield committed fraud and certainly don’t believe the biased reportage of a writer out to get Wakefield. They both have exaggerated and we now get to choose which of the two has good intentions and which has a vendetta.

Fortunately, we don’t have to guess. We can look at facts. So how about you explain why all the facts Deer discovered DON’T prove beyond any shadow of a doubt that Wakefield was in it purely for profit? And manufactured data, lied, and harmed children in pursuit of that?

My friend’s priority is apprehending the assailant. Mine is stabilizing the victim.

I can understand your prioritizing the need to deal with the assailant. Please do us the courtesy of respecting our attention to the victim.

Exactly. I was about to write a long response to Mr. Deer’s slam against Elizabeth, but Prometheus (@35) and you nailed it better than I could think of a way to do.

And now, as much as I hate to have to write this…I must.

Mr. Deer, accusing some of my readers who pointed out that, from a strictly scientific point of view, whether or not Wakefield committed fraud is less important than that he was wrong of either dismissing fraud as unimportant or somehow not caring about the truth is counterproductive and misinterprets what they mean. It really is and does. This is especially true, given that, other than the anti-vaxers and other advocates of quackery and pseudoscience who show up from time to time in my comments to howl their outrage, the vast majority of my readers are great admirers of your work in exposing Andrew Wakefield’s perfidy. So am I. However, my admiration for your dedication and tenacity and wanting to understand your frustration at the abuse you’ve taken over the years because of your work exposing Wakefield will not stop me from telling you when I think you’ve gone too far.

And, make no mistake about it, I really do think that you went too far in your remarks to Elizabeth (@18), particularly the bit about how, if she were “a research scientist in a significant area of public interest” that you would “pull all her papers.” That was completely uncalled for, and I think owe Elizabeth an apology for that remark alone.

If my saying these things means your ire will now be directed at me, so be it. But please consider: These people whom you are criticizing are your allies and fans. They (and I) also agree with you about far more than we disagree about. Can’t we be civil in our disagreements? I understand that you think scientists have downplayed the importance of your work, but, as I said before, I really do think the misunderstanding is primarily a result of two very different ways of looking at the world and not because scientists somehow think your work is without value. I refer readers again to my post from last week for the details:

http://respectfulinsolence.com/2011/01/misdirected_criticism_by_someone_from_wh.php

Any good scientist would agree, again, a study with more authors than subjects should have been a letter to the editor and not a major journal article. The Lancet did their job poorly and now BMJ’s hit man is crying “fraud.” I doubt that Wakefield committed fraud and certainly don’t believe the biased reportage of a writer out to get Wakefield. They both have exaggerated and we now get to choose which of the two has good intentions and which has a vendetta. I choose . . . Dr. W. for the former and Deer for the latter. Anyone here agree?

Dr. Jay. You keep saying you don’t believe Brian Deer. My recent remonstration with him notwithstanding over going a bit too far in criticizing a couple of my readers, I still admire the detail with which Deer has documented each and every one of his conclusions and believe the results of his investigation. Why do I believe them? Because he’s produced the goods. He’s documented everything he’s concluded in agonizing detail.

In comparison, Dr. Jay, all you do is to float in here from time to time, whine about how you think Deer has a vendetta and blithely dismiss his years of work by saying “I don’t believe Deer.” You present no evidence. You present no arguments other than sticking your fingers in your ears and chanting, like a child, “La la la la I can’t hear you!”

Let’s just put it this way. You would be taken a lot more seriously if you were able to produce–oh, you know–some actual evidence for your assertions or if you were able to produce–oh, you know–evidence of actual errors or deficiencies in Deer’s reporting. You can’t, and whenever I call you out on your lack of evidence or valid criticisms, you disappear again for a while, only to reappear later spouting exactly the same nonsense.

Pathetic. Again.

@Jay Gordon:

BMJ’s hit man is crying “fraud.”

So Deer is BMJ’s “hit man”? Meaning the BMJ has it out for Wakefield? What evidence do you have that the BMJ has it out for Wakefield. Besides the fact they accused him of fraud. Unless, of course, the mere accusation of fraud means they must have it out for him.

I doubt that Wakefield committed fraud

Why? Do you know him personally, and he just doesn’t seem like the type of person to do that?

… and certainly don’t believe the biased reportage of a writer out to get Wakefield.

What about it is biased? Or is it that since (according to you) Deer has it out for Wakefield that his reporting must be biased, and hence you don’t even have to bother reading it?

What type of reporter would Dr. Jay not think was biased? One that investigates Wakefield and finds he did not do these fraudulent things? In what way he is biased other than after doing years of research about Wakefield he finds him guilty of many things?

Brian Deer’s reaction to Elizabeth might have been uncalled for, but I think Elizabeth is still wrong.

What if Wakefield’s study had been a huge study (1000s of subjects) with falsified data? From a scientific perspective, a fraudulent study is not the same as an unreproduced study. To clarify, suppose the study met inclusion criteria in a meta-study.

So it’s not just an ethical vs. scientific consideration, as Prometheus argues. If the Bad Guys manage to produce more papers than the Good Guys, a statistical problem results that could, in principle, render science useless. So Brian Deer’s service to the public interest is essential, should not be dismissed, and others like him should be encouraged.

@D.C. — your analogy is extremely British as well. The only thing that differs is what we call “football”. 🙂

Really? So what do you call that game where two teams run around continuously kicking (well, usually) a round black-and-white ball trying to get it into the goal that the other team is defending?

I think ultimately the solution is not to make peer review better able to detect fraud, because that is very difficult to impossible, but to try to somehow get the message out that one small study of ANYTHING is really meaningless except as a possible kicking off point for more studies. It doesn’t really matter from a scientific perspective that Wakefield was fraudulent rather than just wrong. It was a study with twelve subjects and the results were never replicated. (Elizabeth Reid)

In the light of your remarks, if I thought you were a research scientist in a significant area of public interest, I would pull all your papers.

Why would somebody shrug off research fraud? Why would somebody say that Wakefield’s ability to place fraudulent findings on a matter of child health into the Lancet, and keep it there for six years in the face of all the public and professional scrutiny that it received, “doesn’t really matter from a scientific perspective”? (Brian Deer)

Brian, I think I may be able to clarify what is meant here by offering an analogy. I was a contributor to Wikipedia during the Essjay controversy. This was the furor which ensued when a long-time contributor by the pseudonym Essjay, who had posed as a tenured professor with multiple doctorates, turned out to be a college drop-out instead.

Now here’s the thing. In theory, it should have made no difference whatsoever that Essjay’s credentials had been faked — because his credentials should not have been the basis for accepting anything he said. When he said “X and Y and Z is canon law, and I should know because I have a doctorate in canon law,” the response should have been “Fine; if you have a doctorate in canon law then you should have no trouble finding the citations that back up your claim. Until you have those citations, we can’t move forward, because we don’t take anything just on the authority of some editor.” In practice, however, people had said “Well, I’m not so sure about this, but Essjay is absolutely convinced, and he’s an expert, so I guess that’s good enough for me.” Precisely because such credentials were rarely if ever verifiable, Wikipedia’s rules made quite clear: Do not accept a change just on the authority of some other editor. But in practice, people did anyways.

The relevance to the Wakefield affair is that, just as people should not have accepted any edits on the authority of Essjay’s credentials, a tiny study like Wakefield’s should have been at best a suggestion of an avenue of research to explore. Even if some magic oracle could have given us absolute unshakable assurance that Wakefield was completely honest — it was still a tiny study! The ability of such a tiny unreplicated study to prove anything is nearly zero; it should never have been taken as proof of danger in the MMR vaccine or need for a return to single vaccines or anything of the sort. That is what people mean when they say things like “it doesn’t matter that Wakefield’s results were wrong because of fraud” — of course they aren’t saying “oh, it’s okay that he exploited a weakness in the system out of cold-blooded greed and dishonesty,” they’re saying that the weakness in the system is the real problem. Should Wakefield have been exposed as a fraud? Is it important that his fraud be punished severely? Yes and yes; a thousand times yes. But we can’t stop there.

If Wakefield had been 100% sincerely convinced that his research had been accurate and all his results well-supported by his data, he would have led the world just as astray. The damage done by the promotion of his inaccurate results would not be one bit less, just because it wasn’t fraud that led his results to be incorrect.

Everyone is concentrating on the Lancet study, but Wakefield has many more papers. He attempted to strengthen and expand the Lancet paper with the O’Leary et al. paper in Molecular Pathology (PMID: 11950955). Why is there no scrutiny of this paper? Why no calls for retraction? The Mol Path 91 were victims of Wakefield, just like the Lancet 12.

Orac @48 Nicely stated. I think it probably goes without saying the Mr. Deer has suffered a tremendous barrage of hate and disparagement since this series began to appear. No doubt by now he has a collection of death threats as well given past history. I think it is understandable that he may be a bit frayed and sensitive to a perception, however incomplete or off base, that his efforts were somehow not necessary.

To Mr. Deer (in response to no particular comment) I would like to say that indeed we in the scientific community owe him a debt of gratitude for his diligence and persistence in discovering and exposing the full depravity of Wakefield’s many transgressions. The simple fact is we could not have done it. We are not equipped to accomplish such work. He did so in my opinion without generalizing and without paiting all science as fundamentally flawed but he perhaps has some justification for thinking that the medical research community (or a subset thereof) was unhelpful to his efforts and perhaps obstructive. The fact is he, and many people like Orac have for years tried to reassure the public at large that vaccines are both effective and tolerably safe. But I think too we must admit that it is Deer’s work that has driven the proverbial stake through the heart of the vaccines cause autism vampire meme. At least I hope it has.

I think to some extent Mr. Deer may misjudge us in thinking that if the scientific question is settled then we don’t care much about everything that swirls around that. That would be unjust. We do care, quite passionately, but we are constrained by the nature of science from allowing our emotions to determine our actions. In so far as that is possible.

Journalism and scientific reasearch are fundamentally different disciplines. Each bears some culpability in the grim saga of the vaccine scare in my mind because is it not true that without a vociferous and uncritical press response at the beginning of Wakefield’s fraud vaccination rates would have been unaffected? By this I do not in any sense mean to imply that Deer is in culpable for this. Rather that in both professions there are ethical and unethical practioners.

I don’t know to what extent Mr. Deer holds the medical reasearch community in general or some of its members responsible for any sort of ethical lapse or ‘malpractise’. He may in fact not at all do so. He may be just a bit miffed over the ivory tower’s notorious cluelessness.

In all, this particular play points up a few things though in broader context. It has to be admitted that science can be corrupted by lucre. It must also be admitted that some and perhaps much of mainstream journalism is in fact already corrupted by lucre. I think it is also apparent that both science and journalism have the ability to contribute to and benefit humanity, but only if they can retain credibility. Those of us who respect the ideals of both estates must cooperate or fail under the weight of amoral greed.

Wow, I’ve learned a lot from the comments so far, esp. Elizabeth Reid and Prometheus. It seems to be implicit in what they are saying that “from a scientific perspective” doesn’t mean “from a higher perspective” or “from a truer perspective”. In a sense, it means the opposite: taking a more limited perspective where you only care about whether the hypothesis is true. I imagine most scientists in the area are very interested to know that Wakefield committed fraud, as well as very happy that Brian Deer put in the work to catch him. It just doesn’t make much of a difference to their day jobs (except hopefully in the meta sense, of looking that the processes involved in research and publication).

I’m going to be boring and go back to the “improve peer review” discussion.
I like the idea of double blinding, and there doesn’t seem to be much to say against it. OK, so you could sometimes guess who the author is, but you can often guess who the reviewer is now too. On the flip side, I doubt I’d ever review an article again if I thought my name would get attached to it. I know how much animosity it is possible to build up against “That bloody reviewer”, especially when you feel the comments have been harsh, and I really don’t want it directed at me personally! But neither of those will help detect fraud. I think most reviewers feel that, in order to be able to review a paper at all, you have to assume that the authors are honest. Which shifts responsibility onto the editors. But then, if it took Brian Deer months of laborious work to find out the data were fraudulent, I don’t see how we could incorporate that into standard publication routines.

Can’t resist coming back to the subject of contention above: In the narrow focus of the link between MMR and autism, it didn’t matter if Wakers was a lying git or wrong a git, but I think it is highly relevant for science that he was fraudulent rather than wrong as it highlights the big holes in the publication system. Somehow, we need to fix this.

William, you would learn more if you read better, and didn’t just stop as soon as the first idea hit you.

If the detail of why both approaches are important escapes you, try asking questions. But a lot of people have tried to put it simply already.

Personally, I was convinced that Wakefield had committed deliberate fraud before I knew that he had misrepresented patients’ medical data — I figured that since I read the testimony about the Unigenetics lab PCR, and what was done.

Yes, exactly.

Some thoughts regarding replicability, good science, bad science, and fraud:

– Because surprising results tend to get published, and anomalous results tend to average out as numbers grow higher, we do sometimes get the effect that was discussed here recently, that various published “breakthroughs” aren’t borne out over time, even absent fraud or bad work. Someone does a conscientious study on a smallish sample, gets a surprising result, then later studies bring overall results back toward the mean. So not only can it be difficult to tell fraud from garbage, it can be difficult to tell both from good science done on a population that turns out to be anomalous. (Contrast fraud as ham-handed as Wakefield’s, which gets shown to be garbage pretty quickly, though it takes a lot more work to show actual fraud.)

– Back once again to the meaning for “pure” science of bad work vs fraud: I think it’s instructive that Brian Deer used the example of Piltdown Man. Better and worse paleontology has been done on Early Man for well over a hundred years, but Piltdown Man, as an outright fraud, has a special infamy. With the benefit of hindsight, we may congratulate ourselves on the performance of science regarding both Piltdown and Wakefield, but let’s not be too self-congratulatory; reputable scientists were taken in or were more interested in saving their reputations than in seeing scientific fraud exposed.

And the effects on future endeavors in the field are different as well. Science does not take place in a vacuum. Next time there’s a Senate hearing on issues dealing with research funding for childhood vaccines, just how vociferously negative do you think Dan Coats, good politician that he is, will allow himself to be? The exposure of Wakefield should make the research climate and public reception better for people like Paul Offit; perhaps he’ll get a couple fewer death threats this year (let’s hope for none).

So I don’t wonder that Brian Deer sees scientists as having huge blinders on when they say, “Hey, from a science standpoint, we had it handled.”

The problem with expecting work-a-day scientists to catch fraud is that they don’t have the time because they are not funded to do so. Doing peer review isn’t funded. Adding 10x or 100x more unfunded work on top of unfunded peer review isn’t going to work. If you want scientists to look for fraud more, then society has to fund them to look for fraud more, which means doing less new science.

The people who were taken in by Wakefield’s fraud were not scientists. They are quacks like Dr Jay (who apparently is still taken in), opportunists like Jenny McCarthy, cultists like AoA and MDC. Those people are not part of the reality based community. They didn’t arrive at their beliefs through facts and logic. Facts and logic will be unpersuasive at convincing them their beliefs do not correspond with reality.

I have no idea what it would take for Dr Jay to appreciate that Wakefield committed fraud. Trying to convince him is a fool’s errand. Dr Jay is unpersuaded by facts and logic, facts and logic are the only tools that scientists have. Faulting scientists because they can’t convince someone like Dr Jay using facts and logic is unreasonable. The “fault” is not with the scientists.

I am extremely grateful to Brian Deer for uncovering Wakefield’s fraud. Fortunately he was funded to do so. It is unreasonable to expect anyone who is not funded to accomplish such things.

Doing new science and finding new discoveries takes a different skill set than uncovering fraud. James Randi is more effective at uncovering fraud than many scientists because he has the required skill set to detect certain types of fraud. We don’t expect people who make things to be expert at apprehending people who steal things, or people who save lives to be expert at catching people who take lives, or people who make money be expert at catching people who steal money.

I think the largest potential problem is that in the quest to reduce scientific fraud, that scientific error is made into a crime. This is the problem in the criminal justice system, in the zeal of prosecutors to catch and punish someone, they convict innocent people of crimes they never committed. This is because prosecutors are rewarded for prosecutions, not for achieving justice, and not sanctioned for convicting innocent people.

We want and need science to be Byzantine fault tolerant.

http://en.wikipedia.org/wiki/Byzantine_fault_tolerance

That is the source of scientists not worrying as much about fraud as non-scientists think is appropriate. Wakefield’s fraud didn’t lead scientists to change their clinical recommendations. The science heuristic of weighting of evidence, of requiring independent replication, of looking at multiple lines of evidence simultaneously makes science Byzantine fault tolerant.

Scientists want to maximize the rate of progress in the scientific understanding of reality. Unfortunately requirements of the larger society can slow that down. Putting resources in fraud detection means less resources in new science generation. Attempted replication of questionable science is likely to be much cheaper than fraud investigation of questionable science.

I would like to know why there is no investigation of the lawyer who funded Wakefield to commit his scientific fraud? When outside forces (i.e. non-scientists) can commission fraudulent work with impunity, how are scientists supposed to prevent it? What would a scientist have done if Wakefield sued them for libel the way that Brian Deer was sued? The scientist would have had to settle.

I thought Deer’s investigation of Wakefield was very important, and a significant contribution to public health.

But from a scientific perspective, it didn’t tell me anything I didn’t already know. It was already established beyond doubt that Wakefield was wrong. What Deer did was tell us why it was wrong. But knowing why it was wrong does not alter my conclusions on any matter of scientific import.

Wakefield’s findings have been replicated, rendering fraud unnecessary.
See: Gonzalez, L et al (Arch Venez Pueric Pediatrica, 2005).
Balzoa, F. A J of Gastoenterology, 2005
Krigsman, 2009, Autism Insights
Horvath, K. J. pediatrics, 1999
Sabra, the Lancet, 1998
Furlano, R. 1998
Many such papers exist (Torrente, Ashwood) and the connection is being looked at more and more so keep the integrity, whether or not you like the results.

It is worth remembering that Wakefield’s paper was not the only dodgy “research” published by The Lancet. There were also the two papers offering wildly exaggerated figures for the number of casualties in Iraq.

I don’t think the standards of this magazine are very high.

Anon, those have been debunked so many times that I find it disgusting so many people keep bringing them up. Repetition doesn’t make it any more true.

Krigsman: hardly independent of Wakefield. Krigsman was Wakefield’s partner at Thoughtful House, and other authors were their employees.

Balzola: an abstract presented at a meeting 5 years ago, never published as a paper, which does not describe the same GI syndrome which Wakefield purported to have found.

Horvath: author of the long-debunked secretin “therapy”, at least there is no indication that he committed deliberate fraud; he was just shown to be wrong. Also didn’t describe the same GI symptoms or syndrome that Wakefield did.

Sabra: published a case study of two children with ADD and food allergies. Nothing at all to do with autism+GI.

Furlano: talked about the insufficiencies of mucosal defense in the intestines of individuals with some known types of GI diseases; did not confirm Wakefield, only marginally related by subject.

And so on.

In order for papers to support you, they need to be relevant, and, you know, support the assertion being made. FAIL.

Bravo Daedalus2u for a great comment, among many other good ones on the thread.

Re something that was said earlier (by dt, I think), there was no intimation or suspicion of fraud around the original 1998 Wakefield paper, as I remember it. There was, though, some skepticism over what was claimed, and recognition of the ways such small case series studies could be flawed – see e.g. the commentary that ran in the same issue of The Lancet by Robert Chen and Frank DeStefano of the CDC. Unfortunately it is paywalled, so only academic readers will be able to get at it, but among other things it says:

“Alternatively [i.e. if there is no objective laboratory test finding], a clinical or epidemiological study is needed to find out whether the rate of a given syndrome in vaccinated individuals exceeds that expected among unvaccinated controls. Such studies require acquisition of data in an unbaised way. Because of the inherent methodological limitations of epidemiological studies, biological plausibility, consistency, strength, and specificity of association must also be considered in inferring causation…

Is there selection bias [in the study]? The Wakefield report is based on cases referred to a group known to be specially interested in studying the relation of MMR vaccine with IBD [inflammatory bowel disease], rather than a population-based study. A first dose of MMR vaccine is given to about 600 000 children every year in the UK, most during the second year of life, the time when autism first becomes manifest. Not surprisingly, therefore, some cases will follow MMR vaccination. Biased case-ascertainment, as in this study, will exaggerate the association…

Was there recall bias? It is usually difficult to date precisely the onset of a syndrome such as autism. Parents and others may attempt to relate its onset to an unusual event such as coincidental postvaccinal reaction.”

Now, it is absolutely clear that Horton knew the original paper was going to cause controversy. It also seems to be the case (from various things that have leaked out) that the referees were of mixed views about the paper. Horton’s line has always been that he “thought it should be out there so that it could be looked at”. I would tend to see the Chen & DeStefano commentary running in the same issue as a way to contextualise the paper, or provide balance – or, if you prefer, a “CYA” (as in “Cover Your Ass”) tactic.

BTW, as Brian Deer has pointed out before, one of the damning things about Wakers is that he was asked, within days of the original paper and in a high-powered crisis meeting of folk from the MRC (cf US NIH) and the Dept of Health (slightly paraphrased):

“Let’s be quite clear, physician to physician – were these consecutive random referrals? Tell us how they came to you. We need to know to judge if there is a real MMR safety issue. Was there any element of patient recruitment/ selection?

– and he said no. This was one of things for which the GMC busted him, calling his saying this “dishonest” (Para 36 a & b, p 47-48 of the GMC Judgement).

Brian Deer’s latest article, at least the bit about Horton, relates largely to what happened in 2004, when Deer went public with a big chunk of his stuff on Wakefield. The Lancet, as he describes, rapidly published a series of short articles, including ones by Wakefield, by Murch and by Walker-Smith, responding to what Deer had said and denying pretty much all of it. These are the ones which Deer discusses and which show Horton in a bad light, particularly in terms of how they were organised.

Going back to what Daedalus2u said about the lawyer in the original case, the lawyer’s role is directly analogous to what would happen with a vaccine damage case in the US. The lawyer is acting as the parents’ (plaintiffs’) advocate, so it is a tricky point whether there is any duty upon them to be objective about the claims. So unless you thought you could unequivocally demonstrate deliberate intent to fabricate evidence, I would suspect there would be little chance of indicting the lawyer for anything, either in law or under their professional code. S/he is, they would presumably argue, “zealously representing their client’s interests”. Now, there are presumably legal association codes requiring them to advise their clients “in the client’s best interests” – which one might say now was not to be party to the lawsuit – but again I doubt that claiming they had failed in that duty would stick.

NB – IANAL – or an MD…!

Also, Gonzales — found a higher incidence of GI problems in autistic kids than in controls, but no distinct syndromes. Did not replicate Wakefield’s findings.

And if you were thinking of further papers by Furlano — the ones he did that were directly relevant to Wakefield’s assertion, were done with Murch and Walker-Smith, Wakefield’s co-accused — again, not exactly “independent.”

Krigsman: hardly independent of Wakefield. Krigsman was Wakefield’s partner at Thoughtful House, and other authors were their employees.

Also, Wakefield was on the editorial board of Autism Insights, the “journal” where this “study” was published.

And this is how the amazing Krigsman wound up as a doc at Thoughtful House:

Before moving to Texas, Krigsman conducted research related to this speculation [a connection between MMR, intestinal inflammation and autism] at Lenox Hill Hospital in New York City. In 2002, the hospital’s Institutional Review Board became concerned that he was doing unnecessary colonoscopies as part of an unapproved research project. Court documents indicate that he did not fully cooperate with the hospital investigation and ultimately filed suit when the hospital restricted his privileges. In February 2004, the Florida Medical Board assessed an administrative penalty of $1,000, plus $89 in costs, for failure to document continuing medical education required for initial licensure. In April 2004, as noted below, He applied for a Texas license in November and resigned from the hospital staff in December. In 2005, he signed an agreed order under which the Texas Medical Board agreed to license him but assessed a $5,000 penalty for (a) failing to disclose the Florida board action, and (b) while still unlicensed in Texas, misrepresenting himself as being available see patients in Texas.

Anon, any more “scholarly articles” you’d like to cite by doctors fired from New York hospitals, barred from practicing medicine in Florida, and fined for dishonesty in Texas?

In the US, lawyers are considered to be “members of the court”, and so are forbidden to do things like suborn perjury, you know, like pay someone to generate false testimony no matter how much it helps their client. Non-lawyers are not allowed to do this either, but lawyer have “lawyer client privilege” which allows their discussions with their client to be non-discoverable.

The lawyer is not obligated to be objective about the claims, they are required to be truthful. If they cross the line and are not truthful, they should be sanctioned, disbarred and go to prison.

A lawyer can’t advise their client to commit a crime, or they become an accomplice to that crime. Lawyer-client privilege becomes nullified if the lawyer knows of the client’s willingness to commit a crime and allows it to happen unreported and continues with their representation of that client. They are not then engaged in lawyer-client activities, they are engaged in a criminal conspiracy to commit crimes.

Of course when lawyers commit fraud it is more difficult to catch than when scientists commit fraud. There is even less will among lawyers to try and catch other lawyers committing fraud than there is among scientists to catch scientific fraud in my opinion.

Wakefield –

First met Barrs solicitors in Jan 1996, and formulated research study to “prove” MMR damage later that spring.
Lancet paper was published Feb 1998 (submitted earlier).

The time spent on legal work? According to Wakefield, he got £150 per hour, and he accrued £435k as shown by Deer.

That’s 2900 hours, all done within 2 years.

I’d be interested to see the invoice for the work Wakers submitted to the solicitors. I for one cannot envisage how he could do legal work for the equivalent of more than 4 hours a day, for 365 days a year, with no holidays, no weekends off, and all on top of his normal job.

Is someone somewhere telling porkies?
Now either he got far, far more than £150 per hour, or his submission of hours worked to Barrs solicitors was somewhat economical with the truth.
Maybe Barr’s should look at this?

@dt

Was it really all supposed to have been done within those two years? That certainly would be a hell of a lot of hours to cram into Wakers’ working week. From memory, when the jaw-dropping £ 435 K figure first hit the press, Wakers did make some statement along the lines of claiming he had spent “every spare moment” on the case (i.e. all his evenings and weekends) for “several years”. I guess this is on Brian Deer’s website somewhere.

@daedalus2u

The “officer of the court” thing certainly applies in the UK too. But the problem is that you would have to prove the lawyer deliberately mislead the court, or allowed something to go before the court knowing it was misleading. This would be exceptionally hard to prove, I think. The wordings we have, that with hindsight look a bit jarring (“find evidence to prove…” a connection between MMR vaccination and IBD, for instance) all come from proposals to get legal aid (state assistance to people to allow them to bring a case). As these applications go to – presumably – other lawyers and state officials, the language cannot have been that out of line with the usual. I guess the lawyer just has to say:

“Well, based on what my clients told me, I believed, as they did, that there was a case and the evidence would be there. All we were asking for was the money to do the research to substantiate and document the connection we believed was there.”

It might sound hinky to a scientist put like that, but I fear we have to accept that to a lawyer it is nothing special… and it would not be regarded as in any way analogous to (for instance) suborning perjury – not unless they had written invent evidence” rather than “find evidence”.

Incidentally, the lawyer in the case, Richard Barr, still seems very proud of his MMR work and mentions it prominently on his various websites – see e.g. here.

The idea that peer review should be a kind of preemptive fraud investigation is naive. To begin with, it would impose enormous costs. Look at the amount of time Deer had to invest to expose Wakefield, and Wakefield’s misdeeds were not even all that well hidden. Peer review is pro bono work for scientists. It is time-consuming, doesn’t much help our careers, and we don’t get paid to do it.

While reviewers examine data when they review a publication, it is usually in the form of summaries, not raw data. One might imagine that detection of fraud would be improved if reviewers were provided with the raw data, but this is rarely the case. For most types of research, providing the raw data to the reviewers would add very little additional confidence, even if they had the time to reanalyze it in detail (and analysis of raw data tends to be extremely time consuming). Yes, there are a few types of data, such as instrumental data used in climate science, where this is worthwhile. But for most types of research interpreting the results requires more than the raw data. You have to know the provenance of the data: exactly how the experiment was done, and assurance that the data has not been altered along the way. If you aren’t going to trust the author’s stated methods, then there would have to be some kind of procedure for documenting every aspect of data collection (video maybe?) in a manner that was time stamped and watermarked to prevent alteration. You’d need not merely the data that was included in the study, but also some way to verify that no data was improperly discarded, because it is possible to bias a conclusion by leaving out certain data even if the data presented is accurate. And of course, all materials and data generated would have to preserved in a non-tamperable format. The cost and loss in productivity would be enormous. And for the most part, the benefit would be marginal. Even if all of this could be implemented, it is doubtful that it would enhance the progress of science. Identifying fraud can be very time-consuming. Keep in mind that Wakefield’s work had already been shown to be wrong, and discounted by essentially all scientists working in the vaccine or autism field, long before Deer was able to bring his investigation to a conclusion, simply on the basis of conventional science, which failed to confirm or replicate Wakefield’s conclusions.

PS Perhaps the most accessible article on lawyer Richard Barr and his role in the UK MMR case is this one by Dr Michael Fitzpatrick. One thing it discusses, which I had forgotten, was that prior to the MMR business Barr had run “class action” type cases on Gulf War Syndrome and on organophosphate agrochemicals. Both cases subsequently collapsed when lawyers finally admitted there was no chance of winning them in court.

For most types of research, providing the raw data to the reviewers would add very little additional confidence, even if they had the time to reanalyze it in detail (and analysis of raw data tends to be extremely time consuming).

@trill: That’s why I suggested that raw data and source code should be made available to the public, as much as possible. More than allowing people to uncover fraud, it would deter it.

I subscribe to the open-source dictum as stated by Eric S. Raymond: “given enough eyeballs, all bugs are shallow.” (I do disagree completely with some of Raymond’s other views, like those having to do with AGW.)

Comments are closed.

%d bloggers like this: