Things are crazy now for me, both at home and at work. I mean really, really crazy. So crazy that even I, one of the most verbose bloggers out there, am forced to take two or three days off from my little addiction–I mean habit. Consequently, having foreseen that this time would come around these dates, I, Orac, your benevolent (and, above all verbose) blogger have thought of you, my readers. I realize the cries and lamentations that the lack of fresh material inevitably causes. That, I cannot completely obviate. However, I can ease the pain somewhat, and I can do this by continuing my longstanding project of migrating old “classic” (depending upon your definition of the word) posts from the old blog over to the new blog. These are all at least two years old, and thus the vast majority of my readers are unlikely to have seen them. Only the longest-standing readers will recognize these, and it is sincerely hoped that they will enjoy seeing them again. (This particular post first appeared on February 6, 2006, and I see no sign that anything has changed in the nearly two years since the study I discussed was first published.) Truly, Orac is a benevolent blogger, his arrogance, long-windedness, and cantakerousness notwithstanding.
Advocates who are convinced of a link between the mercury in thimerosal used in vaccines as a preservative and autism often point to data derived from the U. S. Vaccine Adverse Event Reporting System (VAERS) as “evidence” that vaccines cause autism. For example, Mark and David Geier, the father-son team of Don Quixotes of the thimerosal/autism movement, have made a veritable career of dumpster-diving the VAERS database and then using the results as their preferred lance to tilt at their windmills–not to mention to use as “evidence” in representing parents suing the Vaccine Injury Compensation Program (VCIP). Publications by the Geiers utilizing the VAERS database are often used by litigants as “evidence” for a link between mercury in vaccines and autism.
Unfortunately, the VAERS database is highly unreliable. The reason is that anyone can submit a report to it, and no one actually verifies the accuracy of the report. Indeed, James Laidler once tested the system by submitting a report that the influenza virus had turned him into The Incredible Hulk. The report was accepted and duly entered into the database. This report was so out of the ordinary that a representative actually contacted him and, amazingly, asked his permission to remove the report from the database (proving that it’s not easy being green). If Laidler had not given it, the report of an adverse reaction in which the flu vaccine turned a man into a huge, immensely powerful green monster would still be in VAERS. Now, via Kathleen Seidel, who alerted me to this, comes more evidence of the corruption of the VAERS database. This evidence comes in the form of a study by Michael J. Goodman and James Nordin published in the most recent issue of the journal Pediatrics, in which the authors examine the question of how much of the seeming increase in autism related to vaccines reported to the VAERS database over the last several years might be related to litigation. Naturally, I couldn’t resist downloading the complete article and reading it.
In the study, the authors, did something incredibly simple that no one had done before. They took data from the VAERS database from 1990 through 2003 and imported it into SAS data files for analysis. Then they searched the database using key words to look for reports associated with litigation, particularly with regards to autism. They searched for records containing “thimerosal,” “mercury,” or “autism” in their fields, especially when coupled with terms like “lawyer,” “legal,” “attorney,” or “litigate,” while excluding records containing “legal” coupled with the term “guardian” that did not relate to litigation. They also excluded cases related to well characterized allergic reactions to thimerosal. Finally, they compared records from nonlitigation cases to those from litigation cases regarding symptomatology reported.
Not surprisingly, beginning in 2001, they noted a dramatic increase in the number of non-Lyme disease VAERS reports related to litigation, from only 7 in 2000 to 213 in 2002 and 108 in 2003. (They attributed the decline in 2003 reports to processing delays in creating public use files.) Next, they examined symptom sets related to symptom sets. For autism, they observed a dramatic increase in the percentage of litigation-related reports from 0% of the reports related to litigation in 1999 to over one-third (35%) in 2002. For records mentioning thimerosal that weren’t related to allergic reactions, the rise was even more dramatic, from 0% of these reports related to litigation in 2000 to 87% in 2002.
This study once again hammers home the inherent unreliability of the VAERS database as a tool for longitudinal studies of the rate of vaccine-related complications. Not only can anyone access it and enter reports without verification, but there is no denominator, which means testing for causality is not even possible with VAERS. Worse, as the authors point out, the rate of reporting of autism as a complication of vaccines is easily influenced by numerous external factors. For example, the authors pointed out that 75% of the autism reports in VAERS between 1990 and 2001 were received not long after the the publication of the the now utterly and completely discredited Wakefield study that claimed to find a link between the MMR vaccine and autism and that 2/3 were received after the American Academy of Pediatrics recommendation that thimerosal be removed from vaccines. And it’s not just autism. For example, in 2002, half the reports to the VAERS database about mental retardation were related to litigation. The authors conclude:
The findings raise an important question about possible misuse of VAERS in the litigation process. When a study is being used to influence important public health decisions, it is important that reviewers and editors fully understand how the data were constructed and their source. Until now, no one has described the magnitude of litigation-related reporting and how these reports might potentially change the results of studies using VAERS data. Longitudinal studies using VAERS data should explicitly take into account changes in reporting sources like the one described in this article.It is impossible to determine the effect of these reports on existing analyses because the existing literature does not describe carefully inclusion and exclusion criteria. For the conditions reviewed here, it is apparent that a large enough percentage of reports are being made related to litigation that failure to exclude these will seriously skew trends. This is important for vaccines that contain thimerosal, and specifically for the MMR vaccine because of the controversy surrounding its relationship to autism. It therefore is incumbent on the authors who use VAERS data to provide detailed methods sections that describe their inclusion and exclusion criteria. To that end, we are making our SAS code available to interested parties. It is not sufficient simply to reference extraction of the VAERS data set.
Indeed it is not. Computer programmers have a famous saying: “Garbage in, garbage out,” meaning that the quality of the results of an analysis can be no better than the quality of the data upon which the analysis relies. Without correction for factors such as the ones for which the authors of this study tried to correct, the VAERS database definitely qualifies as “garbage in” when used to try to follow the incidence of vaccine-related complications over time. The VAERS database may serve a very important function as an early warning system for potential vaccine-related complications that were not picked up in initial clinical trials used to gain FDA approval, but it was never intended to be a means of following the rates of these complications in a longitudinal fashion. Even if it had been, the ease with which the rate of entry of various complications can be influenced by media hype and activists, as well as the indiscriminate use of the database by litigants, long ago destroyed any usefulness that VAERS might have had for such a purpose.
Indeed, it’s even worse than described. The authors themselves recognize that they very well may have underestimated the effect of litigation-related VAERS reports:
Our results are probably conservative. Discussions with VAERS staff at the Centers for Disease Control and Prevention about reports that are generated from the VICP indicate that we may have missed some litigation-related cases because our code identified only a subset of these cases (John Iskander, personal communication, August 6, 2004). We tested code to identify VICP cases but were unable to find a way to identify them. Underascertainment of cases that are related to litigation, however, only strengthens our point. The influence of the litigation process on longitudinal analyses is a serious matter and emphasizes the importance of interpreting VAERS data cautiously.
In other words, the distorting effect of litigation-related cases on reports to the VAERS database is almost certainly worse than this first study suggests. I’m betting that it would be even worse still if the authors had been able to analyze the complete data sets from the years 2004 and 2005, given the firestorm that erupted last year due to the publication of David Kirby’s much hyped conspiracy-mongering “expose” Evidence of Harm and the thoroughly dishonest scaremongering scandal piece by Robert F. Kennedy, Jr. published last summer. Perhaps the authors of this study will update their analysis when VAERS data for 2004 and 2005 become available. I might be going out on a limb here (not really), but I’d predict a huge jump in litigation-related VAERS entries for cases alleging that that vaccines caused autism in the year 2005. I’m also hoping that the authors manage to ovecome the difficulties they encountered and figure out a way to crossreference their data from the VAERS database with the list of litigants going before the VCIP. I predict that the results of such a study would be most illuminating. However, this is a study that, because of HIPAA regulations, will most likely never be done because it would probably require doing what Mark and David Geier once tried to do: Merging data sets in such a way that could risk patient confidentiality.
In the meantime, whenever someone tells you that the VAERS database shows that cases of autism related to vaccines have been rising, just remember two things: (1) garbage in=garbage out; and (2) the VAERS database, sadly, has evolved into a trial lawyer’s best friend.
Just ask Mark and David Geier.
