In the eyes, of anti-vaccine advocates, vaccines bear the brunt of blame for a variety of conditions, including autism, asthma, neurodevelopmental disorders, autoimmune disorders and a wide variety others. Often this link is based on retrospective data, in which parents or patients recall and self-report how many vaccines they’ve had and which ones. This self-recall is then correlated with the health condition under study, and sometimes correlations are found. However, it’s long been known that self-reporting has a tendency to be unreliable, with a tendency to conflate incidents that may or may not be related. That’s why a study in which a patient’s recall is cross-referenced as much as possible with objective and concrete records, such as the study participants’ medical records. Human memory is very fallible, but we human beings often have a very hard time realizing that, admitting it, or accepting it. Each of us tends to think that his or her memory is far above average, at least when it comes to issues like this. Combine the fallibility of human memory with the natural human tendency to find correlations and relations where none exist, and it’s not too surprising that the myth that vaccines cause autism, for instance, is so persistent, all science refuting notwithstanding.
Published just this week in the British Medical Journal is a study that demonstrates how relying on self-reporting can result in investigators finding correlations that are probably not there, particularly when it comes to relating vaccines to health problems. What’s interesting about this study is that it’s not about vaccines and autism. Rather it looks at the relationship between vaccines given to British soldiers serving in Iraq between 2004 and 2006. That no correlation between vaccination status (specifically multiple vaccines) and various health complaints was not particularly surprising, but there was a twist. The authors analyzed the data both using self-reported data and actual medical records about vaccination:
LONDON (Reuters) – A British study has found no link between illness among British troops sent to Iraq and multiple vaccinations. Instead, it says the troops tend to blame poor health on multiple jabs even when they did not have them.
Several studies have linked ill health in forces in Iraq to multiple vaccinations, but Dominic Murphy, a psychologist at King’s College London, said these relied on troops’ own recollections, and should be re-evaluated.
“Multiple vaccinations given to personnel in the UK armed forces in preparation for deployment to Iraq are not associated with adverse health consequences when vaccinations are recorded objectively from medical records,” said the study carried out by Murphy’s team, published in Tuesday’s British Medical Journal.
The twist showed that soldiers tended to blame poor health on multiple vaccines, whether they in fact received multiple vaccines or not. Here’s how the study was set up. The investigators examined a cross section of the British military serving in Iraq between June 2004 and March 2006. Soldiers who had deployed to Iraq either during the 2003 invasion or subsequently were selected for the study, and they completed detailed questionnaires. One of the questions on the questionnaire was: What was the maximum number of any vaccines that you received in one day in preparation for your deployment?” Examples of the specific vaccinations administered to service personnel included vaccines against anthrax, tetanus, typhoid, and yellow fever, all of which, with the exception of the anthrax vaccination, were administered routinely. The exception was the anthrax vaccination, which was offered under a separate program that included information and written consent.
Other questions on the questionnaire were designed to elicit health information by asking about a wide variety of health measures, including a 13 item fatigue scale, a general health questionnaire (GHQ-12), a 53 item physical symptom checklist, self perception of health with a single item from the SF-36, and symptoms of post-traumatic stress disorder measured with a 17 item checklist (PCL-C). In addition, a 10% subset of soldiers was randomly selected for verification of vaccination records. Investigators actually visited military medical facilities and collected data on vaccination, recording the maximum number of vaccinations received on a single day before deployment. To avoid missing data, both deployment medical records and standard medical records were examined.
When investigators looked for agreement between reported vaccinations and actual vaccination status, they found pretty poor agreement. Then, they did their analysis two ways. First they looked for correlations between self-reported vaccination status and a number of health complaints. They found several, with veterans remembering receiving more than one vaccine in a day being more likely to report feelings of stress, fatigue, and other health problems than those who recalled never having more than one vaccine in a day. They then repeated the analysis using objectively determined records of vaccination status and found–well, let the investigators say it:
We repeated the analysis using the number of vaccinations recorded from participants’ medical records (table 4). Without exception, we found no health differences between individuals whose medical records indicated they had received no more than one and those who had received two or more vaccinations on a single day.
The conclusion is that prior reported associations between vaccination and health problems in British veterans are probably due to recall bias. The study is not without its weaknesses, perhaps the most troublesome of which is that the investigators only used one question to assess the subjects’ vaccination status. Also, followup has only been less than four years, which, if there were indeed a link between vaccination and adverse health outcomes in the form of chronic disease, may not sufficient time. Even so, however, this study is pretty strong evidence for an effect of recall bias in British soldiers on prior studies.
The phenomenon of recall bias can occur when there is intentional or unintentional differential recall of information about an exposure or a health outcome of an association by subjects in one group compared to subjects in another group under study. This differential recall can lead to a systematic misclassification of the study subjects with regards to the exposure or outcome variable. This bias can result in a study either finding an association between an exposure and a health outcome when there is none or in failing to find an association that does exist. Indeed, it’s a problem for studies of sun exposure and melanoma, abortion and breast cancer (where recall bias causes a real problem), and, of course, vaccines and a variety of health problems. Unfortunately, for some study designs (sun exposure and melanoma, for example) there is no satisfactory way to get at the exposure data without relying on self-reporting.
Recall bias tends to be the worst when either the exposure or health outcome meet one or more of these conditions:
- The event under study is highly significant in the life of the subject (such as cancer).
- The patient has a preconception that the exposure and the health outcome are related.
- The media reports an association between the exposure and the health outcome.
- The exposure or behavior are socially undesirable or illegal.
Number four, of course, would tend to decrease an association, as people would tend not to recall or report socially undesirable or illegal behaviors. The other three tend very strongly to link a risk factor and a health outcome when there may in fact be no link. Indeed, #1, 2, and 3 are highly applicable to any retrospective study in which self-reporting of vaccinations received by babies by their parents are linked to autism rates. Having a child with autism is a highly significant event in the life of the parent; many parents now have a preconception that vaccines cause autism; and the media, fueled by useful idiots like Jenny McCarthy, is constantly reporting scientifically dubious “associations” between vaccines and autism. Indeed, Generation Rescue’s pseudoscientific telephone survey was nothing if not an exercise in harnessing recall bias to the antivaccinationist cause.
Recall bias is a real potential problem in any retrospective study design that includes self-reporting of exposures or factors that the investigator is testing as a risk factor for the disease in question. It can, however be controlled for. What’s critical in evaluating any retrospective study is to look at (1) whether the investigators admitted the possibility of recall bias and (2) how much effort they took to try to minimize it. The best studies, of course, rely as much as possible on objective data and not patient recall. The medical record and data sources used may have their own problems, but nowhere near the level of problems that recall bias can cause.
REFERENCE:
Murphy, D., Hotopf, M., Wessely, S. (2008). Multiple vaccinations, health, and recall bias within UK armed forces deployed to Iraq: cohort study. BMJ, 337(jun30 1), a220-a220. DOI: 10.1136/bmj.a220
