I have to take this opportunity to express a bit of disappointment in one of my fellow SB’ers.
When I encounter a study that seems to confirm my biases, as a skeptic, I try very hard to be even more skeptical than usual, because I would hate to be caught trumpeting a weak or bogus study as evidence supporting a belief of mine. That would be very embarrassing to me. At the very least, although I might not always succeed, I usually try to be very candid about limitations of studies that I cite. Unfortunately, yesterday, Bora (via Archy) failed to heed that rule. Indeed, he clearly let his politics overwhelm his critical thinking skills when discussing a study. Even Tom Tomorrow fell for it, but I’ve come to expect that of him, given how utterly unskeptical and downright credulous he’s been in the past about, for example, the pseudoscience of the mercury militia. No surprise, either, that Daily Kos gleefully lapped up this twaddle and that the Kossacks are presently merrily gloating over it, with only a precious few comments expressing any critical thinking or skepticism about it at all. Worse, even Sara Robinson over at the usually sane Orcinus fell for this dubious study hook, line and sinker (as did a fellow skeptic Southern Fried Skeptic). It’s obvious that the suppression of skepticism when it comes to comforting studies is not a phenomenon restricted to the right.
Here’s news story about the study that Bora posted about and that’s spreading throughout the liberal blogosphere:
A collective “I told you so” will ripple through the world of Bush-bashers once news of Christopher Lohse’s study gets out.
Lohse, a social work master’s student at Southern Connecticut State University, says he has proven what many progressives have probably suspected for years: a direct link between mental illness and support for President Bush.
Lohse says his study is no joke. The thesis draws on a survey of 69 psychiatric outpatients in three Connecticut locations during the 2004 presidential election. Lohse’s study, backed by SCSU Psychology professor Jaak Rakfeldt and statistician Misty Ginacola, found a correlation between the severity of a person’s psychosis and their preferences for president: The more psychotic the voter, the more likely they were to vote for Bush.
But before you go thinking all your conservative friends are psychotic, listen to Lohse’s explanation.
“Our study shows that psychotic patients prefer an authoritative leader,” Lohse says. “If your world is very mixed up, there’s something very comforting about someone telling you, ‘This is how it’s going to be.'”
The study was an advocacy project of sorts, designed to register mentally ill voters and encourage them to go to the polls, Lohse explains. The Bush trend was revealed later on.
How bad is this? Let me count the ways.
From what I can tell based on the story, this study reeks of really bad design and post hoc analyses, and, sadly, Bora and much of the liberal blogosphere fell for it hook, line, and sinker just because it reinforces their own political beliefs. Let’s take a brief look at the reasons this is almost certainly a worthless study. First, here’s the first:
Rakfeldt says the study was legitimate, though not intended to show what it did.
“Yes it was a legitimate study but these data were mined after the fact,” Rakfeldt says. “You can ask new questions of the data. I haven’t looked at” Lohse’s conclusions regarding Bush, Rakfeldt says.
“That doesn’t make it illegitimate, it just wasn’t part of the original project.”
Aarrrgh! That last statement causes intense pain to my brain! Did this guy ever take a course on the scientific method and statistical analysis? Worse, he’s basically admitting that he hasn’t looked at Lohse’s conclusions about a correlation between psychosis and voting for Bush. Unbelievable! If Lohse were my graduate student, I’d have slapped him down hard for publicizing his new conclusions before I had had a chance to look at them in detail and decide whether I thought they were a valid extension of his original thesis work. After all, my reputation would be on the line as much as his. I might even boot him from my lab.
Rakfeldt is only partially correct at best. The post hoc analysis of data gathered for another purpose and data dredging (a.k.a. data snooping, which occurs when a given set of data is used more than once for purposes of inference or model selection) used to come up with this result don’t make the study “illegitimate.” However, they sure as hell make its conclusions very, very suspect at best, particularly given that there were only 69 patients examined in the study. If Lohse had wanted to do data mining correctly, he’d have to use a technique that we use when looking for patterns in huge data sets of microarrays. Basically, he needs to divide the dataset randomly into two subgroups. Then he needs to use one group for hypothesis generation and then test the hypothesis on the other group. Only if the second group supports the hypothesis generated is it reasonable conclude that the hypothesis might be valid. Ideally, one should then validate this hypothesis again on an entirely different dataset unrelated to the first one. It doesn’t sound to me as though Lohse did any of that, and, even if he did, the number of patients was almost certainly too small to draw valid conclusions about such an issue.
Worse, Lohse completely ignores one major potential explanation for his result. Note that the study looked only at the 2004 election (when Bush was running as an incumbent) and that the other study cited by Lohse as consistent with his results was a 1977 study in which psychotic patients were found to prefer Richard Nixon over George McGovern in the 1972 election (an election when Nixon was running as an incumbent). Couple that with another observation in Lohse’s study:
The study used Modified General Assessment Functioning, or MGAF, a 100-point scale that measures the functioning of disabled patients. A second scale, developed by Rakfeldt, was also used. Knowledge of current issues, government and politics were assessed on a 12-item scale devised by the study authors.
“Bush supporters had significantly less knowledge about current issues, government and politics than those who supported Kerry,” the study says.
Did Lohse ever consider that the true correlation he had detected is not what he thinks it is (i.e., a correlation between support of President Bush and psychosis) but rather a correlation between psychosis and not knowing much about current issues, government, and politics? In other words, did he even bother to consider th alternative hypothesis that, the more psychotic the voter, the more likely he or she wouldn’t know anything about the other candidate in an election, who is not an incumbent? Perhaps the idea that psychotic patients prefer “comforting” or “authoritarian” choices is correct, but a perfectly reasonable alternative explanation could still be: The more psychotic the patient, the more likely that patient is to vote for the incumbent because the incumbent is known to him, already the President, and therefore a more comfortable choice. This could easily be tested by looking at data from the 1996 and 1992 elections, when incumbents were running, and comparing to the 1988 and 2000 election, when no incumbents were running. (I do note that these might not be the best comparisons, because both George H. W. Bush and Al Gore ran as sitting Vice Presidents in 1988 and 2000, respectively.) Or, even better, Lohse could design a prospective study for the 2008 election, which would actually test the hypothesis, rather than relying on crappy post hoc analysis and data dredging of a small study designed to look at an entirely different hypothesis.
Instead, he appears to have published by press release before his work was even accepted by a peer-reviewed journal.
In any case, the story suggests that Lohse didn’t even bother to consider this rather obvious alternate hypothesis. (Believe me, I’m not a social scientist or psychologist; so if this alternate hypothesis is obvious to me it should be mind-numbingly obvious to a real social scientist.) No, Lohse zeroed right in on the correlation he wanted to zero in on, in much the same way that antivaccination zealots zero right in on vaccines as the “cause” for the alleged lower rate of autism in the Amish while ignoring all sorts of other alternative hypotheses about the Amish that could explain such a lower rate (if it even exists). Lohse may be correct that psychotic people prefer an authoritarian, but it could also just be that psychotic people know less about issues and candidates and therefore do what most people ignorant of the candidates and the issues do: Vote for the incumbent. Lohse may be correct, but we’d never know it from this amazingly flawed study. His failure even to consider alternate hypotheses, coupled with the post hoc analysis and data dredging, guarantee that.
No one could ever accuse me of being a fan of our current President, but unless there’s something that is not being reported that makes this study more scientifically sound than it appears at first glance in the the news report, my tentative (and probably correct) conclusion is that this study, to put it bluntly, sucks. And sucks hard.
It’s exactly the sort of study that provokes snorts of justifiable derision from scientists in the “hard” sciences, and that’s exactly the reaction I had when I read the story about this study–that, and a little bit of embarrassment for Bora, Orcinus, Tom Tomorrow, and all the others who were so quick to buy into this because it confirms their preconceived notions.
ADDENDUM: Here‘s the only other skeptical take on this study that I’ve been able to find thus far. Unfortunately it is rather partisan.
ADDENDUM #2: I’ve followed up on this piece.
