Longtime regular readers might remember the various times over the years that I discussed bad epidemiology papers by antivaxxers that claimed to have found associations between various vaccines and bad outcomes. You know the studies, for example, the study ten years ago by Neil Z. Miller and Gary S. Goldman that claimed to find an association between the vaccination schedules of various countries and infant mortality rates. (Yes, they incorrectly claimed that schedules with more vaccines were associated with higher infant mortality rates.) A couple of years before that, antivaccine activist J.B. Handley had published a report that used a cherry picked group of nations to try to argue not only that nations that require more vaccines have higher rates of infant mortality but higher prevalence of autism in children under five. A year before even that, in 2008, the father-son duo of antivaccine quack activists Mark and David Geier had tried to use a similar technique to “show” (incorrectly) that increased vaccine uptake was associated with and increased prevalence of autism. The flaw at the heart of all of these studies was the ecological fallacy. Sure, there were, as you might imagine, many other problems with these studies, such as failure to control for confounders and cherry picking nations used in the analysis (they were done by antivaxxers after all), but the main problem always came back to the ecological fallacy.
Unfortunately, COVID-19 cranks and antivaxxers have also been engaging in the ecological fallacy. For example, believers in hydroxychloroquine as a miracle cure for COVID-19 tried to correlate the use of hydroxychloroquine in various countries with the death rate from COVID-19 in an astonishingly bad pseudo-study. (Well, it was astonishing then. No study, no matter how bad, about COVID-19 or COVID-19 vaccines astonishes me any more.) Later last year, antimaskers and anti-lockdown activists tried to correlate the use of public health interventions with COVID-19 death rates, again falling prey to the ecological fallacy. As I wrote at the time, anti-lockdown ideologues’ “science” had become as bad as antivaccine “science.” Unfortunately, this trend continues, but this time not from antivaxxers but rather from people who should know better but apparently do not. I’m referring to a new paper published in the European Journal of Epidemiology that has gone viral and is being promoted by cranks like “inventor of mRNA vaccines” Robert W. Malone:
I’ve discussed Dr. Malone, his conspiracy theory that Wikipedia is “erasing him,” and his wife’s antics editing his Wikipedia entry to try to falsely portray him as the primary inventor of the technology used to produce mRNA-based COVID-19 vaccines like the Moderna and Pfizer/BioNTech vaccines before. Let’s just say that it’s odd how much he now demonizes those same vaccines as someone who believes that he invented the technology behind them. I will also express some surprise that I was unaware of this study, which had been spreading in the antivaccine and anti-mask crankosphere for at least a few days before I learned of it yesterday morning. I guess there’s just too much misinformation. Indeed, I thought of just letting this cup pass because there is so much misinformation, but I thought that I could add to the discussion what no one else was likely to: A discussion of this study in the context of what antivaxxers have long been doing. Yes, it’s true that a certain ancient reptile who likes to combat pseudoscience has already discussed this study this week, and there is a PubPeer comment that much more succinctly sums up the key problems with this study, but, as my fans know, only rarely have previous criticisms of bad science stopped me (or even slowed me down) when it comes to applying a bit of ultra-Insolence to that very same bad science.
As another prelude, let me just point out that I don’t think that the authors, S. V. Subramanian and Akhil Kumar, are antivax. I do find it rather odd, though, to read that Akhil Kumar is apparently a Canadian high school student at Turner Fenton Secondary School in Brampton, ON. As for S.V. Subramanian, I had never heard of him before; so I Googled him and moseyed on over to his faculty webpage, where I discovered:
S (“Subu”) V Subramanian is a Professor of Population Health and Geography at Harvard University, and chair of the Faculty Advisory Group for the Center for Geographic Analysis at Harvard University. He is a Primary Faculty in the Department of Social and Behavioral Sciences at the Harvard T.H. Chan School of Public Health, a Core Faculty of the Harvard Center for Population and Development Studies, and a Faculty Affiliate of the Harvard Department of Sociology.
These are not exactly what I would consider good qualifications to do a study of this type. He’s a geographer, not a statistician or epidemiologist, much less an infectious disease epidemiologist. Still, he does have a few COVID-19 publications, such as being co-author on this one finding that mobility restrictions were associated with reductions in COVID-19 incidence early in the pandemic and this one urging mass vaccination and other public health interventions in India at the time the pandemic was producing mass illness and death there. In any event, given that Subramanian is faculty at the T.H. Chan School of Public Health at Harvard University, you would have thought that he’d have known better than to mimic the sort of analysis that antivaxxers have long favored since before the pandemic, either to associate harm with vaccines or to imply that they don’t work. Apparently you’d have been mistaken. Unfortunately.
Before I dig into the study by Subramanian and Kumar that Malone was promoting a week ago, let’s revisit the ecological fallacy again, so that you don’t have to click on links to previous discussions by me and others. An ecological trial is a form of epidemiological study in which the unit of analysis is not the individual person, but rather the group. I’ve discussed this issue before (here and here, among other times), in particular the “ecological fallacy,” which states that ecological studies are particularly prone to false positives (although they can also obscure actual correlations as well, depending on the specific analysis).
One of the best explanations of the ecological fallacies I’ve seen is from an epidemiologist by the ‘nym of EpiWonk. Unfortunately, the article is no longer there. Fortunately, there is the almighty Wayback Machine at Archive.org, where EpiWonk defines the ecological fallacy as “thinking that relationships observed for groups necessarily hold for individuals.” It’s worth citing again, even though I have cited this definition before:
The ecological fallacy was first described by the psychologist Edward Thorndike in 1938 in a paper entitled, “On the fallacy of imputing the correlations found for groups to the individuals or smaller groups composing them.” (Kind of says it all, doesn’t it.) The concept was introduced into sociology in 1950 by W.S. Robinson in 1950 in a paper entitled, “Ecological correlations and the behavior of individuals,” and the term Ecological Fallacy was coined by the sociologist H.C. Selvin in 1958. The concept of the ecological fallacy was formally introduced into epidemiology by Mervyn Susser in his 1973 text, Causal Thinking in the Health Sciences, although group-level analyses had been published in public health and epidemiology for decades.
To show you one example of the ecological fallacy, let’s take a brief look at H.C. Selvin’s 1958 paper. Selvin re-analyzed the 1897 study of Emile Durkheim (the “father of sociology”), Suicide, which investigated the association between religion and suicide. Although it’s difficult to find Selvin’s 1958 paper, the analyses are duplicated in a review by Professor Hal Morgenstern of the University of Michigan. Durkheim had data on four groups of Prussian provinces between 1883 and 1890. When the suicide rate is regressed on the percent of each group that was Protestant, an ecologic regression reveals a relative risk of 7.57, “i.e. it appears that Protestants were 7½ times as likely to commit suicide as were other residents (most of whom were Catholic)….ln fact, Durkheim actually compared suicide rates for Protestants and Catholics living in Prussia. From his data, we find that the rate was about twice as great among Protestants as among other religious groups, suggesting a substantial difference between the results obtained at the ecologic level (RR = 7.57) and those obtained at the individual level (RR = 2).” Thus, in Durkheim’s data, the effect estimate (the relative risk) is magnified by 4 by ecologic bias. In a recent methodological investigation of bias magnification in ecologic studies, Dr. Tom Webster of Boston University shows that effect measures can be biased upwards by as much as 25 times or more in ecologic analyses in which confounding is not controlled.
More recently, Gideon Meyerowitz-Katz described the ecological fallacy thusly in the discussion of one of the aforementioned bad studies, and it’s worth citing his description again:
The basic idea of the fallacy is this: you cannot directly infer the properties of individuals from the average of a group. Sounds complicated, but what that means is that if you measure something about lots of people — say, height — you can’t take the average measurement as an indication of any particular person’s status.
There’s a really simple example of this to do with means, or averages. Imagine you’ve got two groups of ten people, A and B. Group A has an average height of 170cm, and group B has an average height of 168cm. If you randomly select one person from each group, who is more likely to be taller, someone from group A or B?
The intuitive reaction is to say that someone from A is going to be taller than B, because the mean height is higher. However, this is not necessarily true. You can have a mean height of 170cm caused by two 200cm giants and eight 162.5cm people, and a mean of 168cm with six 170cm people and four 165cm people. In this case, 80% of group A is shorter than everyone in group B, which means that you’ll almost always get a taller person in group B if you pick randomly.
In other words, the average of a group isn’t always representative of the individuals.
Unsurprisingly, there are a lot of examples of the ecological fallacy from the nutritional literature, as Meyerowitz-Katz noted:
That’s the ecological fallacy in a nutshell. There are dozens of examples, many of them to do with countries and states. It commonly pops up in nutritional epidemiology — if we do a study and find that people who eat vegetarian diets are more likely to be depressed, it actually tells us very little about an individual vegetarian and their risk of depression. Similarly, even though people who eat more red meat tend to be less healthy, we can’t necessarily say that at an individual level eating more red meat is a good or bad thing.
With that in mind, let’s look at what Subramanian and Kumar did in their study. Looking at his faculty webpage, unfortunately I rather can’t escape the conclusion that the ecological fallacy is a big part of what he Subramanian has been doing recently, as the study under discussion seems to use similar methods to a lot of his more recent previous studies.
So let’s look at their findings before going back to discuss the methods:
At the country-level, there appears to be no discernable relationship between percentage of population fully vaccinated and new COVID-19 cases in the last 7 days (Fig. 1). In fact, the trend line suggests a marginally positive association such that countries with higher percentage of population fully vaccinated have higher COVID-19 cases per 1 million people. Notably, Israel with over 60% of their population fully vaccinated had the highest COVID-19 cases per 1 million people in the last 7 days. The lack of a meaningful association between percentage population fully vaccinated and new COVID-19 cases is further exemplified, for instance, by comparison of Iceland and Portugal. Both countries have over 75% of their population fully vaccinated and have more COVID-19 cases per 1 million people than countries such as Vietnam and South Africa that have around 10% of their population fully vaccinated.
And here’s Figure 1:
As our good reptilian buddy noted, the inclusion criteria are not exactly clear. The authors state that they included nations and counties that met the following criteria: “had second dose vaccine data available; had COVID-19 case data available; had population data available; and the last update of data was within 3 days prior to or on September 3, 2021.” As our friendly reptile with sharp pointy teeth noted, though, for some reason France, the United Kingdom, and Germany were not included, and those are some seriously glaring oversights, particularly given that the authors actually mentioned Germany and the UK in the very first paragraph:
Vaccines currently are the primary mitigation strategy to combat COVID-19 around the world. For instance, the narrative related to the ongoing surge of new cases in the United States (US) is argued to be driven by areas with low vaccination rates . A similar narrative also has been observed in countries, such as Germany and the United Kingdom . At the same time, Israel that was hailed for its swift and high rates of vaccination has also seen a substantial resurgence in COVID-19 cases . We investigate the relationship between the percentage of population fully vaccinated and new COVID-19 cases across 68 countries and across 2947 counties in the US.
Even more interestingly (for me, anyway) the references used were not scientific ones, at least references 2 and 3. Reference 2 points to a news article about Germany mulling over restrictions for the unvaccinated as their caseload soared in July, and the article doesn’t even mention the UK other than in passing because there were antimask protests in London. (It does mention France and Greece.) Reference 3 is to an NPR story about how Israel was suffering from a new surge in COVID-19 cases in August, despite a highly vaccinated population. This is generally not how you do scientific papers and references. The article also mentions potential causes: the Delta variant, waning immunity, and how Israel’s population, even though very highly vaccinated, is not vaccinated enough to produce herd immunity against a much more transmissible variant like Delta. I mean, seriously, have Subramanian and Kumar not heard of measles or watched how measles outbreaks could occur in areas of low vaccine uptake, even in a highly vaccinated population?
Then there are issues of potential confounding. Carl Bergstrom pointed this out on Twitter:
The idea is simple: Government entities (be they counties or countries) with better public health infrastructure will have better case reporting and more robust testing. Our reptilian friend noted this too, observing:
They include low GDP countries where vaccination rates are low, but testing levels are similarly low. Using data from these countries may provide with a unreasonably low level of COVID-19.
And, as noted elsewhere:
And then there was this take that called the study “hot garbage” (an accurate characterization):
The paper presents two main arguments. The first is a scatter plot of the percent fully vaccinated against COVID-19 cases per one million in 68 countries. That’s it. It’s a simple bivariate analysis. To be clear, bivariate analysis isn’t inherently wrong, but it depends on the context in which it is used. A published analysis in a prestigious journal needs to do more than simply plot two variables. This excess simplicity ignores the massive variation between countries in testing, quality of vaccines, nonpharmaceutical interventions, and reporting.
Exactly. And don’t even get people started on this issue:
Another concern I have with Subramanian 2021 is the quality of reporting between countries. Does the data coming out of Brazil, India, Libya, and Ukraine match the data quality coming out of Canada, Germany, and the United Kingdom? In other words, is the reported number of COVID-19 cases as a proportion of the true number of COIVD-19 cases equal across the two groups? We already know deaths are underreported around the globe.
Indeed. If you don’t know how cases are reported and compiled in all the countries included in the analysis, you can’t determine if there are systematic flaws in the data that could either produce a false positive or mask an actual correlation. Zero attempt was made to correct for this.
Another epidemiologist made a complementary observation:
Again, remember the lessons from measles. Cases cluster in areas where vaccine uptake is low, and these areas can be smack dab in the middle of a highly vaccinated population, thus skewing the case count for the entire area. For instance, consider an town in the middle of a county (or country) that has low vaccination rates and a high case count. That will artificially inflate the case count for the entire surrounding area (county or country), even if the averaged vaccination rate of the whole county or country is high. Do these people not know basic epidemiology? We saw this sort of thing the year before the pandemic with all the measles outbreaks among populations with low vaccine uptake! We’re just seeing it now on a grander scale. I can forgive Kumar because he’s apparently a high school student, but there’s no excuse for Subramanian not to have gotten a real epidemiologist involved, along with, preferably, a real statistician before undertaking any analysis, not after.
Then there’s this:
Of the top 5 counties that have the highest percentage of population fully vaccinated (99.9–84.3%), the US Centers for Disease Control and Prevention (CDC) identifies 4 of them as “High” Transmission counties. Chattahoochee (Georgia), McKinley (New Mexico), and Arecibo (Puerto Rico) counties have above 90% of their population fully vaccinated with all three being classified as “High” transmission. Conversely, of the 57 counties that have been classified as “low” transmission counties by the CDC, 26.3% (15) have percentage of population fully vaccinated below 20%.
There is but one reaction appropriate to this paragraph:
Why? First, these percentages of vaccinated people are vaccinated adults and do not take into consideration children under 16. Remember, in the US, there is no COVID-19 vaccine yet approved (or even authorized for emergency use) for children under 16. The authors do the same thing with Israel, pointing to it as an example of a highly vaccinated country with rising case counts.
The Raptor will have none of that, and I agree:
Even though Israel had a high rate of vaccination, that was for eligible individuals over the age of 12. Actually, only 58% of the population was vaccinated, which is not high enough. It’s hard to tell what the exact herd immunity level is for COVID-19, but it’s probably close ot 88%. Israel may have been highly vaccinated, but not nearly highly enough.
Multiple critics (and now I) note that the primary endpoint examined is case number per unit population. While vaccines are certainly intended to decrease the case number and transmission of the virus if possible, the primary intent is to prevent people from getting really, really sick, winding up in the ICU, and dying. No attempt is made to analyze the data using these endpoints. Then there is the consideration that various countries were in different phases of the pandemic during the time frame used to examine case counts and vaccination rates. Then, for the county-level analysis, one must remember that states are made up of counties, which mean that groups of counties (in the same state) will tend to cluster because they will have similar testing and other policies. Even so, across the US, different counties (mainly in different states) vary widely in their access to testing, public health interventions, and vaccination rates. That’s not even considering differences in population density that can result in higher transmission in more densely populated areas, or weather. The bottom line is that there is so much wrong with this paper that, to put it kindly, the conclusions are not supported by the data.
As I was wrapping this post up, I came across this:
You know, I shouldn’t have used up that facepalm, because this really does deserve a facepalm—or, better yet, two facepalms:
Anti-vaxxers say they’ve found a smoking gun: a new blue-chip paper that proves COVID vaccines are ineffective.
The vaccine “doesn’t stop you from getting [COVID] at all,” claimed Daniel Horowitz, a senior editor at the Blaze, in a tweet promoting a column he wrote trumpeting the research. The headline: “Harvard researcher finds absolutely no correlation between vax rates and COVID cases globally.” Supporters of Horowitz’s perspective tweeted the piece and posted it on Facebook, where it received more than 4,000 interactions, including 2,600 shares, according to data from CrowdTangle, the Facebook-owned analytics company.
Alas, there’s just one problem for Horowitz and company: S.V. Subramanian, the Harvard professor of population health and geography behind the paper, says the vaccine doubters are completely wrong.
“That conclusion is misleading and inaccurate,” Subramanian told me of Horowitz’s Blaze column over email. “This paper supports vaccination as an important strategy for reducing infection and transmission, along with hand-washing, mask-wearing, and physical distancing.”
Really? It sure didn’t come across that way to me, and I read it before I had seen a lot of antivaccine talking points. Indeed, I’ve never seen or read Horowitz’s article, either. I did do a Twitter search after I had started writing this article, and—surprise! surprise!—virtually every Tweet linking to the study other than the ones I’ve already cited came from antivaxxers touting this study as slam-dunk evidence that COVID-19 vaccines don’t work because there was no correlation between case numbers and percentage of people vaccinated in the country or county analysis. Indeed, what I found was only the top of the iceberg:
Despite the misinterpretation, anti-vaxxers and vaccine doubters like Horowitz have held up and shared Subramanian’s paper as vindication on an array of platforms that have struggled to fight false anti-vaccine information. Horowitz’s own column has been tweeted out to at least half a million users. Posts bringing attention to the paper have done well on anti-vax and right-wing Reddit groups; a summary was posted to more than a dozen subreddit communities with over 34 million followers.
On Facebook, posts sharing a link to the paper’s abstract have also gone viral thanks to similar pages. Bernhard Zimniok, a member of European Parliament representing Germany’s far-right AfD party, shared it to his 24,000 Facebook followers, netting over 1,000 likes, shares, and comments. Slobodny Vysielac, a xenophobic, nativist Slovakian publication which has been likened to Infowars, also shared a link to the study to its 85,000 followers. Across the platform, CrowdTangle analytics show it was shared by pages with over a collective 2 million followers and was interacted with 7,000 times
Subramanian is surprised that his paper was picked up by antivaxxers? How naïve can he be? Also, how clueless can the reporter, Ali Breland, be? He writes the article to frame Subramanian as being unfairly mischaracterized and misinterpreted by antivaxxers. I can see an argument that it is being mischaracterized, but not by a lot. Whether he realizes it or not, Subramanian unwittingly provided a number of antivax-ready quotes in his article.
Go back to the Tweet above by Carl Bergstrom, in which Bergstrom criticizes Subramanian for making prescriptive statements based on a crappy analysis. That whole paragraph basically is, whether Subraminan realizes it or not, parroting an antivaccine talking point, namely that we rely too much on vaccination and that the vaccines are much less effective than advertised. Worse, Subramanian tells us we’re relying too much on vaccination alone (a not unreasonable point)based on an analysis that falls for the ecological fallacy and can very easily be interpreted to suggest that the vaccines don’t work. After all, the key finding of Subramanian’s paper is that there is no correlation between the percentage of the population vaccinated and COVID-19 case counts.
Just look at Subramanian’s concluding paragraph:
In summary, even as efforts should be made to encourage populations to get vaccinated it should be done so with humility and respect. Stigmatizing populations can do more harm than good. Importantly, other non-pharmacological prevention efforts (e.g., the importance of basic public health hygiene with regards to maintaining safe distance or handwashing, promoting better frequent and cheaper forms of testing) needs to be renewed in order to strike the balance of learning to live with COVID-19 in the same manner we continue to live a 100 years later with various seasonal alterations of the 1918 Influenza virus.
Yes, Subramanian states that we should be using standard public health interventions in addition to vaccination to control COVID-19, but then he finishes with a comparison frequently used by antivaxxers, namely likening COVID-19 to seasonal flu and saying that we have to learn to live with it!
If he happens to see this post, I’m sure Subramanian will think I’m being very harsh and very unfair. Maybe. But I don’t think I’m being unfair. Although I don’t think that he or his coauthor are antivaccine, I do think there is considerable value in pointing out how similar the study he did is to the sorts of awful studies prone to the ecological fallacy that antivaxxers did before the pandemic (and still do) to try to attribute harms to vaccines that they don’t cause or imply that they don’t work. Even more importantly, scientists like Subramanian who do studies involving COVID-19 and COVID-19 vaccines really do need to be aware of how their words can be weaponized by antivaxxers and COVID-19 minimizers and conspiracy theorists, so that they don’t craft studies like this one that are so easily co-opted for such purposes.
In fact, I hope that this comparison shocks some scientists. The vast majority physicians and scientists before the pandemic were blissfully unaware of the sorts of distortions of science that antivaxxers routinely engaged in. Some of them were even openly contemptuous of efforts by skeptics to combat health misinformation, quackery, and antivaccine pseudoscience and conspiracy theories, viewing it as beneath them or shrugging it off as unnecessary, not believing that anyone could “believe such nonsense.” I hate to say, “I told you so,” but I told you so. So did a number of others. Unfortunately, we were (and remain) a pitiful band compared to the forces arrayed against us spreading misinformation. It would help if scientists on “our side” didn’t provide ammunition to antivaxxers, which is what Subramanian did.