Gayle DeLong responds to the retraction of her dumpster fire of a study on HPV vaccination and fertility

I have a word of advice for an antivaxxer named Gayle DeLong. If you want to do an epidemiological study examining vaccines, don’t try to do it and write it up alone. The result will be a dumpster fire of a study, as it was a year and a half ago with your study of the effect of the HPV vaccine on female fertility. Last week, I learned that study was retracted. (Why it took a year and a half, I don’t know.) DeLong could have saved herself a lot of embarrassment if she had to heeded the following advice, which, if she ever tries to do a study like that again, I hope she will heed.

The advice? Don’t even start the study until you have (1) a statistician and (2) a real epidemiologist (not an epidemiologist wannabe like Brian Hooker) on board. That’s the bare minimum. It’s also useful to have scientists or medical professionals with real expertise in the area you’re studying. So, for instance, in the case of your study it would have been useful to have a real expert on the HPV vaccine and a real OB/GYN with experience doing population-based studies on board. They’re not essential, but highly desirable for a study of this type. It’s also highly desirable that the epidemiologist with whom you collaborate is familiar with the database that you intend to use, so that its shortcomings and oddities (every database has oddities) are known and accounted for. These bits of advice are especially critical if you are not a scientist or a physician, even more so if you are, as Gayle DeLong is, an Associate Professor of Economics and Finance in the Bert W. Wasserman Department of Economics and Finance at Baruch’s Zicklin School of Business who lists her subject matter expertise as finance. As I noted before, Delong did the analysis basically by herself with the help of an incompetent (David Geier), another economist (Sabastiano Manzano), and a professor of history who is her husband (Jonathan Rose), and she claimed sole authorship.

Of course, DeLong did none of those things, likely through a combination of hubris, ignorance, and having a predetermined result in mind, namely her “finding” (if you can call it that, given the poor quality of the analysis) that HPV vaccination is associated with decreased fertility in women. That’s why it ended up being retracted, as described over at Medscape by Dr. Ivan Oransky of Retraction Watch. Noting criticism from Yours Truly and an microbiologist named Elisabeth Bik. Dr. Oransky then describes how DeLong’s study ended up being retracted:

Today, the journal retracted the paper. “Following review and publication of the article, we were alerted to concerns about the scientific validity of the study,” the journal writes in its retraction notice. “As a result, we sought advice on the methodology, analysis and interpretation from a number of experts in the field.

“All of the post-publication reports we received described serious flaws in the statistical analysis and interpretation of the data in this paper, and we have therefore taken the decision to retract it,” the notice continues.

It’s a decision I agree with that’s taken too long:

The retraction “is the correct decision,” Bik told Medscape Medical News. “This paper was used by many people to ‘prove’ that the HPV vaccine caused infertility in young women, but in reality the paper had some severe flaws. Although the author does not use the word ‘infertility’ to talk about the effects of the vaccine, her study has been used amongst anti-vaxx groups to ‘show’ that the HPV vaccine causes infertility. This made a lot of parents anxious [about getting] their kids vaccinated against HPV.”

Precisely. DeLong’s paper was custom-designed to provide support to the false antivaccine claim that HPV vaccination decreases female fertility and to be consistent with the even more bogus claim that HPV vaccination can result in premature ovarian failure. There’s no evidence that HPV vaccination causes primary ovarian insufficiency (and strong evidence that it is not at all associated with POI), as much as antivaxxers keep producing crappy studies to claim that it does, nor is there evidence that it affects female fertility.

Amusingly, DeLong decided to respond to the retraction yesterday, not in a scientific journal, not in a statement to Retraction Watch or Medscape, but rather in that wretched hive of scum and antivaccine villainy, the antivaccine propaganda blog disguised as an autism advocacy blog, Age of Autism:

On October 23, 2019, I received an email stating that Taylor & Francis and Editor-in-Chief Dr. Sam Kacew had opened an investigation of the paper, based on “several public and private expressions of concern about flaws in analysis.” They gave me two weeks to respond to comments from four post-publication reviewers, and I did so. On December 10, 2019, I received an email from Taylor and Francis stating that despite my comments, the “concerns raised by the reviewers still stand,” and they were retracting the article. Since the retraction, the number of views has increased to 24,227.

I’m amused, yet not amused. She’s basically bragging that the number of views of her paper has increased thanks to the publicity, as in: All publicity is good publicity. Maybe it is—to antivaxxers. Yet, the abstract and paper are still there on the journal’s website, just with a small notice of retraction at the bottom that says at the end that the article will “remain online to maintain the scholarly record, but it will be digitally watermarked on each page as ‘Retracted.'” I’m not sure what the point of doing that is.

DeLong also thinks she’s been treated very, very unfairly:

In the retraction notice (https://www.tandfonline.com/doi/full/10.1080/15287394.2019.1669991), Taylor & Francis state that all post-publication reviews, “described serious flaws in the statistical analysis and interpretation of the data in this paper” without going into any detail. However, one of their reviewers determined that I had been careful with the limitations and conclusions of the paper. That reviewer agreed with me that an open debate concerning the findings of the paper specifically as well as vaccine safety in general is essential.

The manner in which Taylor and Francis is handling these post-publication criticisms is highly unusual. Typically, if a researcher sees a flaw in a published paper, he or she openly writes a letter to the editor, to which the author can reply. Both the critique and the reply are published in the journal.

Sometimes. Maybe that’s the way things are done in economics, finance, and business journals. In science journals, if a paper is fraudulent or so flawed that it is worthless, retraction is generally the course of action taken. It’s a course of action that’s taken far too infrequently, if anything. As for DeLong’s appeal to “open debate” is the same appeal science denialists of all stripes make in order to sow doubt and give the impression that they represent a scientifically valid viewpoint.

Then:

A basic principle of medical ethics holds that if there is evidence that a treatment, drug or vaccine may be dangerous, even if that evidence is not conclusive, we must investigate those possible problems until we have settled the question one way or another. Suppressing studies simply because we disagree with them only stifles legitimate scientific debate. My paper did not claim to offer a final answer to this issue. It simply raised concerns and called for further investigation into a question that may have an enormous impact on the reproductive health of millions of women.

So basically, DeLong is admitting that in her paper she was JAQing off and claiming scientific uncertainty where there really isn’t any. Not only is it biologically implausible that HPV vaccination would decrease fertility or cause primary ovarian insufficiency, HPV is not associated with decreased fertility.

So let’s see DeLong’s responses to specific criticisms. I’m going to go a bit out of order, because one of them stood out to me, her response to the criticism that there could have been selection bias:

Validity questionable: The issues of selection bias, lack of similar observations in USA and Europe, weak biological plausibility and inaccurate statement regarding dose response are all factors that raise doubts regarding the data.

These criticisms are flat out wrong. No selection bias exists: I use every observation where the person surveyed includes answers to all the questions I use in the analysis. There are similar observations in Europe: In a separate study, I show that birth rates are falling in European countries that have implemented wide-spread HPV vaccine programs. That study is here: https://www.tandfonline.com/doi/full/10.1080/21645515.2019.1622977 Not only does biological plausibility exist, studies continue to be published that point to a link between the autoimmunity that vaccines can induce and fertility issues. The statement regarding dose response in the paper is correct: I confirmed my interpretation with two independent statisticians.

First off, if these alleged statisticians were not included as authors on this paper, their “contribution” to me is pretty much meaningless. Does anyone want to bet that these statisticians were business statisticians in her department and don’t have a lot of experience with mining databases like this to do epidemiology? If these statisticians didn’t do enough work to merit authorship on DeLong’s paper, then who knows what they did? If authorship was offered and they didn’t want to put their names on the paper, that tells me that they did very little work or that they don’t have enough confidence in the results to associate their name with the paper. Neither is a good look.

DeLong also clearly doesn’t understand the concept of selection bias, which is a problem in any epidemiological study and can be a problem in a clinical trial. Here’s the thing. Selection bias does not mean that you selected subjects in a biased fashion, at least not intentionally. It is used to refer to the bias that can be introduced such that the group of subjects chosen may differ in ways other than the interventions or exposures under investigation. Sometimes selection bias can be corrected for imperfectly by controlling for confounders, sometimes it can’t. However, to correct for selection bias you actually have to be aware of the possibility that it might exist in how you selected your study population. DeLong is blissfully unaware and only addresses the possibility when it’s pointed out to her.

Be that as it may, her study design also completely ignored at least two huge confounding factors. The first comes from which age groups DeLong decided to look at, as Elisabeth Bik described on PubPeer:

The 2 groups also differ in a very important confounding factor, i.e. % college degree. The HPV vaccinated women had a significantly higher percentage of college degree than the non-vaccinated group. This is a huge confounding factor. Women with a college degree have babies at a higher average age (30.3y) than women without a college degree (23.8y). Here is a graph based on 2016 data from the National Center for Health Statistics data from 2016 (source: https://www.nytimes.com/interactive/2018/08/04/upshot/up-birth-age-gap.html), illustrating this difference:

The author limited here study on women aged 25-29, which is below the average age that women with a college degree have their first baby (see #3). If you limit the study group to women <30 years old, that means that the average women with a college degree did not have their first baby yet. This age group is chosen too narrow to make any correlations with the vaccination status.

To be honest, in my original analysis of this dumpster fire of a paper, I missed this confounder. I am very much chastened and embarrassed that I didn’t pick up on it, but then I’m not the person who did the study. So should DeLong because she is the person who did the study, but instead she just denies that it’s a problem:

Methodological Issues: Potential confounders are not accounted for or simply ignored such as an economic downturn, societal trends including postponing pregnancy, increased women in the workforce, changes in contraceptive use, contraceptive failure rates, etc. All these factors impact pregnancy but were not addressed.

The trends mentioned in this criticism address the decline in the overall birth rate. The value of my study is that each observation identifies whether a woman received the HPV vaccine and whether that woman had ever been pregnant. To confirm that overall trends were not influencing the results, I added time trends to the statistical analysis and the results remained: Women who received the HPV vaccine were less likely to have ever been pregnant.

Again, this is not a persuasive answer, because of the choice of age range. If you have confounders unaccounted for, time trends won’t necessarily correct for those confounders. As for biological plausibility, the publication cited by DeLong is a letter in response to criticism, not original research. It’s basically her putting her spin on fertility data, breaking it down by age, with no statistical analysis and the use of statistics from countries she cherry picked. As for her claim that “studies continue to be published that point to a link between the autoimmunity that vaccines can induce and fertility issues,” I can’t argue with that such studies do continue to be published. They’re crappy studies by antivaccine and antivaccine-adjacent hacks like Yehuda Shoenfeld, Lucija Tomljenovic, and Christopher Shaw.

The other major confounder not accounted for, of course, was birth control use, which I did spend considerable time discussing in my original analysis and which her answer does not convincingly address. In order to save myself some time, I’ll just quote myself:

There’s also another huge problem with this study. One of the most important covariates that could impact pregnancy rates is (obviously) usage of contraception. Yet nowhere in the analysis is there a consideration of contraception usage. Yes, Delong brings up the lack of statistical significance of the results among never-married women by suggesting that maybe most of them want to avoid pregnancy (which could be true), but, again, contraceptive use is an incredibly important factor, which was not even included as a covariate. My first thought was that maybe it was a question that wasn’t asked. It’s possible. Oh, wait. It’s not. The questionnaire asks whether a female has ever used oral contraceptives, if she is taking them now, and how long she’s taken them. So why did Delong not include oral contraceptive use in her analysis? She could have. She doesn’t even really discuss it other than discussion of contraceptive failure rates. I strongly suspect there was a reason for this. I also strongly suspect that a correlation between HPV uptake and oral contraceptive use (which is not unreasonable to hypothesize) could explain the results Delong observed and that correcting for oral contraceptive use in the survey sample would likely have resulted in the results of the logistic regression no longer being statistically significant. In fairness, if the correlation is not positive but negative (i.e., HPV vaccination is associated with less oral contraceptive use), the results could be more robust than what DeLong found.

In any case, I can see only two explanations for Delong’s not having done this analysis, given that the data appear to have been available. Either she was clueless and didn’t even consider it as a covariate, or she did some exploratory analyses and with contraceptive use included the effects that she saw disappeared. After all, they weren’t very robust; so I suspect that it wouldn’t take much. I welcome comments from the epidemiologists who read this blog. After all, existing evidence largely contradicts Delong’s findings, with HPV vaccination having no effect on fertility except in one group. The group? In females with a history of sexually transmitted infections or pelvic inflammatory disease (i.e. a group at high risk of exposure to HPV infection), HPV vaccination made pregnancy more likely.

So what’s her answer? Again, it’s not convincing. In response to a criticism about her using the SAS software package and that “authors do not provide criteria for including or excluding variables which is crucial,” DeLong writes:

The variables I include – age, income, education, and race/ethnicity – are standard demographic and socioeconomic factors that could affect the probability of ever being pregnant.

Concerning contraception, the NHANES database contains fragmentary data. The survey includes questions on only three types of contraception, and many women provided no response to the questions. Certainly, a follow-up study to determine whether contraceptive use influences the results is warranted, but it is beyond the scope of this database and this paper. Certainly, further study is warranted.

I won’t address whether SAS was appropriate for the analysis being done. (Maybe mavens of the various major statistical software packages can chime in.) I will admit to being amused by her response to an observation that “this package cannot judge the validity of the questions asked”:

The data used in the analysis come from the U.S. government. If the responses to questions asked in the course of a national survey sponsored by the U.S. government are not valid, why ask the questions?

I think what the reviewer’s criticism means is simple, and it has nothing to do with whether SAS was the appropriate software to carry out the statistical analysis. If she doesn’t know what she’s doing, any fool can enter any data and covariates into SAS that she wants and run as many analyses comparing them and looking for correlations, whether the analyses have any grounding in reality or not. To SAS, it’s all just numbers. It takes an epidemiologist and statistician to judge the validity of the questions being asked, the variables chosen, and the correlations being tested. Software can only crunch the numbers—and crunch only the numbers entered only exactly as the investigator instructs it to. Only someone who knows what she’s doing can determine if the software is the right tool, if the correct numbers are being entered, and if the correct analyses are being carried out.

As for the missing responses to questions about contraception in the database used, if only there were…statistical methods for accounting for missing responses on surveys.

If only…

I’m sure my statistician friends would be happy to help Ms. DeLong out. In any event, if the data were truly too fragmentary to account for contraceptive use as a confounder, then that should have been explicitly stated, along with what percentage of responses are missing. It should also have been pointed out that the results observed could easily have been due to differences in contraceptive use. Including that discussion would have shown that DeLong had at least thought about these issues. She didn’t mention them, which led me to believe that she either hadn’t thought about them or, worse, had done preliminary attempts to control for contraceptive use as a confounding variable and found the differences in pregnancy rates that she had observed disappear. As for “only three” methods of contraception being used? I’m guessing oral contraceptives, barrier devices (condom, etc.), and IUDs would account for the vast majority of contraceptive methods. Even if they didn’t, contraception use is a confounder that had to be addressed. DeLong didn’t even address it in the paper to explain why the NHANES data couldn’t be used to address it.

In the end, none of DeLong’s responses or explanations is convincing. She’s a hack, a finance expert deciding that she can do epidemiology, which to me makes her even more of a hack than Brian Hooker, a biochemical engineer turned wannabe epidemiologist. (At least Hooker had some scientific training and must have taken some basic biostatistics.) In her arrogance of ignorance, her Dunning-Kruger came to full fruition, and she paid the price. Not that that has changed the mind of antivaxxers. Just read the comments after her AoA article. Or better yet, don’t.

via GIPHY