James Lyons-Weiler whines about his “vaxxed/unvaxxed” study retraction

It is always amusing to me to see the reaction of antivaccine quacks and pseudoscientists when their attempts at achieving some measure of scientific credibility for their ideas in the form of getting their papers published in bottom-feeding (albeit ostensibly “peer-reviewed”) journals ends up biting them in the hindquarters. For example, it happened with Anthony Mawson and his risibly incompetent “vaxxed/unvaxxed” study, which, after a history of disappearing and reappearing, was retracted, much to the sadness of antivaxxers, only to reappear again (I’m guessing after Mawson’s check finally cleared). I was thinking this as I was informed that former actual scientist turned antivaxxer James Lyons-Weiler, who co-authored an equally bad “vaxxed/unvaxxed” study with antivax pediatrician Dr. Paul Thomas (whose license was suspended in December), is very unhappy that it, too, has been retracted—and retracted by MDPI, which is not exactly known for publishing what one would call the best journals.

Indeed, Lyons-Weiler is so upset that he published an epic rant about the retraction entitled MDPI’s “The International Journal of Environmental Research and Public Health” does not publish unbiased research or advances in methodology. Reading the rant, the first thing that disappointed me was that my deconstruction of his awful study back in November apparently had nothing to do with MDPI’s decision to retract the paper. More deliciously, this is apparently what happened (that is, if you believe Lyons-Weiler):

After 250,000 reads, the journal retracted our study, Relative Incidence of Office Visits and Cumulative Rates of Billed Diagnoses Along the Axis of Vaccination (2020 Nov 22;17(22):8674. doi: 10.3390/ijerph17228674) after one reader submitted a poorly written email (with typos and grammatical errors) about an imagined confounder they thought might explain the results.

We asked the Editor-in-Chief and the Editorial Board of the journal to publish the readers’ comments, and our reply. They did not respond to that request. Instead, they decided to retract it without further discussion of our reply.

We had published an Erratum, and wanted to publish a small Corrigendum for an error we had reported to the Editor-in-Chief. These issues were unrelated to the anonymous readers’ concerns.

Lyons-Weiler’s woo-ful whine led me to think a couple of things. First, how is it that, before readers sending me links to his article, hadn’t I known that his and Thomas’ paper had been retracted? Mea culpa. More amusingly, I note how Lyons-Weiler is bragging about how many downloads/reads his incompetent paper had received, as though popularity equals validity in science. We all know that it does not. Indeed, the worst conspiracy mongering and pseudoscience often garner the most engagement on social media, which is the whole reason in the first place that social media companies are coming under criticism for their algorithms that prioritize engagement when deciding what content to show in their users’ feeds.

After his false appeal to popularity, unsurprisingly Lyons-Weiler turns to attacking the person who had written to the MDPI journal’s editor with their concerns, portraying whoever it was as having nefarious motives:

Our concerns are not vainglorius. It is unfortunate that they allowed themselves to be duped by a ghoul. A ghoul is one who haunts journals working to retract only studies that include results they don’t want published). This imparts a ghouling bias, also known as retraction bias, into the literature, preventing unbiased meta-analyses. In our cases, it also prevents advances in the methodology of the analysis of data from vaccine safety studies from being adopted and tried out by other scientists.

We would have preferred rational discourse so the scientific community could make their own assessment of the issues at hand. Instead, we have to assess the journal’s integrity, or at least their ability to discern that they have been defrauded by the ghoul.

I must admit that, before Lyons-Weiler used the term, I hadn’t heard of “ghoul” applied to writers complaining about crappy studies, explaining why they’re crappy, and suggesting retraction, much less “ghouling bias.” Google searches for the term and variants didn’t really help either (although a fair amount of anime content showed up in some of my searches, much to my puzzlement—I’m old). So I figured that maybe the ever-reliable Retraction Watch might be able to help, but, alas, a search of the blog for “ghoul” did not turn up anything, much less “ghouling bias.” I also searched for “retraction bias,” but that particular Google search turned up nothing helpful either, nor did searching Retraction Watch for the term. Basically, Lyons-Weiler sounds more than a little conspiratorial in that he seems to be saying that there are a bunch of “ghouls” out there just waiting to try to get papers like his retracted because they “don’t like” their findings. Of course, what Lyons-Weiler neglects to consider is that it is incredibly difficult to get a paper retracted. Just ask Retraction Watch.

Before I get to Lyons-Weiler’s complaints, let’s just recap a bit about his and Thomas’ study and what made it so bad. As I noted at the time, this was supposedly a retrospective comparison of all patients who were born into Dr. Thomas’ practice between June 1, 2008 and January 27, 2019, with a first visit before 60 days of life and a last visit after 60 days. The inclusion/exclusion criteria whittled down over 21,000 records to 3,324 patients, of whom 2,763 were “variably vaccinated,” defined as “having received 1 to 40 vaccines.” But what is the primary outcome studied? For this, Thomas and Lyons-Weiler make up a brand new metric that I’d never heard of before, the Relative Incidence of Office Visit (RIOV).

Never having heard of RIOV as a validated, I did some PubMed searches and couldn’t find a single paper that used it. Certainly, the authors did not cite any papers that had used this particular made-up metric before to justify its use, to demonstrate its advantages and disadvantages, and in general to provide the sort of information that any clinical investigator would want about an unfamiliar metric. Again, as I said before, I’m always suspicious when I see a metric like RIOV. It strongly suggests to me, particularly in the case of a retrospective study, that the authors tried to do an analysis looking at more defined, traditional primary outcomes and failed to find any statistically significant results. In other words, to me this paper reeked of of p-hacking, the practice of doing comparison after comparison until a “statistically significant” result is tortured out of the data.

One other thing I wrote:

Similarly (and perhaps most importantly) the children in Dr. Thomas’ practice are likely to be quite different than children in a typical pediatrics practice, as I discussed before. For instance, there’s likely to be ascertainment bias, which is the systematic distortion in measuring the true frequency of a phenomenon due to the way in which the data are collected. How could this happen? Think about it. Dr. Thomas believes that vaccines cause autism. That right there introduces unconscious bias that could affect how likely he and his staff are to investigate subtle signs of autism and refer out to for evaluation based on vaccination status and how likely he is to ascribe various diagnoses to “unvaxxed” children compared to “vaxxed” children. One can easily imagine this bias leading to unvaccinated children to be less likely to be given an autism diagnosis than vaccinated children or to be—dare I say?—brought in to the office as often for various conditions that Dr. Thomas attributes to vaccines.

I also noted:

At this point, I’d also like to make a general sort of comment about epidemiological studies (and, make no mistake, that’s what this study is, as it is, in essence, a retrospective cohort study). If you’re going to do a study like this, you need a competent statistician involved before you collect the data. The only two authors listed are Thomas and Lyons-Weiler, and the statistical methods are not well-described. It really also would have behooved them to have an epidemiologist or at least a clinical investigator with experience doing retrospective analyses to help them. Clearly, they did not. Margulis claims that they did, but this “independent statistician” (as she puts it) is not identified anywhere that I can see.

A non-representative study population diagnosed with nonstandard techniques plus no good statistician or epidemiologist who would put his or her name on the paper? What could possibly have gone wrong?

Which brings us back to Lyons-Weiler’s complaint, which quotes one of the “ghoul’s” criticisms:

As he has had a possibility to influence all the essential elements – recommendations for vaccination, information on who is vaccinated, diagnoses made and number of office visits – it is not obvious that the data can be treated as neutral. For example, the diagnosis of autism and ADHD are complex as well as culturally dependent. Unintentional positive bias can emerge for a GP that is known to associate vaccination with autism or ADHD.

Precisely! This is a more concise way of saying what I said about the study! The potential for bias in using a single antivax doctor’s practice, a doctor who claims that there’s virtually no autism in his unvaccinated patients, is huge. I will point out that Lyons-Weiler’s answer most definitely does not address the criticism:

When Dr. Thomas approached Dr. Lyons-Weiler with the request to help design the study, Dr. Lyons-Weiler informed Dr. Thomas that his participation in this design and execution of the study was conditional: that any result found would be published, whether the data indicated that exposure to vaccines were associated with increased, or decreased health. The data were drawn from the electronic medical records according to the reported inclusion/exclusion criteria (as reported in the study); no patients were included or excluded using any criteria beyond that were reported.

One notes that Dr. Lyons-Weiler is neither a biostatistician nor an epidemiologist, and to do a retrospective study like this requires, ideally, both, but at the very least a competent biostatistician. Then, while denying such bias (albeit with a disclaimer that he and Thomas had, in fact, noted that Thomas’s practice might not be representative), Lyons-Weiler goes off with a non sequitur in which he invokes “informed consent” (or, as I like to refer to “informed consent” when invoked by antivaxxers about vaccines, “misinformed refusal”):

The universal adherence by all of the physicians in the practice to Oregon State Law to provide informed consent is, sadly, relatively unique. In spite of laws that protect informed consent, and Federal Regulations that forbid coercion for inclusion in clinical research, the American Academy of Pediatrics supports dismissal of patients from practices (AAP, Edwards, 2016). In so doing, the AAP sanctions coercion, including dismissal of entire families from pediatric practices in violation of state and Federal laws protecting patient rights to informed consent.

After citing the AAP’s position on informed consent and complaining that some pediatrics practices are dismissing patients whose parents refuse to vaccinate them, Lyons-Weiler continues:

From this, it can be seen that the decision to not vaccinate is not undertaken by families without due consideration of stated risks and benefits. It also demonstrates that the parents’ decisions to accept or deny a specific vaccine are not, in the end, in the final control of the physicians of any practice in Oregon providing lawful pediatric medical care. To do otherwise would be unlawful.
The standard of care for the diagnoses of autism and ADHD involves referrals to physician specialists outside the practice, and the final diagnosis was not made by the physicians in Integrative Practice over the 10-year period from which the data were drawn.

As I like to say, “informed consent” according to antivaxxers is in reality “misinformed refusal.” After all, if parents believe the antivaccine misinformation that portrays vaccines as not just ineffective but downright dangerous and causing autism, autoimmune disease, and all manner of other health issues up to and including death, then their refusal of vaccines for their children is not “informed.” It is misinformed, hence my term “misinformed refusal.” True informed consent, which is what doctors who are not antivaccine provide, involves a rigorously science-based discussion of the risks and benefits of any intervention, including vaccines. That is the farthest from what antivaxxers provide.

As for the part about specialists diagnosing autism and ADHD, here Lyons-Weiler is showing how little he knows about the practice of medicine. It is the primary care practitioner who is the gatekeeper and who, in most cases, controls referral to the relevant specialists. If that gatekeeper (like Dr. Thomas) is biased, then his referral pattern will be similarly biased. It’s not just the primary care doctor, though. Parents have to agree. If the antivaccine or highly vaccine-hesitant parents catered to by practices like Dr. Thomas’ don’t believe that their unvaccinated child could have autism, then even if referrals to the appropriate specialists are recommended, they might balk, preferring alternative treatments for whatever diagnosis is given to their children. Then there’s the issue of Dr. Thomas’ referral base, which is likely to consist of parents and physicians who think he’s good, parents recommending him to other parents and physicians referring patients to him. Why would they think Thomas is a good doctor? Most likely it would be because they share Thomas’ antivaccine proclivities and like his use of alternative medicine. Basically, Lyons-Weiler doesn’t address the valid criticisms at all.

The “ghoul” also echoed another one of my criticisms:

The authors have invented a non-standard measure (Relative Incidence of Office Visit (RIOV)) for the health problems. The measure has several weaknesses compared to the diagnoses (sic) data. For example, it is more easily manipulated, the visit can take place because of false expectations of the parent, the link between the reason of the visit and the health problem is vaguer(sic) and at one visit several health problems can be assessed, which complicates the classification of the reason for the visit.

The first part of Lyons-Weiler’s response cracked me up:

In 1934, Sir Ronald Fisher invented a non-standard approach to analysis called null hypothesis testing (Fisher, 1935) In 1973, Nei invented the non-standard method for studying genetic differentiation in populations (Nei, 1973). In 2020, Lyons-Weiler and Thomas invented the non-standard measure RIOV. Science progresses by the introduction of “non-standard methods”. ANONYMOUS claims that the RIOV test has several “weaknesses”; however, they fail to provide any explicit theoretical derivation of “weakness” in the method, nor any data or evidence of any kind to support that these supposed weaknesses are indeed correctly surmised, nor that they apply to the implementation of the RIOV measure in our study.

Yes, but Fisher also validated that approach and showed why it was superior. Lyons-Weiler and Thomas did not validate their approach or show its superiority. It is not up to critics to show “theoretical weakness” of Lyons-Weiler’s construct. It is up to Lyons-Weiler to show its validity and at least equivalence to existing measures. He did not.

Lyons-Weiler appeared particularly incensed at the criticism that his new metric (RIOV) might produce a biased result. You might recall that I said a similar thing, pointing out how it could amplify differences between the groups and suggesting that the reason Lyons-Weiler and Thomas created it was because they couldn’t find any statistically significant differences between the vaccinated and unvaccinated groups—even in a practice as nonstandard as that of Dr. Thomas—without creating such a metric. Lyons-Weiler also seems oblivious to how such a practice could result in a serious selection bias, all his mental contortions to deny selection bias notwithstanding.

I must admit here to feeling a bit of schadenfreude. James Lyons-Weiler was once a legitimate scientist but, for some reason, embraced antivaccine pseudoscience. I don’t know why. Likely only he knows. Whatever his reasons, he has joined antivaccine physicians like Paul Thomas to spread antivaccine disinformation. Worse, in the age of COVID-19, like so many antivaxxers he’s effortlessly pivoted to spreading antivaccine misinformation as well. Indeed, he started doing this in January 2020, nearly two months before the pandemic really hit the US in a big way. MDPI might be a dodgy scientific publisher. Indeed, MDPI has been accused of being a predatory publisher, and, whether it’s predatory or not, definitely publishes low quality journals. That MDPI actually retracted his paper has got to hurt.

Good.