As the pandemic continues to rage out of control in the US, naturally attention has turned to the possibility of a vaccine against COVID-19. Indeed, even a couple of months ago, I expressed concern that “Operation Warp Speed,” the program promoted by the Trump Administration to speed the release of a coronavirus vaccine, might be moving too fast, to the point that safety might be compromised. The problem with the imperative to develop a vaccine as rapidly as possible in order to halt the pandemic and allow life to go back to more or less normal is so powerful that it can even affect people who should know better, people that I otherwise admire, even. I’m referring to this Tweet that I saw yesterday from Steve Salzberg:
You can see from the Tweet what an irresponsible position Steve Salzberg is advocating. My first reaction was shock, as it seemed that he was advocating bypassing phase 3 trials of a coronavirus vaccine. At least the Tweet certainly seemed to read that way, leading to strong pushback and his backpedaling:
It’s always bad form for a writer to blame his readers when they misconstrue what he says. My personal policy is always first to reread what I wrote and ask myself if the fault lies with me that led people to get the wrong message from something I wrote. It’s a difficult exercise that requires a degree of humility, but it’s necessary. Sometimes I do end up concluding that people are willfully misunderstanding or misrepresenting, but I try really hard not to conclude that until I’ve thoroughly read and reread what I’ve written and asked others if they came to the same conclusion about its meaning as those who seemed to be misinterpreting it.
In fairness, a Tweet is easy to screw up and end up saying what you didn’t mean, and I did read the blog post, We Should Consider Starting Covid-19 Vaccinations Now:
Development of new Covid-19 vaccines is proceeding at a furious pace, which is good news for the world. We already have two vaccines in phase 3 trials in the US and Europe; each of these trials which will vaccinate many thousands of people, and then wait to see how many get infected. If the vaccines work, then in a few months’ time we’ll be able to start large-scale production.
But we don’t have to wait. Both of these vaccines (from Moderna and Oxford University/Astra Zeneca) have already been shown, in phase 1 trials, to be safe and probably effective. That’s why the companies are moving ahead and giving each vaccine to 30,000 more people: they are fairly confident that the vaccines are safe. The NY Times reports that 3 other Covid-19 vaccines are also in phase 3 trials: one from BioNTech and Pfizer, and two from Chinese companies, Sinopharm and Sinova Biotech.
So why not start administering millions of doses right now? We should at least consider the possible benefits–and the costs.
There’s so much wrong here. First of all, phase 1 trials do not show efficacy. That’s not their purpose. They are designed to demonstrate safety, but that doesn’t really tell the whole picture. Basically, phase 1 studies are designed to make sure that there aren’t any gross toxicities, but, as I’ve discussed many times with respect to “right-to-try” laws, most phase 1 studies have few subjects in them. Again, the idea is to find severe, frequent toxicities that might show up in such a small sample. The second purpose of phase 1 trials is to find the best dose to test in a phase 2 clinical trial going forward; this is called dose-finding or dose-escalation. The idea is to give the drug (or vaccine) to a group of people until what are called “dose-limiting toxicities” are reached. In any event, it’s a huge misunderstanding to think that phase 1 trials, particularly of a vaccine, will demonstrate efficacy.
To be fair, Salzberg does seem to partially understand this. (I’ll get to that in a moment.) First, let’s see what his primary argument is:
So the current phase 3 plans for these vaccines work like this: identify a large number of people (30,000 in at least one of the trials) and give half of them the vaccine, and give the other half a placebo. Then wait for a few months and see how many people get Covid-19. If the vaccine is working, then we’ll see that significantly fewer people in the vaccinated group get sick.
Great. We should definitely do this, and we are.
But we’re in the midst of the worst pandemic since 1918. The careful, step-by-step vaccine approval regimen wasn’t designed for a global emergency, in which every day of delay means that thousands of people die.
At this point, it occurred to me that these are exactly the same sorts of arguments used by proponents of so-called “right-to-try” laws and that what Salzberg was proposing over the weekend was, in essence, “right-to-try” for vaccines. What do I mean? Regular readers know that I’ve written about the sham that is “right-to-try” for a long time. Basically, “right-to-try” laws purport to allow patients with terminal illnesses access to experimental therapeutics, as long as the drug or treatment had successfully passed at least one phase 1 trial. (I’ve discussed the fallacy of assuming that passing a phase 1 trial means that a drug is safe more times than I can remember.) Of course, the fallacies behind these laws are many. For one thing, there’s no guarantee that the drug will help, and, indeed, it’s more likely to harm than help in most cases. For another thing, right-to-try laws require the patient to pay. That’s not the case in Salzberg’s proposal, but, as you will see, everything else sounds a lot like right-to-try:
We already know that the vaccines in phase 3 trials are safe–otherwise it would be unethical to give the vaccine to 30,000 people, as these trials are doing. (Note: phase 3 trials sometimes uncover safety issues that affect only a small percentage of people–issues that might not appear in smaller phase 1/2 trials. So phase 3 also assesses safety, on a larger scale.) Weighing the risks versus the benefits, I think we should immediately ramp up production, using government funds rather than private money, and then offer these vaccines for free to anyone who wants them.
Even this argument betrays a misunderstanding of clinical trials, particularly clinical trials for vaccines. Here’s what I mean. The standards for vaccines are very different than those for drugs in terms of safety in that the standard for vaccines is much higher. It has to be, because, in contrast to drugs, which are administered to treat disease, vaccines are administered to huge numbers of people without disease in order to prevent disease. The tolerance for adverse reactions in a vaccine is therefore much lower than it is for drugs, particularly drugs for serious illnesses. What that means is that even a phase 3 clinical trial is only the beginning of safety monitoring. As I’ve described many times before, we have multiple postmarketing safety monitoring systems for vaccines, including one passive (Vaccine Adverse Events Reporting System, or VAERS) and three active surveillance systems, including the Vaccine Safety Datalink, the Post-licensure Rapid Immunization Safety Monitoring System (PRISM), and the Clinical Immunization Safety Assessment (CISA) Project.
Any vaccine developed to combat COVID-19 is likely to be administered to hundreds of millions, if not billions, of people, but lets just say that we administer an unproven vaccine to millions, as Salzberg proposed, while doing a phase 3 clinical trial with 30,000 or so subjects, and let’s imagine that we are doing this concurrently. Basically, this would involve administering millions of doses before we know even the frequency and severity of adverse events found in the phase 3 clinical trial. Does that make any sense? Any clinical trialist would tell you that it doesn’t. The same issue applies to efficacy. A plan like Salzberg’s would involve administering millions of doses of a vaccine to anyone who wanted it before we even knew for sure if the vaccine was effective in preventing COVID-19. Again, phase 1 trials will only find common adverse events. Even phase 2 trials, which involve a few hundred people would find only relatively common adverse events. Less common adverse events don’t show up until the phase 3 trials, and rare advere events might not be detected until postmarketing surveillance. (Accepting the possibility of rare adverse events after a vaccine passes phase 3 is a tradeoff that we accept because it is not practical to do randomized controlled trials of hundreds of thousands or millions of people.)
There’s another issue, though. Giving out millions of doses of a vaccine would render the completion of a concurrent phase 3 trial, for all practical intents and purposes, impossible. Why would anyone enroll and risk getting a placebo when they don’t have to? Why wouldn’t everyone conclude from a program like Salzberg’s that the vaccine must work? No matter how much you tell people that we don’t know if the vaccine works, actions speak far louder than words. If you’re giving away millions of doses of a vaccine to anyone who wants it, the message is that it works.
Here are some more excellent considerations, in particular the likelihood that a vaccine that’s made it through phase 1 and 2 trials will show sufficient efficacy and safety in a phase 3 trial to be approved and the issue of what happens if there are severe adverse reactions in the people not on the clinical trial who get the experimental vaccine:
As mentioned above, there’s also the likelihood of undermining trust in the system we have set up to administer vaccinations. The premature deployment of an unproven vaccine would have a not negligible probability of resulting in adverse reactions that could well seriously undermine trust in vaccines, with consequences that go far beyond just COVID-19.
There’s another wrinkle, as a vaccine researcher named Natalie Dean pointed out in a New York Times editorial:
It is possible that some Covid-19 vaccines may not prevent infection entirely, but they could still prepare a person’s immune system so that, if infected, they would experience milder symptoms, or even none at all. That’s similar to the flu vaccine: It’s not perfect, but we advise people to get it because it reduces intensive care admissions and deaths.
How many people need to be protected by a vaccine before it’s recommended for widespread use? Ideally, rates of disease will be 70 percent lower in vaccinated people than in unvaccinated people. The World Health Organization says a vaccine should be at minimum 50 percent effective, averaged across age groups. (We know from influenza that vaccines don’t always work as well on older adults whose immune systems have declined.)
This benchmark is crucial because a weak vaccine might be worse than no vaccine at all. We do not want people who are only slightly protected to behave as if they are invulnerable, which could exacerbate transmission. It is also costly to roll out a vaccine, diverting attention away from other efforts that we know work, like mask-wearing, and from testing better vaccines.
There are ways for vaccines to be approved without definitive efficacy data, based on animal or immune response data instead, but the bar is extremely high, and for good reason. A precondition is that efficacy trials are not possible, typically because the disease is so rare or sporadic that it would require hundreds of thousands of participants to be followed for many years to tell if the vaccine is effective (rabies, for example). That is not the situation here.
These examples are extreme exceptions, not the rule.
Basically, Salzberg, whose work I’ve long admired, allowed the need for a vaccine against COVID-19 to lead him to propose a solution that would be risky at best and actively harmful at worst. He essentially assumed the best case scenario, namely that the vaccines under testing will be effective and safe, based on phase 1 and 2 studies. Indeed, he explicitly said this while defending his post on Twitter:
As anyone who’s ever been involved in drug or vaccine development knows, that’s likely a bad bet, one you’re more likely to lose than win. Sure, we could do what Salzberg says and find out that we’ve guessed right. That would be fantastic—and fairly unlikely. More likely, such a plan would result in adverse reactions in the “right-to-try” group that result in the degradation of trust in vaccines, all while making it very difficult, if not impossible, to actually complete the phase 3 trial needed to properly assess efficacy and safety. It could even result in more, not less, deaths from COVID-19 and other infectious diseases.
Now here’s the part that warmed the cockles of my heart. Salzberg heard the criticism, and he changed his mind, admitting he was mistaken:
In my previous post, I suggested that while we’re pursuing Phase 3 testing of several promising Covid-19 vaccines, we could simultaneously offer those same, unapproved vaccines to a wider community of volunteers, as long as those volunteers were fully informed. The benefits of moving quickly, I argued, would outweigh the risks.
I was wrong. After reading many of the responses to my article, some of them outlining the risks in greater detail, I have concluded that (1) the risks are greater than I presented them, and (2) the benefits are not as great as I had thought.
There are two things we can learn from this incident. First, our bias is to assume that best and that speed in developing a vaccine will necessarily save more lives. Second, and perhaps more important, we learned the value of humility, of quickly admitting error when we make one. Doing so is very, very difficult, and most of us have a hard time doing it.
Also, desperate times call for desperate measures:
That’s right, desperate times call for science-based medicine. And the humility to admit when we are wrong and move on.