The mistake of criminalizing honest scientific mistakes

Given how many bloggers have already weighed in on the story of an Italian court convicting geologists of manslaughter for failing to issue adequate earthquake warnings before an earthquake that devastated the town of L’Aquila, including Steve Novella, Daniela at Skepchicks, Sharon Hill at Skeptic, and even Instapundit, you’d think that even Orac wouldn’t have anything to say about it. You would, of course, be wrong. Orac always has something to say about such things. The question is simply whether he decides he’s interested enough in the story to take the time and effort to compose and let loose one of his typical logorrheic streams of pontification about it. In this case he is, because, like the vast majority of the blogosphere, he considers the conviction of these scientists a horrific miscarriage of justice, particularly their prison sentences. He also can’t resist tweaking someone like Vox Popoli

But first, let’s go back and see what this whole story is about:

Six scientists and a government official were sentenced to six years in prison for manslaughter by an Italian court on Monday for failing to give adequate warning of an earthquake that killed more than 300 people in L’Aquila in 2009.

The seven, all members of a body called the National Commission for the Forecast and Prevention of Major Risks, were accused of negligence and malpractice in evaluating the danger and keeping the central city informed of the risks.

The case has drawn condemnation from international bodies including the American Geophysical Union, which said the risk of litigation may deter scientists from advising governments or even working in seismology and seismic risk assessments.

Ya think? I know that, in the wake of this decision, if I were a geologist in Italy, I would immediately dissociate myself from any government board charged with earthquake risk assessment. The risk would be just too great. Either that, or if I had no choice I would be ridiculously cautious in my assessments, complete with weasel words to cover my posterior in the event I’m wrong. In other words, I’d do like what most geologists in Italy will probably do now, rendering scientific advice on the issue of earthquakes much less useful than it is now. It would go from being precise, clear, and accurate to being a bowl of mush. No one wants to go to jail for six years and face financial ruin.

In the wake of this unbelievably misguided decision, the Italian scientists Franco Barberi, Enzo Boschi, Giulio Selvaggi, Gian Michele Calvi, Claudio Eva and Mauro Dolce as well as Bernardo De Bernardis—a senior official in the Civil Protection Authority—are currently free on appeal. However, if their appeals fail, they will face six years in prison and a $12 million fine. In other words, their lives will be completely ruined for being wrong. They stand convicted of criminal manslaughter and causing criminal injury, all because the court ruled that they had been overly reassuring to the to the population in their statements that a truly devastating earthquake was unlikely. Unfortunately, whether their statements were “too reassuring” or not, at 3:32 AM on April 6, 2009, a devastating earthquake measuring 6.3 on the Richter schale struck L’Aquila, resulting in 300 deaths and more than 1,000 injuries.

While I might not always share Steve Novella’s diligence in finding what he calls the “most charitable interpretation of each side of an argument,” I do feel obligated to dispense with a straw man version of the verdict that I’ve seen floating around the Internet. That is the claim that scientists were convicted for having failed to predict the earthquake. This is clearly not the case, as far as I can tell. The central question in the case was actually the question of what is the responsibility of scientists called upon to provide a risk assessment based on their scientific knowledge if they happen to be wrong. In other words, what sort of diligence is required, and how science-based and scientifically defensible do scientific recommendations. How badly do scientists have to fail at such a task before they can be considered negligent?

As Steve Novella pointed out, these sorts of issues come up in medicine all the time. He provided an excellent example of how medicine is all about probabilities. For example, if there is a diagnosis that is 95% likely to be the correct diagnosis, that means five percent of the time it won’t be and the physician will be wrong. Being wrong doesn’t in and of itself mean that the physician failed to exercise due professional diligence. It doesn’t necessarily mean that a physician committed malpractice. In actuality, many, if not most, diagnoses in medicine are much less certain than 95%, which means the physician will likely be wrong far more than 5% of the time. Of course, in the case of medical diagnoses, physicians will systematically move on down the differential diagnosis as it becomes clear that one diagnosis is not correct, which is a big difference when compared to the geologists who were convicted. They didn’t have a chance to recover from a “wrong” diagnosis and make the “right diagnosis” because once the earthquake happens that’s it.

And the science of earthquake prediction is much more uncertain than most medical diagnoses.

Another way the conviction of the Italian geologists is like medicine is that what they have been convicted of is, in essence, malpractice. Of course, there is one huge difference. In medicine malpractice is a civil offense, not a criminal offense. Physicians are not thrown in jail when a court finds that they have committed malpractice. They pay damages. If their practice repeatedly falls below the standard of care, they might end up before their state medical board, which could potentially strip them of their license to practice. Criminal charges against doctors related to their practice of medicine are much less common than civil penalties; for instance, the sexual abuse of patients.

Building on that analogy, I ask the question, “Did the Italian geologists commit malpractice”? The answer, as far as I’ve been able to tell, is clearly no. For instance, Alan I. Leisher of the AAAS writes to the President of Italy:

The charges against these scientists are both unfair and naive. The basis for those indictments appears to be that that scientists failed to alert the population of an impending earthquake. However, there is no way they could have done that credibly.

Years of research, much of it conducted by distinguished seismologists in your own country, have demonstrated that there is no accepted scientific method for earthquake prediction that can be reliably used to warn citizens of an impending disaster. To expect more of science at this is unreasonable. It is manifestly unfair for scientists to be criminally charged for failing to act on information that the international scientific community would consider inadequate as a basis for issuing a warning.

But is this true? Like Steve, I’ve looked around for accounts of exactly what government officials told the people of L’Aquila. It is said that the Commission provided “incomplete, imprecise and contradictory” information on the danger of an impending earthquake after a meeting on March 31, 2009, a few days before the L’Aquila quake:

Central Italy is continuously shaken by low level tremors, very few of which precede bigger earthquakes and they are generally marked by no more than a brief statement from civil protection authorities.

Key to the dispute is the kind of cautious language, hedged by caveats and reserves which scientists typically use in predicting highly uncertain events, but which can be of limited use as a guideline for the general public.

According to scientific opinion cited by prosecutors, the dozens of lower level tremors seen before the quake were typical of the kind of preliminary seismic activity seen before major earthquakes such as the one that struck on April 6.

Instead of highlighting the danger, they said the experts had made statements playing down the threat of a repeat of the earthquakes which wrecked the town in 1349, 1461 and 1703, saying the smaller shocks were a “normal geological phenomenon”.

In a report in Nature from a couple of years ago, after it was decided that the scientists would have to stand trial, a retrospective analysis of seismic activity in Italy was cited that found that a medium-sized shock in a swarm forecasts a major event within several days only 2% of the time. As pointed out in this analysis, “earthquake swarms” are not uncommon in the L’Aquila region and tend to fade away over time without a major seismic event. Assuming that the L’Aquila region is similar to the regions studied in this analysis, it would not be unreasonable to postulate a risk of similar magnitude—with fairly wide error bars, of course. Let’s make the error bars really big, just to make the case as favorable to the prosecution as possible. So on the one hand, we have science demonstrating that the small shocks that had been occurring before the L’Aquila quake are common and only precede much larger earthquakes maybe 2% of the time, perhaps several percent of the time in the most alarmist assessments. What would you do with this information?

In medicine (again) we have a saying that the retrospectoscope is 100% accurate, but put yourself into the position of the scientists. Based on the earthquake swarms occurring at the time, there was maybe a 2% risk of a larger earthquake. What do you tell the populace? Vincenzo Vittorini, a plaintiff in the case who lost his wife and daughter on that fateful night in April 2009, claims:

“This isn’t a trial against science,” insists Vittorini, who is a civil party to the suit. But he says that a persistent message from authorities of “Be calm, don’t worry”, and a lack of specific advice, deprived him and others of an opportunity to make an informed decision about what to do on the night of the earthquake. “That’s why I feel betrayed by science,” he says. “Either they didn’t know certain things, which is a problem, or they didn’t know how to communicate what they did know, which is also a problem.”

So what would he have done if the Commission had said that, to the best of their ability to estimate, there was a roughly 2% chance of a major earthquake within a few days. Would he have done what he said his parents had had him do in the past and leave the apartment and hang out in the public square overnight? How many days would he have done that before he felt that it was safe to return to his home? These are not easy questions? On the one hand, any warning issued in such a situation would have an approximately 98% chance of being wrong. Being too quick to issue such warnings would have the potential to cause a lot of disruption, piss people off, and, over time, degrade the people’s trust in science and the authorities. Yet failure to issue a warning in such a situation has roughly a 2% chance of being wrong, with potentially tragic results. No matter what the scientific panel decided to do in each case where these issues came up, sooner or later it would get it wrong. Analogies to warnings of terrorist attacks in the wake of 9/11 are inescapable.

Apparently the situation was also complicated by a man named Giampaolo Giuliani, who was making unofficial predictions of an impending large earthquake based on his measurements of radon, whicha according to him fluctuate wildly right before a major earthquake. Theere’s just one problem. His method of earthquake prediction has never been validated as accurate as a short-term predictor of earthquakes, and Giuliani has never published a peer-reviewed paper describing his method. Instead, he maintained a website and issued warnings to the point where national civil-protection officials cited him for procurato allarme; in essence, instigating public alarm or panic. He was causing a lot of unease among the people of L’Aquila, and there’s little doubt that his warnings, which like the proverbial blind squirrel sometimes finding a nut, happened in this case to be right by coincidence, could in retrospect have lead to conspiracy theories and questions about whether the government really knew but didn’t say anything. Then, at the meeting of the Commission on March 31, 2009, the commission was asked specifically if the current seismic swarm could be a precursor to a huge quake like the one that leveled L’Aquila in 1703. According to the meeting minutes, Bosci said, “It is unlikely that an earthquake like the one in 1703 could occur in the short term, but the possibility cannot be totally excluded.” This is, of course, a reasonable statement of the risk, but unfortunately after the meeting, something happened. Accounts differ, but according to Nature:

What happened outside the meeting room may haunt the scientists, and perhaps the world of risk assessment, for many years. Two members of the commission, Barberi and De Bernardinis, along with mayor Cialente and an official from Abruzzo’s civil-protection department, held a press conference to discuss the findings of the meeting. In press interviews before and after the meeting that were broadcast on Italian television, immortalized on YouTube and form detailed parts of the prosecution case, De Bernardinis said that the seismic situation in L’Aquila was “certainly normal” and posed “no danger”, adding that “the scientific community continues to assure me that, to the contrary, it’s a favourable situation because of the continuous discharge of energy”. When prompted by a journalist who said, “So we should have a nice glass of wine,” De Bernardinis replied “Absolutely”, and urged locals to have a glass of Montepulciano.

However, there are problems. There is no mention of any appeal to “continuous discharge of energy” in the minutes. And:

In an interview in the Rome offices of his lawyer, Boschi derided as “absurd” the idea that he in any way played down the risk to L’Aquila. Brandishing a copy of the INGV’s seismic hazard map of Italy, which shows a broad swath of the Apennines in bright hues indicating high risk, the tall, silver-haired geophysicist insisted: “No one can find a single piece of paper where I say, ‘Be calm, don’t worry’. I have said for years that the Abruzzo is the most seismologically dangerous zone in all of Italy. It’s as if I suddenly became an imbecile. I’m accused of being negligent!” He was not invited to participate in the press conference after the meeting, he says, and didn’t even know about it until after his return to Rome.

To me, this is a perfect storm. There was a crank stirring up panic, and the best scientific information available suggested that the risk of a major quake was low. What then followed was a classic conflict between the preference of scientists for nuanced, careful predictions, complete with statements of uncertainty and what we do and don’t know and a public desire for clarity and certainty. We see this conflict pop up all the time; one particular example is communicating the science of anthropogenic global warming (AGW), assessments of whether this chemical or that contributes to cancer, or whether there are any dangers from vaccines. For example, in the case of asking whether vaccines cause autism, scientists often say things such as, “As far as we can tell from multiple large studies, there is no detectable correlation between vaccines and autism.” What the public wants to hear is that vaccines do not cause autism, but it interprets a statement like this to imply that there is still significant uncertainty about the answer to the question. That’s just one example.

If there’s one thing we as scientists do a poor job at sometimes, it’s communicating findings to the public without seeming too wishy-washy. In the case of the Commission meeting regarding L’Aquila, scientists might well have gone too far in trying to calm fears, but their position was scientifically defensible. On a statistical basis, the risk was low. And, to draw another analogy to medicine, when I’m quoting risks to patients I often say something like, “The risk of complication X is 10%, but if it happens to you it’s 100% for you.” Although somewhat nonsensical from a mathematical and statistical standpoint, I find that this really does get across to the patient that, even if the risk of a complication is low, I understand the severity if it happens to her. Unfortunately in the case of L’Aquila, the 2% risk of a major earthquake ended up being 100% in retrospect because it happened.

As horribly misguided and, yes, profoundly antiscientific this verdict is, there remains the question of what scientists should do. One thing that is not an option is something that our old “friend” Vox Day says: Let Science be silent when it cannot predict future events. Vox, unsurprisingly, is being his usual brain dead, antiscientific self. His post demonstrates, again as usual, a profound ignorance of what science is and does. Science can predict some events with incredible accuracy. If it couldn’t, it would not be possible to fly airplanes, land on the moon, send probes to Mars, or predict the decay of radioisotopes. Other things can’t be predicted as precisely because such predictions are based on stochastic models that have a lot of random noise and factors that we don’t yet understand. Some things, like earthquakes, can’t be predicted very well at all because of high variability and our lack of understanding of key mechanisms by which they occur.

What Vox doesn’t seem to understand is that when public policy intersects with science, “remaining silent” is not always an option. Science is messy. Scientists understand that and in their communications to each other communicate that uncertainty. The public often does not; the best that we as scientists can do is to make our recommendations as clear as possible but with an honest assessment of the uncertainty inherent in the prediction. Ironically, the Italian verdict is very likely to inhibit that process—and inhibit it in a big way—in Italy. Scientists will likely either refuse to serve on the Commission or will become hypercautious, as I suggested above. The end result will be that throwing a few scientists in jail for having been wrong about what sort of warning should be issued is far more likely to harm the very process the prosecutors claim to be championing: More accurate earthquake predictions.

In the meantime, the Italian courts richly deserve the condemnation and ridicule directed at them. Hopefully the protests will have an effect, and the conviction of these scientists will be overturned on appeal.