Surgical checklists as a strategy to reduce morbidity and mortality

About a year ago, I discussed an article by Dr. Atul Gawande describing a quality improvement initiative that appeared to have been stalled by the Office for Human Research Protections and its apparent tendency to apply human subjects research protection rules to initiatives that are not exactly research using human subjects. The problem appeared to be an excessively legalistic and a “CYA” attitude more than a genuine concern for protecting human subjects. At the time, I was more concerned with the ethical and policy implications of the story rather than the actual research itself. After all, what was being examined was not something new or experimental. It was nothing more than a checklist designed to remind doctors placing central venous catheters to use rigorous sterile technique, and, not surprisingly, its use was associated with a dramatic decrease in catheter infections. Given that the checklist contained nothing more than tried-and-true infection control measures, it was not surprising that catheter infections dropped. What was somewhat surprising is how much they dropped.

One question that arises from these results is: Do these lists have a more general utility? In last week’s issue of the New England Journal of Medicine, a study published by Gawande and numerous collaborators takes the use of checklists to a whole other level.

The study, A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population, represents the fruit of a multinational collaboration that tested a very simple intervention, namely a surgical checklist designed to verify a number of factors, such as patient identity, surgical site, and others. Some elements of this sort of checklist have been in place in most U.S. hospitals, elements such as a “time out” to verify patient identity, operation being performed, and correct surgical site before the patient is placed under anaesthesia, but the list examined in this study goes much further than most such lists in the U.S., or anywhere else, for that matter.

The checklist was tested at eight hospitals scattered throughout the world, both in developed countries and Third World countries. Sites included London, Seattle, Toronto, Tanzania, India, Jordan, the Philippines, and New Zealand. These hospitals had varying safety measures in place before the study, and none of them were as extensive as the checklist introduced as part of the study, as described here:

The intervention involved a two-step checklist-implementation program. After collecting baseline data, each local investigator was given information about areas of identified deficiencies and was then asked to implement the 19-item WHO safe-surgery checklist (Table 1) to improve practices within the institution. The checklist consists of an oral confirmation by surgical teams of the completion of the basic steps for ensuring safe delivery of anesthesia, prophylaxis against infection, effective teamwork, and other essential practices in surgery. It is used at three critical junctures in care: before anesthesia is administered, immediately before incision, and before the patient is taken out of the operating room. The checklist was translated into local language when appropriate and was adjusted to fit into the flow of care at each institution. The local study team introduced the checklist to operating-room staff, using lectures, written materials, or direct guidance. The primary investigators also participated in the training by distributing a recorded video to the study sites, participating in a teleconference with each local study team, and making a visit to each site. The checklist was introduced to the study rooms over a period of 1 week to 1 month. Data collection resumed during the first week of checklist use.

The checklist itself involves some fairly basic things. It’s divided into three sections. The “sign-in” section has typical elements, such as verifying patient identity; making sure that surgical site is properly marked with a marking pen; making everyone aware of any patient allergies; checking the pulse oximeter; and determining if there is a risk of aspiration or significant blood loss. The preop “time out” involves such things as once again verifying the patient identity and surgical site; making sure that any preop antibiotics have been given less than 60 minutes prior to skin incision; confirming that all appropriate imaging results are available, correct, and show what they are reported to have shown; and reviewing anticipated critical events. Finally, the “sign out” phase involves the typical surgical necessity of making sure that the sponge and instrument counts are correct before waking the patient up; checking any specimens to make sure they are labeled correctly; and reviewing aloud concerns about any issues that might interfere with the recovery of the patient.

Pretty simple, isn’t it? Moreover, none of this is anything that is not only common sense (verification of surgical site) or reiteration of science-based surgery (preoperative antibiotics that are given after the skin incision is made are completely useless and if they are given more than 60 minutes prior to surgery they are likely to be less effective). The only difference is that these practices were systematized, and the health care teams were required to verify that they were being done appropriately. Perhaps that’s why the results were so striking and unexpected. No, it wasn’t unexpected that the introduction of this checklist would probably be associated with a decrease in complications and possibly even surgical mortality. That wasn’t a huge surprise. What was a huge surprise was how much this checklist appeared to decrease morbidity and mortality. Mortality decreased by one half (1.5% to 0.8%) and complications by a third (11% to 7%).

The results were so striking that the principal investigator of the catheter infection study that I mentioned at the beginning of this post was quoted thusly:

The results were so dramatic that Dr. Peter Pronovost, a Johns Hopkins University doctor who proved in a highly influential study a few years ago that checklists could cut infection rates from intravenous tubes, said he was skeptical of the findings.

One possible flaw, he said, is that “you had people who bought into the system collecting their own data.”

I find this explanation for the results to be pretty unlikely, given the description of the methodology in the paper, but I can’t rule it out. I also find it strange that Dr. Pronovost would say such a thing, given that the results from his own study were even more dramatic than this one, with catheter sepsis decreased by two thirds and overall catheter infections decreased from 4% to zero. One aspect of these results is that the results at “low income” sites were more striking when compared with the results at “high income” sites in the developed world, with no significant decrease in the death rate in the wealthiest countries due to this list. This, too, is not that surprising, as most of the wealthiest countries already have systems in place to assure that measures designed to decrease or eliminate wrong site surgery, wound infections, and other complications, at least more so than hospitals in more impoverished nations. However, even in the wealthiest countries, most existing checklists are not as comprehensive.

So was it the introduction of checklists that resulted in this improvement? That’s difficult to say definitively from just this study. Certainly it is possible, but exactly how is harder to ascertain. One possibility is that the introduction of such checklists leads to a systemic improvement in how care is delivered driven by adherence to the principles embodied in the checklist. As the authors speculate in the discussion:

Whereas the evidence of improvement in surgical outcomes is substantial and robust, the exact mechanism of improvement is less clear and most likely multifactorial. Use of the checklist involved both changes in systems and changes in the behavior of individual surgical teams. To implement the checklist, all sites had to introduce a formal pause in care during surgery for preoperative team introductions and briefings and postoperative debriefings, team practices that have previously been shown to be associated with improved safety processes and attitudes and with a rate of complications and death reduced by as much as 80%. The philosophy of ensuring the correct identity of the patient and site through preoperative site marking, oral confirmation in the operating room, and other measures proved to be new to most of the study hospitals.

This study also had some significant drawbacks. For example, it was a “before and after” study with the same staff, which introduces some potential for bias in that there was no randomization. The authors themselves concede:

This study has several limitations. The design, involving a comparison of preintervention data with postintervention data and the consecutive recruitment of the two groups of patients from the same operating rooms at the same hospitals, was chosen because it was not possible to randomly assign the use of the checklist to specific operating rooms without significant cross-contamination. One danger of this design is confounding by secular trends. We therefore confined the duration of the study to less than 1 year, since a change in outcomes of the observed magnitude is unlikely to occur in such a short period as a result of secular trends alone. In addition, an evaluation of the American College of Surgeons’ National Surgical Quality Improvement Program cohort in the United States during 2007 did not reveal a substantial change in the rate of death and complications (Ashley S. personal communication, http://acsnsqip.org). We also found no change in our study groups with regard to the rates of urgent cases, outpatient surgery, or use of general anesthetic, and we found that changes in the case mix had no effect on the significance of the outcomes. Other temporal effects, such as seasonal variation and the timing of surgical training periods, were mitigated, since the study sites are geographically mixed and have different cycles of surgical training. Therefore, it is unlikely that a temporal trend was responsible for the difference we observed between the two groups in this study.

They also concede that there may be a bit of the “Hawthorne effect” contributing to the improvement, namely the phenomenon that people tend to perform differently, usually better, when they know they are being watched. On the other hand, who cares if the results are due to the Hawthorne effect if there was a real improvement? In that case, keep watching, I say.

One interesting aspect of this initiative (interesting to me, at least) is the level of hostility to the results of the catheter study and this study that I have encountered. Doctors tend to resist checklists. I don’t know if this is anything unique to physicians, but given that airline pilots have used preflight checklists for years I’ve always had a hard time understanding the visceral reaction that such lists seem to provoke in physicians. It’s as though too many of us consider ourselves so superior to our fellow human beings that the utility of such lists do not apply to us, we never forget to do anything, and never need reminding to do what we should be doing in the first place. To some extent, I can understand in that I used to make fun of what I used to disparage as the “mindless ritual” of my having to mark which breast I was going to operate on and to do a “time out” to verify it with the entire surgical staff in my operating room. I even used to say that once something is made into a rigid policy or checklist, it’s an excuse to stop thinking and mindlessly go through the motions. To some extent, I still think it’s important to guard against that tendency, but I’m far more supportive of checklists than I used to be, to the point that it disappoints me to see a fellow surgeon (from the town where I did my surgery residency, yet!) dismiss these results so blithely:

Certainly, there is something to be said for meticulous routines when it comes to surgery or other procedures. But do we need mandatory 19 item checklists? Why stop there? Why not make it a 40 item checklist? Why not make the attending surgeon write an essay on how to avoid complications before every case? Or how about having the surgeon and all assistants read the chapter corresponding to the proposed operation from the textbook out loud together (alternating paragraphs) prior to making the incision?

It’s good to be organized and precise in surgery. Limited checklists are useful in this regard. We ought to mark our initials on the correct side of the hernia repair. Point taken. Nothing groundbreaking here. We don’t want to be operating on the wrong leg or leaving sponges inside bellies. But it’s rather a ridiculous leap to think that death rates can be halved just by following a series of irritating instructions on a laminated list.

I was disturbed to see in this reaction the arrogance that I used to have when it came to these lists, with a huge dash of the logical fallacy known as the slippery slope argument taken to the extreme of ridicule. Certainly, such lists can be irritating. Certainly, I myself have been irritated by them on occasion, even to the point of making fun of them. Certainly, these lists can be made too long and onerous. However, as harsh as this may sound, the objection above is simply nothing more than the logical fallacy of argument from incredulity, the same argument frequently used by creationists to dismiss evolution because they cannot conceive how evolution might have formed an eye. To dismiss these results simply because one can’t imagine how they might be true or valid does not demonstrate that they are not true or valid; it only demonstrates a lack of imagination and a bit of stubbornness. If this study were the first one that showed that surgical checklists and various means of verifying operative site can significantly reduce complications and wrong site surgery mistakes, then that would be one thing. But it’s not. There’s lots of other evidence in the medical and surgical literature (and cited by Gawande’s paper) that various checklists do contribute to decreased complication rates in various settings. It is not all that surprising that such a checklist would do so for surgery.

The problem, I think, is that surgeons like Buckeye Surgeon have a distressing tendency to view such studies as assaults on them as surgeons, as accusations that they are not good surgeons if they do not follow such lists, the implication being that they are being told they need reminders to do what they know they should do. That is the wrong attitude to take. No one is accusing surgeons like Buckeye of being negligent or insufficiently conscientious. However, there is a significant body of literature that indicates that doctors do not do what they think they do a significant proportion of the time. For example, in studies of hand-washing, physicians often self-report that they always or almost always wash their hands between patients. Objective observations by investigators often show otherwise. Human psychology fills in the gaps. There is no reason to think that the same thing is not also true of surgeons. Indeed, even at the Seattle site, prior to the introduction of the checklist some of the basics on the list were performed considerably less frequently than one would think they should be in such an advanced hospital. In addition, there is a culture inculcated in surgeons that we are always the captains of the ship. Consequently, the collaborative nature of such checklist systems, which require surgeons to submit themselves to nurses and other nonphysicians who verify compliance with the list, often grates, particularly when combined with surgeons’ general dislike of bureaucracy:

Surgical safety is always paramount when I do an operation. But to use the results of this study as definitive proof that by simply implementing Dr Gawande’s 19 point checklist will save thousands of lives is misguided. This was a non-randomized, non-blinded study. It’s not hard science. Long, indepth checklists are only going to complicate health care. Will we need different checklists depending on the operation? Will there be separate checklists for doctors vs nurses vs anesthesia staff? Who will be in charge of determining each checklist? A subcommittee of the AMA? A national bureaucracy>

Common sense and moderation, as usual, ought to be our guiding principles. The article is useful in the sense that it highlights the potential benefits of a checklist; but let’s not fall over ourselves thinking that we’ve found some sort of panacea….

These are not arguments based on a sound understanding of science. Indeed, the remark that, because this study was not randomized or blinded, it is not “hard” science is really off the mark and epitomizes what irritates me the most about how many doctors have come to view evidence-based medicine. That view seems to be that, if it is not a randomized double blind study, it is meaningless or bad science. That is far too narrow a view of evidence-based medicine. After all, some interventions, for example many surgical interventions, often can’t be studied using blinded studies, and systems studies often can’t be blinded or randomized. That doesn’t mean such issues can’t be studied rigorously; it simply means that different methods are needed to study them and that it’s more difficult to control for confounding features.

Checklists are not a panacea. Indeed, they even have the potential to do harm if they are too long, too onerous, or filled with items that have nothing to do with patient safety and reducing complications. However, the slippery slope argument against them is unconvincing. There is no inevitability to their growing to the ridiculous proportions used to mock them. Indeed, research is what will allow investigators to figure out what items do and don’t belong on such lists. Nor is the argument that we don’t know exactly how these lists contribute to improved outcomes, in essence an argument from ignorance, particularly compelling either. It’s quite possible that such lists, when properly designed and using science-based principles, contribute to systematizing best practices that hospitals and surgeons should be doing anyway. If so, why should that be an argument against them? Finally, the argument from consequences opponents of these lists use, namely that they will somehow erode the magic professionalism of surgeons and reduce them to just technicians following a list, remains unconvincing as well. Are airline pilots considered any less “professional” than surgeons? They use checklists all the time. What about clinical care pathway protocols, many of which are more detailed than this surgical checklist? Such protocols have been in place at most hospitals for many years.

There may well be rational, science-based reasons to oppose the creeping influx of checklists and protocols. It may also be that there are right ways and wrong ways to introduce such checklists and that wrong ways could even do harm. However, none of these are reasons not to introduce into more hospitals lists that have been validated in studies as being useful and let the data show whether these lists work when used more widely. Unfortunately, a lot of the resistance I sense from my colleagues seems to derive more from an emotional reaction to what they perceive as being told what to do or as an erosion of their autonomy than from science, risk/benefit ratios, or economic concerns about the expense of introducing such lists.

REFERENCE:

A. B. Haynes, T. G. Weiser, W. R. Berry, S. R. Lipsitz, A.-H. S. Breizat, E. P. Dellinger, T. Herbosa, S. Joseph, P. L. Kibatala, M. C. M. Lapitan, A. F. Merry, K. Moorthy, R. K. Reznick, B. Taylor, A. A. Gawande (2009). A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population New England Journal of Medicine DOI: 10.1056/NEJMsa0810119