In recent years, there has been a lot of interest in improving surgical outcomes. One strain of research tends to examine the “volume-outcome” relationship, which in essence asked the question if the volume of cases that a surgeon or hospital does has a relationship outcome. In other words, are mortality rates lower, survival rates better, or the correction of symptoms more reliable for a given surgical procedure in the hands of surgeons who do more of them per year or hospitals in which more of them per year are done? On the surface, it would seem self-evident that the answer must be yes, but the situation is not as simple as you may think, and the volume-outcome relationship doesn’t hold up for all procedures. Basically, for certain complex procedures, there is a correlation between volume and better outcomes, but it is not always consistent and often has at least as much to do with the volume of the hospital as the surgeon.
One consequence of this emphasis on outcomes is that more and more, databases are being developed and maintained that track mortality and complications by hospital, often mandated by state and local governments. Increasingly, these results are being reported to the public, sometimes even showing up in newspapers as tables of mortality rates for various procedures. If a hospital shows up on these lists with a mortality rate significantly higher than that of surrounding hospitals, the consequences can be severe, ranging from the loss of patients to even investigations. But if you’re a patient who needs a certain procedure, how useful are these measures? In other words, how predictive of your outcome is a hospital’s mortality rate? Intuitively, it would seem that hospitals that report zero mortality for a procedure in a given year would be a good bet, but is that correct?
Justin B. Dimick and H. Gilbert Welch, in an article hot off the presses in the most recent Journal of the American College of Surgeons, examined just that question, and the answer is not what you would predict, so much so that their article is entitled The Zero Mortality Paradox in Surgery.
This article is short and sweet. It asks and tries to answer a very simple question, which is whether hospitals that have had zero mortality from several index procedures over the preceding three year period have a lower than average mortality in the subsequent year for those same procedures. The rationale was as follows:
Policymakers and health care payers believe public disclosure of provider performance will help patients choose the best hospitals. Beginning in 1990, New York State began publicly reporting risk-adjusted mortality rates for cardiac surgery at every hospital. Pennsylvania followed shortly thereafter, and now several states track and publicly report cardiac surgery mortality. More recently, the Agency for Healthcare and Research Quality proposed the use of mortality rates as quality indicators for a broader group of six other noncardiac operations. The underlying idea is appealing; by choosing a low mortality hospital, patients can improve their chances of surviving their operation.
The problem with this approach, however, is as the authors point out, that in hospitals that have a low volume of a given procedure, it is quite possible that, by the vagaries of chance alone, a hospital that may not be so great could have zero mortality. Unless an operation has a very high mortality in all hands, it is quite possible for a low volume hospital to have no mortalities for a considerable period of time for reasons that have little or nothing to do with the true rate of mortality. Most “high mortality” operations, however, have mortalities under 10%.
The study design was quite straightforward. The investigators obtained the Medicare records necessary to identify all hospitals with zero mortality for several procedures from 1997 to 1999. The procedures examined included coronary artery bypass grafting, elective abdominal aortic aneurysm repair, carotid endarterectomy, colon cancer resection, pulmonary lobectomy, and pancreatic resection. The hospitals had to have preformed at least one during the time periods under study. They then examined the mortality rates in these same hospitals for these same procedures for the year 2000. These mortality rates had to be adjusted for risk factors using a multivariable logistic regression model for various factors, including patient demographics (age, gender, and race), urgency of admission (elective, urgent, emergent), socioeconomic status, comorbid diseases, and socioeconomic status. The results were also just as straightforward. In the year after a three year stretch of no mortality for these procedures, no mortality hospitals had mortality rates that were no different from any other hospital in the following year. Indeed, for pancreatic resection, the results were worse in the subsequent year.
Truly, past performance is no guarantee of future results.
There are several possible explanations for these results highlighted in the discussion. One is that the zero mortality hospitals are truly better, such that their average mortality over the entire period is still better than average. This may be true for coronary artery bypass, for which there was a trend towards lower mortality in the year after the zero mortality period. Another possible explanation was that hospitals with zero mortality were actually average but did lower case volumes for given operations studied, with random chance alone making it more likely that they would go longer periods of time without a mortality. This possibility is supported by the second result of this study, which is that the zero mortality hospitals did far fewer of each procedure than the other hospitals. Indeed, for one operation (colon cancer resection), the zero mortality hospitals did only one quarter as many as the other hospitals. The final possibility is that zero mortality hospitals actually have worse performance, which may actually have been the case for pancreatic resections, for which mortality was higher in the period following the zero mortality period. Finally, it is possible that performance at the zero mortality hospital deteriorated during the time of observation, although such deterioration would be unlikely to happen in enough of the zero mortality hospitals at the same time to account for this result.
Overall, this was a provocative study, but my sense is that it’s probably close to correct. Its strengths include consistency across all hospitals over four years and and a known database. However, the Medicare database does, as has been pointed out, include a subset of mainly older patients. Unfortunately, an all-payer database does not always cover the same hospitals from year to year. Another problem is that mortality rates are at best a very rough measure of quality when numbers are low. However, mortality is an unequivocal result that is easy to measure that is recorded in most databases, which is why it’s often used.. All of these issues confound the interpretation of the study’s results, but not enough to invalidate them. The authors conclude:
Patients and families may reasonably be attracted to a hospital with zero deaths. A reported finding of zero events has a qualitative impact exceeding its quantitative meaning, as others have noted; people tend to focus on numerators and ignore the size of the denominator. Although the problems with small samples are widely known among the statisticians and epidemiologists, they may not be readily apparent to patients. We concluded that patients considering where to have surgery should not choose hospitals just because they have reported mortality rates of zero. Otherwise, they may miss out on the potential benefits of going to a hospital with better performance.
This is exactly right. If you don’t know the denominator, just seeing a report of 0% mortality for a procedure is almost meaningless. There’s a huge difference between 0% mortality for one procedure done and 0% mortality for two hundred procedures performed (and, I might add, a 0% mortality for 1,000 procedures performed). There’s also lot more to determining a patient’s risk than just looking at the mortality rates. Choosing a hospital based on a spuriously low mortality rate may actually be riskier than choosing a hospital with a higher mortality rate but that does a much higher number of a given operation per year.
