Do negative clinical trials change practice?

One of the central themes of this blog from the very beginning is that all medicine, regardless of where it comes from or how it was developed, should be held to a single science-based standard with regards to efficacy, effectiveness, and safety. I tend to focus primarily on “complementary and alternative medicine” (CAM), now more commonly known as “integrative medicine,” (1) because I believe it to be undermining the scientific basis of medicine and allowing outright quackery (or, as I like to refer to it, quackademic medicine) to infiltrate medical academia, which is fast becoming medical quackademia and (2) because when it comes to the other threats to the scientific basis of medicine as it is practiced today, such as pharma influence and various medical dogmatism, there are a whole host of active critics better equipped and more energetic than I am who can do the job (usually) better. That is why, whenever I hear advocates of CAM/”integrative medicine” attack us for not spending enough time on various corruptions of clinical trial processes or the perfidy of big pharma, I tend to gently tell them in my characteristically diplomatic manner that that’s what I like to call the “Why don’t you blog about what I think is important and interesting instead of what you think is important and interesting?” criticism, then I refer them to my posts on John Ioannidis, overdiagnosis and overtreatment, the shortcomings of mammography, or any number of other posts I’ve done through the years pointing out where current medical practice falls short.

In particular, though, I like to point out a post I did on vertebroplasty as a treatment for vertebral compression fractures (VCFs) due to osteoporosis. Basically, I discussed then recent evidence showing how vertebroplasty for such fractures is, basically, placebo medicine, no better than acupuncture. Indeed, I even called it “modern acupuncture,” in which small, pilot studies appear to be positive, but then the follow-up rigorous randomized clinical trials fail to find a benefit greater than that of placebo. It turns out that a rather telling study regarding vertebroplasty was published earlier this year that I somehow missed that addresses a problem we have in “conventional” medicine.

I became aware of this new study when I was perusing the latest issue of the Journal of the American College of Surgeons (JACS), hot off the presses. (OK, in reality it wasn’t the latest issue. It was the September issue; I’m more than a month behind in my perusing of the medical literature.) This study, however, had been published online back in April. It must not have gotten much news coverage then, but it’s an important study, because it is one of the few studies out there that look explicitly at the effect of randomized clinical trials with negative results. The study is by Fabrice Smieliauskas, PhD, Sandi Lam, MD, MBA, and David H. Howard, PhD, and is entitled “Impact of Negative Clinical Trial Results for Vertebroplasty on Vertebral Augmentation Procedure Rates.” Smieliauskas et al. set the stage by describing what vertebroplasty is. Basically, it’s a procedure that’s done by either spine surgeons or radiologists in which bone cement (polymethelmethacrylate, or PMMA) is injected into the fracture site under radiologic guidance. The intent is to stabilize the fracture and relieve the pain. Kyphoplasty is a variant of vertebroplasty in which a device, such as a balloon, is inflated or expanded in the vertebral body. In theory, this restores vertebral body height and creates a cavity for the PMMA to enter. These procedures are considered similar enough that they are often referred to as “vertebral augmentation procedures” (VAPS).

Now here’s the real issue:

In August 2009, the New England Journal of Medicine published 2 randomized controlled trials testing vertebroplasty vs sham surgery for patients with osteoporotic VCFs. Both found that patients randomized to vertebroplasty did not experience decreases in pain or disability relative to patients in the placebo arm. In these studies, both groups showed improvement over time, suggesting a natural history for reduced pain and/or placebo response. The trials were covered in the lay press in newspapers, internet articles, and television news reports. In September 2010, the American Academy of Orthopaedic Surgeons (AAOS) issued a guideline that advised against the use of vertebroplasty for patients with osteoporotic VCFs. The guideline listed kyphoplasty as a treatment option but noted that evidence to support its use is “limited.” Many payers have restricted coverage of VAPs to patients whose symptoms have failed to resolve after a course of optimal medical therapy, but these types of coverage restrictions can be difficult to enforce in practice.

These are the studies that I blogged about five years ago, as well as four years prior to that, when I used the whole question of VAPS as an example of a scientific delusion in reference to earlier studies of VAPS and the anecdotal evidence used to justify it, which I characterized as testimonials not unlike alternative medicine testimonials. Of course, given that randomized clinical trials (RCTs) are the “gold standard,” what’s of most interest is whether practice changes after major negative RCTs, like the two negative RCTs (linked to here and here). I discussed them in detail at the time. Suffice to say, taken together, they were pretty definitive evidence that VAPS for osteoporotic vertebral compression fractures do not work. Besides having been published in a journal with as high a visibility as the NEJM, these two RCTs got a lot of publicity in the lay press, including a major article in the New York Times. The question that remained to be answered is whether physicians changed their practice in response to two resoundingly negative clinical trials that capped off a series of smaller studies questioning the value of VAPS.

The answer is: Yes and no, at least in Florida. Also, as they say in Facebook status updates, it’s complicated, as you will see.

So what did Smieliauskas et al. do? They examined quarterly trends in VAPS use using the Florida hospital inpatient discharge and ambulatory surgery databases from 2005 to 2012, which capture a 100% sample of surgical spine procedures performed in the state in hospitals, hospital surgery centers, and freestanding surgery centers. They chose Florida because it is large and sociodemographically diverse. Discharge data was supplemented with data on physician specialty from the Florida Practitioner Profile, and population counts were obtained from the 2000 and 2010 Censuses and US population counts from the 2012 Census Population Estimates. This allowed the investigators to adjust for population growth during that period. So, yes, one weakness of this study is that it only covers Florida, and, as already mentioned, there is a huge regional variation in the use of VAPS throughout the US. The investigators also cautioned that it’s possible that some of the decline in some procedures was due to the recession due to the financial meltdown that occurred in 2008, but if that’s true, then these results would look even worse.

First up, we see that the average age of patients was 76 years, with Medicare being the primary payer in 85% of cases. Procedures were done by interventional radiologists (27.9%); orthopedic surgeons (32.2%); neurosurgeons (22.3%); other specialties (7.1%) and “unidentified” (10.6%). Interventional radiologists performed 57% of vertebroplasties and 18% of kyphoplasties, while orthopedic surgeons performed 13% of vertebroplasties and 38% of kyphoplasties. A little more than half of the vertebroplasties were performed on an outpatient basis, while most kyphoplasties were performed in an inpatient setting.

Here’s the money figure. The TRIALS line indicates quarter before publication of vertebroplasty trials in the NEJM. The AAOS line indicates quarter before release of the American Academy of Orthopaedic Surgeons statement on vertebral augmentation procedures:

Looking at the graph, it looks as though VAPS rates did decline after the studies. Indeed, as the authors reported:

The interrupted time series estimates of the effect of the trials generally confirm the observations in the figures (Table 2). Overall, the vertebroplasty rate declined by 51.5% (column 1) and the kyphoplasty rate declined by 40.0% (column 2). Both results were statistically significant (p < 0.010). Similar declines (53.8% and 38.9%; columns 3 and 4, respectively) were observed for these procedures when restricting the sample to patients with osteoporotic fractures only. The change in the vertebroplasty rate was driven by a time trend effect, while the change in kyphoplasties was driven by a level effect and a time trend effect.

The interesting observation really comes when the investigators break down declines in VAPS by specialty:

The striking differences by physician specialty were confirmed in the regression results. Interventional radiologists responded to the trials (column 7) with an increase in time trend of the vertebroplasty rate (+0.019, p = 0.046) and a percentage change after the trials that was not significantly different from 0. This was very different (p < 0.010) from physicians in other specialties, whose vertebroplasty rate declined by 73.1% after the trials (column 8). Interestingly, radiologists exhibited a greater decline in the kyphoplasty rate (−46.9%) than did other specialists (−21.7%, p < 0.010) (columns 9 and 10).

Remember, according to this database, interventional radiologists do far more vertebroplasties than they did kyphoplasties, while surgeons do more kyphoplasties than vertebroplasties. Also, kyphoplasty is viewed as a more “invasive” procedure than vertebroplasty, mainly because of the inflation of the balloon in the vertebral body. So it might be that radiologists were more willing to shy away from kyphoplasty, leaving that to surgeons, than they were vertebroplasty. Moreover, surgeons, viewing the “more invasive” kyphoplasty procedure as more “surgical” in nature, might be less willing to give it up, given that the two studies didn’t directly address kyphoplasty. However, it is disturbing that the rates of vertebroplasty done by interventional radiologists haven’t declines in Florida at all since these 2009 studies, while among other specialties have decreased their use of vertebroplasty markedly.

So, yes, as I’ve described before, physicians really do abandon the use of procedures that are shown to be ineffective in large randomized clinical trials, but the process is messy and often arguably takes longer than it should. One other aspect of this process is that specialty society recommendations matter. Note that it took only a year for the AAOS to issue its guideline that strongly advised against the use of vertebroplasty for patients with osteoporotic VCFs. And so vertebroplasty use among specialties other than interventional radiology (the vast majority of whom were orthopedic surgeons or neurosurgeons) declined by nearly three-quarters. However, the guideline noted that evidence to support the use of kyphoplasty is “limited” but still retained it as an option for the treatment of osteoporotic VCFs. As a result, the decline in use of kyphoplasty was much more modest among surgeons. The authors note that:

Perhaps not coincidentally, the specialty societies’ practice guidelines after the trials interpreted the clinical trial evidence differently. The AAOS guidelines strongly recommended against vertebroplasty for osteoporotic VCFs, implying that physicians should follow the recommendation “unless a clear and compelling rationale for an alternate approach is present.” In contrast, kyphoplasty was described as a treatment option with a limited strength of recommendation. Other professional specialty organizations such as the North American Spine Society, the Society of Interventional Radiologists, and other groups did not issue recommendations against vertebroplasty or vertebral augmentation.

There are also practice patterns to consider, as the authors note. Interventional radiologists invented vertebroplasty, while orthopedic surgeons invented kyphoplasty, with each specialty retaining their dominance in the practice of the procedures they each invented. Indeed, for a time, for a time kyphoplasty training was restricted to surgeons. History, turf, and politics all too frequently intrude to prevent the rapid widespread adoption of guidelines, particularly for procedures that just seem so damned plausible. However, as I said in my article in Trends in Molecular Medicine last month and mentioned in my recent article in Nature Reviews Cancer on integrative oncology, although a prior plausibility that is clearly close to zero (as for homeopathy, for example) is a good predictor that a treatment will not be shown to work in RCTs, plausibility by itself is not anything close to a guaranteed that a procedure will be shown to work in RCTs.

Also, physicians practicing science-based medicine are human beings as well. They’re just as prone to biases and not wanting to admit that something they believed in before isn’t helping patients. This leads to a phenomenon described by the authors, but not explicitly addressed:

Although publication of the trials and the AAOS guideline influenced the use of vertebroplasty and kyphoplasty, our results indicate that both procedures are still widely used to treat patients with osteoporotic spinal fractures. Both trials were subject to criticism over their design and execution, and the appropriateness of vertebral augmentation continues to be widely debated. Multiple studies, analyses, and systematic reviews have been published since 2009 without consensus. In light of the controversy of the trials’ design and conflicting evidence from other studies, many physicians may feel free to ignore the “inconveniently negative” results from the trials and continue to offer VAPs to patients with osteoporotic spinal fractures.

You’ll note a similarity here. Remember what happened in response to the report of the Canadian National Breast Screening Study (CNBSS), which failed to find a benefit in terms of breast cancer survival due to mammographic screening. I put it all into context as best I could, but when I did that I not well-grounded in science and nearly all of it coming from radiologists and their societies

Like CAM practitioners, some have even argued that the placebo response is a legitimate endpoint for these procedures. If you don’t believe me, check out this article in Radiology by authors of one of the VAPS trials in the NEJM entitled “Vertebroplasty and the placebo response“, where such arguments are actually described. The authors conclude:

In conclusion, our interpretation of the evidence for vertebroplasty has distinctive implications for clinical practice and health policy. If the benefits of vertebroplasty derive from the placebo response, it is difficult to justify the continuing use of this procedure in clinical practice. The reason for this is not that benefit from the placebo response is lacking in therapeutic value; rather, it is doubtful that the placebo response benefits from vertebroplasty are sufficiently large to justify the risks. Moreover, although it is unknown what the outcomes would be for patients who are candidates for vertebroplasty and informed about the evidence that this procedure is no different from a sham intervention without injection of cement, they are likely to be lower than the benefits observed in clinical practice or in the Vertos II trial. Hence, on the basis of the placebo response hypothesis, the risk-benefit ratio of vertebroplasty does not appear favorable. Finally, whether continued evaluation of vertebroplasty within clinical trials might be justified is debatable—indeed, we are not in consensus on this issue.

What is being described here is the justification of a procedure that has no specific benefits but has a placebo response, which the authors refute. How many times have we here at SBM refuted the exact same sorts of arguments? It just goes to show that even ostensibly evidence-based physicians can fall prey to such arguments. Indeed, even the authors of one of the VAPS studies have implicitly accepted such arguments. The reason they reject them is not because they are not clinically significant but rather because they deem the risks of vertebroplasty not to be worth the placebo benefit.

I’ve been following the vertebroplasty story intermittently since 2005. It struck me an example of just how easily we in “conventional” medicine can fall prey to the same sort of fallacious thinking that drives the advocacy of modalities such as homeopathy or acupuncture. It’s an example that helped me in my journey towards skepticism and science-based medicine. We all like to assume that negative well-designed RCTs will result in physicians abandoning a procedure, and in general they do. However, the lesson of VAPS tells us that the process is…complicated. It’s not straightforward, and it’s messy. (There are still radiologists out there advertising vertebroplasty as if it were a miracle cure.) Still, five years on, the process for VAPS procedures appears to be well under way. Just try comparing that to the continued use of acupuncture, where no amount of negative evidence causes its abandonment, even after a period of time much, much longer than five years.

As I said, it’s complicated. But medicine does eventually eliminate practices that are not supported by evidence. It’s a huge contrast to alternative medicine.