The Source for Neurovascular News and Education

March 29, 2024

 

The focus should be on building better doctors, not better models, Frank Veith advises.

 

Prediction models do a poor job of estimating short-term and long-term outcomes for patients undergoing carotid revascularization for symptomatic internal carotid artery stenosis, according to an external validation study published online recently in Stroke.

 

“Over the past decade, many prediction models for outcome after carotid revascularization have been developed,” write the authors, led by Eline J. Volkers, MD (University Medical Center Utrecht, the Netherlands). They point out, however, that “only a few of these prediction models were validated in independent patient populations.” External validation, they add, “is an essential step in prediction model development that should be performed before a model can be implemented in clinical practice.”

 

Volkers and colleagues identified 30 prediction models based on readily available patient characteristics that were known before the initiation of the index procedure. Of these, 23 predicted short-term outcomes and seven predicted long-term outcomes.

 

To validate the models, they used data from four randomized trials that compared the use of carotid artery stenting (CAS) and carotid endarterectomy (CEA) for the treatment of symptomatic internal carotid artery stenosis (EVA-3S, SPACE, ICSS, and CREST). These trials involved 2,184 patients who underwent CAS and 2,261 who underwent CEA.

 

Short-term models were externally validated for their ability to predict stroke or death within 30 days of the index procedure, in addition to the original outcome measures used in the model development studies. Long-term models were externally validated only for their ability to predict the outcomes used in the model development studies.

 

Overall, stroke or death within 30 days of the index procedure occurred in 158 patients after carotid artery stenting (7.2%) and in 84 patients after carotid endarterectomy (3.7%).

 

Most of the 23 models that predicted short-term outcome after CAS (n = 4) or CEA (n = 19) had poor discriminative performance, with C statistics ranging from 0.49 to 0.64. They were also found to have poor calibration, with small absolute risk differences between the lowest and highest risk groups, and overestimation of risk in the highest risk groups.

 

For the seven long-term outcome models that were evaluated, performance was slightly better, with C statistics ranging from 0.59 to 0.67 and what the authors estimated to be “reasonable calibration.”

 

Predictive Ability of Long-Term Outcome Models

 

Model Development Study

External Validation

 

C Statistic (95% CI) in Development Cohort

Original Outcome Measure

C Statistic (95% CI) for Original Outcome Measure

Endarterectomy Studies

Hoke 2012

0.79 (not reported)

Mortality

0.67 (0.63-0.71)

Cheng 2016

0.66 (not reported)

Stroke, myocardial infarction, or death

0.61 (0.57-0.65)

                                                                  Stenting Studies

van Lammeren 2012

0.69 (0.64–0.73)

Stroke, myocardial infarction, or death from cardiovascular causes

0.59 (0.54-0.63)

Conrad 2013

0.250 (0.118-0.388)

Mortality

0.67 (0.63-0.71)

Wallaert 2013

0.74 (not reported)

Mortality

0.66 (0.59-0.73)

Gates 2015

Not reported

Short-term stroke, MI, or death, long-term ipsilateral stroke or death from neurological causes

0.60 (0.55-0.65)

Endarterectomy and Stenting

Alcocer 2013

Not reported

Mortality

0.60 (0.57-0.64)

 

“Current models did not reliably predict outcome after carotid revascularization in a trial population of patients with symptomatic carotid stenosis,” conclude the authors. “In particular, prediction of short-term outcome seemed to be difficult. Further external validation of existing prediction models or development of new prediction models is needed before such models can be used to support treatment decisions in individual patients.”

 

Focus on Cultivating Good Physicians

 

Commenting on the study for Neurovascular Exchange, Frank J. Veith, MD (NYU Langone Medical Center, New York, NY, and Cleveland Clinic, OH), said the findings are not surprising.

 

“To quantitate and predict outcomes is intrinsically an imperfect venture,” he explained, because there are many subjective patient characteristics that are either not included in such models or are difficult to quantify. The latter include things like patient frailty and the presence of a bull neck. In addition, he noted, the models typically fail to take into account operator skill and experience, which are extremely important predictors of outcome. Finally, the studies themselves that were used to validate the models have their own limitations and are becoming somewhat outdated.

 

Unfortunately, he noted, in the case of internal carotid artery stenosis, “the patients who need procedures because they are high risk if you do nothing, are usually the ones who are high risk if you do something. And you take risk when the benefit outweighs [that] risk. That’s what medicine is all about, weighing risks using your judgment, experience, etc.”

 

Rather than developing better models, said Veith, the focus should be on “cultivating better doctors with better judgment and sometimes perhaps with better motivation,” meaning doctors should not be incentivized, as they may be in some medical systems, to perform interventions for monetary purposes, he concluded.

 


 

Source:

Volkers EJ, Algra A, Kappelle LJ, et al. Prediction models for clinical outcome after a carotid revascularization procedure: an external validation study. Stroke. 2018;Epub ahead of print.

 

Disclosures:

Volkers and Veith report no relevant conflicts of interest.