Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Improving quality of stroke care through benchmarking center performance: why focusing on outcomes is not enough

BMC Health Services Research volume 20, Article number: 998 (2020) Cite this article

Abstract

Background

Between-center variation in outcome may offer opportunities to identify variation in quality of care. By intervening on these quality differences, patient outcomes may be improved. However, whether observed differences in outcome reflect the true quality improvement potential is not known for many diseases. Therefore, we aimed to analyze the effect of differences in performance on structure and processes of care, and case-mix on between-center differences in outcome after endovascular treatment (EVT) for ischemic stroke.

Methods

In this observational cohort study, ischemic stroke patients who received EVT between 2014 and 2017 in all 17 Dutch EVT-centers were included. Primary outcome was the modified Rankin Scale, ranging from 0 (no symptoms) to 6 (death), at 90 days. We used random effect proportional odds regression modelling, to analyze the effect of differences in structure indicators (center volume and year of admission), process indicators (time to treatment and use of general anesthesia) and case-mix, by tracking changes in tau2, which represents the amount of between-center variation in outcome.

Results

Three thousand two hundred seventy-nine patients were included. Performance on structure and process indicators varied significantly between EVT-centers (P < 0.001). Predicted probability of good functional outcome (modified Rankin Scale 0–2 at 90 days), which can be interpreted as an overall measure of a center’s case-mix, varied significantly between 17 and 50% across centers. The amount of between-center variation (tau2) was estimated at 0.040 in a model only accounting for random variation. This estimate more than doubled after adding case-mix variables (tau2: 0.086) to the model, while a small amount of between-center variation was explained by variation in performance on structure and process indicators (tau2: 0.081 and 0.089, respectively). This indicates that variation in case-mix affects the differences in outcome to a much larger extent.

Conclusions

Between-center variation in outcome of ischemic stroke patients mostly reflects differences in case-mix, rather than differences in structure or process of care. Since the latter two capture the real quality improvement potential, these should be used as indicators for comparing center performance. Especially when a strong association exists between those indicators and outcome, as is the case for time to treatment in ischemic stroke.

Peer Review reports

Background

Endovascular treatment (EVT) has been shown to be a highly effective treatment for patients with ischemic stroke due to a proximal intracranial occlusion in the anterior circulation [1,2,3]. EVT is defined as arterial catheterization with a micro-catheter to the level of the occlusion, followed by mechanical thrombectomy or thrombus aspiration, or both, with or without delivery of a thrombolytic agent. Currently EVT is widely implemented in routine clinical practice, and the challenge is how to continuously improve the quality of this service.

Worldwide, healthcare systems and practices are being reorganized with a strong focus on measuring and improving outcomes of care. The quality of care for patients with a specific medical condition is judged by the achieved outcomes that are relevant for those patients. A central aspect of this development is benchmarking, comparing quality of care and specifically outcomes between healthcare providers, in this case EVT centers. If a specific center A has better outcomes for a certain condition than center B, this suggests that center B should copy the medical management strategy of center A in order to improve the quality of care in that center. An important problem of this approach, however, is the variability in baseline characteristics of patients (‘case-mix’) that often exists between centers. If center B treats more severely affected patients than center A, this might (partly) explain the better outcomes of center A. Moreover, especially if centers are relatively small, between-center differences in outcome may be caused by chance (‘random variation’). Therefore, between-center comparisons of outcome should be adjusted for case-mix and random variation. If not done properly, such comparisons are likely to miss their purpose and could even be counterproductive as clinicians may base their decisions on flawed information. Furthermore, following Donabedian’s framework for evaluating healthcare quality, the quality improvement potential is especially captured by variation in structures (‘How is care organized?’) and processes (‘What is done?’) of care [4]. For many diseases, however, it is unknown whether between-center variation in outcome reflects true differences in quality of care, captured by this framework.

Using data from a large nation-wide registry, the aim of this study was to assess the effect of structure and process indicators on between-center variation in outcome for ischemic stroke patient treated with EVT, while adjusting for case-mix and random variation.

Methods

Study design and patients

For this study we used data collected between March 2014 and November 2017 from the MR CLEAN Registry, a prospective, observational study in all 17 centers that perform EVT in the Netherlands (Supplementary Figure 1) [5]. This registry is unique since it includes clinical and neuro-imaging data of all patients treated with EVT in one country during a multi-year period and thereby reflects clinical practice. All patients undergoing EVT for acute ischemic stroke have been registered. Inclusion criteria were: age 18 years and older, treatment in a center that participated in the MR CLEAN trial, and proximal intracranial vessel occlusion in the anterior circulation (internal carotid artery (ICA), internal carotid artery terminus (ICA-T), middle (M1/M2) cerebral artery, or anterior (A1/A2) cerebral artery), as shown by computed tomography angiography (CTA). Details on the study design and objectives of the MR CLEAN Registry have been described elsewhere [5]. Overall, data from 3279 patients were included for the current analysis (Supplementary Figure 2).

The MR CLEAN Registry was approved by the ethics committee of the Erasmus University MC, Rotterdam, The Netherlands (MEC-2014-235). With this approval it was approved by the research board of each participating center. At UMC Utrecht, approval to participate in the study has been obtained from their own research board and ethics committee.

Case-mix indicators

For case-mix adjustment we used the following patient and neuro-imaging characteristics: age, sex, relevant medical history (i.e. previous stroke, atrial fibrillation, myocardial infarction, peripheral arterial disease, hypertension, diabetes mellitus, hypercholesterolemia), pre-stroke score on the modified Rankin Scale (mRS), the baseline score on the National Institute of Health Stroke Scale (NIHSS) as a stroke-related neurologic deficit score, and occlusion location and collateral grade on CT angiography. These characteristics were selected based on clinical knowledge and previous studies [6, 7]. Two additional characteristics were considered as case-mix indicators, because these are strongly associated with outcome but not influenceable by the centers: time between stroke onset and arrival at the emergency department (ED) of the intervention center, and whether or not the patient had been admitted to another center before being transferred to the intervention center [8, 9]. Stroke onset was defined as the time point when the sudden appearance of stroke symptoms was witnessed by the patient or an observer. In cases the time of first symptoms was unknown, onset was defined as the moment the patient was last seen well.

Quality of care indicators

Quality of care indicators were defined using Donabedian’s framework comprising indicators of structure, process, and outcome [4]. Both center volume and year of admission reflect the experience of a center with EVT and were used as structure indicators. Center volume was defined as the percentage of all EVT-patients treated in each center relative to all EVT-patients treated in the Netherlands in the study period. In stroke care, high center volume was found to be associated with lower stroke-related mortality [10]. In other studies, higher volume stroke centers showed better outcomes on average [11,12,13]. Since EVT is a relatively new treatment in stroke care, we hypothesized (overall) performance to increase with calendar year and therefore added year of admission as an additional structure indicator.

Two process indicators were defined: time from arrival at the emergency department of the intervention center to groin puncture and the use of general anesthesia (yes/no). A significant negative association between ‘time-to-groin’ and outcome was found in previously published research, indicating that time delays before initiation of EVT have a negative effect on the likelihood of independent functional recovery at 90 days [8, 13,14,15,16,17]. So far, no differences in outcomes between general anesthesia and conscious sedation were observed in previous randomized controlled trials [18,19,20]. In several meta-analyses of observational studies, conscious sedation was associated with better outcomes than general anesthesia [21,22,23]. Although general anesthesia reduces the risk of patient agitation, unnecessary use of general anesthesia increases time delay in the total process of care, intra-procedural complications, and may result in cerebral hypoperfusion (e.g. through fluctuations in blood pressure on general anesthesia induction and abnormal cerebral auto-regulation) [24, 25]. Overall, we expected the use of general anesthesia to influence patient outcome, and therefore defined this as an additional process indicator.

We used the modified Rankin Scale (mRS) score as the outcome indicator [26]. The mRS score is a commonly used measure of patients’ functional outcome after ischemic stroke, and ranges from 0 (no symptoms) to 6 (death). The mRS score was assessed at 90 days after EVT (± 14 days). Good functional outcome, defined as mRS 0–2, was used as secondary outcome (see below).

Statistical analyses

Descriptive analyses

We used Pearson’s chi-square statistic and the non-parametric Kruskal Wallis test for a univariable comparison of centers on case-mix and quality of care indicators. The predicted probability of good functional outcome can be considered an overall measure of each centers’ case-mix. To calculate this, we first fitted an individual-level logistic regression model including all case-mix indicators as predictors and yes/no good functional outcome (mRS 0–2 at 90 days) as the dependent variable. The predicted patient-level probabilities by this model were then used to calculate the median predicted probability per center.

Random effect regression models

In order to adjust for random variation and assess the effects of adjusting for case-mix and performance on structure and process indicators on between-center variation in outcome, we used random effect proportional odds regression modelling. A random center effect (intercept) accounts for the fact that the observed outcomes for lower-volume centers can take extreme values due to random variation. A proportional odds model exploits the full ordinal nature of the mRS as an outcome scale with more than two possible categories [27].

In all analyses, we used the inverse of the mRS score for each patient. Doing so allows us to interpret the estimates as the effects on the likelihood of a more favorable outcome, since a higher score on the inverse of the mRS means more favorable outcome (see above). We estimated common odds ratios with 95% confidence intervals on the patient level using four proportional odds regression models [27]. First, we fitted an ‘empty’, unadjusted model including only a random center effect, providing insight in between-center variation in outcome accounting only for random variation. In the second model, in addition to the random center effect, we adjusted for the individual-level fixed effects of case-mix indicators on outcome. In the third model, we added the fixed effects of the structure indicators to the model. Finally, the fourth model also contains the individual-level fixed effects of the process indicators.

Between-center variation

To assess the relative impact of adjusting for case-mix and performance on structure and process indicators on between-center variation in outcome, we compared the variance of the random center effect (tau2) across models. Essentially, tau2 reflects the amount of between-center variation in outcome. In addition, separately for each model, we constructed forest plots to visualize this variation using estimates of center-specific outcome (i.e. the random center effects). The predictive power of the four models was compared using Akaike’s Information Criterion (AIC), in which a lower AIC value indicates a higher predictive power for outcome [28].

Missing data

Multiple imputation was used to deal with missing data, which ranged between 0.7% (previous diabetes) to 6.3% (collateral grade) [5]. We fitted regression imputation models [29, 30] and imputed data five times, using the following variables: age, sex, medical history, pre-stroke mRS score, location of occlusion, collateral grade, baseline NIHSS score, whether patient transferred from other hospital, time intervals from onset to arrival at the ED, center volume per year, year of admission, time intervals from onset to groin puncture, and use of general anesthesia. Each imputed dataset was analyzed separately, after which the results were pooled.

All statistical analyses were performed with R statistical software version 3.4.3 (R Foundation for Statistical Computation, Vienna, Austria), using the clmm module in the ordinalimputation package. Statistical significance was assessed at P < 0.05 in all analyses.

Results

Descriptive analyses

At the center level, the median patient age ranged from 68 to 77 years, with statistically significant differences between centers (Table 1). Differences between centers were also statistically significant for previous stroke (range 0–26%), atrial fibrillation (13–37%), peripheral arterial disease (4–22%), hypertension (41–67%), hypercholesterolemia (15–50%), pre-stroke mRS, location of occlusion, collateral grade, baseline NIHSS score (range median score per center 13–17) and percentage of transferred patients from another hospital (0–77%). Median time of stroke onset to arrival at the ED of the intervention center ranged from 52 to 160 min across centers (P < 0.001). Importantly, the median predicted probability of good functional outcome (mRS 0–2 at 90 days), which can be interpreted as an overall measure of a center’s case-mix, varied between 17 and 50% across centers (P = 0.004) (Table 1 and Supplementary Table 1). The patient-level effect estimates of case-mix variables on outcome shown in Supplementary Figure 3 are comparable to the results of prior research [31].

Table 1 Case-mix characteristics of patients treated in intervention centers in MR CLEAN Registry
Full size table

Relative to the total number of patients in our data, the number of EVT-patients varied significantly between centers across the 4 years (Fig. 1). Median time from arrival at the ED of the intervention center to groin puncture also varied substantially: between 74 and 125 min for non-transferred patients (P < 0.001) and between 20 and 60 min for transferred patients (P < 0.001). Variation in the use of general anesthesia (0–99%) was also statistically significant. Crude differences in outcome were statistically significant (P < 0.001) across centers for mRS values 0–6: no symptoms (1–18%), no significant disability (7–26%), slight disability (4–29%), moderate disability (4–20%), moderately severe disability (7–31%), severe disability (2–11%), and death (21–36%) (Table 2 and Supplementary Table 2).

Fig. 1
figure 1

Center volume in each intervention year. Center volume is defined as percentage of all EVT patients treated in each center relative to all EVT patients treated in the Netherlands in that year

Full size image
Table 2 Quality of care indicators of all 17 intervention centers in the Netherlands
Full size table

Predictive power for outcome

Model 1 generated an AIC of 11,158, which dropped to 10,050 after adding case-mix variables in model 2, suggesting a considerably improved predictive power (Table 3). Adding structure (AIC = 10,043) and process indicators (AIC = 10,012) only slightly improved the predicted power further (Table 3).

Table 3 Results from the random effect proportional odds regression analysis using the inverse of the modified Rankin Scale at 90 days as the dependent variable
Full size table

Between-center variation in outcome

In model 1, which only adjusts for random variation, the amount of the between-center variation in outcome (tau2) was 0.040 (Table 3). The tau2 represents the amount of variability in outcome between centers. This estimate more than doubled after adding case-mix indicators (model 2, tau2: 0.086), while adding structure indicators (model 3) and process indicators (model 4) left it almost unaffected (tau2: 0.081 and 0.089, respectively). This indicates that only a small amount of between-center variation was explained by variation in performance on structure and process indicators. This finding is also reflected in the forest plots (Fig. 2), which for each model show the estimated effects of all 17 centers on the likelihood of favorable outcome. Between-center variation increased particularly after case-mix indicators were added (compare Fig. 2b with Fig. 2a), while variation remained rather constant after adding structure (Fig. 2c) and process (Fig. 2d) indicators.

Fig. 2
figure 2

Forest plots reporting random center effect (odds ratios and 95% confidence intervals) on inverse of modified Rankin Scale at 90 days in four models using random effect proportional odds regression analysis. a: Model 1 (unadjusted model); b: Model 2 (case-mix adjusted model); c: Model 3 (case-mix and structure indicators adjusted model); d: Model 4 (case-mix, structure and process indicators adjusted model)

Full size image

Discussion

This study focused on assessing variation in outcome between centers treating ischemic stroke with EVT in the Netherlands, and the impact of differences in case-mix and in performance on structure and process indicators. Our results show that differences in case-mix have a much larger impact on between-center differences in outcome for stroke patients treated with EVT than differences in structure and processes of care across centers. Therefore, an (unadjusted) measure of functional outcome may not be a valid indicator for quality of care when comparing stroke centers.

Outcome indicators as measures of quality of care

Benchmarking initiatives increasingly use outcome measures to compare quality of care between centers. However, as found in this study, significant differences in performance on evidence- and consensus-based measures of (processes of) care may not be reflected in between-center differences in outcome. An important explanation for this is the observational nature of the data that are typically used in such benchmarking initiatives. When assessing the effect of interventions or processes of care on outcome in such data, confounding by indication is a major issue. For example, physicians might treat more severely affected patients faster than less severely affected patients [32, 33]. As disease severity is likely to influence outcome, in observational data it can confound the estimated relation with outcome insofar severity is insufficiently accounted for. Although we adjusted for disease severity in several ways using measured case-mix variables, we cannot preclude the possibility that our estimated effect of time to treatment on outcome is to some extent biased. This could partly be because of a residual unmeasured case-mix effect, as well as unaccounted for technical parameters. Knowing the association of center-level outcomes with process indicators and structural characteristics allows decision-makers to understand the determinants of performance and implement improvement strategies; in other words, all indicators should be retained to get an empirical (i.e., context specific) evaluation of the quality of care. If a center is an “outlying provider” even after accounting for patient case mix and process/structural characteristics, in-depth analyses and audit activities should be put in place to understand the reasons for this center’s outlying performance. Given that detecting a beneficial effect of effective interventions on outcome can be difficult even in large methodologically sound randomized trials, observing effects of evidence- and consensus-based indicators of (processes of) care on between-center differences in outcome may be even harder in benchmarking initiatives using observational registry data. In other words, when observational data are used for benchmarking, which in practice is typically the case, good performance on structure and process indicators may not be reflected in favorable outcome due to unmeasured confounding factors. Therefore, identifying and benchmarking performance on indicators with a proven contribution to favorable outcomes should be an important future direction for stroke care quality assessment and improvement initiatives.

Between-center differences in outcome after case-mix adjustment

The observed difference of the unadjusted and adjusted estimated center effect is because of statistical consideration and is a result of both imbalance and stratification [34, 35]. Good outcomes will generally be more difficult to achieve for patients who are more severely affected at baseline. Therefore, the observed outcomes of centers with a relatively ‘severe’ case-mix will on average be less favorable (i.e. biased downwards) as compared to those of centers with a relatively ‘mild’ case-mix (i.e. biased upwards). In general, adjusting for the imbalance in case-mix would then reduce observed variation in outcome between centers. However, in this study an opposite pattern is observed. After adjusting for case-mix, actual between-center differences in outcome become visible. Apparently, centers with a more severe case-mix tend to have relatively good outcomes and vice versa. Although counterintuitive, this still underlines the necessity of appropriate case-mix adjustment in benchmarking quality performance using observational data. Besides the effect of adjusting for the imbalance, more extreme center effect after adjustment could be a result of the stratification effect. Although “stratification” usually refers to conditioning on categoric subgroups (e.g. on sex), we also use this term when continuous variables are involved, for example, age. Adjustment will generally increase standard errors, and the stratification (adjustment) effect will lead to more extreme effect estimates [34].

Strengths and limitations

In previous studies [6, 7], a dichotomized version of the mRS was used (mRS ≥3 was considered as ‘poor outcome’) and analyzed using binary logistic regression models. In order to exploit the full ordinal nature of the mRS score, we used proportional odds regression analysis. In addition, the use of random effect analyses allowed us to estimate (the variance of) center outcomes adjusted for random variation and various other factors (i.e. case-mix, structure, and process indicators).

A first limitation of this study is the unavailability of other potentially contributing factors to between-center outcome differences, e.g. unmeasured patient characteristics, care processes, and center characteristics. A second limitation is that missing values may have introduced some bias, although we believe to have mitigated this issue considerably using multiple imputation, which is the preferred method over complete case analysis [36, 37]. A third limitation is low number of second-level units, with resulting lack of precision in tau2 estimates. A final limitation is that we only analyzed one outcome. Although the mRS is used as an outcome in virtually all modern stroke trials, our conclusions might have been different if we had used other outcomes that are relevant to stroke patients treated with EVT, like patient-reported outcomes such as quality of life. In addition, even though the mRS is an appropriate tool for assessing patient disability after stroke care, it may not be easily transferrable to outcomes research. After all, in many countries other than the Netherlands, the mRS is not routinely collected for all acute stroke patients. However, if we would have used clinical outcomes that are commonly registered in administrative databases, such as short-term mortality or readmission, we would have less certainty about our estimations because of the much smaller number of events in the context of acute stroke treatment. Moreover, from a clinical perspective these outcome measures are far less relevant in the stroke context compared to the mRS. Therefore, we used the mRS as the most powerful and clinically relevant outcome for our study.

Conclusions

In this study, we have demonstrated that between-center differences in performance on structure and process indicators have a small impact on functional outcome of ischemic stroke patients treated with EVT, while differences in case-mix affects this variation substantially. Thus, outcome indicators may not be valid and useful for comparing and improving the quality of stroke care based on observational data. Since variation in performance on structure and process indicators captures real quality improvement potential, these indicators should be used in future benchmarking initiatives. This is especially true when a strong association exists between those indicators and outcome, as is the case for time to treatment in ischemic stroke.

Availability of data and materials

The dataset analyzed during the current study is available from the MR CLEAN Registry upon request as a study proposal. Requests should be directed to the executive committee (mrclean@erasmusmc.nl). Detailed analytic methods and study materials, including log files of statistical analyses, will be made available to other researchers on request to the corresponding author (m.amini@erasmusmc.nl).

Abbreviations

EVT:

Endovascular treatment

ICA:

Internal carotid artery

ICA-T:

Internal carotid artery terminus

M1/M2:

Middle cerebral artery

A1/A2:

Anterior cerebral artery

CTA:

Computed tomography angiography

mRS:

Modified Rankin Scale

NIHSS:

National Institute of Health Stroke Scale

ED:

Emergency department

AIC:

Akaike’s Information Criterion

IQR:

Interquartile range

PAD:

Peripheral arterial disease

MI:

Myocardial infarction

HCL:

Hypercholesterolemia

References

  1. Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016;387(10029):1723–31.

    Article  PubMed  Google Scholar 

  2. Albers GW, Marks MP, Kemp S, Christensen S, Tsai JP, Ortega-Gutierrez S, et al. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N Engl J Med. 2018;378(8):708–18.

    Article  PubMed  Google Scholar 

  3. Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. N Engl J Med. 2018;378(1):11–21.

    Article  PubMed  Google Scholar 

  4. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743–8.

    Article  CAS  PubMed  Google Scholar 

  5. Jansen IGH, Mulder M, Goldhoorn RB, Investigators MCR. Endovascular treatment for acute ischaemic stroke in routine clinical practice: prospective, observational cohort study (MR CLEAN Registry). BMJ. 2018;360:k949.

    Article  PubMed  Google Scholar 

  6. Lingsma HF, Dippel DW, Hoeks SE, Steyerberg EW, Franke CL, van Oostenbrugge RJ, et al. Variation between hospitals in patient outcome after stroke is only partly explained by differences in quality of care: results from the Netherlands stroke survey. J Neurol Neurosurg Psychiatry. 2008;79(8):888–94.

    Article  CAS  PubMed  Google Scholar 

  7. Lingsma HF, Steyerberg EW, Eijkemans MJ, Dippel DW, Scholte OP, Reimer WJ, Van Houwelingen HC, et al. Comparing and ranking hospitals based on outcome: results from the Netherlands stroke survey. QJM. 2010;103(2):99–108.

    Article  CAS  PubMed  Google Scholar 

  8. Mulder M, Jansen IGH, Goldhoorn RB, Venema E, Chalos V, Compagne KCJ, et al. Time to endovascular treatment and outcome in acute ischemic stroke: MR CLEAN registry results. Circulation. 2018;138(3):232–40.

    Article  PubMed  Google Scholar 

  9. Venema E, Groot AE, Lingsma HF, Hinsenveld W, Treurniet KM, Chalos V, et al. Effect of Interhospital transfer on endovascular treatment for acute ischemic stroke. Stroke. 2019;50(4):923–30.

    Article  PubMed  Google Scholar 

  10. Saposnik G, Baibergenova A, O'Donnell M, Hill MD, Kapral MK, Hachinski V. Hospital volume and stroke outcome. Does it matter? Neurology. 2007;69(11):1142–51.

    Article  CAS  PubMed  Google Scholar 

  11. Gupta R, Horev A, Nguyen T, Gandhi D, Wisco D, Glenn BA, et al. Higher volume endovascular stroke centers have faster times to treatment, higher reperfusion rates and higher rates of good clinical outcomes. J Neurointerv Surg. 2013;5(4):294–7.

    Article  PubMed  Google Scholar 

  12. Tsugawa Y, Kumamaru H, Yasunaga H, Hashimoto H, Horiguchi H, Ayanian JZ. The association of hospital volume with mortality and costs of care for stroke in Japan. Med Care. 2013;51(9):782–8.

    Article  PubMed  Google Scholar 

  13. Lorenzo R, Waleed B, Alejandro AR. Transfer to high-volume centers associated with reduced mortality after endovascular treatment of acute stroke. Stroke. 2017;48(5):1316–21.

    Article  Google Scholar 

  14. Aghaebrahim A, Granja MF, Agnoletto G, Aguilar-Salinas P, Cortez GM, Santos R, et al. Workflow optimization for ischemic stroke in a community-based stroke center. World Neurosurg. 2019;129:e273–8.

    Article  Google Scholar 

  15. Fransen PS, Berkhemer OA, Lingsma HF, Beumer D, van den Berg LA, Yoo AJ, et al. Time to reperfusion and treatment effect for acute ischemic stroke: a randomized clinical trial. JAMA Neurol. 2016;73(2):190–6.

    Article  Google Scholar 

  16. Goyal M, Almekhlafi MA, Fan L, Menon BK, Demchuk AM, Yeatts SD, et al. Evaluation of interval times from onset to reperfusion in patients undergoing endovascular therapy in the interventional Management of Stroke III trial. Circulation. 2014;130(3):265–72.

    Article  CAS  PubMed  Google Scholar 

  17. Saver JL, Goyal M, van der Lugt A, Menon BK, Majoie CB, Dippel DW, et al. Time to treatment with endovascular Thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA. 2016;316(12):1279–88.

    Article  Google Scholar 

  18. Lowhagen Henden P, Rentzos A, Karlsson JE, Rosengren L, Leiram B, Sundeman H, et al. General anesthesia versus conscious sedation for endovascular treatment of acute ischemic stroke: the AnStroke trial (anesthesia during stroke). Stroke. 2017;48(6):1601–7.

    Article  Google Scholar 

  19. Schonenberger S, Uhlmann L, Hacke W, Schieber S, Mundiyanapurath S, Purrucker JC, et al. Effect of conscious sedation vs general anesthesia on early neurological improvement among patients with ischemic stroke undergoing endovascular Thrombectomy: a randomized clinical trial. JAMA. 2016;316(19):1986–96.

    Article  Google Scholar 

  20. Simonsen CZ, Yoo AJ, Sorensen LH, Juul N, Johnsen SP, Andersen G, et al. Effect of general anesthesia and conscious sedation during endovascular therapy on infarct growth and clinical outcomes in acute ischemic stroke: a randomized clinical trial. JAMA Neurol. 2018;75(4):470–7.

    Article  PubMed  Google Scholar 

  21. Brinjikji W, Murad MH, Rabinstein AA, Cloft HJ, Lanzino G, Kallmes DF. Conscious sedation versus general anesthesia during endovascular acute ischemic stroke treatment: a systematic review and meta-analysis. AJNR Am J Neuroradiol. 2015;36(3):525–9.

    Article  CAS  PubMed  Google Scholar 

  22. Wan TF, Xu R, Zhao ZA, Lv Y, Chen HS, Liu L. Outcomes of general anesthesia versus conscious sedation for stroke undergoing endovascular treatment: a meta-analysis. BMC Anesthesiol. 2019;19(1):69.

    Article  PubMed  Google Scholar 

  23. Jing R, Dai HJ, Lin F, Ge WY, Pan LH. Conscious sedation versus general anesthesia for patients with acute ischemic stroke undergoing endovascular therapy: a systematic review and meta-analysis. Biomed Res Int. 2018;2018:2318489.

    PubMed  Google Scholar 

  24. John N, Mitchell P, Dowling R, Yan B. Is general anaesthesia preferable to conscious sedation in the treatment of acute ischaemic stroke with intra-arterial mechanical thrombectomy? A review of the literature. Neuroradiology. 2013;55(1):93–100.

    Article  CAS  PubMed  Google Scholar 

  25. Berkhemer OA, van den Berg LA, Fransen PSS, Beumer D, Yoo AJ, Lingsma HF, et al. The effect of anesthetic management during intra-arterial therapy for acute stroke in MR CLEAN. Neurology. 2016;87(7):656–64.

    Article  CAS  PubMed  Google Scholar 

  26. van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke. 1988;19(5):604–7.

    Article  PubMed  Google Scholar 

  27. Roozenbeek B, Lingsma HF, Perel P, Edwards P, Roberts I, Murray GD, et al. The added value of ordinal analysis in clinical trials: an example in traumatic brain injury. Crit Care. 2011;15(3):R127.

    Article  PubMed  Google Scholar 

  28. Akaike H. Information theory and an extension of the maximum likelihood principle. 2nd International Symposium on Information Theory; 1973.

    Google Scholar 

  29. Rubin DB, Schenker N. Multiple imputation in health-care databases: an overview and some applications. Stat Med. 1991;10(4):585–98.

    Article  CAS  PubMed  Google Scholar 

  30. Schenker N. Inference with imputed conditional means AU - Schafer, Joseph L. J Am Stat Assoc. 2000;95(449):144–54.

    Article  Google Scholar 

  31. Venema E, Mulder M, Roozenbeek B, Broderick JP, Yeatts SD, Khatri P, et al. Selection of patients for intra-arterial treatment for acute ischaemic stroke: development and validation of a clinical decision tool in two randomised trials. BMJ. 2017;357:j1710.

    Article  PubMed  Google Scholar 

  32. Bosco JL, Silliman RA, Thwin SS, Geiger AM, Buist DS, Prout MN, et al. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. J Clin Epidemiol. 2010;63(1):64–74.

    Article  PubMed  Google Scholar 

  33. Sjoding MW, Luo K, Miller MA, Iwashyna TJ. When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study. Crit Care. 2015;19:195.

    Article  PubMed  Google Scholar 

  34. Steyerberg EW, Bossuyt PM, Lee KL. Clinical trials in acute myocardial infarction: should we adjust for baseline characteristics? Am Heart J. 2000;139(5):745–51.

    Article  CAS  PubMed  Google Scholar 

  35. van Leeuwen N, Walgaard C, van Doorn PA, Jacobs BC, Steyerberg EW, Lingsma HF. Efficient design and analysis of randomized controlled trials in rare neurological diseases: an example in Guillain-Barre syndrome. PLoS One. 2019;14(2):e0211404.

    Article  PubMed  Google Scholar 

  36. van der Heijden GJ, Donders AR, Stijnen T, Moons KG. Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. J Clin Epidemiol. 2006;59(10):1102–9.

    Article  Google Scholar 

  37. Steyerberg EW, van Veen M. Imputation is beneficial for handling missing data in predictive models. J Clin Epidemiol. 2007;60(9):979.

    Article  Google Scholar 

Download references

Acknowledgements

We thank the MR CLEAN Registry Investigators-group authors:

Executive committee.

Diederik W.J. Dippel1;Aad van der Lugt2;Charles B.L.M. Majoie3;Yvo B.W.E.M. Roos4;Robert J. van Oostenbrugge5;Wim H. van Zwam6;Jelis Boiten14;Jan Albert Vos8.

Study coordinators.

Josje Brouwer 4;Sanne J. den Hartog1,2,40;Wouter H. Hinsenveld 5,6;Manon Kappelhof3;Kars C.J. Compagne2;Robert- Jan B. Goldhoorn5,6; Maxim J.H.L. Mulder1,2; Ivo G.H. Jansen3.

Local principal investigators.

Diederik W.J. Dippel1;Bob Roozenbeek1;Aad van der Lugt2;Adriaan C.G.M. van Es2;Charles B.L.M. Majoie3;Yvo B.W.E.M. Roos4;Bart J. Emmer3;Jonathan M. Coutinho4;Wouter J. Schonewille7;Jan Albert Vos8; Marieke J.H. Wermer9;Marianne A.A. van Walderveen10;Julie Staals5;Robert J. van Oostenbrugge5;Wim H. van Zwam6;Jeannette Hofmeijer11;Jasper M. Martens12;Geert J. Lycklama à Nijeholt13;Jelis Boiten14;Sebastiaan F. de Bruijn15;Lukas C. van Dijk16;H. Bart van der Worp17;Rob H. Lo18;Ewoud J. van Dijk19;Hieronymus D. Boogaarts20;J. de Vries22;Paul L.M. de Kort21; Julia van Tuijl21;Jo Jo P. Peluso26;Puck Fransen22;Jan S.P. van den Berg22;Boudewijn A.A.M. van Hasselt23;Leo A.M. Aerden24;René J. Dallinga25;Maarten Uyttenboogaart28;Omid Eschgi29;Reinoud P.H. Bokkers29;Tobien H.C.M.L. Schreuder30;Roel J.J. Heijboer31;Koos Keizer32;Lonneke S.F. Yo33;Heleen M. den Hertog22;Emiel J.C. Sturm35; Paul Brouwers34.

Imaging assessment committee.

Charles B.L.M. Majoie3(chair);Wim H. van Zwam6;Aad van der Lugt2;Geert J. Lycklama à Nijeholt13;Marianne A.A. van Walderveen10;Marieke E.S. Sprengers3;Sjoerd F.M. Jenniskens27;René van den Berg3;Albert J. Yoo38;Ludo F.M. Beenen3;Alida A. Postma6;Stefan D. Roosendaal3;Bas F.W. van der Kallen13;Ido R. van den Wijngaard13;Adriaan C.G.M. van Es2;Bart J. Emmer,3;Jasper M. Martens12; Lonneke S.F. Yo33;Jan Albert Vos8; Joost Bot36, Pieter-Jan van Doormaal2; Anton Meijer27;Elyas Ghariq13; Reinoud P.H. Bokkers29;Marc P. van Proosdij37;G. Menno Krietemeijer33;Jo P. Peluso26;Hieronymus D. Boogaarts20;Rob Lo18;Dick Gerrits35;Wouter Dinkelaar2Auke P.A. Appelman29;Bas Hammer16;Sjoert Pegge27;Anouk van der Hoorn29;Saman Vinke20.

Writing committee.

Diederik W.J. Dippel1(chair);Aad van der Lugt2;Charles B.L.M. Majoie3;Yvo B.W.E.M. Roos4;Robert J. van Oostenbrugge5;Wim H. van Zwam6;Geert J. Lycklama à Nijeholt13;Jelis Boiten14;Jan Albert Vos8;Wouter J. Schonewille7;Jeannette Hofmeijer11;Jasper M. Martens12;H. Bart van der Worp17;Rob H. Lo18.

Adverse event committee.

Robert J. van Oostenbrugge5(chair);Jeannette Hofmeijer11;H. Zwenneke Flach23.

Trial methodologist.

Hester F. Lingsma40.

Research nurses / local trial coordinators.

Naziha el Ghannouti1;Martin Sterrenberg1;Corina Puppels7;Wilma Pellikaan7;Rita Sprengers4;Marjan Elfrink11;Michelle Simons11;Marjolein Vossers12;Joke de Meris14;Tamara Vermeulen14;Annet Geerlings19;Gina van Vemde22;Tiny Simons30;Cathelijn van Rijswijk21;Gert Messchendorp28;Nynke Nicolaij28;Hester Bongenaar32;Karin Bodde24;Sandra Kleijn34;Jasmijn Lodico34; Hanneke Droste34;Maureen Wollaert5;Sabrina Verheesen5;D. Jeurrissen5;Erna Bos9;Yvonne Drabbe15;Michelle Sandiman15;Marjan Elfrink11;Nicoline Aaldering11;Berber Zweedijk17;Mostafa Khalilzada15;Jocova Vervoort21;Hanneke Droste34;Nynke Nicolaij2;Michelle Simons11;Eva Ponjee22;Sharon Romviel19;Karin Kanselaar19;Erna Bos9;Denn Barning10.

PhD / Medical students.

Esmee Venema40; Vicky Chalos1,40;Ralph R. Geuskens3; Tim van Straaten19;Saliha Ergezen1; Roger R.M. Harmsma1; Daan Muijres1; Anouk de Jong1;Olvert A. Berkhemer1,3,6;Anna M.M. Boers3,39; J. Huguet3;P.F.C. Groot3;Marieke A. Mens3;Katinka R. van Kranendonk3;Kilian M. Treurniet3;Ivo G.H. Jansen3;Manon L. Tolhuisen3,39;Heitor Alves3;Annick J. Weterings3,Eleonora L.F. Kirkels3,Eva J.H.F. Voogd11;Lieve M. Schupp3;Sabine Collette28,29;Adrien E.D. Groot4;Natalie E. LeCouffe4;Praneeta R. Konduri39;Haryadi Prasetya39;Nerea Arrarte-Terreros39;Lucas A. Ramos39.

List of affiliations.

Department of Neurology1, Radiology2, Public Health40, Erasmus MC University Medical Center;

Department of Radiology and Nuclear Medicine3, Neurology4, Biomedical Engineering & Physics39, Amsterdam UMC, University of Amsterdam, Amsterdam;

Department of Neurology5, Radiology6, Maastricht University Medical Center and Cardiovascular Research Institute Maastricht (CARIM);

Department of Neurology7, Radiology8, Sint Antonius Hospital, Nieuwegein;

Department of Neurology9, Radiology10, Leiden University Medical Center;

Department of Neurology11, Radiology12, Rijnstate Hospital, Arnhem;

Department of Radiology13, Neurology14, Haaglanden MC, the Hague;

Department of Neurology15, Radiology16, HAGA Hospital, the Hague;

Department of Neurology17, Radiology18, University Medical Center Utrecht;

Department of Neurology19, Neurosurgery20, Radiology27, Radboud University Medical Center, Nijmegen;

Department of Neurology21, Radiology26, Elisabeth-TweeSteden ziekenhuis, Tilburg;

Department of Neurology22, Radiology23, Isala Klinieken, Zwolle;

Department of Neurology24, Radiology25, Reinier de Graaf Gasthuis, Delft;

Department of Neurology28, Radiology29, University Medical Center Groningen;

Department of Neurology30, Radiology31, Atrium Medical Center, Heerlen;

Department of Neurology32, Radiology33, Catharina Hospital, Eindhoven;

Department of Neurology34, Radiology35, Medical Spectrum Twente, Enschede;

Department of Radiology36, Amsterdam UMC, Vrije Universiteit van Amsterdam, Amsterdam;Department of Radiology37, Noordwest Ziekenhuisgroep, Alkmaar;

Department of Radiology38, Texas Stroke Institute, Texas, United States of America.

Funding

No funding was received in support of this study.

Author information

Authors and Affiliations

  1. Department of Public Health, Erasmus MC University Medical Center, P.O. Box 2040, 3000, CA, Rotterdam, The Netherlands

    Marzyeh Amini, Nikki van Leeuwen, Jan Hazelzet & Hester F. Lingsma

  2. Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The Netherlands

    Frank Eijkenaar

  3. Department of Neurology, Erasmus MC University Medical Center, Rotterdam, The Netherlands

    Maxim J. H. L. Mulder, Diederik W. J. Dippel & Bob Roozenbeek

  4. Department of Neurology, St. Antonius Hospital, Nieuwegein, The Netherlands

    Wouter Schonewille

  5. Department of Radiology, Medical Center Haaglanden, The Hague, The Netherlands

    Geert Lycklama à Nijeholt

  6. Maastricht University Medical Center and Cardiovascular Research Institute Maastricht (CARIM), Maastricht, The Netherlands

    Wouter H. Hinsenveld & Robert-Jan B. Goldhoorn

  7. Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands

    Pieter Jan van Doormaal

  8. Department of Radiology, Radboud University Medical Center, Nijmegen, The Netherlands

    Sjoerd Jenniskens

Authors
  1. Marzyeh Amini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikki van Leeuwen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank Eijkenaar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maxim J. H. L. Mulder
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wouter Schonewille
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Geert Lycklama à Nijeholt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wouter H. Hinsenveld
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Robert-Jan B. Goldhoorn
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Pieter Jan van Doormaal
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sjoerd Jenniskens
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jan Hazelzet
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Diederik W. J. Dippel
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Bob Roozenbeek
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Hester F. Lingsma
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

on behalf of the MR CLEAN Registry Investigators

  • Diederik W. J. Dippel
  • , Aad van der Lugt
  • , Charles B. L. M. Majoie
  • , Yvo B. W. E. M. Roos
  • , Robert J. van Oostenbrugge
  • , Wim H. van Zwam
  • , Jelis Boiten
  • , Jan Albert Vos
  • , Josje Brouwer
  • , Sanne J. den Hartog
  • , Wouter H. Hinsenveld
  • , Manon Kappelhof
  • , Kars C. J. Compagne
  • , Robert-Jan B. Goldhoorn
  • , Maxim J. H. L. Mulder
  • , Ivo G. H. Jansen
  • , Bob Roozenbeek
  • , Adriaan C. G. M. van Es
  • , Bart J. Emmer
  • , Jonathan M. Coutinho
  • , Wouter J. Schonewille
  • , Marieke J. H. Wermer
  • , Marianne A. A. van Walderveen
  • , Julie Staals
  • , Jeannette Hofmeijer
  • , Jasper M. Martens
  • , Geert J. Lycklama à Nijeholt
  • , Sebastiaan F. de Bruijn
  • , Lukas C. van Dijk
  • , H. Bart van der Worp
  • , Rob H. Lo
  • , Ewoud J. van Dijk
  • , Hieronymus D. Boogaarts
  • , J. de Vries
  • , Paul L. M. de Kort
  • , Julia van Tuijl
  • , Jo Jo P. Peluso
  • , Puck Fransen
  • , Jan S. P. van den Berg
  • , Boudewijn A. A. M. van Hasselt
  • , Leo A. M. Aerden
  • , René J. Dallinga
  • , Maarten Uyttenboogaart
  • , Omid Eschgi
  • , Reinoud P. H. Bokkers
  • , Tobien H. C. M. L. Schreuder
  • , Roel J. J. Heijboer
  • , Koos Keizer
  • , Lonneke S. F. Yo
  • , Heleen M. den Hertog
  • , Emiel J. C. Sturm
  • , Paul Brouwers
  • , Marieke E. S. Sprengers
  • , Sjoerd F. M. Jenniskens
  • , René van den Berg
  • , Albert J. Yoo
  • , Ludo F. M. Beenen
  • , Alida A. Postma
  • , Stefan D. Roosendaal
  • , Bas F. W. van der Kallen
  • , Ido R. van den Wijngaard
  • , Joost Bot
  • , Pieter-Jan van Doormaal
  • , Anton Meijer
  • , Elyas Ghariq
  • , Marc P. van Proosdij
  • , G. Menno Krietemeijer
  • , Jo P. Peluso
  • , Rob Lo
  • , Dick Gerrits
  • , Wouter Dinkelaar
  • , Auke P. A. Appelman
  • , Bas Hammer
  • , Sjoert Pegge
  • , Anouk van der Hoorn
  • , Saman Vinke
  • , H. Zwenneke Flach
  • , Hester F. Lingsma
  • , Naziha el Ghannouti
  • , Martin Sterrenberg
  • , Corina Puppels
  • , Wilma Pellikaan
  • , Rita Sprengers
  • , Marjan Elfrink
  • , Michelle Simons
  • , Marjolein Vossers
  • , Joke de Meris
  • , Tamara Vermeulen
  • , Annet Geerlings
  • , Gina van Vemde
  • , Tiny Simons
  • , Cathelijn van Rijswijk
  • , Gert Messchendorp
  • , Nynke Nicolaij
  • , Hester Bongenaar
  • , Karin Bodde
  • , Sandra Kleijn
  • , Jasmijn Lodico
  • , Hanneke Droste
  • , Maureen Wollaert
  • , Sabrina Verheesen
  • , D. Jeurrissen
  • , Erna Bos
  • , Yvonne Drabbe
  • , Michelle Sandiman
  • , Nicoline Aaldering
  • , Berber Zweedijk
  • , Mostafa Khalilzada
  • , Jocova Vervoort
  • , Eva Ponjee
  • , Sharon Romviel
  • , Karin Kanselaar
  • , Denn Barning
  • , Esmee Venema
  • , Vicky Chalos
  • , Ralph R. Geuskens
  • , Tim van Straaten
  • , Saliha Ergezen
  • , Roger R. M. Harmsma
  • , Daan Muijres
  • , Anouk de Jong
  • , Olvert A. Berkhemer
  • , Anna M. M. Boers
  • , J. Huguet
  • , P. F. C. Groot
  • , Marieke A. Mens
  • , Katinka R. van Kranendonk
  • , Kilian M. Treurniet
  • , Ivo G. H. Jansen
  • , Manon L. Tolhuisen
  • , Heitor Alves
  • , Annick J. Weterings
  • , Eleonora L. F. Kirkels
  • , Eva J. H. F. Voogd
  • , Lieve M. Schupp
  • , Sabine Collette
  • , Adrien E. D. Groot
  • , Natalie E. LeCouffe
  • , Praneeta R. Konduri
  • , Haryadi Prasetya
  • , Nerea Arrarte-Terreros
  •  & Lucas A. Ramos

Contributions

HFL, BR and WJD designed the study. MA undertook analyses and interpretation of study findings and wrote the first draft of the manuscript. NvL, HFL, FE, BR, WJD contributed to the interpretation of study findings, reviews, and revision of the manuscript. MJHLM, WS, GLN, WHH, RJBG, PJD, SJ, JH contributed to the findings review and revision of the manuscript. All the authors critically reviewed the various versions of the full paper and approved the final manuscript.

Corresponding author

Correspondence to Marzyeh Amini.

Ethics declarations

Ethics approval and consent to participate

Not required.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table S1.

Case-mix characteristics of patients treated in intervention centers in the MR CLEAN Registry. Table S2. Quality of care indicators characteristics of the intervention centers in the MR CLEAN Registry database. Figure S1. Specialized EVT centers in the Netherlands. Adapted from MR CLEAN Registry study (https://www.mrclean-trial.org/mr-clean-registry/participating-centers.html). Figure S2. Flowchart of patient selection in the MR CLEAN Registry. Figure S3 Forest plot reporting odds ratios with 95% confidence intervals of the fixed effects of the case-mix variables on the inverse of the modified Rankin Scale at 90 days using random effect proportional odds regression analysis in the case-mix adjusted model.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amini, M., van Leeuwen, N., Eijkenaar, F. et al. Improving quality of stroke care through benchmarking center performance: why focusing on outcomes is not enough. BMC Health Serv Res 20, 998 (2020). https://0-doi-org.brum.beds.ac.uk/10.1186/s12913-020-05841-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12913-020-05841-y

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us