Validity of comorbidity adjustment scores in estimating in-hospital mortality in individual subgroups of race/ethnicity

Using data from the SID, Healthcare Cost and Utilization Project, Agency for Healthcare Quality and Research for the states of CA (2007–2011), FL, NY, MD and KY (2007–2014). We derived and tested composite scores of the Elixhauser comorbidities by individual categories of race/ethnicity: White, Black, Hispanic, other (which includes: Asian or Pacific Islander, Native American or other). The SID is a census of more than 95% of all hospital discharges in the USA and includes a range of patient- and hospital-level information, including patient race/ethnicity and data on diagnoses and procedures, with present-on-admission indicators for each diagnosis to distinguish comorbidities from in-hospital complications [9]. Each record in the SID represents an individual hospital admission [9].

For comparability to van Walraven et al.’s methods, we excluded from our analysis sample population same-day surgeries and obstetric, pediatric and interhospital transfer admissions [1]. We randomly divided our sample population into derivation (66.7%) and validation (33.3%) samples. Using the derivation sample, we re-derived Elixhauser comorbidity summary measures by categories of race/ethnicity using the methods similar to those from van Walraven et al. [1] and Thompson et al. [8]. For each population, we created logistic regression models predicting in-hospital mortality, including as covariates only the 29 Elixhauser comorbidities. Through backwards stepwise elimination, we retained variables in subsequent logistic regression models with a p-value of <0.05, until we arrived at a model containing only co-efficients that met this threshold. For modeling the outcome in the validation dataset, the comorbidities that were not retained received a comorbidity weight of 0; retained covariates were assigned weights by taking the covariate’s co-efficient divided by the absolute value of the smallest co-efficient in the final model. To create a summary comorbidity measure, each co-efficient was multiplied by its weight.

In models stratified by category of race/ethnicity, we modeled our category-specific comorbidity measures on the outcome of in-hospital mortality in our validation sample. We also modeled the data with binary indicators of all 29 comorbidities, the VW score and Thompson’s SID 29 and SID 30 (without cardiac arrhythmia) for each category. Model calibration was assessed graphically and with a likelihood ratio test, and discrimination was assessed with the concordance (C) statistic with a 95% CI. Analyses were performed in Stata 16 SE (TX, USA). Discrimination and calibration were assessed using the roccomp and calibrationbelt packages, respectively [10]. All study-related activities were approved by the Institutional Review Board at Weill Cornell Medical College.

Prevalence of individual comorbidities and their respective weights used in analyses are found in Table 1. In our overall validation sample (N = 9,564,277), our index (c = 0.80; 95% CI: 0.79–0.80) discriminated better than the VW score (c = 0.79; 95% CI: 0.79–0.79), SID 29 (c = 0.78; 95% CI: 0.78–0.79) and SID 30 (c = 0.78; 95% CI: 0.78–0.78), but was not superior to the binary indicators (c = 0.80; 95% CI: 0.80–0.80).

In populations of White patients (N = 5,867,762), our index (c = 0.80, 0.80–0.80) performed slightly better than the VW score (c = 0.78; 95% CI: 0.78–0.78), SID 29 (c = 0.78; 95% CI: 0.78–0.78) and SID 30 (c = 0.77; 95% CI: 0.77–0.78), but had no difference in discriminatory power with the binary indicators (c = 0.80; 95% CI: 0.79–0.80; Figure 1).

Among Black patients (N = 1,563,454), our index (c = 0.82; 95% CI: 0.82–0.82) again performed slightly better than the VW score (c = 0.80; 95% CI: 0.80–0.81), SID 29 (c = 0.80; 95% CI: 0.80–0.80) and SID 30 (c = 0.80; 95% CI: 0.80–0.80), but was not different than the binary indicators (c = 0.82; 95% CI: 0.82–0.82; Figure 2). Our index also performed well in Hispanic patients (N = 1,341,827): c = 0.81; 95% CI: 0.80–0.81 and in patients of other races/ethnicities (N = 644,927): c = 0.81; 95% CI: 0.80–0.81.

All models displayed weak calibration in each category of race/ethnicity, notably at high predicted probabilities with few observations; in each model, the hypothesis of good calibration is rejected (p < 0.01 for all.) Figure 3 shows calibration belts for all validated models of in-hospital mortality for White and Black patients separately.

Using SID data from CA, FL, NY, MD and KY over a span of 8 years, we derived composite scores of the Elixhauser comorbidities by individual categories of race/ethnicity and found that race/ethnicity-specific indexes performed slightly better than the overall VW score, SID 29 and SID 30 at discriminating in-hospital mortality. However, our index did not have superior discriminatory power to the binary Elixhauser measures.

To our knowledge, this is the first study to examine the validity and performance of composite comorbidity indexes in estimating in-hospital mortality in individual subgroups of race/ethnicity. As with any study using administrative data, the prevalence of comorbidities in our sample population may not be completely captured due to variations in coding practices. However, SID data are validated by quality control measured established by Healthcare Cost and Utilization Project to ensure the accuracy and internal consistency of data [9].

In sum, we found only modest differences by race/ethnicity in the discriminative validity of comorbidity adjustment scores, and no model was well calibrated. Existing measures remain valid and useful tools for researchers to adjust for patient risk factors among racial/ethnic subgroups in population-based health outcomes studies, but our race/ethnicity-specific index could provide a useful tool for parsimonious comorbidity risk adjustment in racial and ethnic health disparities research. However, our score and others suffered from poor calibration, notably at high probabilities. For this reason, we recommend that future research explore recalibrating race/ethnicity-specific scores, pointing to the better calibration observed in Thompson’s study, whose baseline model included additional covariates [8]. We also recommend the development of comorbidity indexes specific to other categories of social determinants of health, such as insurance payer and household income level.

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.