Validation of an all-source composite mortality endpoint in the US population
Abstract
Aim: To validate ConcertAI’s All Source Composite Mortality Endpoint (ASCME), which combines information from electronic health records, obituary, government and administrative claims. To compare overall survival (OS) estimates using ASCME versus a National Death Index (NDI) dataset across clinical cohorts. Materials & methods: Retrospective study of oncology real-world data, reporting sensitivity, specificity, positive predictive value and negative predictive value compared with the NDI standard, plus 5, 7 and 15-day concordance on date of death. Additional comparisons included Kaplan–Meier OS measured by ASCME versus NDI. Data sources included ConcertAI’s Patient360™ dataset, and a 2022 annual finalized NDI dataset. The sample included cancer patients prevalent from 1 April 2014 to 31 December 2022 in any of 10 of ConcertAI’s solid tumor-specific datasets. Results: Of 32,358 study patients, 14,241 (44.0%) were deceased as defined by an NDI true match. Sensitivity was 95.0% overall (an incremental 5.2% due to claims) and ranged from 92.7% to 97.8% across clinical cohorts. Overall specificity was 96.5%, with positive predictive value and negative predictive value of 95.8% each. ASCME's 5-day concordance was 97.9%, with 7-day and 15-day concordance of 98.2% and 99.1%, respectively. ASCME and NDI-based median OS estimates differed by 12.2 days among non-metastatic cohorts, and 4.5 days among metastatic cohorts. Conclusion: Results show ConcertAI’s ASCME death indicator to provide high completeness and accuracy, producing OS estimates largely indistinguishable from NDI-based estimates. Findings show the importance of including claims in composite mortality indicators and demonstrate the value of real-world data in assessing OS outcomes in metastatic and non-metastatic cancer patient populations.
Plain language summary: Checking the accuracy of a multi-source method for tracking deaths in the US
What is this article about?
This study checked the accuracy of a method for tracking deaths in cancer patients who are treated in community oncology practices. The method combines information about deaths from patient health records, obituaries, billing for medical care and government records. This matters because studies of cancer patients often track how long patients live, and this requires complete and accurate death information.
How was the study done?
The study used a dataset from ConcertAI that included information from more than 30,000 patients, including information about whether patients had died, and when. The study compared that dataset to information from another dataset that is considered a gold standard. The study reported how often patients shown as deceased by the gold standard appeared deceased in the dataset, how often the dataset accurately showed patients to still be living, and how closely dates of death in the two sources matched.
What were the results? What do they mean?
The results showed that of every 100 patients shown as deceased in the gold standard, 95 appeared deceased in the dataset. Of every 100 who were living according to the gold standard, 96 appeared living in the dataset. Also, among patients shown as deceased in the dataset and the gold standard, more than 95 of 100 had dates of death that matched within 5 days. These findings show that including billing for medical care improves the accuracy of death information, that the overall accuracy of death information in this dataset is very good, and that it is suitable for research with cancer patients treated in community practices.
Overall survival (OS) is a standard endpoint in both prospective and retrospective cancer research [1,2]. In clinical trials, patients are followed over time, allowing death information to be collected prospectively. However, in retrospective real-world data (RWD) studies, assessment of OS depends on the availability of death information within the clinical record or in linkable external sources.
Electronic health records (EHR) are the foundation of oncology RWD studies examining OS, because they are the richest source of clinical information about patients and their disease. EHRs may contain mortality information in structured fields, unstructured clinical records identified through chart abstraction or scanned copies of death certificates. However, EHR mortality information is usually not complete enough to support research-grade OS analysis. As a result, additional sources of mortality information that can be linked with the personally identifiable information (PII) of patients are required.
Other linkable mortality sources that are publicly available for commercial use include: the Limited Access Death Master File (LADMF), obituaries and administrative claims data. The LADMF contains death records for people with a Social Security number whose deaths were reported to the Social Security Administration. It is the principal US Government source of publicly available death records. However, its utility for mortality research was limited by a 2011 restriction in the inclusion of state-reported deaths [3], and by a mandated 3-year delay in the release of newly reported deaths in publicly accessible versions of the data [4].
Obituary data include burial records and death notices published by newspapers or funeral homes. Because these data are more fragmented than US Government death records, research requires multiple data vendors or an aggregator of these data (e.g., ObituaryData.com) [5] to cover the population. Nevertheless, obituary data have been found reliable and valid for those covered [6], and with adequate coverage, become a valuable data source.
Open and adjudicated claims capture inpatient deaths from ‘disposition at discharge’ reporting [7,8]. If a patient dies during a hospitalization, the claim record documents the death with essentially comprehensive capture of the date of death [7]. If that patient’s death is not documented in other available death records, the claim record will enhance the completeness of death information. Either open or adjudicated claims can be used, but open claims often provide nationwide coverage, and tend to be available sooner without the delay of adjudication. Although individual claims sources have limitations, including the introduction of bias in OS estimates, the use of a composite constructed from multiple sources may overcome these limitations [9–12].
An effective and unbiased RWD mortality endpoint should exhibit accuracy and completeness. Inaccurate death dates can bias survival estimates. Incomplete death data can result in misclassification of patients and overestimates of survival, with the degree of overestimate depending on censoring rules for death-related outcomes [13].
Death certificates represent the cardinal source evidence that a death has occurred. However, the costs and requirements for obtaining death certificates vary by state and create barriers to routine use, especially large-scale research. The National Death Index (NDI) aggregates death certificates [14]. Although the NDI data is not perfectly complete [15], and cannot be used for commercial research [14], its central aggregation allows it to serve as the gold standard for evaluating other mortality sources [8,16,17]. Many studies have compared RWD oncology datasets to the NDI [8,9,11,18,19]. However, none included mortality information from all four sources cited above: EHR, government data, obituaries and claims. In this study, we compare the All-Source Composite Mortality Endpoint (ASCME) available in the ConcertAI Patient360™ research dataset with NDI to evaluate the impact of claims data on a composite mortality endpoint, and to assess ASCME’s completeness and accuracy.
Materials & methods
Data source
The data source for this study was the ConcertAI Patient360™ dataset, and an annual finalized NDI dataset, National Death Index (RRID:SCR_016369). ConcertAI’s Patient360™ dataset is a comprehensive, representative, de-identified oncology dataset of human-abstracted, patient-level, RWD drawn from academic and community EHRs with full data provenance. It contains structured EHR data and information sourced from unstructured clinical records linked to open claims, with linked dates of death for more than 225,000 patients with confirmed cancer diagnosis across most solid tumors.
Inclusion in each tumor-specific dataset requires a histologically confirmed cancer, verified through human review, and an identified year of initial diagnosis. Structured dates of death are entered into the EHR when patients expire while under care. Manually abstracted dates of death are typically drawn from obituaries and communications from next of kin. ConcertAI receives daily mortality data updates from external sources, including obituary and burial records, government death records and administrative claims. When multiple death sources are available, priority is established by source-level 5-day concordance. The linkage of EHR data to external death records requires PII, including first and last names, sex, date of birth and other identifiers.
The NDI was established by the National Center for Health Statistics (NCHS) in 1981 [10], and is the most comprehensive single source of death information in the US [14,20]. Although participation is voluntary, all US states and territories submit death records to the NCHS through the Vital Statistics Cooperative Program. NDI data may not be used for commercial purposes. The NDI releases an annual finalized file of US deaths at least 12 months after the end of the most recent included year [14,20].
The study did not use biological samples, organisms, antibodies, cell lines or plasmids. All data used in this study were secondary data. There was no contact with patients, and no patient attrition, given the design and research questions.
Patients
Eligible patients included those represented in ConcertAI datasets for bladder cancer, breast cancer, gastroesophageal cancer, hepatocellular carcinoma (HCC), melanoma, non-small cell lung cancer, pancreatic cancer, prostate cancer, renal cell carcinoma (RCC) and small cell lung cancer. Except for prostate cancer, patients were classified as non-metastatic and/or metastatic. Non-metastatic cohorts included patients who presented with stage I–III disease, including loco-regional disease. Patients who later developed distant metastases, along with patients with de novo metastatic disease, were included in the metastatic cohorts. The prostate cohort was limited to metastatic castrate resistant prostate cancer (mCRPC) because patients with early-stage disease or castrate sensitive disease are more often treated in urology than oncology practices, which provide the majority of ConcertAI data. Stage at initial diagnosis varies across cancer types and, excluding breast cancer, appears correlated with the management of early-stage disease by non-oncology specialties (e.g., urology, dermatology), and inversely correlated with referral to oncology practices. Consequently, the distribution of non-metastatic and metastatic disease reported in this study may not match the US cancer population for some cancer types.
Patients were required to have a prevalent cancer on or after 1 January 2014, and an initial date of diagnosis on or before 31 December 2021. This allowed for a minimum of 12 months of follow-up at the study’s data end date, 31 December 2022, to align with the 2022 finalized NDI data file available to ConcertAI in March 2024.
Procedure
Sample size determination
A subset of all eligible patients was selected to manage the cost of NDI Plus Search services required for patients with a known date of death [14]. Sample sizes for non-metastatic and metastatic cohorts in each tumor type were estimated separately, assuming sensitivity of the ASCME endpoint based on existing literature regarding other mortality endpoint [11], and evidence regarding death prevalence as indicated in the ConcertAI data. Sample sizes were identified such that the lower bound of a one-tailed 95% CI would be within 5% of the estimated sensitivity with 80% likelihood, given our assumptions as to sensitivity and death prevalence. Additional patients were sampled where possible to account for attrition during sample matching to NDI data. Each tumor/setting sample was drawn randomly from the respective Patient360™ cancer-specific dataset.
NDI matching
A list of randomly selected, study-eligible patients was submitted to NDI along with all relevant PII for NDI to conduct patient matching. PII included Social Security numbers, the identifier assigned the largest weight for matching by NDI, plus patient names and dates of birth. Matching was based on NDI’s established and calibrated two-step method [14]. Patients retained post-match were those with an NDI class 1 match, or a class 2 to 4 match with weighted scores exceeding the cutoff that NDI considers indicative of ‘true matches’ [14].
Study measures
The study evaluated the completeness of data capture and the accuracy of recorded dates of death. Completeness was assessed as the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of ASCME compared with the NDI database. Accuracy was assessed as the 5-day, 7-day and 15-day concordance between ASCME and NDI dates of death. Note that exact date concordance was not evaluated (or expected) because dates of death are typically shifted to maintain the deidentified status of the data in commercially available datasets. Shifting of dates to the nearest Sunday, for example – a maximum 4-day shift, provides week of death granularity and greatly reduces the risk of reidentification. Median OS based on the ASCME mortality endpoint was an additional study measure, assessed for accuracy compared with median OS as indicated by the NDI date of death.
Statistical analysis
Each study measure was evaluated descriptively overall, by mortality data source, and by cancer type and setting (non-metastatic and metastatic). Because the study was descriptive and had no comparator groups, there was no blinding as to cohort membership. However, random sampling of patients within the cohort was employed to reduce the potential for bias associated with non-representativeness. Descriptive results were generated across demographic characteristics such as age, sex and race. Kaplan–Meier survival analysis was employed to assess OS in subgroups crossing tumor with non-metastatic versus metastatic status, to evaluate the incremental contribution of claims data to accurate assessment of median OS compared with the NDI reference standard. Analysis of OS was restricted to patients with an initial diagnosis on or after 1 January 2014 – a subset of the broader study sample – to avoid a survival bias. Specifically, patients diagnosed before 2014 who remained under care after that date (a study inclusion requirement) would, by definition, represent long-term survivors, artificially inflating OS estimates.
The time origin for assessment of OS was initial diagnosis for the non-metastatic cohorts, date of metastatic disease for the metastatic cohorts and the date of mCRPC diagnosis for prostate cancer patients. Date of death was as indicated in the respective sources: NDI, ASCME or EHR + obituary + government (i.e., ASCME excluding claims). Cases with no record of death in the applicable source were censored at the date of last activity in the EHR or date of last service in the claims record. For example, if a case had no ASCME death record, but did have an NDI death record, it would be censored at the date of last activity or service in the ASCME dataset, but would have an event at the applicable date in the NDI dataset. This approach served to assess the impact of the completeness and accuracy of death information on sample estimates of OS.
The protocol for this study underwent review by Advarra Institutional Review Board, Columbia, MD, and was determined to be exempt as secondary research for which consent is not required under 45 CFR 46.104(d)(4). All statistical analysis was conducted with R (version 4.4.2; R Project for Statistical Computing, RRID:SCR_001905).
Results
Sample characteristics
A total of 32,358 patients across tumor types were submitted to NDI for matching. Of these, 14,241 (44.0%) had a true match as defined in the Methods, and were classified as NDI-deceased in the data, and 13,529 (41.8% of the total) were classified as deceased in both ASCME and NDI datasets. The sample was 84.5% White, 7.2% Black or African American and 8.3% Other race; 56.5% female, evenly split between those older than 65 years (50.6%) versus 65 or younger. Most patients were drawn from community centers (90.9%) rather than academic institutions (see Table 1). Tumor/setting cohort sample sizes ranged from 412 for non-metastatic RCC to 8823 for non-metastatic breast cancer (see Table 2). NDI-defined death prevalence ranged from 17.3% for non-metastatic breast cancer to 90.3% for metastatic pancreatic cancer.
| Group | Total patients | Percent deceased† | Sensitivity (%) | Specificity (%) | PPV (%) | NPV (%) | 5-day concordance (%) |
|---|---|---|---|---|---|---|---|
| All patients | 32,358 | 44.0 | 95.0 (94.6, 95.3) | 96.5 (96.2, 96.7) | 95.8 (95.4, 96.1) | 95.8 (95.5, 96.1) | 97.9 (97.6, 98.1) |
| Source | |||||||
| Academic | 2946 | 49.7 | 95.2 (94.1, 96.3) | 97.0 (96.0, 97.8) | 97.1 (96.1, 97.9) | 95.1 (93.8, 96.1) | 97.3 (96.4, 98.1) |
| Community | 29,412 | 43.4 | 95.0 (94.6, 95.3) | 96.4 (96.1, 96.7) | 95.6 (95.2, 95.9) | 95.9 (95.6, 96.2) | 97.9 (97.7, 98.2) |
| Sex‡ | |||||||
| Male | 14,072 | 57.4 | 95.7 (95.2, 96.1) | 94.6 (94.0, 95.1) | 96.3 (95.9, 96.7) | 93.7 (93.0, 94.3) | 97.8 (97.5, 98.1) |
| Female | 18,285 | 33.7 | 94.1 (93.5, 94.7) | 97.4 (97.1, 97.7) | 95.1 (94.5, 95.6) | 96.9 (96.5, 97.2) | 98.0 (97.6, 98.3) |
| Age (years) | |||||||
| ≤65 | 15,987 | 34.8 | 94.5 (93.9, 95.1) | 97.6 (97.3, 97.9) | 95.5 (94.9, 96.1) | 97.0 (96.7, 97.3) | 97.2 (96.7, 97.6) |
| >65 | 16,371 | 54.4 | 95.3 (94.8, 95.7) | 94.9 (94.4, 95.4) | 95.9 (95.5, 96.3) | 94.1 (93.6, 94.7) | 98.3 (98.0, 98.6) |
| Race | |||||||
| Black or African–American | 2341 | 50.4 | 94.0 (92.6, 95.3) | 95.4 (94.0, 96.6) | 95.9 (94.7, 97.0) | 93.3 (91.6, 94.7) | 96.8 (95.7, 97.7) |
| White | 27,332 | 43.4 | 95.5 (95.1, 95.8) | 96.8 (96.5, 97.0) | 96.0 (95.6, 96.3) | 96.4 (96.0, 96.6) | 98.0 (97.7, 98.2) |
| Other | 2685 | 45.1 | 91.6 (89.9, 93.0) | 94.0 (92.6, 95.2) | 93.5 (92.0, 94.8) | 92.3 (90.8, 93.6) | 97.8 (96.8, 98.5) |
| US region | |||||||
| Midwest | 7902 | 47.4 | 96.7 (96.1, 97.2) | 94.9 (94.2, 95.5) | 94.5 (93.8, 95.2) | 96.9 (96.3, 97.4) | 98.4 (97.9, 98.7) |
| Northeast | 7242 | 39.4 | 96.2 (95.4, 96.8) | 97.8 (97.3, 98.2) | 96.6 (95.9, 97.3) | 97.5 (96.9, 97.9) | 97.5 (96.8, 98.0) |
| South | 12,475 | 44.2 | 94.7 (94.1, 95.3) | 96.6 (96.1, 97.0) | 95.9 (95.3, 96.4) | 95.6 (95.1, 96.1) | 97.8 (97.3, 98.1) |
| West | 3926 | 43.9 | 90.1 (88.6, 91.5) | 96.9 (96.1, 97.6) | 95.9 (94.8, 96.8) | 92.5 (91.3, 93.5) | 97.8 (96.9, 98.4) |
| Not reported | 813 | 49.7 | 95.3 (93.3, 96.8) | 93.3 (88.8, 96.4) | 97.8 (96.3, 98.8) | 86.2 (80.8, 90.6) | 98.1 (96.6, 98.9) |
†
Percent of patients confirmed deceased in NDI.
‡
One patient had unknown sex.
ASCME: All-Source Composite Mortality Endpoint; NDI: National Death Index; NPV: Negative predictive value; PPV: Positive predictive value.
| Group | Total patients | Percent deceased† | Sensitivity (%) | Specificity (%) | PPV (%) | NPV (%) | 5-day concordance (%) |
|---|---|---|---|---|---|---|---|
| Bladder non-met | 919 | 53.9 | 94.0 (91.6, 95.9) | 94.5 (91.8, 96.5) | 95.4 (93.1, 97.0) | 92.9 (90.1, 95.2) | 97.4 (95.6, 98.5) |
| Breast non-met | 8823 | 17.3 | 92.7 (91.3, 93.9) | 98.4 (98.1, 98.7) | 92.4 (90.9, 93.7) | 98.5 (98.2, 98.7) | 97.1 (96.0, 97.8) |
| GEC non-met | 2169 | 53.0 | 94.5 (93.1, 95.8) | 95.4 (93.9, 96.6) | 96.0 (94.7, 97.1) | 93.7 (92.1, 95.1) | 97.6 (96.6, 98.4) |
| HCC non-met | 683 | 69.8 | 94.2 (91.7, 96.1) | 89.0 (83.8, 93.0) | 95.4 (93.1, 97.1) | 86.4 (81.0, 90.8) | 98.5 (96.8, 99.2) |
| Melanoma non-met | 4650 | 23.5 | 95.2 (93.7, 96.4) | 98.1 (97.6, 98.5) | 93.9 (92.3, 95.2) | 98.5 (98.1, 98.9) | 97.4 (96.2, 98.2) |
| NSCLC non-met | 4224 | 46.9 | 94.2 (93.1, 95.2) | 96.1 (95.2, 96.9) | 95.7 (94.7, 96.5) | 94.8 (93.8, 95.7) | 97.4 (96.6, 98.0) |
| Pancreatic non-met | 1698 | 65.5 | 95.1 (93.7, 96.3) | 93.7 (91.3, 95.5) | 96.8 (95.5, 97.7) | 90.6 (88.0, 92.9) | 97.7 (96.6, 98.4) |
| RCC non-met | 412 | 52.7 | 94.4 (90.5, 97.1) | 93.4 (88.9, 96.4) | 94.0 (90.0, 96.8) | 93.8 (89.5, 96.8) | 98.0 (95.1, 99.2) |
| SCLC non-met | 433 | 65.6 | 94.7 (91.5, 97.0) | 90.5 (84.6, 94.7) | 95.1 (91.9, 97.3) | 89.9 (83.9, 94.3) | 98.9 (96.8, 99.6) |
| Bladder met | 860 | 79.2 | 95.5 (93.7, 96.9) | 86.5 (80.5, 91.3) | 96.6 (95.0, 97.8) | 82.7 (76.3, 87.9) | 98.6 (97.4, 99.3) |
| Breast met | 1409 | 62.2 | 95.6 (94.1, 96.9) | 95.0 (92.7, 96.7) | 97.0 (95.7, 98.1) | 92.7 (90.1, 94.7) | 97.8 (96.5, 98.6) |
| GEC met | 1103 | 84.7 | 96.6 (95.2, 97.7) | 83.0 (76.4, 88.4) | 97.0 (95.7, 98.0) | 81.1 (74.3, 86.7) | 97.9 (96.7, 98.6) |
| HCC met | 456 | 85.3 | 94.7 (92.1, 96.7) | 80.7 (68.1, 90.0) | 97.2 (95.0, 98.6) | 68.7 (56.2, 79.4) | 97.6 (95.5, 98.7) |
| Melanoma met | 1650 | 59.8 | 97.2 (96.0, 98.2) | 94.9 (93.0, 96.5) | 96.6 (95.2, 97.6) | 95.9 (94.1, 97.3) | 98.3 (97.3, 99.0) |
| NSCLC met | 1173 | 78.2 | 95.5 (93.9, 96.7) | 86.6 (81.8, 90.6) | 96.4 (95.0, 97.5) | 83.6 (78.5, 87.9) | 98.1 (96.9, 98.8) |
| Pancreatic met | 1069 | 90.3 | 95.9 (94.5, 97.1) | 59.6 (49.8, 68.9) | 95.4 (93.9, 96.7) | 62.5 (52.5, 71.8) | 98.6 (97.6, 99.2) |
| mCRPC | 857 | 77.9 | 97.8 (96.3, 98.7) | 92.6 (87.8, 95.9) | 97.9 (96.5, 98.8) | 92.1 (87.2, 95.5) | 97.5 (96.1, 98.5) |
| RCC met | 1486 | 62.9 | 96.4 (94.9, 97.5) | 93.3 (90.9, 95.3) | 96.0 (94.6, 97.2) | 93.8 (91.5, 95.7) | 98.2 (97.1, 98.9) |
| SCLC met | 1100 | 86.8 | 95.2 (93.6, 96.4) | 76.0 (67.7, 83.1) | 96.8 (95.4, 97.8) | 67.6 (59.3, 75.1) | 98.5 (97.5, 99.1) |
†
Percent of patients confirmed deceased in NDI.
Metastatic and non-metastatic cohorts may overlap within tumor type, because the metastatic cohorts include early-stage patients with metastatic recurrence.
ASCME: All-Source Composite Mortality Endpoint; GEC: Gastro-esophageal cancer; HCC: Hepatocellular carcinoma; mCRPC: Metastatic castrate-resistant prostate cancer; Met: Metastatic; NDI: National Death Index; Non-Met: Non-metastatic; NPV: Negative predictive value; NSCLC: Non-small cell lung cancer; PPV: Positive predictive value; RCC: Renal cell carcinoma; SCLC: Small cell lung cancer.
Completeness
Table 1 reports the completeness of the data overall and by patient characteristics. As shown, the overall sensitivity of the ASCME endpoint was 95.0%, with 96.5% specificity and PPV and NPV of 95.8%. Sensitivity exceeded 90.0% in every patient characteristic class across patient source, age, sex, race and geography.
Table 2 reports the completeness of the death information by tumor/setting status. As shown, sensitivity exceeded 92.0% in every cohort and ranged from 92.7% to 97.8% across the samples, with a median of 95.0%. Specificity ranged from 59.6% to 98.4% across groups, with a median of 95.6%. Notably, lower specificity statistics were observed in the cohorts with the highest prevalence of death, with Spearman ρ = -0.958, measured as the correlation between specificity and prevalence across the n = 19 tumor/setting cohorts described in Table 2. The specificity of a measure should, in theory, be unrelated to condition prevalence [21], but in this dataset, they appeared to be strongly, negatively related. This suggests that the true ASCME specificity values in this study are underestimated and reflect an artifact of imperfect capture of death in the NDI database, a condition described in a separate publication [22]. Such an artifact would impact the measurement of specificity even in cohorts with low death prevalence, but would be most evident where the prevalence of death is high [22]. PPV ranged from 92.4% to 97.9% across groups, and was 96.0% or greater in most groups. NPV ranged from 62.5% to 98.5%, with lower estimated NPV values occurring in the cohorts with the highest death prevalence (Spearman p = -0.953, n = 19), consistent with the known effect of condition prevalence on NPV [21].
The study also examined the completeness of mortality ascertainment as additional sources were added to EHR-only death data. The additional sources were government and obituary data (combined), and then claims. As shown in Table 3, EHR-only mortality data identified 75.5% of patient deaths but missed 24.5%. The addition of government and obituary records increased sensitivity to 89.8%, primarily due to the obituary records. The inclusion of claims further increased the sensitivity of ASCME to 95.0%, reducing the proportion of missed deaths from 10.2% to 5.0% – a reduction of more than half. Specificity and PPV were excellent in all three composite mortality data sources, but as shown, the addition of claims enhanced ASCME sensitivity without meaningfully reducing specificity. This means that deaths uniquely identified by claims were nearly all supported by the NDI record and included very few that appeared as false positives to the NDI database. Essentially, claims enhanced sensitivity without a tradeoff in specificity. NPV for EHR alone was modest, at 82.7%, due to false negative cases (deceased patients interpreted by EHR data as living), but increased with additional sources of mortality data, and reached 95.8% for the ASCME mortality endpoint.
| Group | Total patients† | Sensitivity (%) | Specificity (%) | PPV (%) | NPV (%) |
|---|---|---|---|---|---|
| EHR (structured and abstracted data) | 32,358 | 75.5 (74.8, 76.1) | 98.2 (98.0, 98.4) | 97.3 (97.0, 97.6) | 82.7 (82.2, 83.2) |
| EHR + Gov't + obituary | 32,358 | 89.8 (89.3, 90.2) | 96.8 (96.5, 97.1) | 95.9 (95.6, 96.2) | 91.9 (91.5, 92.2) |
| ASCME: EHR + Gov't + obituary + claims | 32,358 | 95.0 (94.6, 95.3) | 96.5 (96.2, 96.7) | 95.8 (95.4, 96.1) | 95.8 (95.5, 96.1) |
†
Fourteen thousand two hundred forty-one patients with a matched NDI date of death.
ASCME: All-Source Composite Mortality Endpoint; EHR: Electronic health record; Gov't: Government; NDI: National Death Index; NPV: Negative predictive value; PPV: Positive predictive value.
Accuracy
There were 712 patients shown as deceased in the NDI data for whom there was no ASCME death record, leaving 13,529 patients for whom a death was reported in both ASCME and NDI. Table 1 reports the accuracy of matched death records in that sample as the 5-day concordance between ASCME and the NDI date of death, by patient characteristics. As shown, the 5-day concordance was 97.9% overall, indicating a very high degree of accuracy in the ASCME death endpoint. The 5-day concordance was at least 96.8% in every patient characteristic class across patient source, age, sex, race and geography. The overall 7-day and 15-day concordance percentages were 98.2% and 99.1%, respectively (not shown).
Table 2 reports the 5-day concordance of ASCME and NDI dates of death by tumor and non-metastatic versus metastatic setting. As shown, the 5-day concordance exceeded 97.0% in every cohort, ranging from 97.1% to 98.9% across samples, indicating consistent accuracy across relevant tumor cohorts as well as across patient characteristics. Supplementary Table 1 shows the 5-day, 7-day and 15-day concordance by tumor and non-metastatic versus metastatic setting.
Overall survival
Kaplan–Meier estimates of OS based on the ASCME and NDI dates of death are presented by non-metastatic tumor in Figure 1, through 96 months from the date of initial diagnosis. ASCME survival curves in the figure are depicted in color. The corresponding NDI-based survival curves, which closely match the ASCME estimates, are shown in black. As shown, OS varied widely across non-metastatic cohorts, with medians not reached in breast cancer and melanoma, and with NDI medians ranging from 18.4 months in pancreatic cancer to 67.4 months in RCC across the remaining tumors. The Kaplan–Meier estimates of OS for metastatic tumors are presented in Figure 2. As shown, the NDI median OS ranged from 5.5 months for HCC to 34.3 months for breast cancer. Table 4 reports median OS for both non-metastatic and metastatic tumors, and includes the medians for NDI, ASCME and a median based on EHR + government + obituary sources (but without claims). In addition, for each, the number of death events is reported. As shown in the figures and the table, OS estimates as determined by the ASCME death indicator are hardly distinguishable from the NDI-based estimates, with a median absolute difference of 0.4 months (about 12.2 days) for non-metastatic tumors, and a median of 0.15 months (about 4.5 days) for metastatic tumors. Values in Table 4 show that EHR + government + obituary (excluding claims) identified fewer deaths than NDI, and produced a systematically longer median. Supplementary Figures 1–19 show each tumor/setting separately, and include the survival curves based on NDI, ASCME and EHR + government + obituary-based indicators of death.
Figure 1. Overall survival, by All-Source Composite Mortality Endpoint versus National Death Index death endpoint, by non-metastatic tumor.
ASCME: All-Source Composite Mortality Endpoint; GEC: Gastro-esophageal cancer; HCC: Hepatocellular carcinoma; NDI: National Death Index; Non-Met: Non-metastatic; NSCLC: Non-small cell lung cancer; RCC: Renal cell carcinoma; SCLC: Small cell lung cancer.
Figure 2. Overall survival, by All-Source Composite Mortality Endpoint versus National Death Index death endpoint, by metastatic tumor.
ASCME: All-Source Composite Mortality Endpoint; GEC: Gastro-esophageal cancer; HCC: Hepatocellular carcinoma; mCRPC: Metastatic castrate-resistant prostate cancer; Met: Metastatic; NDI: National Death Index; NSCLC: Non-small cell lung cancer; RCC: Renal cell carcinoma; SCLC: Small cell lung cancer.
| Tumor/setting group | Patients | NDI events | NDI median (95% CI) | ASCME events | ASCME median (95% CI) | EHR + Gov't + Obit events | EHR + Gov't + Obit median (95% CI) |
|---|---|---|---|---|---|---|---|
| Bladder non-met | 671 | 354 | 39.6 (34.0, 49.9) | 348 | 40.0 (34.8, 50.6) | 324 | 44.0 (38.0, 53.2) |
| Breast non-met | 4378 | 610 | NR (NR, NR) | 604 | NR (NR, NR) | 559 | NR (NR, NR) |
| GEC non-met | 1818 | 1,002 | 36.4 (33.0, 41.3) | 978 | 37.1 (34.1, 42.5) | 929 | 40.0 (35.4, 46) |
| HCC non-met | 642 | 453 | 24.0 (21.0, 28.6) | 442 | 24.2 (21.2, 28.6) | 402 | 26.6 (22.9, 30.5) |
| Melanoma non-met | 3170 | 738 | NR (95.5, NR) | 726 | NR (100.9, NR) | 676 | NR (NR, NR) |
| NSCLC non-met | 3367 | 1,584 | 53.0 (49.9, 58.3) | 1,546 | 55.3 (51.1, 59.0) | 1,429 | 59.6 (55.9, 62.3) |
| Pancreatic non-met | 1547 | 1,078 | 18.4 (17.2, 20.5) | 1,051 | 18.8 (17.4, 20.9) | 996 | 20.2 (18.0, 22.3) |
| RCC non-met | 228 | 116 | 67.4 (58.6, 75.7) | 117 | 67.0 (58.6, 74.4) | 111 | 69.6 (59.0, 76.2) |
| SCLC non-met | 390 | 259 | 25.7 (21.0, 31.3) | 257 | 25.6 (20.6, 30.6) | 246 | 26.4 (20.6, 33.1) |
| Bladder met | 496 | 406 | 10.6 (8.9, 12.2) | 399 | 10.6 (8.9, 12.1) | 377 | 11.0 (9.1, 12.2) |
| Breast met | 690 | 431 | 34.3 (31.3, 38) | 428 | 33.8 (31.1, 37.4) | 402 | 35.0 (31.9, 39.0) |
| GEC met | 974 | 832 | 9.1 (8.0, 10.1) | 828 | 9.1 (8.0, 10.1) | 794 | 9.4 (8.4, 10.2) |
| HCC met | 368 | 322 | 5.5 (4.9, 6.6) | 311 | 6.0 (5.2, 6.8) | 294 | 6.1 (5.4, 7.1) |
| Melanoma met | 921 | 556 | 20.2 (17.4, 24.8) | 556 | 20.2 (17.4, 24.8) | 540 | 21.4 (18.1, 25.6) |
| NSCLC met | 993 | 782 | 11.3 (10.1, 12.5) | 771 | 11.4 (10.2, 12.8) | 737 | 11.6 (10.8, 13.6) |
| Pancreatic met | 1012 | 915 | 5.6 (5.0, 6.5) | 916 | 5.4 (5.0, 6.4) | 879 | 5.8 (5.2, 6.5) |
| mCRPC | 683 | 519 | 19.2 (17.6, 22.0) | 518 | 19.2 (17.6, 22.0) | 493 | 19.9 (17.9, 22.8) |
| RCC met | 931 | 604 | 26.9 (24.0, 30.6) | 604 | 27.2 (24.0, 30.7) | 577 | 28.9 (25.7, 32.0) |
| SCLC met | 995 | 883 | 8.2 (7.4, 8.9) | 866 | 8.4 (7.6, 8.9) | 828 | 8.5 (7.8, 9.0) |
Metastatic and non-metastatic cohorts may overlap within tumor type, because the metastatic cohorts include early-stage patients with metastatic recurrence.
Analysis of overall survival limited inclusion to patients with a date of initial diagnosis 1 January 2014 or later, to avoid a survival bias associated with study patients who were initially diagnosed in earlier years. Tumor/setting-specific samples are therefore smaller here than in Table 2.
ASCME: All-Source Composite Mortality Endpoint; EHR: Electronic health record; GEC: Gastro-esophageal cancer; Gov't: Government; HCC: Hepatocellular carcinoma; mCRPC: Metastatic castrate-resistant prostate cancer; Met: Metastatic; NDI: National Death Index; Non-Met: Non-metastatic; NR: Not reached; NSCLC: Non-small cell lung cancer; RCC: Renal cell carcinoma; SCLC: Small cell lung cancer.
Discussion
This study validated the completeness and accuracy of ConcertAI’s ASCME against an annual finalized NDI dataset in a sample of 32,358 patients drawn from ConcertAI’s Patient360™ dataset. Results showed incremental sensitivity of 5.2% associated with the inclusion of claims, with overall sensitivity of 95.0% and 5-day concordance of 97.9%. The study showed that completeness and accuracy were maintained across demographic cohorts, and across solid tumor by treatment setting subsets. The study showed remarkable alignment between ASCME and NDI-based Kaplan–Meier OS curves.
To our knowledge, this is the first NDI-validated study of EHR-focused data in oncology to include claims as a source of mortality information, and to have reported on the incremental contribution of claims records in the completeness of death reporting. The study’s attention to mortality outcomes in patients with non-metastatic cancer is also important. The minimum sensitivity and minimum 5-day accuracy estimates in the non-metastatic cohorts (92.7% and 97.1%, respectively) demonstrate the viability of assessing OS with RWD in this population.
The 97.9% accuracy of ASCME assessed as 5-day concordance surpasses the accuracy measured as 15-day concordance in other datasets that are commercially available outside of proprietary environments [11,18,19]. The effect of the completeness and accuracy of ASCME is evident in Figures 1 & 2, which show OS as measured by ASCME that is not meaningfully distinguishable from what is indicated by the NDI reference standard. The consistency is evident across tumor cohorts and across the non-metastatic versus metastatic treatment setting. The addition of claims as a source of death information reduced the missingness of death records by more than half, and should imply the inclusion of claims-based mortality information as the new industry standard. The resulting 95.0% overall sensitivity matches or exceeds any reported in commercially available data.
Findings from this study show that death information that draws on all four reported sources: EHR, government, obituary and claims can provide an effective real-world research solution for evaluation of survival outcomes even in early-stage cancer populations. More generally, results of this validation study show that an accurate, trustworthy and reliable assessment of the gold-standard outcome of OS can be provided by high-quality RWD. However, as reported here, the addition of claims had a measurable impact on the completeness of the data, and on the assessment of OS. Investigators should therefore carefully consider the sourcing of death information in EHR-based real-world studies of OS during the planning phase of their research.
This study had several limitations and a number of strengths. First, patients represented in the data were treated predominantly in community oncology settings. We are not aware of evidence that the completeness or accuracy of death information differs for patients treated in academic settings or health systems. However, academic medical centers may have different data integration workflows, including more centralized EHR systems and registry infrastructure, while also serving as referral centers whose patients may receive care outside the health system. Therefore, mortality ascertainment may differ somewhat in these settings. Second, data regarding the completeness and accuracy of death information were limited to patients who received cancer care in 2014 or later and may not hold equally for patients who died in earlier years. Also, as noted, patients with certain early-stage cancers that are treated predominantly in non-oncology settings were either not included or were underrepresented. Third, although the NDI is a widely used reference standard for evaluating the completeness of mortality information [8,9,11,18,19], it is recognized as imperfectly complete [15]. To the extent that true death events identified by ASCME were not captured by NDI, the analysis understates the true specificity of ASCME [22], an effect that would be most evident among metastatic cohorts. Note, however, that the measured sensitivity of ASCME would be unaffected by this artifact. More generally, incomplete mortality ascertainment can affect survival analyses regardless of the mortality data source used. If patients who are lost to follow-up are at greater risk of death than patients who remain under observation, informative censoring may occur, and survival may be modestly overestimated. However, the high sensitivity of ASCME relative to NDI and the close agreement in survival estimates between the two mortality sources suggest that the practical impact of informative censoring is limited.
Key strengths of this validation study included its use of a large, geographically and demographically diverse population of patients across all major solid tumors. The study reported ASCME data completeness validation metrics broken out by demographic characteristics and by tumor and metastatic versus non-metastatic treatment setting. Finally, the study examined the accuracy of the date of death as measured by concordance, but also as measured practically by the difference in OS outcomes.
Conclusion
This study demonstrated the validity of the ASCME mortality indicator used in ConcertAI’s RWD. Results reported here show ASCME to have excellent completeness and accuracy, and to produce OS estimates in metastatic cohorts that match NDI-based estimates within 5 days on average. ASCME’s use of claims as an additional source of death information was pivotal, reducing the remaining proportion of unidentified deaths by more than half, and suggesting that inclusion of claims in composite death indicators should be considered the new industry standard. Finally, this study demonstrates that real-world oncology data are an appropriate source for evaluating OS and allow for accurate assessment of outcomes in patients with early and advanced disease.
Summary points
•
Overall survival (OS) is a key endpoint of interest in real-world oncology studies, but requires a reliable death indicator.
•
This study validated an All-Source Composite Mortality Endpoint (ASCME) drawn from electronic health records, obituary, government, and claims data, against the National Death Index (NDI).
•
The sample included 32,358 cancer patients treated from 1 January 2014 to 31 December 2022 across ten solid tumor datasets.
•
The study assessed ASCME’s sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and 5-day date-of-death concordance compared with NDI.
•
The study also compared median OS according to ASCME versus NDI.
•
ASCME showed an overall 95.0% sensitivity, with claims contributing 5.2%, a specificity of 96.5% and PPV and NPV of 95.8% each.
•
Sensitivity ranged from 92.7% to 97.8% across clinical cohorts.
•
ASCME’s 5-day concordance with NDI was 97.9%.
•
Median OS estimates differed by 12.2 days among non-metastatic cohorts, and 4.5 days among metastatic cohorts.
•
Findings show that ASCME provides high completeness and accuracy and can support OS analysis in metastatic and non-metastatic cancer populations.
Author contributions
Y Natanzon was involved in conceptualization, methodology, project administration, supervision, validation, writing original draft, writing review and editing. L Slipski was involved in methodology, formal analysis, validation, writing review and editing. MS Walker was involved in methodology, supervision, validation, writing original draft, writing review and editing.
Acknowledgments
The authors thank Amanda Johns (ConcertAI, LLC) for assisting with manuscript preparation, and Jessica Ken (ConcertAI, LLC) for her support of analytic work on the study. Part of this research was presented as a poster at ISPOR Europe 2024.
Financial disclosure
This work was supported by ConcertAI, LLC (MA, USA). ConcertAI, LLC, provided financial support for the conduct of the research and preparation of the article.
Competing interests disclosure
Y Natanzon and L Slipski were employed by ConcertAI, LLC during the conduct of this research. MS Walker is a paid consultant for ConcertAI, LLC. The authors have no other competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript apart from those disclosed.
Writing disclosure
No funded writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
The study was determined by the Advarra Institutional Review Board, Columbia (MA, USA), to be exempt as secondary research for which consent is not required.
Data sharing statement
The data that support the findings of this study are available from ConcertAI, LLC but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data may however be available from ConcertAI, LLC upon reasonable request (https://www.concertai.com/contact-us/).
Data transparency statement
The study protocol was prespecified, but it is not publicly available. The study was not preregistered, because it was not a real-world comparative effectiveness study, but rather a validation study of a real-world effectiveness endpoint. Restrictions apply to the availability of the underlying study data, which were used under license for the current study, and so are not publicly available. The analytical code is limited because the study was entirely descriptive and is not sharable because the code operated within the restricted data environment. Some elements of the STROBE and RECORD guidelines were not applicable because the study is descriptive/validation rather than inferential/outcomes. However, the study broadly adheres to the applicable elements of STROBE and RECORD.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
Supplementary Material
File (supplementary materials.docx)
- Download
- 4.90 MB
References
Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Friends of Cancer Research. Establishing a framework to evaluate real-world endpoints. Friends of Cancer Research, DC, USA (2018). https://friendsofcancerresearch.org/wp-content/uploads/RWE_FINAL-7.6.18_1.pdf
• This multi-dataset pilot demonstrates the feasibility of deriving robust real-world endpoints for assessing outcomes in advanced lung cancer.
2.
Purna Singh A, Shahapur PR, Vadakedath S et al. Research question, objectives, and endpoints in clinical and oncological research: a comprehensive review. Cureus 14(9), e29575 (2022).
• Provides a broad overview of research design, analytic approaches and endpoint selection in oncology clinical trials, offering context for interpreting real-world evidence.
3.
Levin MA, Lin HM, Prabhakar G, McCormick PJ, Egorova NN. Alive or dead: validity of the Social Security Administration Death Master File after 2011. Health Serv. Res. 54(1), 24–33 (2019).
4.
United States Congress. Bipartisan Budget Act of 2013, Public Law 113–167. US Government Printing Office, DC, USA (2013).
5.
ObituraryData.com. ObituraryData.com: The Nation's Leading Death Record Resource. https://www.obituarydata.com/default.asp
6.
Soowamber ML, Granton JT, Bavaghar-Zaeimi F, Johnson SR. Online obituaries are a reliable and valid source of mortality data. J. Clin. Epidemiol. 79, 167–168 (2016).
7.
HCUP National Inpatient Sample (NIS). Healthcare Cost and Utilization Project User Support. Agency for Healthcare Research and Quality, MD, USA (2025). https://hcup-us.ahrq.gov/db/vars/siddistnote.jsp?var=died&utm.com
8.
Jamal-Allial A, Sponholtz T, Vojjala SK et al. Validation of mortality data sources compared to the National Death Index in the Healthcare Integrated Research Database. Pragmat. Obs. Res. 16, 19–25 (2025).
• Validates a composite mortality indicator against the National Death Index (NDI) in an advanced cancer population, showing substantially improved sensitivity over individual real-world sources.
9.
Lerman MH, Holmes B, St Hilaire D et al. Validation of a mortality composite score in the real-world setting: overcoming source-specific disparities and biases. JCO Clin. Cancer Inform. 5, 401–413 (2021).
• Validates a composite mortality indicator against the NDI, with consistent performance across demographic and socioeconomic subgroups despite gaps in individual sources.
10.
Sohn MW, Arnold N, Maynard C, Hynes DM. Accuracy and completeness of mortality data in the Department of Veterans Affairs. Popul. Health Metr. 4, 2 (2006).
11.
Zhang Q, Gossai A, Monroe S, Nussbaum NC, Parrinello CM. Validation analysis of a composite real-world mortality endpoint for patients with cancer in the United States. Health Serv. Res. 56(6), 1281–1287 (2021).
12.
Slipski L, Dub B, Yu Y, Walker M, Natanzon Y. RWD23 Addition of open administrative claims significantly improves capture of mortality in electronic health record (EHR)-focused real-world data (RWD). Value Health 27(12), S576 (2024).
13.
Hsu WC, Crowley A, Parzynski CS. The impact of different censoring methods for analyzing survival using real-world data with linked mortality information: a simulation study. BMC Med. Res. Methodol. 24(1), 203 (2024).
14.
Centers for Disease Control and Prevention. National Death Index: user's guide (2021). https://www.cdc.gov/nchs/data/ndi/ndi_users_guide.pdf
•• NDI User’s Guide outlines NDI structure, linkage procedures, match criteria, and limitations and is the authoritative reference for US mortality verification.
15.
Jacobs EJ, Newton CC, Wang Y, Campbell PT, Flanders WD, Gapstur SM. Ghost-time bias from imperfect mortality ascertainment in aging cohorts. Ann. Epidemiol. 28(10), 691–696.e3 (2018).
16.
Calle EE, Terrell DD. Utility of the National Death Index for ascertainment of mortality among cancer prevention study II participants. Am. J. Epidemiol. 137(2), 235–241 (1993).
17.
Lash TL, Silliman RA. A comparison of the National Death Index and Social Security Administration databases to ascertain vital status. Epidemiology 12(2), 259–261 (2001).
18.
Curtis MD, Griffith SD, Tucker M et al. Development and validation of a high-quality composite real-world mortality endpoint. Health Serv. Res. 53(6), 4460–4476 (2018).
• Shows that a composite electronic health records (EHR)-based mortality indicator increases sensitivity across multiple cancers and aligns survival estimates more closely with NDI benchmarks.
19.
Dong S, Kansagra A, Kaur G et al. Validation of a composite real-world mortality variable among patients with hematologic malignancies treated in the United States. JCO Clin. Cancer Inform. 9, e2400233 (2025).
• Validates a multisource composite mortality variable in hematologic malignancies, showing high accuracy and strong date concordance compared with the NDI.
20.
National Center for Health Statistics. Centers for Disease Control and Prevention, U.S. Department of Health and Human Services. National Death Index Atlanta, GA, USA (2024). Available from: https://www.cdc.gov/nchs/ndi/index.html
21.
Gordis L. Epidemiology. (Fifth Edition). Elsevier, PA, USA (2014).
•• This is a foundational epidemiology text outlining core principles of study design, bias, confounding and causal inference – essential context for evaluating RWE studies.
22.
Walker M, Slipski L, Natanzon Y. Behavior of test specificity under an imperfect gold standard: findings from a simulation study and analysis of real-world oncology data. BMS Med. Res. Methodol. 25(1), 151 (2025).
Information & Authors
Information
Published In
Copyright
© 2026 ConcertAI, LLC. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 4 December 2025
Accepted: 23 July 2026
Published online: 31 August 2026
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Validation of an all-source composite mortality endpoint in the US population. (2026) Journal of Comparative Effectiveness Research. DOI: 10.57264/cer-2025-0198
Export citation
Select the citation format you wish to export for this article or chapter.