Bridging the gap between oncology clinical trials and real-world data: evidence on replicability of efficacy results using German claims data

Of 46,621 patients with ≥1 inpatient or confirmed outpatient diagnosis of PC in the inclusion period, 8918 had at least one inpatient or outpatient diagnosis of a metastasis within 365 days from a PC diagnosis (Figure 1). Of these, 4408 had at least one prescription or inpatient administration of docetaxel or ADT agent after first diagnosis of a metastasis and within the inclusion period. After implementation of the inclusion and exclusion criteria based on the CHAARTED trial criteria, the authors included 441 patients (median follow-up: 22.3 months). 75 patients were included in the docetaxel+ADT group (median age: 67 years) and 366 patients in the ADT mono group (median age: 76 years).

Comparing the baseline characteristics of patients in the two groups (Table 1) showed that mean age was significantly lower in the docetaxel+ADT group (66.5 years) compared with the ADT mono group (74.4 years; p < 0.001). In addition, patients in the docetaxel+ADT group had on average a lower number of outpatient visits during baseline (6.8) compared with patients in the ADT mono group (7.4; p = 0.04). Patients in the two groups did not differ significantly in terms of other baseline characteristics.

When doing an unadjusted comparison of the two unmatched groups in terms of OS (Table 2), median survival was not reached in the docetaxel+ADT RWD group, and it was 35.2 months in the ADT mono RWD group, with an HR of 0.68 (95% CI: 0.45–1.03) (Table 2 & Figure 2). In comparison (Table 2), the CHAARTED trial included younger patients (median age: 64 years), and at a median follow-up of 53.7 months, the median OS was 57.6 months for the docetaxel+ADT arm versus 47.2 months for the ADT mono arm (HR: 0.72; 95% CI: 0.59–0.89).

After estimating the PS through a logistic regression (c-statistic = 0.8), patients in the two groups were matched using a 1:1 nearest-neighbor approach, resulting in two equally sized matched cohorts of 74 patients each. Patients in the matched cohorts did not differ in any of the baseline characteristics (Table 1), and the mean and median age in the docetaxel+ADT group (66.7 and 67 years, respectively) matched the mean and median age in the ADT mono group (66.1 and 65 years). In comparison, the median age of the CHAARTED trial was 64 years (Table 3). In the matched RWD cohorts (Table 3), the median survival was not reached in the docetaxel+ADT RWD group, and it was 37.2 months in the ADT mono RWD group. The HR was 0.71 (95% CI: 0.42–1.19) (Figure 3 & Table 3). In comparison (Table 3), the median follow-up in the CHAARTED trial was 53.7 months, and the median OS was 57.6 months for the docetaxel+ADT arm versus 47.2 months for the ADT mono arm (HR: 0.72; 95% CI: 0.59–0.89).

With the mandate of the 21st Century Cures Act, the US FDA established the RWE Program to foster the scientific evaluation of RWD in support of regulatory approvals and post-approval safety studies [16,17]. In Europe, the EMA formed the HMA/EMA Joint Big Data Task Force to establish a roadmap for the use of RWD in the regulatory assessment of drugs [18]. The framework for the FDA's RWE Program suggests that the replication of RCTs using RWD may provide insights into the opportunities and limitations of observational studies [17].

This work investigated the replicability of an oncology trial in mHSPC using German claims data. After implementation of inclusion and exclusion criteria closely aligned to the trial protocol, the real-world population included in the study was significantly older and had a shorter median OS compared with the trial population. However, the HR was concordant with the CHAARTED trial and, after adjusting for baseline characteristics with PSM, the point estimate from the RWD (0.71) was remarkably similar to the estimate in the trial (0.72). The results were robust to different types of adjustments (multivariate adjustment and IPSW) for baseline confounders, but PSM achieved the closest point estimate of the efficacy estimate in the trial. This is partially explained by the better balance in observable covariates achieved by PSM compared with IPSW and specifically the inability of weighting to balance cohorts in terms of age, with the weighted age in the treated group being significantly lower than the weighted age in the comparator group (p = 0.011). While in general PSM may result in greater unbalance compared with other methods [19,20], prior studies have shown that PSM can perform better than IPSW [21,22].

Prior studies focused primarily on electronic health records (EHRs) and registry data [9,23]. Compared with claims, EHRs and registries are more likely to provide clinical data and important confounders and thus represent a better source for comparative effectiveness research [1]. The current findings further contribute to the literature, providing an example on the replicability of RCTs in oncology using claims data, which are particularly disadvantageous for this purpose.

The existing literature reached mixed conclusions regarding the replicability of oncology trials using RWD, with some studies achieving the replicability of RCT results and other studies identifying significant discrepancies. The present results are in line with prior studies [7,24,25] that found largely similar results and sizes of treatment effects between observational studies and RCTs and an association between the results of non-randomized versus randomized studies.

In contrast, a systematic MEDLINE search by Soni et al. [10] found that only 62% of the observational study HRs fell within the 95% CIs of the randomized trials. Kumar et al. [9] examined 141 RCTs covering eight tumor types. For each trial, they performed their own observational study using the National Cancer Database registry. Propensity-matched HRs for OS fell outside the trial CI 36% of the time. However, Kumar et al. included RCT for surgical procedures versus medical treatment, placebo control trials and phase II trials, which may be more difficult to replicate in the real world [1,12] and for which there might be good reasons for different effect estimates other than bias. Like Kumar et al., this analysis found that confounding adjustment through PS can reduce bias and help replicate trial results.

The authors acknowledge that this analysis has some limitations. First, due to the stringency of the inclusion criteria adopted, the analytical sample in claims data was small, which reduced statistical power and led to an estimate non-significant at the 5% significance level. However, the proximity of the point estimate to the point estimate in the trial suggests that these results are concordant with the trial and are less precise merely due to sample size. Second, claims data do not provide important prognostic variables such as ECOG score, blood test results and PSA levels. Despite potential residual bias from unobservable confounders, after matching the authors were able to replicate RCT estimates of the treatment effect. These findings need further validation with additional data sources and larger sample sizes, as well as within different indications and treatments. If validated, these results suggest that after adjusting for observable confounders, claims data have the potential to emulate relative efficacy for OS from trial results, despite residual systematic differences between trial and real-world populations. In contrast, the lack of important prognostic factors impacts the estimate of absolute OS. Thus, these results confirm that the use of RWD to directly extrapolate absolute survival is not a proper endeavor. The use of claims for synthetic control arms in oncology is also discouraged based on these findings, while the use of RWD is encouraged to estimate relative efficacy in OS within Bayesian hierarchical network meta-analyses and to measure effectiveness in outcome-based reimbursement agreements. The results of this analysis indicate that using early comparative evidence from RWD sources for regulatory and HTA decision making it useful.

Conception and design of the study: all; acquisition of the data: M Ghiani; analysis of the data: M Ghiani; interpretation of the data: all; drafting of the article: M Ghiani; review and finalization of the article: all. All authors have made substantial contributions to the study and writing of the article and have given their final approval of the version to be submitted.

M Ghiani is a staff member of IPAM. U Maywald is an employee of AOK PLUS. T Wilke and B Heeg have received honoraria from several pharmaceutical/consultancy firms. No funding was received to conduct this research. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.