Comparing prospectively assigned trial and real-world lung cancer patients

In this study, we compare probable RCT patients with non-contemporaneous RW patients. Our reference RCT is “Nivolumab versus docetaxel in advanced squamous-cell non-small-cell lung cancer” [11]. In this phase III trial, patients diagnosed with advanced (stage IIIB or IV) squamous-cell NSCLC who have disease progression during or after first-line platinum-containing chemotherapy were assigned to the intervention drug, nivolumab (n = 135), or the standard of care control, docetaxel (n = 137). The study reported that median overall survival (OS), the primary study end point, was 9.2 months for nivolumab (95% confidence interval [CI]: 7.3 to 13.3) versus 6.0 months (95% CI: 5.1 to 7.3) with docetaxel. Nivolumab – first approved and made commercially available on 22 December 2014 to treat unresectable or metastatic melanoma [12] – was approved for use in NSCLC on 4 March 2015 [13].

We used structured patient-level data sourced from electronic medical records (EMR) through ConcertAI, LLC. As with the reference trial, OS was the primary study end point. EMR-based OS has been evaluated against multiple external sources of mortality and, in the context of this study, was measured in the same way between the cohorts [14]. We defined OS as time from treatment initiation (which was assumed to be equivalent to randomization for the presumed trial patients) until death. Patients were censored at 24 months after treatment initiation, the index date in this study.

Given that nivolumab was first commercially available on 22 December 2014, we identified patients who were treated with nivolumab whose primary diagnosis was NSCLC prior to this date in the EMR dataset. The sample size for the probable CT cohort is 49. While we do not have flags in the data specifically denoting trial participation, these patients represent highly probable patients from the reference trial given that all 49 satisfied trial inclusion criteria observable in the data and that these patients were treated prior to commercial availability [11]. The RECIST 1.1 criteria and the availability of tumor samples for biomarker evaluation were not available and the lone trial criteria not included. We then identified NSCLC patients who started a nivolumab monotherapy regimen in the RW since approval (e.g., since 4 March 2015). Mirroring the reference trial inclusion criteria, we further only included patients with stage IIIB or IV disease aged 33 to 85 (e.g., the range from the final trial sample; though, the youngest actual patient meeting this criterion was 39) with Eastern Cooperative Oncology Group (ECOG) 0 or 1, squamous histology, treated with a platinum-based therapy in the first-line setting (i.e., cisplatin, carboplatin, oxaliplatin, nedaplatin and lobaplatin) and treated in the second-line setting resulting in a final RW sample of n = 160. All patients in the probable CT sample were treated before 2015 while the RW comparison group was comprised of the following treatment year distributions: 2015 (n = 49), 2016 (n = 62), 2017 (n = 46) and 2018 (n = 10).

We compared the RW patients (e.g., our treatment group) with the probable trial patients (e.g., our control group) to determine if there were statistically significant differences in OS between the groups. Following the methods utilized in the reference trial, OS was analyzed using a two-sided log-rank test. The HRs were estimated using a Cox proportional-hazards model with different matching methods. KM curves estimate the OS as well, accounting for the various weighting methodology. Details regarding the matching and propensity scores (PS) are available in the next section. Each regression estimate was calculated with a corresponding 95% CI. We adopted an alpha of 0.05 for the purpose of assessing statistical significance.

We assessed the differences in covariate distributions between the two groups using ANOVA – Equal Variances for continuous variables and Chi-square tests for categorical variables. The covariates assessed included: age at index in years as well as in categorical terms (i.e., under 65, 65 to under 75 and 75 and older), brain metastasis, age category with and without brain metastases, smoking status for current or former smokers versus never or undocumented, gender, gender interacted with smoking status, race (Black or African–American, White, or other), stage at index date (IIIB vs IV), practice region (American South vs other regions) and time from completion of most recent prior systemic regimen (less than 3 months vs 3 or more months).

We then conducted multiple matching methods to assess their relative performance in achieving balance and ultimate concordance between probable trial and RW populations. These include: Cox proportional-hazards model weighted by inverse probability of treatment weighting (IPTW), adjusting for PS in Cox proportional-hazards model, Cox proportional-hazards model stratified on PS deciles, PS 1:4 greedy nearest neighbor matching with caliper of 0.5 and directly adjusting for all covariates in Cox proportional-hazards model. We balanced on the following variables: age at index, brain metastasis, gender, race, smoking status and two interaction items – age with brain metastasis and gender with smoking status (Table 1). The variables were selected for their clinical relevance, and the interaction variables were included to improve the fit of the model.

The unadjusted covariate means were statistically significantly different at the 95% confidence level for two (notably, related) of the thirteen evaluated covariates: smoking (RWD n [%] = 145 [90.6%]; probable CT n [%] = 28 [57.1%]; omnibus Chi-square p-value < 0.0001) and gender interacted with smoking (as an example: RWD n (%) male current or former smoker = 100 (62.5%); probable CT n (%) = 16 (32.7%); omnibus Chi-square p-value < 0.0001). Following the greedy match procedure, while statistically closer in distributions, the distributions remained statistically significantly different for smoking (omnibus Chi-square p = 0.0010) and its interaction with gender (omnibus Chi-square p = 0.0100) (Table 1).

Mean (Standard deviation [SD]) OS was 12.2 (0.7) for the RWD arm and 12.5 (1.3) for the probable CT arm with a Log rank p-value equal to 0.4771. The median OS (95% CIs) were 10.2 (8.1–12.4) and 11.3 (5.9 to 17.0) and overlap (Table 2). The HRs generated under all six assessed matching methods (including without adjustment), using the probable CT arm as the reference group, were not statistically significantly different at the 95% confidence level: Unadjusted HR: 0.880 (95% CI: 0.613–1.263), IPTW HR: 0.851 (95% CI: 0.591–1.224), PS HR: 0.743 (95% CI: 0.486–1.137), PS Decile HR: 0.920 (95% CI: 0.265–3.191), Greedy Match HR: 0.804 (95% CI: 0.502–1.289), and Multivariate HR: 0.739 (95% CI: 0.483–1.132) (Figure 1 & Table 3 for more detail). The similarity of these survival patterns is reflected in the KM OS curves that were generated for the unadjusted comparison and for the greedy match, IPTW and IPTW adjusted models (Figure 2).

Our study has several limitations, starting with our CT cohort. Since we did not have direct access to this group in its entirety, we instead identified probable CT patients based on their pre-market access to nivolumab in our database. It is possible that some of these patients received care through other pre-market channels such as compassionate use via expanded access programs or Right to Try [15]. We utilized inclusion and exclusion criteria based on the CT's criteria, and 30 patients were excluded (21 due to failing criteria and 9 due to missing variables necessary to determine inclusion). While we believe that those included are likely to have received pre-market nivolumab through the CT, we cannot completely rule out that some in-sample patients did not participate in the reference CT. Relatedly, having only 49 cases in one of the groups challenges our PS modelling and highlights the potential difficulties of this approach with few observations, limits or risks overfitting the classification model and raises the possibility of our overall sample of 209 being insufficiently powered.

Though likely drawn from it, this sample is also likely non-representative of the underlying CT population whose trial-reported median OS duration is roughly two months shorter than our CT sub-population (though, the 95% CIs of these populations overlap). This potential upward bias in median OS among our probable CT population may be reflected in our results comparing survival against the RW population where, qualitatively speaking as none were statistically significant, all HRs for the RW population were less than one. We similarly find that our probable CT population smoked less and was more likely to be female than the overall published CT population. Relatedly, while both probable trial and RW patient data were generated from the same EMR environment, the probable CT patients were likely managed under CT settings that are generally more tightly controlled and monitored than RW settings. This could result in different patient management and care patterns between groups stemming from differences in the clinical environment rather than differences in the underlying clinical benefits stemming from the assigned treatment.

Finally, our aim in this study was to evaluate the comparability between probable CT patients and RW patients treated with the same therapy. This focuses our attention on evaluating null results and on discussing the limits of one's ability to reject a null hypothesis. Were we able to compare the relative performance of a probable CT treatment arm against either a probable CT control arm or a RW control arm, we would be discussing statistically significant differences instead of nonsignificant results. However, this was not possible in this setting as we were unable to identify probable CT control arm cohorts given that the standard of care control arm therapy, docetaxel, was widely commercially available during our study timeframe.