Considerations for emulations of randomized controlled trials using real-world data: learnings from an emulation of MONALEESA-2

This cohort study used secondary, de-identified, individual-level patient data. The MONALEESA-2 trial was selected because of several factors favoring its replication in RWD: use of an active control arm (letrozole) since placebo arms are difficult to mimic in RWD, outcome of overall survival (OS) which is objectively measurable in routine clinical practices unlike other oncology-specific outcomes such as progression, and a sufficiently historical US FDA approval date allowing for follow-up in the data given the expected data lag in RWD. This analysis was granted Institutional Review board exemption.

A data feasibility assessment was conducted to determine the most fit for purpose data for this analysis using the SPIFD2 framework [11]. The data feasibility was based on data manuals, data dictionaries, published validation analyses and discussions with data vendors. Patient-level data samples were not analyzed as part of the assessment. After comparing five US-based data sources, Optum’s de-identified Market Clarity Data (Optum^® Market Clarity) was selected on the basis of sample size, data elements available and administrative considerations such as cost and contracting timelines.

Optum^® Market Clarity contains linked data from claims and electronic health records (EHR) for over 1.8 million US patients. All patients within Optum^® Market Clarity have events from EHR and a subset have information from claims data. The database comprises enrollees of commercial health plans, Medicare, Medicaid and uninsured patients who received healthcare across all 50 states. EHR data are sourced from integrated delivery networks consisting of ambulatory only facilities and hospital networks and includes clinical data from laboratories. The EHR data contain drug information from prescriptions written or medications administered in the outpatient setting. A subset of patients has prescription fill information from external outpatient pharmacy dispensing records. This dataset contains a combination of structured and natural language processed unstructured data (e.g., provider notes) from the EHR. The database provides a comprehensive, multipayer view of the demographics, diagnoses, procedures performed during outpatient visits or inpatient stays and outpatient prescription records. Death information is provided at month-year granularity and is aggregated from the EHR, claims, obituary data sources, death master file and Centers for Medicare & Medicaid Services.

A subset of Optum^® Market Clarity patients with clinical and provider notes as well as other unstructured documents from large health systems suitable for extraction are included in oncology-specific data tables. The Optum enriched oncology dataset is a group of tables that can supplement the EHR dataset. It contains additional cancer-specific information on a subset of the EHR population that have at least one solid or nonsolid tumor diagnosis. In addition to structured data obtained from medical records, the enriched oncology tables provide additional clinical information extracted from provider notes using natural language processing (NLP) machine learning models (for example, tumor progression, histology, etc.). Variables obtained from NLP-derived fields included biomarker status and Eastern Cooperative Oncology Group (ECOG) performance status. Please see Supplementary Tables A1 & A2 for more information on each variable.

Optum oncology data initiatives include enriching the data by extracting essential information from the oncology patient’s medical records and making it usable for researchers. Specific oncology concepts important in understanding the progression of the disease are often not available in structured formats. The dataset framework of deriving valuable data elements from the unstructured data within EHRs are carried out by a collaborative effort of Optum’s data scientists, clinical experts and product executives.

For the quality controls around the enriched oncology dataset: Once completed, each concept within the enriched oncology dataset goes through a manual validation process on a brand-new sample of notes, and precision and recall scores associated with the NLP algorithms are calculated based on that sample (our minimum precision requirement is 80%). Once the concept meets this requirement, the NLP-derived data goes through the following curation steps before inclusion in the enriched oncology deliverable: concept normalization (transforming the raw values into more standardized options for ease of interpretation, while ensuring no loss of information or context), exclusion of data output with little clinical context or value, de-duplication of records that provide the same result on the same note date and creation of a data dictionary that includes lookup tables, includes value breakdowns for all variables.

First, an RWD population was identified in Optum^® Market Clarity. All trial inclusion and exclusion criteria were applied where possible (except those deemed to be related to safety outcomes based on clinical expertise) to identify a patient population reflecting that of the MONALEESA-2 trial (Figure 1). The detailed approach for operationalizing the eligibility criteria employed in MONALEESA-2 in RWD are described in Supplementary Tables A1 & A2.

Women were required to have evidence of advanced breast cancer defined as recurrent or metastatic breast cancer based on two claims-based algorithms [12,13] between 13 March 2017 (the approval date of ribociclib in the US) [14] and 30 September 2023 (the Optum^® Market Clarity data cut end date). The date of the first prescription for ribociclib or letrozole was assigned as the index date (see Exposure section below for more detail on capture of treatment status). Patients were required to be postmenopausal women at least 18 years old on the index date. Because of missingness, patients with no evidence of hormone receptor (HR)-negative histology were considered HR-positive (i.e., no HR test results at all or no HR-negative tests), and patients with no evidence of human epidermal growth factor receptor 2 (HER)-positive disease were considered HER-negative. Women were required to have an ECOG performance status of 0 or 1, which was operationalized as no evidence of ECOG greater than 1. The number of missing values for each variable is reported.

In addition to the eligibility criteria applied in MONALEESA-2, an observability criterion was applied to increase the accuracy of the measurements of baseline characteristics in both populations. Patients were required to have 180 days of continuous claims enrollment or EHR activity prior to and including the index date, allowing for 30-day gaps.

The exposure of interest was the new initiation of ribociclib in combination with letrozole, defined as a prescription written, medication administration, medical claim, or prescription claim for ribociclib and letrozole, within 60 days of one another, with no prior prescriptions/claims of either medication. The comparator group was patients who had at least one prescription/claim for letrozole and no prescriptions/claims for ribociclib in the following 60 days (referred to throughout this manuscript as ‘letrozole’). Patients were allowed to initiate other treatments following the index date, which is consistent with another MONALEESA-2 trial emulation [15]. The outcome of interest was OS.

Patient characteristics from the MONALEESA-2 trial population were identified from published study findings [16]. Demographic characteristics (age, race, region) are reported. Age was measured at index date while race and region are summarized as the most frequently recorded value in a patient’s records. Clinical characteristics including previous treatments and metastases based on diagnosis codes were measured in the 365 days prior to and including index date based on claims and EHR records. Some characteristics were measured in the RWD population despite not being captured in the trial to better describe a patient's disease severity.

To assess the generalizability of the trial participants, the impact of each inclusion/exclusion criterion was determined by the proportion of patients excluded at that point in the attrition table and by comparing the final sample size to the number of patients who initiated one of the exposures of interest. The distributions of demographic and clinical characteristics were compared between the ribociclib plus letrozole trial population and the RWD population using absolute standardized differences (ASDs). An ASD > 0.10 was considered a meaningful difference [17]. ASDs were also used to compare the RWD exposure and comparator groups.

To estimate the association between ribociclib plus letrozole treatment and OS, Cox proportional hazards models were fit comparing the ribociclib plus letrozole arm to the letrozole group. To control for confounding, inverse probability of treatment weighting (IPTW) and 1:1 propensity score matching were used. The following characteristics were included in the propensity score model: age, race, ethnicity, region, ECOG performance status, progesterone receptor (PgR) status, Elixhauser Comorbidity Score, evidence of ECG, prescription/claim for opioids and previous antineoplastic treatments. These variables were included to adjust for demographic differences, general health status and disease burden/characteristics which may be associated with treatment assignment and OS. For categorical variables with missing values, missing was considered a level of the variable (i.e., a missing indicator was created). Patient characteristics were considered balanced if the post-weighting/post-matching ASDs were <0.10 [17].

Following the initial findings, several post hoc analyses were undertaken to explore and contextualize the results. After noting the baseline differences between the ribociclib plus letrozole and letrozole groups, two additional analyses were taken: the creation of historical control groups and a subgroup analysis of patients with metastatic disease based on the presence of a diagnosis code for secondary malignancy. First, to evaluate the impact of using other potential comparator groups, two additional time frames were used to select letrozole patients: initiated letrozole from 1 January 2013 through 2 February 2015 (prior to the approval of palbociclib, the first cyclin-dependent kinase 4/6 [CDK4/6] inhibitor approved in the US; Historical Letrozole Group 1) and initiated letrozole from 3 February 2015 through 12 March 2017 (after palbociclib approval but prior to ribociclib approval; Historical Letrozole Group 2) (Figure 2). The goal of the historical controls was to replicate the treatment landscape prior to ribociclib approval.

Second, given the discrepancy between the proportion of patients with metastases in the two arms and because evidence of metastasis was not included in the propensity score model, a subgroup analysis of those with evidence of metastatic disease was conducted. The goal of the subgroup analysis was to adjust for uncontrolled confounding in disease stage, which would be associated with OS, between the comparator groups. Finally, to remove any potential immortal time bias resulting from patients in the ribociclib plus letrozole arm having to live long enough to receive two treatments compared with only requiring evidence of one treatment in the letrozole arm, the analysis was limited to patients with at least 60 days on follow-up with follow-up time starting on day 61 (ribociclib plus letrozole n = 97 and letrozole n = 3687).

There were 132,406 patients meeting the exposure or comparator group definitions during the study period. For transparency/replicability purposes, the numbers and proportions of patients excluded for each individual criterion presented in Figure 3 reflect the order in which the criteria were applied. A different order of application may result in different numbers and proportions; however, some trends are noteworthy. After excluding patients without sufficient baseline observability or previous use of the medications of interest, there were 55,189. A small number of patients were excluded for male or missing gender, or for being less than 18 years old. The application of the criteria for recurrent or metastatic breast cancer and postmenopausal status reduced the sample by approximately 70%, which was expected given the subsequent approvals for ribociclib during our study period [18]. Other exclusion criteria that resulted in a large reduction in sample size were prior systemic anti-cancer therapy in the 1 year prior to index date, diagnosis of another malignancy within 3 years prior to index date or on index date, active cardiac disease or history of cardiac dysfunction recorded in the 1 year prior to index date or on index date, and use of prohibited medications such as cytochrome P450 3A4 or 3A5 (CYP3A4/5) inducers or inhibitors and medications with a risk to prolong the QT interval. The final patient sample consisted of 3912 patients: 106 ribociclib plus letrozole patients and 3806 letrozole patients. Approximately one-quarter of ribociclib plus letrozole patients and 12% of patients treated with letrozole had another anticancer therapy within the 60 days after index date.

The baseline characteristics for MONALEESA-2 trial participants and patients in the RWD population after applying the full set of inclusion/exclusion are presented in Tables 2 & 3. There were differences in most characteristics between the trial patients and real-world patients who received ribociclib plus letrozole. The trial participants tended to be younger than RWD patients among whom more than 60% were over 65. While the distribution of race was similar when classifying patients/participants as Asian versus non-Asian, the ribociclib plus letrozole arm of the RWD population was 9.4% Black whereas only 2.5% of the MONALEESA-2 population was Black, a limitation acknowledged in the publication [16]. The distribution of US region could not be compared as the trial was multinational and only the national-level distribution was presented. While prior use of chemotherapy was similar between the two groups, prior use of tamoxifen was more common among trial participants, while nonsteroidal aromatase inhibitor use was more common among RWD patients. Larger proportions of patients in the trial were noted as having liver or lung involvement than in the RWD population. The proportion of patients with bone lesions only approximately 20% among trial participants; whereas nearly 38% of RWD ribociclib plus letrozole patients had a diagnosis of bone metastasis without diagnosis of liver, lung, or lymph node metastasis. The proportion of trial patients with new metastatic disease was lower than the proportion of patients with a diagnosis code for any metastatic disease (new or previously diagnosed) within the prior year. Large amounts of missing data in the RWD population precluded valid comparisons on ECOG, PgR status and hormone receptor status.

While age was similar between the two treatment groups in RWD population, there were differences noted in race and US Census region: a larger proportion of ribociclib plus letrozole patients were Asian and a larger proportion were from the Midwest compared with letrozole patients (Table 1). There were also differences noted in the index year with smaller proportions of ribociclib plus letrozole patients entering the analysis in 2020 and 2021. Ribociclib plus letrozole patients had a larger comorbid disease burden as evidenced by a higher mean comorbidity score (Table 2). There were smaller proportions of patients with previous chemotherapy, surgery, and radiation compared with letrozole patients. A diagnosis code indicating metastatic disease in the past year was nearly twice as common among patients treated with ribociclib plus letrozole than patients treated with letrozole (85 vs 45%).

Comparisons of key characteristics between the ribociclib plus letrozole group and the three letrozole comparator groups are shown in Table 3. Surgery and radiation therapy were consistently more common in all three letrozole groups compared with the ribociclib plus letrozole group. A diagnosis of any metastasis (secondary nonbreast malignant neoplasm) was more common among ribociclib plus letrozole group compared with all three of the letrozole comparator groups; however, the proportion of women in the letrozole comparator group with any metastasis decreased over time, with the highest proportion of women with any metastases among Historical Letrozole Group 1. The same trend was observed for bone metastasis. Characteristics after weighting and matching, along with ASDs, are shown in Supplementary Tables A3 & A4 for primary comparison of ribociclib plus letrozole versus the contemporaneous letrozole group and the comparison within the subgroup of women with a secondary nonbreast malignancy. In the primary comparison, most characteristics were considered balanced with ASDs < 0.1; however, it is notable that prior treatments, comorbidity score, and diagnosis of secondary nonbreast malignancy were not balanced. In part, this prompted the subgroup analysis where we still found imbalances in certain disease-related characteristics such as prior surgical intervention and type of secondary malignancy. Characteristics for the historical letrozole groups after weighting and matching are shown in Supplementary Tables A5 & A6 and similar trends were noted in terms of imbalances.

In the MONALEESA-2 trial, a significant survival benefit was found for women treated with ribociclib plus letrozole compared with letrozole plus placebo (hazard ratio [HR] = 0.76, 95% CI: 0.63–0.93). However, in the RWD emulation, treatment with ribociclib plus letrozole was associated with an increased risk of all-cause mortality with a HR of 3.07 (95% CI: 1.89–5.01) in the crude analysis (Table 4). The HR moved closer to the null after adjusting for baseline characteristics but was still above 1.0. The HRs were consistently >1.0 across all post hoc analyses. Kaplan-Meier curves from the primary comparisons of ribociclib plus letrozole versus contemporaneous and historical letrozole groups can be found in Supplementary Figures A1–A6. There is evidence that the assumptions are not met as we see divergence of the curves as time progresses. In the MONALEESA-2 trial, a divergence of the curves between ribociclib plus letrozole and letrozole plus placebo was also seen at around 2 years where ribociclib plus letrozole begins to look protective against all-cause mortality.

One challenge in emulations is identifying the most appropriate data sources, a process in which researchers must weigh the benefits and limitations of available data to arrive at the most fit-for-purpose dataset for their research question. Considerations include availability and completeness of clinical variables, recency of data and cost of acquiring the data source [11]. Often data feasibility assessments rely on available meta-data such as data dictionaries because it is not always possible to obtain sample data or detailed counts/explorations from data vendors. This was the case in this analysis and therefore, the level of missingness for key variables was not understood until this data source was already selected. This is a major challenge in RWD when purchasing commercialized datasets. In the event more detailed data explorations are possible, they should be conducted to determine the distributions and amount of missingness in key variables.

The outcome of interest in the emulation was OS. OS is a commonly used outcome in trial emulations because tumor response outcomes are not assessed in routine clinical practice as they are in trials. Work is ongoing to develop algorithms to capture these other outcomes in EHR and other sources of RWD [19]. In this analysis, we observed an absolute probability of death of 7%, which was much lower than expected based on the trial (absolute probability of death of 54% in the ribociclib arm and 66% in the placebo group). This could be due in part to incomplete follow-up and capture of death in the sources included in Optum^® Market Clarity. Complete capture of death is challenging in RWD sources like claims and EHR, especially deaths occurring outside of a hospital setting. While there are data sources with more complete death information, there are trade-offs when utilizing such data sources. For example, in the US, the National Death Index is considered the gold standard for measuring mortality; however, there is a long data lag and a time-consuming process to obtain and link the data to other real-world sources with the relevant study variables. To our knowledge, there are no publicly available estimates of the completeness of mortality information in Optum^® Market Clarity.

Because a small number of the patients included in this analysis had information in the oncology-specific tables created using NLP, we relied heavily on the EHR and claims information to operationalize the trial inclusion/exclusion criteria and patient characteristics. For example, the trial included only patients with advanced breast cancer defined as those with “locoregionally recurrent or metastatic breast cancer not amenable to curative therapy”. In our RWD source, detailed staging information was not available; therefore, we included patients who met one of two accepted claims-based algorithms. The first required at least two records with a breast cancer diagnosis at least 30 days apart followed by at least two records with a diagnosis for secondary nonbreast malignancy at least 30 days apart to define patients with metastatic disease [12] while the second defined recurrent disease as meeting at least one of the following criteria: at least 1 record for a secondary nonbreast malignancy greater than 180 days after a diagnosis of primary breast cancer, a mastectomy greater than 180 days after a diagnosis of primary breast cancer or radiation therapy after a diagnosis of primary breast cancer [13]. Therefore, it is possible we captured a slightly different patient population than the trial population. Similarly, we noted that a large number of real-world patients were excluded from our sample based on having prior systemic anticancer therapy in the prior year to initiating ribociclib plus letrozole or letrozole. The trial excluded patients with prior systemic anti-cancer therapy for advanced breast cancer, except for (neo) adjuvant therapy and <=14 days of letrozole or anastrozole treatment. Given the complexities of identifying adjuvant therapy in this RWD source, our exclusion likely removed patients who would have been trial-eligible.

We also noted high levels of missingness in biomarkers and ECOG performance status. In analyses of treated patients, lack of information on biomarker status may be less impactful given the medication's indication. In this case, ribociclib plus letrozole is only approved for HR-positive, HER-negative cancer. With the exception of potential off-label use, we are fairly confident of the biomarker status of women treated with ribociclib but far less so for the women treated with letrozole without evidence of ribociclib treatment. For performance status, only 10% of ribociclib plus letrozole patients had ECOG recorded in the +/-3 months around treatment initiation. ECOG is an important predictor of survival [20] and therefore, the substantial level of missingness in this analysis is a key limitation. Since medical record data are captured during clinical practice, they are less complete than clinical trial data regarding factors such as performance status. This highlights the importance of exploratory analyses during the feasibility stage to confirm the extent and pattern of missingness of key variables in the patient population of interest (which may be lower or higher than the general missingness reported for the data source as a whole). While some RWD sources with more complete notes or those that are oncology-specific may have better capture of ECOG status if providers record that information, real-world analyses conducted in claims data sources need to rely entirely on algorithms to predict performance status [21].

The primary rationale for conducting this analysis was to determine if there was an impact to the HR for OS by relaxing certain trial inclusion/exclusion criteria that may be overly restrictive. Therefore, we identified patients for cohort entry at the time of drug initiation for follow-up and capture of death. With this study design, we were able to compare the patients who received ribociclib plus letrozole in the real world to those in the trial in terms of a range of characteristics, including those that were not trial entry criteria. However, we were not able to adequately answer other questions about the generalizability of the trial population to the broader group of real-world patients who would be eligible for a trial or describe what types of patients were excluded. To address this question, patients would need to be selected first based on evidence of advanced HR-positive, HER2-negative breast cancer diagnosis, rather than treatment with ribociclib. Within that group of patients, we could have assessed the prevalence of conditions that were used to include/exclude patients from the trial.

For the design of the emulation itself, the exposure group was assigned based on the evaluation of all prescriptions/claims on or within 60 days of the first evidence of ribociclib or letrozole. In studies where the primary inclusion criterion includes multiple elements (such as initiating both ribociclib and letrozole), researchers should consider immortal time bias. Immortal time is a period of follow-up where the outcome of the study cannot occur because of the exposure definition [22]. In this case, patients who initiated both ribociclib and letrozole needed to live long enough to initiate both therapies. This could result in longer follow-up times for ribociclib plus letrozole patients, biasing the HR toward the null and potentially below 1. While we do not believe immortal time bias impacted our primary analysis because of the direction and magnitude of the HR, we conducted a sensitivity analysis requiring all patients were alive on day 60 after index date and began follow-up on day 61. The HRs after weighting and matching were similar to the primary analysis.

The ability of methods like IPTW and propensity score to control for confounding is determined by the accuracy in which confounding variables are measured and the extent to which all possible confounding variables are measured at all. The lack of availability of clinical information likely resulted in unmeasured confounding. Assuming the HR from the MONALEESA-2 trial of 0.76 is the true causal effect and our HR of 3.07 from the primary model, we calculated an E-value of 7.5 [23,24]. This indicates that unmeasured confounders would need to have a strength of association of at least 7.5 with both treatment with ribociclib plus letrozole and with OS to result in our HR. This seems plausible given the importance of several variables that were either missing or incompletely captured in this data source. One piece of evidence supporting the presence of unmeasured confounding is that after limiting the RWD population to those with a diagnosis of secondary nonbreast malignancy, the HRs were closed to the null, indicating there was uncontrolled confounding in the overall population. Following weighting and matching, balance was not achieved on all characteristics and these differences persisted within the subgroup analysis. While these variables could have further been adjusted for in multivariable regression, given the data source limitations described above, this step was not pursued. Additional useful information includes disease stage and provider and patient preferences.

In this analysis, letrozole without ribociclib was the comparator group, serving as the proxy for letrozole plus placebo. However, the approval of an effective therapy can dramatically change the landscape of treatments available to patients. This can make identifying a valid comparator group difficult. There are currently three CDK4/6 inhibitors approved for HR+, HER2- breast cancer: palbociclib (IBRANCE) approved in 2015 [25], ribociclib, and abemaciclib (Verzenio) approved in 2017 [26]. The approvals of these drugs dramatically altered the treatment landscape for advanced breast cancer and were considered “game-changers”, resulting in changes to the National Comprehensive Cancer Network treatment guidelines [27]. Patients who may be eligible for but do not receive effective therapies after their approval may not be an appropriate comparator to mimic the randomization in an RCT.

We hypothesize that the approval and introduction of these highly effective therapies for women with advanced HR-positive, HER2-negative breast cancer resulted in the substantial differences in the characteristics of RWD patients who received ribociclib plus letrozole and RWD patients who received letrozole. Differences were found in region, index year, comorbidities, previous treatments and diagnoses of metastatic disease. Differences in index year may be explained by the impact of the COVID-19 pandemic on healthcare patterns and comfort of providers in initiating certain therapies without the same level of monitoring as occurred earlier in the pandemic. Metastatic disease was more common among ribociclib plus letrozole patients while treatment with chemotherapies and other modalities such as surgery or radiation were less common. This may indicate that patients with more advanced disease stage and poorer prognosis received ribociclib plus letrozole. In attempt to account for this, we create two historical control groups where patients were selected prior to the approval of any CDK4/6 inhibitors (Historical Letrozole Group 1) and prior to ribociclib (Historical Letrozole Group 2). While the historical control groups were more similar to the ribociclib plus letrozole patients than the contemporaneous letrozole patients on some characteristics, there were still important differences in the prevalence of metastatic disease burden.

Despite the challenges, trial emulation is still important for understanding best practices for robust real-world study design and may be feasible in certain circumstances. In the context of this work, the unexpected results of this analysis demonstrate the importance of conducting feasibility analyses and descriptive studies of the baseline characteristics and treatment patterns of the real-world indicated population prior to doing trial emulation. A more thorough understanding of utilization of ribociclib in the real world would have potentially led to different study design decisions or possibly the conclusion that an emulation was not possible. First, our sample size was impacted by the subsequent expanded approval of ribociclib in pre- and peri-menopausal women. The total number of ribociclib patients in initial sample counts with a prescription/claim for ribociclib included those women as well as post-menopausal women who would have been eligible for MONALEESA-2. Second, the median follow-up time within the RWD population overall was substantially shorter than the trial population (27 vs 80 months) and was only 15 months for patients treated with ribociclib plus letrozole. The Kaplan Meier curve included in the MONALEESA-2 publication shows that the OS benefit of ribociclib began to emerge at approximately 20 months and continued to increase with longer follow-up, as indicated by survival at 5 years and 6 years [16]. This makes it unlikely that we would have found the same HR as the trial even with better balance between the exposure groups on confounders and complete death capture. Third, the duration of therapy was also much shorter in the real world than in the trial: a median of 6 months for the ribociclib plus letrozole group and 5 months for letrozole group compared with 20 months in the RCT. With a shorter duration of exposure in the emulation, patients may not have had enough time to experience the full therapeutic effects of the treatments. Information about sample size, follow-up, and duration of therapy should be obtained through descriptive analyses to provide insights before designing a trial emulation. Outside of trial emulations, there are many other relevant and important research questions that can be answered in RWD. Other important questions that can be answered in RWD include characterizing the patient population receiving ribociclib, describing treatment patterns of women with advanced breast cancer prior to and following the approval of ribociclib, understanding healthcare utilization and costs among women treated with ribociclib and characterizing the rates of safety events. Continuous evidence generation after approval asking new relevant questions with RWD is needed.