Replication of randomized clinical trial results using real-world data: paving the way for effectiveness decisions

Despite efforts to approximate and apply the eligibility criteria from the RCT, important differences may remain between patient populations in the target RCT and the observational study. First, it can be challenging to apply trial eligibility criteria to real-world data sources and criteria that involve clinical data or physiological measurements will not be measurable in administrative claims data, or even in electronic medical records [18]. Even after applying similar eligibility criteria, there may be differences between the RCT and real-world populations in the underlying distributions of baseline patient characteristics, such as age or prevalence of comorbid conditions. RCT DUPLICATE researchers made the decision not to reweight the real-world populations to match the reference RCT population in primary analyses; however, this approach may be explored in sensitivity analyses. It is possible that the patient populations from which the real-world study samples are drawn (e.g., Medicare and commercial claims databases) may have different baseline risks for the primary outcome compared with the RCT patients or even to each other, as observed in a recent database study predicting the results of the CAROLINA trial [19]. The distributions of treatment effect modifiers also might differ between the RCT and real-world population, or there may be heterogeneity of treatment effect based on population variables that may be unknown or unaccounted for in each dataset. Finally, geographic differences need to be considered. Most pivotal, registrational RCTs are global studies and patients from the USA represent a subset of the overall study population. Replication efforts based on US data only should conduct sensitivity analyses to compare results to the treatment effect estimates for US trial patient subgroups where possible.

Differences in end point measurement also may impact agreement. The outcome definitions and severity of events that are captured in RCTs may differ from those used in observational studies based on secondary data, meaning that the end points could represent somewhat different concepts. Claims-based studies require rules or algorithms to identify cohorts, exposures and outcomes events. This speaks to the desirability of using validated algorithms with known operating characteristics in RWE studies. The capture of outcome events may only occur for events that required medical care or were queried about during a medical encounter. The surveillance periods, duration of follow-up and timing of end point measurement may differ as well. Trials evaluate end points at protocol-specified time intervals, while in clinical practice patients are monitored according to usual care, which varies across practices, health systems and regions and it is not possible to determine the intent of observed tests and procedures (e.g., routine surveillance vs suspicion of disease).

Finally, there may be calendar time differences. In some cases, the observational study will require a later time period than the corresponding pivotal Phase III RCT because the drug was either not yet on the market or only used off-label at the time the RCT was conducted. Improvements in clinical practice and changes in the treatment landscape or patient population may occur over time, which can impact treatment effect estimates.

When comparing treatment effect estimates from the RCTs and observational studies, it is important to consider differences in data analysis and the causal effect measures under study. RCTs typically conduct intent-to-treat analysis based on how patients were randomly assigned into treatment groups, whereas observational studies group patients into treatment groups based on what was observed (as used) rather than what was intended (intent to treat) and conduct the observational analog of a ‘per protocol’ analysis [20]. Direct comparisons of effect estimates between RCTs and observational studies may require reanalysis of the RCT data to estimate the per-protocol effect and make adjustments for adherence and loss to follow-up [20]. Patterns of intercurrent events such as treatment adherence, dose changes, and treatment augmentation or switching are expected to vary between RCTs and RWD. In an RCT, the study protocol defines rules for treatment changes and adherence is closely monitored, while in clinical practice there is less oversight of patients and much greater variability in treatment changes and the influence of other factors, such as cost or insurance coverage and social determinants of health. RCT DUPLICATE investigators plan to censor patients when they discontinue the study treatment or switch treatments [14]. It is not clear what other approaches may be used to address ITT versus per-protocol analysis or account for treatment changes during the follow-up period in the observational studies, but this issue highlights the potential complementarity of RCTs and RWD.

It is well known that observational studies may be affected by bias and unmeasured confounding. Real-world data sources are often missing key sociodemographic and clinical variables that may confound the treatment-outcome relationship. There may be channeling bias related to patient, physician and healthcare system factors influencing prescribing and use of a newly approved drug [21] or other biases known to harm the internal validity of observational studies, such as confounding by indication, confounding by frailty and the healthy user effect [22]. There also may be misclassification or measurement error in the exposure and baseline predictors or covariates that can impact the estimation of treatment effects and alter their interpretation [23].

None of these issues have a simple or guaranteed solution that can offer the reassurance provided by randomization. Comparative analyses based on observational data are challenging. Standard bias control methods rely on the assumption of no unmeasured confounding, which may be untenable in many observational studies of treatment effectiveness. However, one is not without some recourse. The literature is rich with methods and evaluations of methods for adjusting for confounders, such as the use of matching, stratification, weighting and regression-based methods often through use of the propensity score [4,7,8,24–27]. Emerging methods such as doubly robust approaches, double matching (propensity and prognostic scores) and even machine learning approaches such as model averaging may provide improved estimators of treatment effects [28–31]. Recent review papers are emphasizing the use of an expanding toolkit of sensitivity analysis methods to assess the potential impact of unmeasured confounding [32–35]. It is expected that competent researchers will avoid common study design flaws [1] and apply accepted methods to adjust for measured confounding, as well as more novel methods to address unmeasured confounding. It is critical to conduct extensive sensitivity analyses where researchers diligently seek and include all factors that can address any underlying bias or confounding.

The impact of known but unmeasured confounding and measurement error could be minimized even further by the selection of real-world data sources that are ‘fit for purpose’ for replicating the treatment effect from the RCT. The decision to use claims data for the RCT DUPLICATE project was logistical rather than scientific. It is likely that for some RCTs, administrative claims data alone will be insufficient to support replication since it may not be able to match the inclusion or exclusion criteria or have the necessary variables to adequately find and correct the bias or confounding. Linkage with laboratory data or electronic health records data may be necessary to provide richer information. If linking is not possible, then replication in different types of real-world data sources may be helpful.

The efficacy–effectiveness gap refers to the longstanding observation that products perform differently in clinical practice than in RCTs. There is considerable literature on this topic [36], but it has not been directly addressed in recent discussions on replications of RCTs using RWD. Imperfections of the healthcare system and delivery of care may contribute to differences in outcomes between RCTs and observational studies [36]. In routine practice, there are barriers to accessing healthcare resources, variability in testing, diagnosis, and treatment, and physician behavior and patient adherence are not optimal [36]. True differences in outcomes may be expected between highly protocolized care of RCTs and usual care in real-world settings. The efficacy–effectiveness gap may also be the result of complex interactions between the drug’s biologic effects and patient, provider and healthcare-related factors [36]. While the methodological considerations discussed earlier in this paper may partially explain the efficacy–effectiveness gap, it is also possible that observational study results represent part of a continuum of truth about a treatment effect.

Disagreement in results of these RCT replication efforts could be evidence of challenges with emulating the target trials using RWD, differences in the data analysis and effect measures, bias or confounding in the observational study, or the efficacy–effectiveness gap. It is likely that all these factors will contribute in some way to any observed discrepancies in effect estimates. Arguably, the most important work will involve disentangling the reasons for differences to understand the results of the exercises and the implications for confidence in the internal validity of observational studies. A recent paper by the lead investigators of RCT DUPLICATE considers the potential challenges in emulating a target trial with RWD and provides a list of measures of emulation differences related to study populations, treatment strategies and outcomes (e.g., proportion of patients with labs available and the length of follow-up in RWE vs RCT) that are observable [37]. These measures, along with targeted sensitivity analyses, may help investigators understand the potential impact of emulation differences between the RCT and the RWE replication. Lodi and colleagues [20] describe a systematic approach to improve the comparison of effect estimates from RCTs and observational studies based on RWD, based on harmonization of study protocols and data analysis to target the same causal effect and the same estimand and sensitivity analyses to investigate remaining discrepancies.

We encourage other researchers to build upon the foundation of current RCT replication efforts and conduct and publish similar replication exercises in additional therapeutic classes, with careful consideration of data and methods in the context of fitness for purpose, with attention to describing agreement and when results are discrepant, to identifying potential explanations for differences. We recommend that future efforts are expanded to include not only health insurance claims but also electronic health records data, registries, linked data sources and other clinically rich data sources. It would be most useful to focus on clinical and regulatory contexts where observational studies could demonstrate substantial evidence of treatment effect to support product effectiveness determinations and labeling changes, especially since this is an important goal identified in the 21st Century Cures Act. While FDA has accepted observational studies to support effectiveness determinations in limited instances in the past, there are a growing number of therapeutic areas where there appears to be an acceptable level of risk, particularly in oncology, in which observational studies may provide primary or supportive evidence sufficient for a label change. These may include long-term effectiveness within an already approved indication, additional claims or end points within an approved indication, changes in the combination therapy, or changes in the indicated patient population. Finally, it will be important to consider other approaches, in addition to RCT replication efforts, to address FDA concerns regarding establishing causality with observational studies, for example, the application of methods to address unmeasured confounding and work to advance methodological and statistical approaches to support causal conclusions. There is a growing literature on causal frameworks and causal estimation in observational studies; however, there is more work to be done to develop and evaluate novel statistical approaches and to advance regulatory understanding and confidence in observational studies of treatment effectiveness. Just as the requirements for an acceptable RCT have become recognized and codified, it will be necessary to develop consensus on the characteristics of high-quality observational research that could meet the standard of an ‘adequate and well-controlled investigation’ in order for these studies to rise to the level of substantial evidence [38,39].