Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal

Traditionally, randomized controlled clinical trials (RCTs) are considered as the established method for providing information pertaining to the efficacy and safety of health interventions. However, the highly controlled conditions characteristic of RCTs may not always accurately represent clinical practice [5]. Although RCT patient populations are highly selected and homogenous, with a protocol-driven patient follow-up, patient populations seen in clinical practice are typically heterogeneous and  often present with comorbidities. Moreover, RCTs often have short follow-up durations, preventing the detection of rare or long-term adverse events of interventions. Surrogate end points measured in RCTs, such as progression-free survival in oncology patients may also be less relevant to decision-making than overall survival. Furthermore, clinical practice may vary on a regional or national level. These differences can lead to a discrepancy between the observed efficacy of interventions in RCTs and their effectiveness in clinical practice, a phenomenon often referred to as the efficacy–effectiveness gap [6,7].

RWE, broadly defined as evidence generated based on health data collected outside the context of RCTs [3], may help identify, quantify and address this efficacy–effectiveness gap in treatment effects where needed. For example, RWE may supplement RCT data to improve estimates of treatment effects in the real-world setting through evidence synthesis or the use of predictive modeling techniques. RWE may also provide information on parameters not examined in clinical trials, such as adherence to treatment, rare adverse events and resource use in clinical settings. The insights provided from RWE may have significant implications for drug developers, regulators and particularly, HTA agencies and payers whose decisions rely on evidence of comparative effectiveness [1].

Methodologies for the statistical analysis, synthesis and critical appraisal of RWE have developed considerably in the past 20 years, including formal checklists for assessing risk of bias, propensity scoring techniques, instrumental variable analyses, multivariable regression analyses and advanced meta-analysis methods [8–11]. These methods can begin to address a number of important shortcomings with RWE that are particularly problematic for its use in CER, such as the lack of randomization of patients which can result in a lack of comparability between treatment groups, the presence of missing observations on relevant patient outcomes or covariates and the presence of confounders. Therefore, the implementation of such methods can be used to increase the robustness of estimates derived from analyses using RWE in a number of CER scenarios.

Summary data (also known as aggregate data), whether from RCTs or RWE, such as estimates of comparative treatment effect are often of limited value for CER. For instance, a major drawback of aggregate data (AD) is the limited ability to explore individual patient characteristics which may influence or confound treatment outcomes. Therefore, in order to conduct robust CER that can inform decision-making, HTA agencies and payers often require more sophisticated analyses to be conducted whereby researchers can adjust for individual patient characteristics to generate more accurate estimates of effectiveness [1,12]. Several strategies may be employed by research teams to do so and will be discussed further below. Importantly, such strategies require analyses based on individual patient-level data (IPD), whether by the researchers themselves, or an alternative party.

The Innovative Medicines Initiative (IMI)-GetReal project was a 3-year project exploring the use of RWE to improve drug effectiveness research throughout the life cycle of drug development. The project was a public–private partnership with a multi-stakeholder constituency including industry, regulators, HTA agencies, academia and patient organizations. In total, the project comprised 5 Work Packages (WP1–5), each addressing different objectives [2]. Work conducted for two work packages (WP1 & WP4) involved attempts to access and use IPD from RWE and/or RCTs. Work Package 1 (WP1) conducted a series of case studies in multiple disease areas, aiming to explore methods for using RWE to improve effectiveness estimates and to examine the acceptability of these methods among relevant stakeholders through stakeholder workshops. Meanwhile, Work Package 4 (WP4) explored best practices for evidence synthesis from RWE and/or RCTs through literature reviews and a series of case studies. Some of the methods explored included extrapolation of long-term outcomes beyond trial durations, enrichment of network meta-analyses (NMAs) with RWE and generalization of RCT results to real-world populations through propensity scoring techniques. Together, the case study teams, each jointly co-lead by a public and industry partner, sought to access IPD from both RWE repositories and RCTs to conduct these analyses.

In total, seven case studies were conducted as part of WP1 and WP4 work, spanning multiple disease indications and each lasting approximately 1.5 years. RWE repositories approached included 12 indication registries and eight observational studies. Eventually, case study co-leads managed to secure access to IPD from 4/12 registries and 3/8 observational studies, indicating that IPD access from RWE sources succeeded in only 35% of cases. On the other hand, IPD was requested from 43 RCTs and granted in 41/43 studies, indicating that IPD retrieval from RCTs exceeded 95%.

Experiences encountered with accessing IPD from RWE repositories varied per case study. A positive example relates to a combined WP1/4 case study whereby co-leads secured access to IPD from registries in two different countries. Moreover, the registries actively informed the case study team of upcoming data updates [9,13]. In four instances across different case studies, registry holders and observational study authors initially indicated their willingness to provide access to IPD. However, they eventually communicated that their datasets were not research-ready within project timelines due to an extensive need for cleaning and trimming [9,13,14]. A negative example relates to a WP1 case study, whereby one registry refused to discuss possibilities for collaboration upfront, due to being approached by an industry co-lead [14]. Meanwhile, prolonged negotiations lasting 16 months with another registry were abandoned when representatives iterated that access to IPD would be refused until all PhD students associated with the registry completed their dissertations, in fear that they would otherwise lose ownership of findings based on the data [14]. The same registry indicated earlier in negotiations that access to a tailored portion of IPD based on the research proposal submitted could also be bought for a fee. However, the considerable amount of this fee (surpassing €100,000) acted as a direct barrier to IPD retrieval.

In summary, IMI-GetReal's experiences in accessing IPD from RWE repositories were disparate. In general, only a third of all requests for IPD access from RWE repositories submitted across all case studies were successful. For half of the case studies, IPD was accessed from registries and observational studies. Furthermore, co-leads iterated that data sharing agreements and structures did not pose considerable problems for those case studies. However, for the remaining case studies, access to IPD was denied. Reasons for inaccessibility mostly related to datasets not being research-ready within project timelines or unwillingness to share data. These reasons raise important questions regarding general competence in generating datasets of sufficient quality to be readily available for research, as well as data ownership, respectively.

As an alternative to accessing IPD, case study teams explored options for using AD from registries and observational studies. To do so, case study teams either requested that registries run prescripted analyses on IPD and report the aggregate results back to the team [14,15], or attempted to use AD as reported in literature [9,14,15]. The AD retrieved from both approaches was subsequently used in several ways to perform CER, for example by simulating patient-level data or as direct input for effect estimates in NMA models. Access to AD through both approaches was relatively easier. Importantly, AD generated by prescripted analyses on IPD provided relevant insights for conducting CER (e.g., by illustrating the distribution of covariates within patient populations thus allowing for more accurate simulations of the original patient population). However, this approach requires considerable expertise to implement and relies heavily on cooperation from registry holders to run the requested analyses. On the other hand, the absence of information on patient covariates within AD retrieved from literature limited the robustness of health outcome estimates generated from such data. Therefore, although AD can be easily obtained from literature, it is often of limited usefulness, mostly lending itself to descriptive statistical analyses rather than to analysis of treatment effects across different settings and populations [9,14].

Another point worth noting is that although accessibility of IPD from RWE repositories was a prominent issue encountered in using RWE for CER in IMI-GetReal case studies, it was not the only one [9,13]. For example, in order to make use of IPD accessed, the case study teams often had to invest considerable time and effort in making datasets research-ready (e.g., by trimming the dataset or imputing missing data values). Occasionally, observational studies only investigated treatment patterns, rather than treatment outcomes, making them of little use to analyses involving head-to-head comparisons of effectiveness. Moreover, where treatment outcomes were recorded, varying definitions of the outcome measures across different studies often complicated the synthesis of IPD from RWE and RCT sources. These issues raise additional methodological and practical concerns in applying RWE to CER, some of which have been addressed in scientific literature and should be considered by all stakeholders attempting to undertake similar efforts [10,16]. However, in subsequent sections we focus on the issue of accessibility to IPD from RWE repositories and its implications for using RWE for CER and decision-making.

IMI-GetReal case study workshops demonstrated considerable variability in external stakeholders’ views on the acceptability of RWE use in CER and subsequent decision-making. The reasons behind such controversy are multifactorial, yet generally hinged on two inter-related aspects: a lack of trust in the robustness of findings based on RWE compared with RCT data, as well as a lack of experience with using RWE in currently available methods to address questions relating to (comparative) drug effectiveness. Numerous ongoing initiatives aim to address the former aspect through guideline development on topics including: good practices to ensure data quality and standardized core outcomes datasets within registries to inform CER [17,18], statistical analysis of RWE [19–21] and the reporting of results from observational studies [22,23]. On the other hand, the latter aspect implies a lack of published examples exploring advanced methods for RWE use in CER and subsequent feedback on these methods from relevant decision-makers.

Despite, the multi-stakeholder nature of IMI-GetReal case study teams, adherence to application procedures for data access, as well as the necessary disclaimers to registry owners and study authors approached, accessibility to IPD from RWE repositories proved to be challenging. Consequently, insufficient data were available to thoroughly explore novel methods for RWE use in almost half of the case studies. More importantly, the consortium's experience with inaccessibility of IPD RWE for research purposes was echoed by many external stakeholders present in stakeholder workshops, implying that access to IPD RWE remains a persistent issue beyond the IMI-GetReal consortium. Arguably, this inaccessibility to IPD RWE both contributes to the lack of concrete examples demonstrating the potential added value of RWE use in CER and the wide lack of trust among decision-makers regarding the robustness of findings based on RWE.

Bearing in mind that CER aims to shed light on the ideal implementation of healthcare interventions to achieve maximum societal benefits, inaccessibility to IPD from RWE repositories adversely affects society as a whole. Moreover, as RWE is generated by patients within routine healthcare, it is essential that patients benefit from the use of this data; increased accessibility to IPD RWE to improve CER and decision-making should benefit all patients, not just those who control access to such data. Consequently, the dynamic, multistakeholder nature of the healthcare sector warrants a collaborative approach to solving issues pertaining to governance of RWE repositories, including accessibility to IPD.

An important aspect to enable collaborative efforts is a general understanding among all stakeholders of the patient-centered goals behind healthcare in general, as well as RWE collection and analysis to improve healthcare. In this regard, the role RWE can play in pursuing patient-centered goals will be best understood if healthcare stakeholders make a strong commitment to involve all key actors in setting-up and developing procedures to enable access to registries. This requires that, contrary to current practice, all relevant stakeholders participate in steering committees of these registries, whereby a spirit of joint action is crucial for success.

Furthermore, registries are currently set up based on undisclosed contracts, leading to situations where it is difficult to deduce why accessibility is difficult and which stakeholders are involved in deciding on data requests. Therefore, making such contracts transparent is another important step to increase clarity in the wider community about governance issues such as data ownership, gate keepers for data access, funding sources and conflicts of interests.

Developments on other fronts may provide additional potential solutions. For example, the EU Clinical Trial Directive was recently established, whereby sponsors of RCTs conducted for marketing authorization applications agreed to provide access to all patient-level clinical reports of trial subjects online [24]. Presently, no equivalent initiative exists for the publication of similar IPD for RWE generated through observational studies and may be a worthwhile endeavor for the future. However, bearing in mind that patient-level data is subject to strict privacy rules, such endeavors should not preclude the review of research protocols by relevant committees to guarantee that such data are not misused and that the scientific rigor of analyses exploiting the data are guaranteed through transparent publication of the analysis protocols.

Another example relates to the US FDA-Sentinel initiative, whereby external researchers can send standardized data queries to multiple nodes of a decentralized network of participating databases [25]. In this model, databases can opt-in or out of the sentinel initiative without having to relinquish complete access to their IPD, yet still run external research queries. The main advantage of such a model is its ability to circumvent sensitivities relating to full-fledged access to IPD while delivering the required information for furthering scientific pursuits. This approach toward remote data querying has demonstrated potential in IMI-GetReal case studies [14,15]. Moreover, similar frameworks have been implemented in other fields of research, such as DataSHIELD to conduct international research as part of the Healthy Obese Project and the Environmental Core Project (BioSHaRE-EU) [26]. Other initiatives exploring such frameworks include the IMI-Big Data for Better Outcomes (IMI-BD4BO) [27]. Arguably, equivalent systems for existing registries would bring RWE use in CER a long way.

In conclusion, the current state of accessibility to RWE experienced during IMI-GetReal case studies and by stakeholders beyond the consortium poses a considerable barrier to furthering the use of RWE in CER and healthcare decision-making. Bearing in mind that such data is generated by patients in clinical practice, this barrier diminishes the potential benefit of using RWE to provide critical insights on the effectiveness of treatments for all patients in real practice; insights that RCTs are often not designed to provide. An array of potential solutions lend themselves to overcoming this persistent inaccessibility to RWE and maximizing societal gain from its use in CER. However, the choice regarding which path to take, addressing trade-offs associated with such a choice, as well as its implementation, requires a collaborative effort spanning all relevant stakeholders; from decision-makers, to industry and patient representatives.

The research leading to these results was conducted as part of Work Package 1 and 4 of the GetReal consortium. For further information, please refer to the IMI-GetReal website [28]. This paper only reflects the personal views of the stated authors.

The work leading to these results has received support from the Innovative Medicines Initiative joint undertaking under grant agreement no [115546], resources of which are composed of financial contribution from the European Union's Seventh Framework Program (FP7/2007–2013) and EFPIA companies’ in kind contribution. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.