Round-up of the ISPOR Real-World Evidence Summit 2024

Joanne Walker and Laura Dormer of The Evidence Base attended the ISPOR—The Professional Society for Health Economics and Outcomes Research (ISPOR) Real-World Evidence Summit 2024 on November 17, 2024, to gather insights on practical approaches to utilizing real-world evidence (RWE) in research, decision-making, and policy implementation. Held as a precursor to the ISPOR Europe 2024 conference, this one-day event provided a platform for sharing strategies and best practices. Here they summarize the discussions and key takeaways from the day's presentations.

Session 1: Unleashing the Latent Power of Real-World Evidence in Decision-Making

Shirley Wang (Brigham & Women's Hospital, Harvard Medical School, USA) opened the first session of the ISPOR RWE Summit 2024 by highlighting the growing importance of RWE, reflected in over 50 new regulatory and health technology assessment (HTA) guidance documents since 2017. She emphasized the harmonization efforts under International Council for Harmonisation (ICH) draft guideline, ICH M14, providing guidance on the planning, design, and analysis of pharmacoepidemiologic studies using RWD for medicine safety assessment. Wang stressed the importance of aligning the triad of study question, design, and data for generating fit-for-purpose RWD. She also emphasized the need for early regulatory engagement and adherence to guidelines such as the HARPER guidance. Wang then introduced the panelists: Ashley Jaksa (Aetion, Inc., USA), Maria Kamusheva (Sofia Medical University, Bulgaria), and Steve Williamson (National Institute for Health and Care Excellence [NICE], UK).

Maria Kamusheva began by outlining the differences between RWD and RWE, emphasizing varied international definitions. ISPOR, for example, define RWD as: “data used for decision-making that are not collected in conventional randomized controlled trials (RCTs).” She discussed initiatives like DARWIN-EU and the European Medicines Agency’s reflection paper on RWE. From a HTA perspective, she noted the uncertain role of RWE in joint clinical assessments (JCAs) under the EU HTA Regulation, pointing out the need for greater clarity on national implementation. Kamusheva underscored the potential of RWE in HTA.

“Nowadays, we have the expert knowledge and technological capacity to integrate RWD/RWE in HTA. What we need now is the will and the dedication to make it happen.”

Steve Williamson discussed NICE's use of RWE, sharing insights from four case studies. He highlighted the NICE RWE Framework and described applications of RWE, including validation of economic models, assessing the safety of medical technologies, and supporting system implementation, particularly for high-cost medicines. RWE is also used in managed access to address uncertainties. NICE sources RWD from NHS registries, academic networks, patient groups, and pharmaceutical companies, evaluating factors like completeness, population coverage, and data accuracy to ensure fitness for purpose. While RWE helped resolve uncertainties in the case studies, there were also notable committee concerns in some of the case studies shared, underlining the need for improvements in RWE use and quality.

Ashley Jaksa discussed overcoming barriers to using RWD in HTA, focusing on data quality, transportability, and lifecycle management. Data quality was identified as a critical challenge, with inconsistent standards limiting RWD use in decision-making. Jaksa emphasized the need for harmonized quality standards, transparency from data providers, and efficient data collection processes. Through the RWE4Decisions initiative, she highlighted the role of stakeholders—including pharma companies, payers, and HTA bodies—in collaborating to improve RWD quality, such as through unified registries and common frameworks.

Transportability was also explored, where statistical methods are used to 'transport' an effect estimate from one (original) population to another (target) population – such a from a US population to a UK population. Transportability of RWE addresses challenges such as unavailable or poor-quality RWD, limited accessibility across jurisdictions, and industry constraints preventing RWE studies in every target region for access purposes. To advance the use of transportability, demonstration projects and shared standards are needed across stakeholders.

Jaksa closed by discussing lifecycle management—using RWE in evidence requirements throughout a drug's lifecycle, including post-launch reassessments. Jaksa shared that while RWE helps address uncertainties, it is often used alongside RCT evidence. This was shown in a recent study exploring HTA reassessments across six major HTA agencies, where it was shown that RWE was primarily used to address clinical uncertainties alongside RCT evidence, with no de novo comparative effectiveness RWE studies conducted. The path forward includes creating comprehensive RWE generation plans, with prioritized research questions and a timeline for data readiness. Jaksa also called for coordinated efforts between stakeholders, encouraging industry engagement in scientific advice meetings and alignment of post-launch evidence requirements to enhance the utility of RWE in reassessments.

Session 2: Methods for Causal Inference Using Real-World Data

The second session of the ISPOR RWE Summit, moderated by Uwe Siebert (UMIT TIROL – University for Health Sciences and Technology Hall in Tirol, Austria and Harvard Chan School of Public Health Harvard University, USA), brought together a distinguished panel to discuss innovative methods for causal inference using RWD in healthcare. Siebert opened the session with an overview of the evolving landscape of causal inference in RWD. He emphasized the necessity of understanding assumptions underlying causal relationships and the importance of identifying and mitigating biases in observational data.

To frame the session, Siebert introduced tools such as directed acyclic graphs (DAGs), which help researchers map data-generating processes and identify variables that should or should not be used in adjustments. He also discussed the role of target trial emulation in avoiding self-issues such as immortal time bias or selection bias. Siebert noted that these methods, previously the domain of RCTs, are now increasingly being applied to RWD to answer pressing healthcare questions.

The first panelist, Barbra Dickerman (Harvard TH Chan School of Public Health, USA), spoke on, ‘From healthcare data to decisions: the target trial framework’, emphasizing the value of target trial emulation in using RWD to answer causal questions in healthcare. The target trial framework involves designing an observational study to emulate a hypothetical RCT that addresses a specific causal question. This approach ensures that the study incorporates the same protocol components as an RCT, including eligibility criteria, treatment strategies and follow-up plans.

Dickerman illustrated the importance of this framework through a case study comparing statin use and cancer risk. Certain earlier observational studies had suggested that statin use reduced cancer risk, while RCTs had found no such effect. Using the target trial framework, her team reanalyzed the observational data to align it with RCT design principles, such as eliminating biases like immortal time bias (arising from misclassifying treatment duration) and selection bias (from including prevalent users). The results demonstrated that statins had no significant effect on cancer risk, highlighting how biases, rather than confounding alone, can skew observational study results.

Concluding her talk, Dickerman argued that adopting the target trial framework as a standard for causal inference in RWD can help address biases and discrepancies, ultimately strengthening the evidence base for healthcare decision-making.

Building on Dickerman’s presentations, the second speaker, Nicholas Latimer (University of Sheffield & Delta Hat Limited, UK), spoke on, ‘The use of causal inference methods in health technology assessment: addressing hypothetical estimands in clinical trials’. Latimer explored the use of G-methods, such as inverse probability weighting and rank-preserving structural failure time models (RPSFTMs), in HTA and how these methods can be adapted for RWD. He began by contextualizing these methods within HTA’s need for hypothetical estimands, particularly in scenarios like treatment switching in RCTs. Treatment switching, where patients transition from control treatments to experimental ones, is common in oncology trials and complicates the interpretation of trial outcomes for decision-making. Latimer detailed how G-methods are used to estimate counterfactual outcomes, which represent what would have happened in the absence of treatment switching. By demonstrating these techniques in RCTs, Latimer provided a foundation for their application to RWD, which often faces similar methodological challenges.

Concluding, Latimer highlighted the need for rigorous reporting and benchmarking to improve the acceptance of G-methods in HTA. He stressed that while RWD introduces additional challenges like data quality and missingness, these methods are viable tools for addressing causal questions if applied with transparency and methodological rigor.

The final speaker Sebastian Schneeweiss (Brigham & Women's Hospital, Harvard Medical School, USA) spoke on, ‘External control arm analyses with calibration and hybrid designs’, focusing on the growing use of external control arms (ECAs) derived from RWD in regulatory decision-making. ECAs provide a comparator group for single-arm trials (SATs) in cases where randomization is unfeasible.

He presented an example involving the use of ECAs for FDA approval of blinatumomab for acute lymphoblastic leukemia. The SAT demonstrated strong efficacy, contextualized against an ECA constructed from electronic health records (EHRs) and registries. While the FDA accepted the analysis, Schneeweiss highlighted the inherent challenges of using secondary data, such as incomplete measurement of key confounders and differential information quality between trial and ECA data.

To address these challenges, he proposed strategies like linking trial participants' historical EHR data to calibrate measurements and improve comparability. He also introduced hybrid designs, which combine underpowered internal control arms with ECAs, down-weighting the latter using Bayesian methods. These approaches balance the strengths of randomized and observational data while mitigating biases inherent in secondary datasets.

Schneeweiss concluded by advocating for early integration of RWD strategies into drug development. By investing in data quality and aligning RWD analyses with trial designs through frameworks like target trial emulation, researchers can enhance the credibility of ECAs. He emphasized that ECAs are not a shortcut to lower evidence standards but a complementary tool to traditional methods.

Session 3: Embracing Diversity and Tackling Heterogeneity in Data, Methods, and Jurisdictions

Patrice Verpillat (European Medicines Agency [EMA], The Netherlands) led the next session exploring heterogeneity in RWD. To achieve more accurate outcomes, diverse datasets and methodologies from multiple jurisdictions are employed, introducing variability—termed heterogeneity—that poses challenges in interpreting results. This session explored the sources of heterogeneity in RWD and strategies to manage it. The discussants included Daniel Prieto-Alhambra (Oxford University, UK), Chantal Quinten (EMA, The Netherlands), and Daniel Rosenberg (Johnson & Johnson, Basel, Switzerland).

Chantal Quinten discussed potential sources of heterogeneity in multi-database studies and analytical methods to manage and report this variability. She illustrated how heterogeneity can arise from variations in study protocol, study quality, differences in interventions received, and variations in treatment covariate interactions.

Quinten highlighted that heterogeneity is not inherently negative: it can be either informative or unwanted. Methodological heterogeneity, such as variations in data recording, is often undesirable, whereas clinical heterogeneity can provide useful insights into differences in patient populations or healthcare systems.

Quinten shared an example involving an orphan drug application at EMA, where RWE was required to validate orphan status. This study aimed to determine the prevalence of acute liver injury using five RWD sources from different countries. However, the observed prevalence varied significantly across databases due to differences in coding systems (e.g., ICD-10 codes), healthcare systems, and data quality. Variability in data collection methods—such as how conditions were classified or recorded—introduced significant heterogeneity. In this context, harmonization efforts attempted to address these issues, yet some heterogeneity persisted. The key takeaway was that the presence of heterogeneity must be acknowledged, and appropriate decisions must be made regarding whether to use stratified or pooled estimates.

Daniel Prieto-Alhambra provided insights from the DARWIN EU network, a federated data network designed to support decision-making throughout the product lifecycle by generating RWE. Now approaching its third year, DARWIN EU includes numerous database partners across Europe, covering EHRs, cancer registries, biobanks, and more. Given Europe's diverse healthcare landscape, data heterogeneity is inevitable, with differences in disease prevalence and healthcare systems contributing to variability.

To address heterogeneity, DARWIN EU converts data into a common data model (OMOP CDM), standardizing it across all sources to enable federated analytics. This allows researchers to conduct consistent analyses across different jurisdictions. Prieto-Alhambra highlighted the benefits of using standardized data models and data analytics tools that have led to a catalogue of standard data analyses to accommodate all requested study designs. DARWIN EU groups its studies by their anticipated level of complexity: Off-the-shelf and Complex. The off-the-shelf studies primarily involve characterization questions that can be executed with a generic protocol, including studies on disease epidemiology (e.g., estimating prevalence or incidence of health outcomes) and drug utilization at the population or patient level. More complex studies may require customized study designs, tailored protocols, analytics, and phenotype definitions, including safety and effectiveness evaluations of medicines and vaccines. He also emphasized the importance of data provenance to understand the sources and their impact on research outcomes.

Daniel Rosenberg concluded the session with a discussion on the advantages and challenges of using CDMs to harmonize RWD. He emphasized the need for trustworthy, transparent, and reproducible RWE, noting that data fragmentation across regions, limited accessibility, and diverse healthcare settings create significant challenges. Federated approaches and pooled analyses, supported by CDMs, help overcome these obstacles.

Rosenberg elaborated on how CDMs facilitate the transformation of diverse datasets into a common format that supports reproducibility and efficient analysis. He noted that although all data can be harmonized into a single format, inherent differences in the original data, such as source methodologies and geographic variations, often lead to persistent heterogeneity.

He shared a specific example in which three registry databases were successfully mapped from STDM to OMOP CDM to answer research questions within a federated network. Although the datasets were successfully harmonized, the underlying data collection methods varied, introducing heterogeneity even after mapping. This highlighted that whilst the data format may be homogeneous, the data quality and methodology may still differ, depending on how and why the data was collected.

Rosenberg also discussed the importance of feasibility assessments when using multiple datasets. These assessments involve evaluating data provenance, patient numbers, key characteristics, availability, treatment information, and timelines. The aim is to determine how representative the dataset is, its generalizability, and whether it can support stratified or pooled analyses. Decisions regarding eligibility criteria, the feasibility of temporal stratifications, and data pooling also rely heavily on these assessments. In multi-database research, understanding and addressing heterogeneity through feasibility assessments is critical for generating reliable and valid results.

Rosenberg concluded by describing the OMOP CDM as one of the best-performing models for observational health data, as it harmonizes data formats and allows for standardized programming code and outputs, enabling more consistent and transparent analyses. He also emphasized that both qualitative and quantitative feasibility assessments are crucial for evaluating data heterogeneity and ensuring reliable, reproducible, and transparent results. Such assessments are essential for determining the suitability of datasets for specific research questions.

Session 4: Overcoming Obstacles: Charting the Path Ahead

The fourth and final session was moderated by François Meyer (FIPRA, Paris) and delved into practical approaches to enhance the use of RWE in HTA. Together, the speakers addressed challenges and actionable solutions to optimize the generation and application of RWE for healthcare decision-making.

Meyer introduced the session, highlighting its focus on the integration of RWE into HTA processes to support payer and regulatory decisions. He emphasized the need to complement clinical trial data with robust RWE, particularly for innovative medicines, and described ongoing efforts to align evidence requirements across jurisdictions. Meyer also outlined the history of RWE4Decisions, a payer-led, multi-stakeholder learning network, focusing on highly innovative medicines that often have immature clinical evidence and high prices. The initiative examines how RWE could be used to demonstrate value – filling gaps in evidence from clinical development and support conditional reimbursement. Meyer also touched on JCAs and joint scientific consultations, along with other European and regional initiatives such as: JNHB, Beneluxa, Darwin EU, IDERHA, Real4Reg, Data Saves Lives and GetReal.

Piia Rannanheimo (Finnish Medicines Agency, Kuopio, Finland) detailed an updated framework from RWE4Decisions for stakeholder actions to improve RWE generation and utilization in HTA and payer decision-making. Key updates since 2020 included enhanced data governance, analytical methods and harmonization efforts. She emphasized the need for collaborative stakeholder engagement to align data needs and optimize evidence generation. Actions were categorized under four pillars: data availability, governance and quality; methodology design and analysis; policy and partnerships; and trust and transparency. Rannanheimo urged stakeholders to operationalize these recommendations and leverage tools such as the European Health Data Space.

The ISPOR RWE Summit Program Committee co-chair, Massoud Toussi (Cytel, Paris, France), presented a novel alignment matrix developed to address the complexity of matching evidence needs with appropriate study designs and data sources. This matrix aims to guide decision-makers in selecting optimal methodologies for specific research questions, addressing limitations in traditional evidence hierarchies. Toussi underscored the importance of adaptability, as the matrix evolves with advancements in methods and data sources. He also highlighted its potential applications in HTA modeling, evidence strategy planning, and stakeholder dialogues.

Speaking next, Conor Teljeur (Health Information and Quality Authority, Dublin, Ireland) and outlined challenges in utilizing RWE for complex public health interventions, such as vaccination programs and screenings. He emphasized the importance of context-specific evaluations and the need to integrate epidemiological and health-related quality-of-life data. Teljeur advocated for pragmatic approaches, leveraging the best available evidence to inform timely decision-making.

Follwing Teljeurt, Jana Hlavacova brought a patient-centric perspective, stressing the need for high-quality, relevant RWE to bridge data gaps and support expedited access to innovative treatments.

“Patients need RWE to be used, and to be used right.”

Hlavacova emphasized the role of patient organizations in shaping and disseminating RWE standards, and highlighted the importance of collaborative efforts to ensure data is used efficiently and inclusively.

Finally, Niklas Hedberg reflected on Sweden’s approach to integrating RWE in decision-making for drug pricing and reimbursement. He acknowledged challenges in addressing long-term outcomes for innovative treatments but emphasized the necessity of leveraging national patient registries and secondary data. Hedberg also advocated for methodological innovation through cross-border collaborations and pilot projects.