Skip to main content
Search
Search
Advanced Search
Skip main navigation
Open access
Research Article
12 August 2024

The maze of real-world evidence frameworks: from a desert to a jungle! An environmental scan and comparison across regulatory and health technology assessment agencies

Authors: Grammati Sarri https://orcid.org/0000-0001-5536-8038 and Luis G Hernandez https://orcid.org/0000-0002-9532-546X [email protected]Author Info & Affiliations
Volume 13, Number 9
https://doi.org/10.57264/cer-2024-0061
PDFFormats

Abstract

Aim: Regulatory and health technology assessment (HTA) agencies have increasingly published frameworks, guidelines, and recommendations for the use of real-world evidence (RWE) in healthcare decision-making. Variations in the scope and content of these documents, with updates running in parallel, may create challenges for their implementation especially during the market authorization and reimbursement phases of a medicine's life cycle. This environmental scan aimed to comprehensively identify and summarize the guidance documents for RWE developed by most well-established regulatory and reimbursement agencies, as well as other organizations focused on healthcare decision-making, and present their similarities and differences. Methods: RWE guidance documents, including white papers from regulatory and HTA agencies, were reviewed in March 2024. Data on scope and recommendations from each body were extracted by two reviewers and similarities and differences were summarized across four topics: study planning, choosing fit-for-purpose data, study conduct, and reporting. Post-authorization or non-pharmacological guidance was excluded. Results: Forty-six documents were identified across multiple agencies; US FDA produced the most RWE-related guidance. All agencies addressed specific and often similar methodological issues related to study design, data fitness-for-purpose, reliability, and reproducibility, although inconsistency in terminologies on these topics was noted. Two HTA bodies (National Institute for Health and Care Excellence [NICE] and Canada's Drug Agency) each centralized all related RWE guidance under a unified framework. RWE quality tools and checklists were not consistently named and some differences in preferences were noted. European Medicines Agency, NICE, Haute Autorité de Santé, and the Institute for Quality and Efficiency in Health Care included specific recommendations on the use of analytical approaches to address RWE complexities and increase trust in its findings. Conclusion: Similarities in agencies' expectations on RWE studies design, quality elements, and reporting will facilitate evidence generation strategy and activities for manufacturers facing multiple, including global, regulatory and reimbursement submissions and re-submissions. A strong preference by decision-making bodies for local real-world data generation may hinder opportunities for data sharing and outputs from international federated data networks. Closer collaboration between decision-making agencies towards a harmonized RWE roadmap, which can be centrally preserved in a living mode, will provide manufacturers and researchers clarity on minimum acceptance requirements and expectations, especially as novel methodologies for RWE generation are rapidly emerging.

Shareable abstract

Confused by RWD/RWE guidance for healthcare decision-making? Review and comparison of guidance documents by regulatory and reimbursement agencies, as well as research groups, revealed the need for a harmonized roadmap and standards across agencies! #RWD #RWE #DrugDevelopment #EvidenceGeneration #Healthcare #Regularory #HTA

Plain language summary

What is the article about?

Real-world data (RWD) is collected outside clinical trial settings, from sources such as patient medical records from routine clinical practice. The knowledge gained from analyses of RWD is called real-world evidence (RWE) and it can be used alongside clinical trials to assess how effective and safe treatments perform in the real world. Organizations that assess the value of medicines have provided and continue to provide guidance on how to generate and use RWE for healthcare decisions, such as market authorizations and reimbursement. In this article we aimed to identify and summarize the guidance on RWE developed by regulatory and reimbursement agencies, as well as other organizations focused on healthcare decision-making, and present similarities and differences.

What are the results?

We looked at guidance documents from regulatory agencies (such as FDA and EMA), from organizations that assess the value of healthcare technologies compared with standard of care for pricing and reimbursement decisions (such as NICE), and from organizations focused on healthcare decision making (such as ISPOR, ISPE). Overall, the documents covered similar topics when talking about issues of data quality and of transparency in results reporting, but the documents differed in the level of detail and specificity in their recommendations. In preparation for regulatory and health technology assessment submissions, this maze of RWE frameworks and guidance can be unclear and cumbersome to navigate for manufacturers and researchers.

What do the results of the study mean?

The findings revealed a duplication of effort during the development of guidance documents and the lack of a uniform, clear set of guidelines and expectations. We believe more collaboration between the organizations is needed to improve clarity and efficiency of everyone involved. Ideally, a central resource with up-to-date information and standardized guidance and approaches should be established.

Supplementary Material

File (supplementary materials.docx)

References

Papers of special note have been highlighted as: • of interest; •• of considerable interest
2.
Nishioka K, Makimura T, Ishiguro A et al. Evolving acceptance and use of RWE for regulatory decision making on the benefit/risk assessment of a drug in Japan. Clin. Pharmacol. Ther. 111(1), 35–43 (2022).
PubMed
Google Scholar
3.
Canadian Agency for Drugs and Technologies in Health. Guidance for Reporting Real-World Evidence 2023. https://www.cadth.ca/sites/default/files/RWE/MG0020/MG0020-RWE-Guidance-Report-Secured.pdf
Google Scholar
•• This framework aims to promote standardization in the reporting of real-world evidence (RWE) studies to inform reimbursement decision-making in Canada.
4.
National Institute for Health and Care Excellence. NICE real-world evidence framework 2022. https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837#:~:text=The%20framework%20aims%20to%20improve,the%20section%20on%20NICE%20guidance
Google Scholar
•• This framework describes best practices for planning, conducting and reporting RWE studies with the aim to increase the quality and transparency of this type of evidence.
5.
International Coalition of Medicines Regulatory Authorities. ICMRA statement on international collaboration to enable real-world evidence (RWE) for regulatory decision-making 2022. https://www.icmra.info/drupal/sites/default/files/2022-07/icmra_statement_on_rwe.pdf
Google Scholar
6.
Duke-Margolis Institute for Health Policy. Real-World Evidence Collaborative. https://healthpolicy.duke.edu/projects/real-world-evidence-collaborative
Google Scholar
7.
Real-world Evidence Alliance. https://rwealliance.org/
Google Scholar
8.
US Food & Drug Administration. Guidance document. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices. Guidance for Industry and Food and Drug Administration Staff 2017. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices
Google Scholar
9.
Faria R, Hernandez Alava M, Manca A et al. NICE DSU Technical Support Document 17: the use of observational data to inform estimates of treatment effectiveness in technology appraisal: Methods for comparative individual patient data 2015. http://www.nicedsu.org.uk
Google Scholar
10.
European Medicines Agency. European medicines agencies network strategy to 2025. https://www.ema.europa.eu/en/documents/report/european-union-medicines-agencies-network-strategy-2025-protecting-public-health-time-rapid-change_en.pdf
Google Scholar
•• Outlines areas of strategic focus to handle the rapidly changing environment surrounding how to ensure patient access to safe, effective medicines in the European Union.
13.
European Medicines Agency. Technical workshop on real-world metadata for regulatory purposes. https://www.ema.europa.eu/en/events/technical-workshop-real-world-metadata-regulatory-purposes
Google Scholar
14.
European Medicines Agency. Data Analysis and Real World Interrogation Network (DARWIN EU). https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-eu
Google Scholar
15.
European Medicines Agency. Learnings initiative webinar for optimal use of big data for regulatory purpose. https://www.ema.europa.eu/en/events/learnings-initiative-webinar-optimal-use-big-data-regulatory-purpose
Google Scholar
17.
US Food & Drug Administration. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. Draft Guidance for Industry 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory
Google Scholar
• This guidance, issued as part of the 21st Century Cures Act, aims to provide relevant stakeholders with considerations on an evidence base that relies on electronic health records or medical claims data in clinical studies.
18.
US Food & Drug Administration. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products. Guidance document 2023. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-design-and-conduct-externally-controlled-trials-drug-and-biological-products
Google Scholar
21.
Haute Autorité de Santé. Real-world studies for the assessment of medicinal products and medical devices (2021). https://www.has-sante.fr/jcms/p_3284524/en/real-world-studies-for-the-assessment-of-medicinal-products-and-medical-devices
Google Scholar
22.
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG). Concepts for the generation of routine practice data and their analysis for the benefit assessment of drugs according to §35a Social Code Book V (SGB V) 2020. https://www.iqwig.de/download/a19-43_routine-practice-data-for-the-benefit-assessment-of-drugs_extract-of-rapid-report_v1-0.pdf
Google Scholar
• Outlines and assesses concepts related to data collected outside a clinical trial setting, specifically in regard to suitability, quality, methodology and reporting.
23.
European Network for Health Technology Assessment (EUnetHTA). Individual Practical Guideline Document: D4.3.1: Direct and Indirect Comparisons 2022. https://www.eunethta.eu/wp-content/uploads/2022/12/EUnetHTA-21-D4.3.1-Direct-and-indirect-comparisons-v1.0.pdf
Google Scholar
25.
Wang SV, Pottegård A, Crown W et al. HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: a good practices report of a joint ISPE/ISPOR task force. Pharmacoepidemiol. Drug Saf. 32(1), 44–55 (2023).
PubMed
Google Scholar
• Provides a template to guide the development of clear, reproducible RWE study protocols.
26.
Orsini LS, Berger M, Crown W et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing-why, what, and how: recommendations and a road map from the Real-World Evidence Transparency Initiative. Value Health 23(9), 1128–1136 (2020).
PubMed
Google Scholar
27.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183(8), 758–764 (2016).
PubMed
Google Scholar
28.
Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med. Res. Methodol. 8, 70 (2008).
PubMed
Google Scholar
29.
Lash TL, Fox MP, MacLehose RF et al. Good practices for quantitative bias analysis. Int. J. Epidemiol. 43(6), 1969–1985 (2014).
PubMed
Google Scholar
30.
Sarri G, Patorno E, Yuan H et al. Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a systematic review and meta-analysis for healthcare decision making. BMJ Evid. Based Med. 27(2), 109–119 (2022).
PubMed
Google Scholar
31.
Wang SV, Pinheiro S, Hua W et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ 372, m4856 (2021).
PubMed
Google Scholar
32.
European Network for Health Technology Assessment (EUnetHTA). Vision paper on the sustainable availability of the proposed Registry Evaluation and Quality Standards Tool (REQueST). https://www.eunethta.eu/wp-content/uploads/2019/10/EUnetHTAJA3_Vision_paper-on-REQueST-tool.pdf
Google Scholar
33.
Franklin JM, Patorno E, Desai RJ et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE Initiative. Circulation 143(10), 1002–1013 (2021).
PubMed
Google Scholar
34.
Burns L, Roux NL, Kalesnik-Orszulak R et al. Real-world evidence for regulatory decision-making: guidance from around the world. Clin. Ther. 44(3), 420–437 (2022).
PubMed
Google Scholar
35.
Jaksa A, Wu J, Jónsson P et al. Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance. J. Comp. Eff. Res. 10(9), 711–731 (2021).
PubMed
Google Scholar
36.
Beyrer J, Abedtash H, Hornbuckle K et al. A review of stakeholder recommendations for defining fit-for-purpose real-world evidence algorithms. J. Comp. Eff. Res. 11(7), 499–511 (2022).
PubMed
Google Scholar
38.
Kent S, Burn E, Dawoud D et al. Common problems, common data model solutions: evidence generation for health technology assessment. Pharmacoeconomics 39(3), 275–285 (2021).
PubMed
Google Scholar
39.
Claire R, Elvidge J, Hanif S et al. Advancing the use of real world evidence in health technology assessment: insights from a multi-stakeholder workshop. Front. Pharmacol. 14, 1289365 (2023).
PubMed
Google Scholar
|