Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance
Publication: Journal of Comparative Effectiveness Research
Abstract
Decision-makers have become increasingly interested in incorporating real-world evidence (RWE) into their decision-making process. Due to concerns regarding the reliability and quality of RWE, stakeholders have issued numerous recommendation documents to assist in setting RWE standards. The fragmented nature of these documents poses a challenge to researchers and decision-makers looking for guidance on what is ‘high-quality’ RWE and how it can be used in decision-making. We offer researchers and decision-makers a structure to organize the landscape of RWE recommendations and identify consensus and gaps in the current recommendations. To provide researchers with a much needed pathway for generating RWE, we discuss how decision-makers can move from fragmented recommendations to comprehensive guidance.
Supplementary Material
File (supplementary material.pdf)
- Download
- 161.33 KB
References
Papers of special note have been highlighted as: • of interest
1.
National Institute for Health and Care Excellence. Increasing use of health and social care data in guidance development (2020). www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-guidelines/how-we-develop-nice-guidelines/data-and-analytics-statement-of-intent
2.
US Food and Drug Administration. Framework for FDA's Real-World Evidence Program (2018). www.fda.gov/media/120060/download
• FDA's RWE Framework outlines key considerations that it will explore when developing guidance documents to assist manufacturers interested in using real-world evidence (RWE).
3.
European Medicines Agency. EMA Regulatory Science to 2025 (2020). www.ema.europa.eu/en/documents/regulatory-procedural-guideline/ema-regulatory-science-2025-strategic-reflection_en.pdf
• The European Union's focus on understanding and using Big Data in regulatory and HTA decision-making is a cornerstone of the EMA's strategic focus through 2025.
4.
Office of the Commissioner. Coronavirus (COVID-19) Update: FDA Collaborations Promote Rigorous Analyses of Real-World Data to Inform Pandemic Response. FDA (2020). www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-collaborations-promote-rigorous-analyses-real-world-data-inform
5.
Collins R, Bowman L, Landray M, Peto R. The magic of randomization versus the myth of real-world evidence. N. Engl. J. Med. 382(7), 674–678 (2020).
6.
Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Cardiovascular disease, drug therapy, and mortality in Covid-19. N. Engl. J. Med. 382(25), e102 (2020).
7.
Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. The Lancet (2020). www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31180-6/abstract
8.
Avorn J. In Defense of Pharmacoepidemiology — Embracing the Yin and Yang of Drug Research. N. Engl. J. Med. 357(22), 2219–2221 (2007).
9.
Griffiths EA, Macaulay R, Vadlamudi NK, Uddin J, Samuels ER. The role of noncomparative evidence in health technology assessment decisions. Value Health J. Int. Soc. Pharmacoeconomics Outcomes Res. 20(10), 1245–1251 (2017).
10.
Makady A, Ham RT, de Boer A et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health J. Int. Soc. Pharmacoeconomics Outcomes Res. 20(4), 520–532 (2017).
11.
Morton SC, Costlow MR, Graff JS, Dubois RW. Standards and guidelines for observational studies: quality is in the eye of the beholder. J. Clin. Epidemiol. 71, 3–10 (2016).
12.
IQWiG. Development of scientific concepts for the generation of routine practice data and their analysis for the benefit assessment of drugs according to §35a Social Code Book V – rapid report (2020). www.iqwig.de/en/projects-results/projects/drug-assessment/a19-43-development-of-scientific-concepts-for-the-generation-of-routine-practice-data-and-their-analysis-for-the-benefit-assessment-of-drugs-according-to-35a-social-code-book-v-rapid-report.11901.html
13.
Duke-Margolis Center for Health Policy. Understanding the need for non-interventional studies using secondary data to generate real-world evidence for regulatory decision making, and demonstrating their credibility. (2019). https://healthpolicy.duke.edu/publications/understanding-need-non-interventional-studies-using-secondary-data-generate-real-world
14.
Franklin JM, Glynn RJ, Martin D, Schneeweiss S. Evaluating the use of nonrandomized real-world data analyses for regulatory decision making. Clin. Pharmacol. Ther. 105(4), 867–877 (2019).
15.
US Food and Drug Administration. Demonstrating substantial evidence of effectiveness for human drug and biological products (2019). www.fda.gov/media/133660/download
16.
European Medicines Agency. Choice of control group in clinical trials (2001). www.ema.europa.eu/en/documents/scientific-guideline/ich-e-10-choice-control-group-clinical-trials-step-5_en.pdf
17.
Friends of Cancer Research. Characterizing the use of external controls for augmenting randomized control arms and confirming benefit (2019). www.focr.org/sites/default/files/Panel-1_External_Control_Arms2019AM.pdf
18.
Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML. Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: The International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report—Part II. Value Health. 12(8), 1053–1061 (2009).
19.
Franklin JM, Schneeweiss S, Solomon DH. Assessment of confounders in comparative effectiveness studies from secondary databases. Am. J. Epidemiol. 185(6), 474–478 (2017).
20.
Gatto NM, Reynolds RF, Campbell UB. A structured preapproval and postapproval comparative study design framework to generate valid and transparent real-world evidence for regulatory decisions. Clin. Pharmacol. Ther. 106(1), 103–115 (2019).
21.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available: table 1. Am. J. Epidemiol. 183(8), 758–764 (2016).
22.
Canada H. Elements of real world data/evidence quality throughout the prescription drug product life cycle (2020). www.canada.ca/en/services/health/publications/drugs-health-products/real-world-data-evidence-drug-lifecycle-report.html
23.
Center for Drug Evaluation and Research. Best practices for conducting and reporting pharmacoepidemiologic safety studies using electronic healthcare data sets. US Food Drug Adm (2020). www.fda.gov/regulatory-information/search-fda-guidance-documents/best-practices-conducting-and-reporting-pharmacoepidemiologic-safety-studies-using-electronic
24.
Developing a protocol for observational comparative effectiveness research: a user's guide. Velentgas P, Dreyer NA, Nourjah P, Smith SR, Torchia MM. (Eds). Agency for Healthcare Research and Quality (US), MD, USA, www.ncbi.nlm.nih.gov/books/NBK126190/
25.
International Society for Pharmacoepidemiology. Guidelines for good pharmacoepidemiology practices (GPP) (2015). www.pharmacoepi.org/resources/policies/guidelines-08027/
26.
Berger ML, Sox H, Willke RJ et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol. Drug Saf. 26(9), 1033–1039 (2017).
27.
Newhouse R, Goodman S, Aronson N et al. PCORI Methodology Report (2019). www.pcori.org/sites/default/files/PCORI-Methodology-Report.pdf
28.
European Medicines Agency. Guidance for the format and content of the protocol of non-interventional post-authorisation safety studies (2012). www.ema.europa.eu/en/documents/other/guidance-format-content-protocol-non-interventional-post-authorisation-safety-studies_en.pdf
29.
European Network of Centres for Pharmacoepidemiology and Pharmacovigilance. ENCePP Checklist for Study Protocols (2018). www.encepp.eu/standards_and_guidances/checkListProtocols.shtml
30.
HMA/EMA Joint Task Force on Big Data. Observational data (Real World Data) Subgroup report (2019). www.ema.europa.eu/en/documents/report/observational-data-real-world-data-subgroup-report_en.pdf
31.
US Food and Drug Administration. Use of electronic health record data in clinical investigations guidance for industry (2018). www.fda.gov/media/97567/download
32.
Medicines & Healthcare Products Regulatory Agency. Medicines & Healthcare products Regulatory Agency (MHRA) ‘GXP’ Data Integrity Guidance and Definitions (2018). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/687246/MHRA_GxP_data_integrity_guide_March_edited_Final.pdf
33.
CanREValue. Mapping Canadian Provincial Data Assets to Conduct Real-World Studies on Cancer Drugs CanREValue Collaboration Data Working Group Interim Report 2020 (2020). https://cc-arcc.ca/wp-content/uploads/2020/04/The-CanREValue-Data-WG-Interim-Report-Revision_Final_v1.pdf
34.
EUNetHTA JA3. Registry Evaluation and Quality Standards Tool (REQueST®). EUnetHTA (2019). https://eunethta.eu/request-tool-and-its-vision-paper/
35.
Duke-Margolis Center for Health Policy. Characterizing RWD Quality and Relevancy for Regulatory Purposes (2018). https://healthpolicy.duke.edu/sites/default/files/2020-08/Characterizing%20RWD%20for%20Regulatory%20Use.pdf
36.
Duke-Margolis Center for Health Policy. Determining Real-World Data's Fitness for Use and the Role of Reliability (2019). https://healthpolicy.duke.edu/sites/default/files/2019-11/rwd_reliability.pdf
37.
US Food and Drug Administration. Use of real-world evidence to support regulatory decision-making for medical devices - guidance for industry and food and drug administration staff (2017). www.fda.gov/media/99447/download
38.
Hall GC, Sauer B, Bourke A et al. Guidelines for good database selection and use in pharmacoepidemiology research. Pharmacoepidemiol. Drug Saf. 21(1), 1–10 (2012).
39.
Faria R, Alava MH, Manca A, Wailoo AJ. The use of observational data to inform estimates of treatment effectiveness in technology appraisal: methods for comparative individual patient data (2015). http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD17-DSU-Observational-data-FINAL.pdf
40.
Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U. Good Research Practices for Comparative Effectiveness Research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report–Part III. Value Health 12(8), 1062–1073 (2009).
41.
Schneeweiss S. Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiol. Drug Saf. 15(5), 291–303 (2006).
42.
Schneeweiss S, Rassen JA, Glynn RJ, Avorn J, Mogun H, Brookhart MA. High-dimensional propensity score adjustment in studies of treatment effects using health care claims data. Epidemiology 20(4), 512–522 (2009).
43.
Schneeweiss S, Rassen JA, Brown JS et al. Graphical depiction of longitudinal study designs in health care databases. Ann. Intern. Med. 170(6), 398 (2019).
44.
Wang SV, Schneeweiss S, Berger ML et al. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Pharmacoepidemiol. Drug Saf. 26(9), 1018–1032 (2017).
45.
Orsini LS, Berger M, Crown W et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing—why, what, and how: recommendations and a road map from the real-world evidence transparency initiative. Value Health 23(9), 1128–1136 (2020).
46.
Benchimol EI, Smeeth L, Guttmann A et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLOS Med. 12(10), e1001885 (2015).
47.
Langan SM, Schmidt SA, Wing K et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ k3532 (2018).
48.
Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335(7626), (2007).
49.
Sterne JA, Hernán MA, Reeves BC et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. i4919 (2016).
50.
GRACE Principles: Recognizing High-Quality Observational Studies of Comparative Effectiveness. AJMC. www.ajmc.com/journals/issue/2010/2010-06-vol16-n06/ajmc_10jundreyer_467to471
51.
Motheral B, Brooks J, Clark MA et al. A checklist for retrospective database studies–report of the ispor task force on retrospective databases. Value Health 6(2), 90–97 (2003).
52.
Berger ML, Martin BC, Husereau D et al. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: An ISPOR-AMCP-NPC Good Practice Task Force Report. Value Health 17(2), 143–156 (2014).
53.
Duke-Margolis Center for Health Policy. A framework for regulatory use of real-world evidence (2017). https://healthpolicy.duke.edu/sites/default/files/2020-03/rwe_white_paper_2017.09.06.pdf
54.
JA3 Work Package 5 - Lifecycle approach to improve evidence generation. EUnetHTA. https://eunethta.eu/ja3-archive/work-package-5-life-cycle-approach-to-improve-evidence-generation/
55.
IMI GetReal - Workpackage 3. www.imi-getreal.eu/About-GetReal/Workpackage-3
56.
Hampson G, Towse A, Dreitlein WB, Henshall C, Pearson SD. Real-world evidence for coverage decisions: opportunities and challenges. J. Comp. Eff. Res. 7(12), 1133–1143 (2018).
57.
Garrison LP, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR real-world data task force report. Value Health 10(5), 326–335 (2007).
58.
Bell H, Wailoo AJ, Hernandez M et al. The use of real world data for the estimation of treatment effects in NICE decision making (2016). http://nicedsu.org.uk/wp-content/uploads/2018/05/RWD-DSU-REPORT-Updated-DECEMBER-2016.pdf
59.
RWE4Decisions. Real-world evidence to support HTA/payer decisions about highly innovative technologies (2019). https://8c3e11d9-5f36-452f-abe3-c95befd6e85d.filesusr.com/ugd/e1a359_56d20ef27bba4a949a4adf7cae77cd35.pdf
60.
Randomized Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology (RCT DUPLICATE). RCT Duplic. www.rctduplicate.org/
• FDA funded demonstration project to duplicate 37 RCTs using RWE. Results from all 37 replication studies are expected at the end of 2020.
61.
Reproducible Evidence: Practices to Enhance and Achieve Transparency (REPEAT). REPEAT Initiat. www.repeatinitiative.org/
62.
Gustafsson S, Johansson P, Ponten J, Stromgren A, Viber A. Follow-up of drug utilisation and treatment effects in clinical practice (2018). www.tlv.se/in-english/reports/arkiv/2020-12-09-follow-up-of-drug-utilisation-and-treatment-effects-in-clinical-practice.html
63.
Zhang AD, Puthumana J, Downing NS, Shah ND, Krumholz HM, Ross JS. Assessment of Clinical Trials Supporting US Food and Drug Administration Approval of Novel Therapeutic Agents, 1995-2017. JAMA Netw. Ope 3(4), e203284 (2020).
64.
Committee for Medicinal Products for Human Use (CHMP). Assessment report: blincyto (2015). www.ema.europa.eu/en/documents/assessment-report/blincyto-epar-public-assessment-report_en.pdf
65.
Center for Drug Evaluation and Research. Statistical Review(s): Application Number 125557Orig1s000 (2014). www.accessdata.fda.gov/drugsatfda_docs/nda/2014/125557Orig1s000StatR.pdf
66.
EHDEN. EHDEN Vision, mission & objectives. ehden.eu (2020). www.ehden.eu/vision-and-mission/
67.
Basch E, Deal AM, Dueck AC et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318(2), 197 (2017).
68.
Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)= (2020). https://healthcaredelivery.cancer.gov/pro-ctcae/
69.
PROMIS: Clinical Outcomes Assessment (2019). https://commonfund.nih.gov/promis/index
70.
EQ-5D instruments – EQ-5D (2021). https://euroqol.org/eq-5d-instruments/
71.
Real-World Evidence | Friends of Cancer Research. www.focr.org/rwe
72.
Chamova J. Europäische Kommission, Generaldirektion Gesundheit und Lebensmittelsicherheit. Mapping of HTA national organisations, programmes and processes in EU and Norway (2017). https://doi.org/10.2875/5065
73.
European Commission, EMA and FDA agree new priorities to strengthen their collaboration on medicines. Eur. Med. Agency (2020). www.ema.europa.eu/en/news/european-commission-ema-fda-agree-new-priorities-strengthen-their-collaboration-medicines
74.
IMI GetReal - New methods for RWE collection and sythesis (2021). www.imi-getreal.eu/About-GetReal/Overall-objectives
75.
Joint ISPE-ISPOR Special Task Force (2021). www.ispor.org/member-groups/task-forces/joint-ispe-ispor-special-task-force
76.
Evidence Accelerator (2021). https://evidenceaccelerator.org/
77.
Global regulatory workshop on COVID-19 real-world evidence and observational studies. Eur. Med. Agency (2020). www.ema.europa.eu/en/news/global-regulatory-workshop-covid-19-real-world-evidence-observational-studies
• With the urgent need to understand the implications of the COVID-19, observational studies have become a prominent tool of researchers. Global regulators are collaborating to use RWE in studying vaccines, treatments and cohorts typically not included in RCTs (e.g., pregnant women).
Information & Authors
Information
Published In
Pages: 711 - 731
PubMed: 33928789
Copyright
© 2021 Ashley Jaksa. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 21 October 2020
Accepted: 18 March 2021
Published online: 30 April 2021
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance. (2021) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2020-0228
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Katsiaryna Bykov, Ashley Jaksa, Jennifer L. Lund, Jessica M. Franklin, Cynthia J. Girman, Madlen Gazarian, Hongbo Yuan, Stephen Duffield, Seamus Kent, Elisabetta Patorno, APPRAISE: A Tool for Appraising Potential for Bias in Real-World Evidence Studies on Medication Effectiveness or Safety, Value in Health, 10.1016/j.jval.2025.07.024, 28, 12, (1849-1856), (2025).
- I. R. Svechkareva, A. H.-M. Valeeva, T. I. Busheva, A. S. Kolbin, Real-world data: basic definitions, New St. Petersburg Medical Records, 10.24884/1609-2201-2025-104-2-11-18, 2, (11-18), (2025).
- Jonathan Hasian Haposan, Ajeng Viska Icanervilia, Emma Watts, Suci Ardini Widyaningsih, Sarah Ulfa, Prita Widayanti, Ida Safitri Laksanawati, Jarir At Thobari, Julie E. Bines, Real-world vaccine effectiveness of typhoid conjugate vaccine in children and adolescents: a systematic review and meta-analysis, Vaccine, 10.1016/j.vaccine.2025.127872, 67, (127872), (2025).
- Yanqi Shao, Xinyu Zhao, Zunchun Xie, You Xie, Shunyuan Guo, Efficacy and safety of idebenone tablets in patients with post-stroke cognitive impairment: a real-world study, Frontiers in Neurology, 10.3389/fneur.2025.1599501, 16, (2025).
- Eric Barrette, William H. Crown, Melinda Hanisch, Rafael Alfonso-Cristancho, Sunny Sheth, Samuel C.Z. Gautier, Andy Surinach, Mark Cziraky, Jon D. Morrow, Ami R. Buikema, Research Method, Conduct, and Reporting Considerations for Improving the Quality of Non-Hypothesis-Evaluating Treatment Effectiveness Analyses Using Real-World Data: An ISPOR Working Group Report, Value in Health, 10.1016/j.jval.2025.02.017, 28, 7, (979-987), (2025).
- Ashley Jaksa, Patrick Joseph Arena, Melinda Hanisch, Mark Marsico, Use of Real-World Evidence in Health Technology Reassessments Across 6 Health Technology Assessment Agencies, Value in Health, 10.1016/j.jval.2025.02.012, 28, 6, (898-906), (2025).
- Alexa Vidalis, Oscar Dumoulin, Mrunmayee Godbole, Catia C Proenca, The role and value of real-world evidence in health technology decision-making in France, Germany, Italy, Spain, and the UK: insights on external control arms, International Journal of Technology Assessment in Health Care, 10.1017/S0266462324004720, 41, 1, (2025).
- Olmo R. van den Akker, Robert T. Thibault, John P. A. Ioannidis, Susanne G. Schorr, Daniel Strech, Transparency in the secondary use of health data: assessing the status quo of guidance and best practices, Royal Society Open Science, 10.1098/rsos.241364, 12, 3, (2025).
- Casper van der Zee, Heshow Jamal, Marc Muijzer, Laurence Frank, Gerko Vink, Robert Wisse, Real world data on digital remote refraction in a healthy population of 14,680 eyes, npj Digital Medicine, 10.1038/s41746-025-01453-0, 8, 1, (2025).
- Olga E. Hart, Amanda Wachtel, Katherine Jones, Tony Jimenez, Peter Gregory, David Tam, Beyond money: A social burden framework to enhance resilience valuation for tribal communities in the United States, Energy Research & Social Science, 10.1016/j.erss.2025.103934, 120, (103934), (2025).
- See more