The use of nonrandomized evidence to estimate treatment effects in health technology assessment
Publication: Journal of Comparative Effectiveness Research
Abstract
Health technology assessment (HTA) is increasingly informed by nonrandomized studies, but there is limited guidance from HTA bodies on expectations around evidence quality and study conduct. We developed recommendations to support the appropriate use of such evidence based on a pragmatic literature review and a workshop involving 16 experts from eight countries as part of the EU’s Horizon-2020 IMPACT-HTA program (work package six). To ensure HTA processes remain rigorous and robust, HTA bodies should demand clear, extensive and structured reporting of nonrandomized studies, including an in-depth assessment of the risk of bias. In recognition of the additional uncertainty imparted by nonrandomized designs in estimates of treatment effects, HTA bodies should strengthen early scientific advice and engage in collaborative efforts to improve use of real-world data.
Formats available
You can view the full content in the following formats:
References
1.
Makady A, ten Ham R, de Boer A et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health 20, 520–532 (2017).
2.
Hatswell AJ, Baio G, Berlin JA et al. Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999–2014. BMJ Open 6, e011666 (2016).
3.
Crispi F, Naci H, Barkauskaite E et al. Assessment of devices, diagnostics and digital technologies: a review of NICE Medical Technologies Guidance. Appl. Health Econ. Health Pol. 17, 189–211 (2019).
4.
Grimes DA, Schulz KF. An overview of clinical research: the lay of the land. Lancet 359, 57–61 (2002).
5.
Hatswell AJ, Freemantle N, Baio G. Economic evaluations of pharmaceuticals granted a marketing authorisation without the results of randomised trials: a systematic review and taxonomy. Pharmacoeconomics 35, 163–176 (2017).
6.
ISPOR. Improving transparency in non-interventional research for hypothesis testing – why, what, and how: considerations from the real-world evidence transparency initiative (draft White Paper) (2019). https://www.ispor.org/docs/default-source/strategic-initiatives/improving-transparency-in-non-interventional-research-for-hypothesis-testing_final.pdf?sfvrsn=77fb4e97_6
7.
Orsini LS, Berger M, Crown W et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing-why, what, and how: recommendations and a road map from the real-world evidence transparency initiative. Value Health 23, 1128–1136 (2020).
8.
Collins R, Bowman L, Landray M et al. The magic of randomization versus the myth of real-world evidence. N. Engl. J. Med. 382, 674–678 (2020).
9.
Naci H, Davis C, Savović J et al. Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014–16: cross sectional analysis. BMJ 366, l5221 (2019).
10.
Eichler HG, Koenig F, Arlett P et al. Are novel, nonrandomized analytic methods fit for decision making? The need for prospective, controlled, and transparent validation. Clin. Pharmacol. Ther. 107, 773–779 (2019).
11.
Goring S, Taylor A, Müller K et al. Characteristics of non-randomised studies using comparisons with external controls submitted for regulatory approval in the USA and Europe: a systematic review. BMJ Open 9, e024895 (2019).
12.
Nicod E, Annemans L, Bucsics A et al. HTA programme response to the challenges of dealing with orphan medicinal products: process evaluation in selected European countries. Health Pol. 123, 140–151 (2019).
13.
Facey K, Rannanheimo P, Batchelor L et al. Real-world evidence to support Payer/HTA decisions about highly innovative technologies in the EU – actions for stakeholders. Int. J. Technol. Assess. Health Care 36, 459–468 (2020).
14.
Gerstein HC, McMurray J, Holman RR. Real-world studies no substitute for RCTs in establishing efficacy. Lancet 393, 210–211 (2019).
15.
Asche CV, Seal B, Kahler KH et al. Evaluation of healthcare interventions and big data: review of associated data issues. Pharmacoeconomics 35, 759–765 (2017).
16.
Woolacott N, Corbett M, Jones-Diette J et al. Methodological challenges for the evaluation of clinical effectiveness in the context of accelerated regulatory approval: an overview. J. Clin. Epidemiol. 90, 108–118 (2017).
17.
Franklin JM, Patorno E, Desai RJ et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE Initiative. Circulation 143(10), 1002–1013 (2021).
18.
Makady A, van Veelen A, Jonsson P et al. Using real-world data in health technology assessment (HTA) practice: a comparative study of five HTA agencies. Pharmacoeconomics 36, 359–368 (2018).
19.
Anderson M, Naci H, Morrison D et al. A review of NICE appraisals of pharmaceuticals 2000–2016 found variation in establishing comparative clinical effectiveness. J. Clin. Epidemiol. 105, 50–59 (2019).
20.
Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst. Rev. 4, MR000034 (2014).
21.
Woolacott N, Corbett M, Jones-Diette J et al. Methodological challenges for the evaluation of clinical effectiveness in the context of accelerated regulatory approval: an overview. J. Clin. Epidemiol. 90, 108–118 (2017).
22.
ICH. Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials E9(R1). International Conference on Harmonization (2019). https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdf
23.
Agency for Healthcare Research and Quality. Developing a protocol for observational comparative effectiveness research: a user's guide. (2013). https://effectivehealthcare.ahrq.gov/products/observational-cer-protocol/research
24.
Bartlett VL, Dhruva SS, Shah ND et al. Feasibility of using real-world data to replicate clinical trial evidence. JAMA Netw. Open 2, e1912869 (2019).
25.
Franklin JM, Glynn RJ, Martin D et al. Evaluating the use of nonrandomized real-world data analyses for regulatory decision making. Clin. Pharmacol. Ther. 105, 867–877 (2019).
26.
Eichler HG, Bloechl-Daum B, Bauer P et al. “Threshold-crossing”: a useful way to establish the counterfactual in clinical trials? Clin. Pharmacol. Ther. 100, 699–712 (2016).
27.
Gatto NM, Reynolds RF, Campbell UB. A structured preapproval and post approval comparative study design framework to generate valid and transparent real-world evidence for regulatory decisions. Clin. Pharmacol. Ther. 106, 103–115 (2019).
28.
Moseley J, Vamvakas S, Berntgen M et al. Regulatory and health technology assessment advice on postlicensing and postlaunch evidence generation is a foundation for lifecycle data collection for medicines. Br. J. Pharmacol. 86, 1034–1051 (2020).
29.
Ng T, Ziomek J, Delaitre-Bonnin C et al. PHP276 – a review of the impact of integrated scientific advice for the optimisation of evidence generation for HTA appraisals. Value Health 21, S197 (2018).
30.
The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). The European Network of Centres for pharmacoepidemiology and pharmacovigilance (ENCePP) guide on methodological standards in pharmacoepidemiology (revision 8) (2020). http://www.encepp.eu/standards_and_guidances/documents/GuideMethodRev8.pdf
31.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183, 758–764 (2016).
32.
Schneeweiss S, Gagne JJ, Glynn RJ et al. Assessing the comparative effectiveness of newly marketed medications: methodological challenges and implications for drug development. Clin. Pharmacol. Ther. 90, 777–790 (2011).
33.
European Medicines Agency. Workshop on single-arm trials in oncology (2016). https://www.ema.europa.eu/en/events/workshop-single-arm-trials-oncology
34.
Institute for Quality and Efficiency in Health Care. [A19-43] Development of scientific concepts for the generation of routine practice data and their analysis for the benefit assessment of drugs according to §35a Social Code Book V – rapid report (2020). https://www.iqwig.de/en/projects-results/projects/drug-assessment/a19-43-development-of-scientific-concepts-for-the-generation-of-routine-practice-data-and-their-analysis-for-the-benefit-assessment-of-drugs-according-to-35a-social-code-book-v-rapid-report
35.
Schneeweiss S. Real-World Evidence of Treatment Effects: The Useful and the Misleading. Clin. Pharmacol. Ther. 106, 43–44 (2019).
36.
Lawlor DA, Tilling K, Smith GD. Triangulation in aetiological epidemiology. Int. J. Epidemiol. 45, 1866–1886 (2016).
37.
Sammon CJ, Leahy TP, Gsteiger S et al. Real-world evidence and nonrandomized data in health technology assessment: use existing methods to address unmeasured confounding? J. Comp. Eff. Res. 9(14), 969–972 (2020).
38.
Berger ML, Sox H, Willke RJ et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Value Health 20, 1003–1008 (2017).
39.
HTAi Global Policy Forum. Real-world evidence in the context of health technology assessment processes – from theory to action (2018). https://htai.org/wp-content/uploads/2019/02/HTAiGlobalPolicyForum2019_BackgroundPaper.pdf
40.
The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). ENCePP Resources Database. (2020). http://www.encepp.eu/encepp/resourcesDatabase.jsp
41.
Wang SV, Pinheiro S, Hua W et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ 372, m4856 (2021).
42.
Wang SV, Schneeweiss S, Berger ML et al. Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0. Value Health 20, 1009–1022 (2017).
43.
D'Andrea E, Vinals L, Patorno E et al. How well can we assess the validity of non-randomised studies of medications? A systematic review of assessment tools. BMJ Open 11, e043961 (2021).
44.
Sterne JA, Hernán MA, Reeves BC et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355, i4919 (2016).
45.
Deeks JJ, Dinnes J, D'Amico R et al. Evaluating non-randomised intervention studies. Health Technol. Assess. 7, 1–186 (2003).
46.
Kanavos P, Ferrario A, Tafuri G et al. Managing risk and uncertainty in health technology introduction: the role of managed entry agreements. Global Pol. 8, 84–92 (2017).
47.
Wenzl M, Chapman S. Performance-based managed entry agreements for new medicines in OECD countries and EU member states: how they work and possible improvements going forward (2019) (OECD Health Working Papers, No. 115). https://www.oecd-ilibrary.org/social-issues-migration-health/performance-based-managed-entry-agreements-for-new-medicines-in-oecd-countries-and-eu-member-states_6e5e4c0f-en
48.
Facey K, Whittal A, Drummond M et al. IMPACT HTA WP10 HTA appraisal framework suitable for rare disease treatments (2021). https://8c3e11d9-5f36-452f-abe3-c95befd6e85d.filesusr.com/ugd/e1a359_70bcc08bb2584a10b1717000e5d14fc9.pdf
49.
Head of Medicines Agencies and European Medicines Agency. HMA-EMA Joint Big Data Taskforce Phase II report: ‘Evolving Data-Driven Regulation’ (2020). https://www.ema.europa.eu/en/documents/other/hma-ema-joint-big-data-taskforce-phase-ii-report-evolving-data-driven-regulation_en.pdf
50.
National Institute for Health and Clinical Excellence. Increasing use of health and social care data in guidance development (2019). https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-guidelines/how-we-develop-nice-guidelines/data-and-analytics-statement-of-intent
Information & Authors
Information
Published In
Pages: 1035 - 1043
PubMed: 34279114
Copyright
© 2021 National Institute for Health and Care Excellence, UK. This work is licensed under the Creative Commons Attribution 4.0 License
History
Received: 4 May 2021
Accepted: 28 June 2021
Published online: 19 July 2021
Keywords:
Topics
Authors
Funding Information
Horizon 2020 Framework Programme: 779312
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
The use of nonrandomized evidence to estimate treatment effects in health technology assessment. (2021) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2021-0108
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Nensi Bralić, Ružica Bandić, Miro Vuković, Danira Matijaca, Renata Valsami, Panagiotis Zoumpoulakis, Kalle Schnitzer, Mari Levula, Risto Roine, Antti Viitala, Antti Pihlava, Olivia McDermott, Ana Marušić, Methodologies for generating and evaluating clinical and performance evidence for high-risk and innovative medical devices and in vitro diagnostics: a scoping review, Frontiers in Medical Technology, 10.3389/fmedt.2026.1857401, 8, (2026).
- Thomas P. Leahy, Craig Keenan, Emily Robertshaw, Megan McStravick, Sylvaine Barbier, Stephen Duffield, Seamus Kent, From Framework to Practice: Evaluating the Uptake of Sensitivity Analyses in The National Institute for Health and Care Excellence Real-World External Control Arm Evidence, Value in Health Regional Issues, 10.1016/j.vhri.2026.101651, (101651), (2026).
- Gemma Cupples, Emanuel Krebs, Deirdre Weymann, Cheryl Ho, Dean A. Regier, Adopting life-cycle HTA: a tumor-agnostic precision oncology index economic evaluation from publicly available reimbursement reviews, International Journal of Technology Assessment in Health Care, 10.1017/S0266462325100111, 41, 1, (2025).
- Olmo R. van den Akker, Robert T. Thibault, John P. A. Ioannidis, Susanne G. Schorr, Daniel Strech, Transparency in the secondary use of health data: assessing the status quo of guidance and best practices, Royal Society Open Science, 10.1098/rsos.241364, 12, 3, (2025).
- Danilo Di Maio, S. A. Mitchell, S. Batson, E. Keeney, Howard H. Z. Thom, Matching plus regression adjustment for the estimation of the average treatment effect on survival outcomes: a case study with mosunetuzumab in relapsed/refractory follicular lymphoma, BMC Medical Research Methodology, 10.1186/s12874-025-02456-x, 25, 1, (2025).
- Stefania Manetti, Elisa Guidotti, Federico Vola, Milena Vainieri, A systematic literature review of real-world evidence (RWE) on post-market assessment of medical devices, Health Economics, Policy and Law, 10.1017/S1744133124000148, 21, 2, (181-213), (2025).
- Konstantinos Zisis, Elpida Pavi, Mary Geitona, Kostas Athanasakis, Real-world data: a comprehensive literature review on the barriers, challenges, and opportunities associated with their inclusion in the health technology assessment process, Journal of Pharmacy & Pharmaceutical Sciences, 10.3389/jpps.2024.12302, 27, (2024).
- Li Jiu, Michiel Hartog, Junfeng Wang, Rick A Vreman, Olaf H Klungel, Aukje K Mantel-Teeuwisse, Wim G Goettsch, Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis, BMJ Open, 10.1136/bmjopen-2023-075173, 14, 2, (e075173), (2024).
- Thomas P. Leahy, Isabelle Durand-Zaleski, Laura Sampietro-Colom, Seamus Kent, York Zöllner, Doug Coyle, Gianluigi Casadei, The role of quantitative bias analysis for nonrandomized comparisons in health technology assessment: recommendations from an expert workshop, International Journal of Technology Assessment in Health Care, 10.1017/S0266462323002702, 39, 1, (2023).
- Leah Burns, Nadege Le Roux, Robert Kalesnik-Orszulak, Jennifer Christian, Jennifer Dudinak, Frank Rockhold, Sean Khozin, John O’Donnell, Real-world evidence for regulatory decision-making: updated guidance from around the world, Frontiers in Medicine, 10.3389/fmed.2023.1236462, 10, (2023).