Advancing the role of real-world evidence in comparative effectiveness research
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: Comparative effectiveness research (CER) is essential for making informed decisions about drug access. It provides insights into the effectiveness and safety of new drugs compared with existing treatments, thereby guiding better healthcare decisions and ensuring that new therapies meet the real-world needs of patients and healthcare systems. Objective: To provide a tool that assists analysts and decision-makers in identifying the most suitable analytical approach for answering a CER question, given specific data availability contexts. Methods: A systematic literature review of the scientific literature was performed and existing regulatory and health technology assessment (HTA) guidance were evaluated to identify and compare recommendations and best practices. Based on this review a methods flowchart that synthesizes current practices and requirements was proposed. Results: The review did not find any papers that clearly identified the most appropriate analytical approach for answering CER questions under various conditions. Therefore, a methods flowchart was designed to inform analyst and decision makers choices starting from a well-defined scientific question. Conclusion: The proposed methods flowchart offers clear guidance on CER methodologies across a range of settings and research needs. It begins with a well-defined research question and considers multiple feasibility aspects related to CER. This tool aims to standardize methods, ensure rigorous and consistent research quality and promote a culture of evidence-based decision-making in healthcare.
Shareable abstract
This study introduces a methods flowchart to guide comparative effectiveness research (CER). It helps identify the best analytical approach, ensuring standardized, high-quality, evidence-based healthcare decisions. #CER #decision-making #comparative-effectiveness, #evidence-based
Plain language summary
What is this article about?
This article discusses how researchers and healthcare decision-makers can determine the best way to compare the effectiveness of different drugs. This type of research, known as comparative effectiveness research (CER), helps to make better healthcare decisions by providing information on how new drugs perform compared with existing treatments. The article aims to offer a tool that guides analysts in choosing the right method for their CER based on the data they have.
What were the results?
The study found that there are no existing papers that clearly explain which method to use for different CER questions under various conditions. To address this gap, the authors created a tool - a methods' flowchart - that will facilitate a transparent way of choosing which method should be used for a specific CER question. This tool helps researchers start with a specific question and then choose the best method to answer it, rather than forcing a one-size-fits-all approach.
What do the results of the study mean?
The results mean that researchers and decision-makers now have a clear guide to help them choose the most appropriate methods for their CER questions. This new tool aims to make CER more standardized and consistent, which can lead to higher quality research and better, evidence-based decisions in healthcare. Ultimately, this can improve patient care by ensuring that new therapies meet real-world needs.
Supplementary Material
File (supplementary material.docx)
- Download
- 77.89 KB
References
1.
Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann. Intern. Med. 151(3), 203–205 (2009).
2.
Luke JJ. The role of comparative effectiveness research in developing clinical guidelines and reimbursement policies. Virtual Mentor 13(1), 52 (2011).
3.
Price-Haywood EG. Clinical comparative effectiveness research through the lens of healthcare decisionmakers. Ochsner J. 15(2), 154–161 (2015).
4.
Tugwell P, Knottnerus JA. Is the ‘Evidence-Pyramid’ now dead? J. Clin. Epidemiol. 68(11), 1247–1250 (2015).
5.
Luce BR, Drummond M, Jönsson B et al. EBM, HTA, and CER: clearing the confusion. Milbank Q. 88(2), 256–276 (2010).
6.
Wang SV, Schneeweiss S, Gagne JJ et al. Using real-world data to extrapolate evidence from randomized controlled trials. Clin. Pharmacol. Ther. 105(5), 1156–1163 (2019).
7.
Doebele RC, Perez L, Trinh H et al. Comparative effectiveness analysis between entrectinib clinical trial and crizotinib real-world data in ROS1+ NSCLC. J. Comp. Eff. Res. 10(17), 1271–1282 (2021).
8.
Gong Y, Kehl KL, Oxnard GR, Khozin S, Mishra-Kalyani PS, Blumenthal GM. Time to treatment discontinuation (TTD) as a pragmatic endpoint in metastatic non-small cell lung cancer (mNSCLC): a pooled analysis of 8 trials. J. Clin. Oncol. 36(Suppl. 15) (2018).
9. .
Berger M, Daniel G, Frank K. A Framework for Regulatory Use of Real-World Evidence. Duke Margolis Center for Health Policy White Paper (2017).https://healthpolicy.duke.edu/sites/default/files/2020-08/rwe_white_paper_2017.09.06.pdf
10. .
FDA. Food & Drug administration work plan and proposed funding allocations of FDA innovation account (2017). https://www.fda.gov/downloads/RegulatoryInformation/LawsEnforcedbyFDA/SignificantAmendmentstotheFDCAct/21stCenturyCuresAct/UCM562852.pdf
11. .
FDA. Framework for FDA's Real-World Evidence Program (2018). https://www.fda.gov/media/120060/download?attachment
12. .
FDA. Considerations for the Use of Real-World Data and Real-World Evidence To Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry (2023). https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug
13.
Nicotera G, Sferrazza G, Serafino A, Pierimarchi P. The iterative development of medicines through the European Medicine Agency's Adaptive Pathway Approach. Front. Med. 6, 148 (2019).
14.
Cave A, Kurz X, Arlett P. Real-world data for regulatory decision making: challenges and possible solutions for Europe. Clin. Pharmacol. Ther. 106(1), 36–39 (2019).
15. .
EMA. HMA-EMA Joint Big Data Taskforce—summary report (2019). https://www.ema.europa.eu/en/documents/minutes/hmaema-joint-task-force-big-data-summary-report_en.pdf
16.
Nishioka K, Makimura T, Ishiguro A, Nonaka T, Yamaguchi M, Uyama Y. Evolving acceptance and use of RWE for regulatory decision making on the benefit/risk assessment of a drug in Japan. Clin. Pharmacol. Ther. 111(1), 35–43 (2022).
17.
Burns L, Roux NL, Kalesnik-Orszulak R et al. Real-world evidence for regulatory decision-making: guidance from around the world. Clin. Ther. 44(3), 420–437 (2022).
18.
Facey KM, Rannanheimo P, Batchelor L, Borchardt M, de Cock J. Real-world evidence to support Payer/HTA decisions about highly innovative technologies in the EU—actions for stakeholders. Int. J. Technol. Assess. Heal. Care 36(4), 459–468 (2020).
19.
Cook R, Lovell A, Khanzada A, Barham L. Value of real-world evidence in health technology assessment: lost in translation? Economist Impact (2022). https://impact.economist.com/projects/rwe-in-hta/
20. .
EU HTA CG. Practical Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons (2024). https://health.ec.europa.eu/latest-updates/practical-guideline-quantitative-evidence-synthesis-direct-and-indirect-comparisons-2024-03-25_en
21.
Page MJ, McKenzie JE, Bossuyt PM et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021).
22.
Cooke A, Smith D, Booth A. Beyond PICO. Qual. Heal. Res. 22(10), 1435–1443 (2012).
23.
Alemayehu D, Cappelleri JC. Revisiting issues, drawbacks and opportunities with observational studies in comparative effectiveness research. J. Eval. Clin. Pr. 19(4), 579–583 (2013).
24.
Neely JG, Sharon JD, Graboyes EM et al. Practical guide to understanding comparative effectiveness research (CER). Otolaryngol. Head Neck Surg. 149(6), 804–812 (2013).
25.
Luce BR, Drummond MF, Dubois RW et al. Principles for planning and conducting comparative effectiveness research. J. Comp. Eff. Res. 1(5), 431–440 (2012).
26.
Guise J, Viswanathan M. Overview of best practices in conducting comparative-effectiveness reviews. Clin. Pharmacol. Ther. 90(6), 876–882 (2011).
27.
Tsertsvadze A, Maglione M, Chou R et al. Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program. J. Clin. Epidemiology 64(11), 1208–1215 (2011).
28. .
PBAC. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (PBAC) Section 2 Clinical evaluation (2016). https://pbac.pbs.gov.au/section2-clinical-evaluation.html
29. .
CADTH. Indirect Evidence: Indirect Treatment Comparisons in Meta-Analysis (2009). https://www.cadth.ca/sites/default/files/pdf/H0462_itc_tr_e.pdf
30. .
EUnetHTA. Levels of Evidence: internal validity of randomised controlled trials (2015). https://www.eunethta.eu/wp-content/uploads/2018/01/16_WP7-SG3-GL-int_val_RCTs_amend2015.pdf
31. .
EUnetHTA. Internal validity of non-randomised studies (NRS) on interventions (2015). https://www.eunethta.eu/wp-content/uploads/2018/01/Internal-validity-of-non-randomised-studies-NRS-on-interventions_Guideline_Final-Jul-2015.pdf
32.
EUnetHTA. Comparators & Comparisons: direct and indirect comparisons (2024). https://www.eunethta.eu/wp-content/uploads/2018/03/Direct_comparators_comparisons.pdf
33.
EUnetHTA. Levels of Evidence. Applicability of evidence for the context of a reletive effectiveness assessment (2015). https://www.eunethta.eu/wp-content/uploads/2018/01/Levels-of-Evidence-Applicability-of-evidence-for-the-context-of-a-relative-effectiveness-assessment_Amended-JA1-Guideline_Final-Nov-2015.pdf
34.
EU HTA CG. Methodological Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons (2024). https://health.ec.europa.eu/latest-updates/methodological-guideline-quantitative-evidence-synthesis-direct-and-indirect-comparisons-2024-03-25_en
35.
IQWiG. General Methods Version 7.0 (2023). https://www.iqwig.de/methoden/general-methods_version-7-0.pdf
36.
HAS. Indirect comparisons. Methods and validity (2019). https://www.has-sante.fr/upload/docs/application/pdf/2011-02/summary_report__indirect_comparisons_methods_and_validity_january_2011_2.pdf
37.
HAS. Choices in methods for economic evaluation – HAS (2024). https://www.has-sante.fr/upload/docs/application/pdf/2020-11/methodological_guidance_2020_-choices_in_methods_for_economic_evaluation.pdf
38.
HAS. Doctrine de la commission de la transparence (CT). Principes d'évaluation de la CT relatifs aux médicaments en vue de leur accès au remboursement (2023). https://www.has-sante.fr/upload/docs/application/pdf/2021-03/doctrine_ct.pdf
39.
Vanier A, Fernandez J, Kelley S et al. Rapid access to innovative medicinal products while ensuring relevant health technology assessment. Position of the French National Authority for Health. BMJ Évid.-Based Med. 29(1), 1–5 (2024).
40. .
Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 2: A Generalised Linear Modelling Framework for Pairwise and Network Meta-Analysis of Randomised Controlled Trials (2014). https://www.ncbi.nlm.nih.gov/books/NBK310366/pdf/Bookshelf_NBK310366.pdf
41.
Dias S, Welton NJ, Sutton AJ, Ades A. NICE DSU Technical Support Document 1: Introduction To Evidence Synthesis For Decision Making (2012). https://www.ncbi.nlm.nih.gov/books/NBK395870/pdf/Bookshelf_NBK395870.pdf
42.
Ades A, Caldwell DM, Reken S, Welton NJ, Sutton AJ, Dias S. NICE DSU Technical Support Document 7: Evidence Synthesis Of Treatment Efficacy In Decision Making: A Reviewer's Checklist (2012). https://www.sheffield.ac.uk/sites/default/files/2022-02/TSD7-reviewer-checklist.final_.08.05.12.pdf
43.
Dias S, Welton NJ, Sutton AJ, Caldwell DM, Lu G, Ades A. NICE DSU Technical Support Document: Inconsistency In Networks of Evidence Based On Randomised Controlled Trials (2014). https://www.sheffield.ac.uk/sites/default/files/2022-02/TSD4-Inconsistency.final_.15April2014.pdf
44.
Faria R, Alava MH, Manca A, Wailoo AJ. NICE DSU Technical Support Document: The Use Of Observational Data To Informa Estimates Of Treatment Effectiveness In Technology Appraisal: Methods For Comparative Individual Patient Data (2015). https://www.sheffield.ac.uk/nice-dsu/tsds/observational-data
45.
Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support Document 18: Methods for Population-Adjusted Indirect Comparisons In Submissions To NICE (2016). https://www.sheffield.ac.uk/nice-dsu/tsds/population-adjusted
46.
NICE. CHTE2020 sources and synthesis of evidence (2020). https://www.sheffield.ac.uk/nice-dsu/methods-development/chte2020-sources-and-synthesis-evidence
47.
NICE. NICE real-world evidence framework (ECD9) (2022). https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837
48.
ICER. A Guide to ICER's Methods for Health Technology Assessment (2020). https://icer.org/wp-content/uploads/2021/01/ICER_HTA_Guide_102720.pdf
49.
ENCePP. The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) Guide on Methodological Standards in Pharmacoepidemiology. EMA/95098/2010 Rev.11 (2010). https://encepp.europa.eu/document/download/f6e403a6-8033-4c22-a5ff-195ba3666299_en?filename=01.ENCePPMethodsGuideRev.11.pdf
50.
FDA. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products Guidance for Industry (2023). https://www.fda.gov/media/164960/download
51.
Richardson WS, Wilson MMC, Nishikawa MJ, Hayward MRSA. The well-built clinical question: a key to evidence-based decisions. ACP J. Club. 123(3), A12–A13 (1995).
52.
Bhatt A. Evolution of clinical research: a history before and beyond James Lind. Perspect. Clin. Res. 3(4), 149–149 (2010).
53.
Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin. Neurosci. 13(2), 217–224 (2011).
54. .
Propes C, Sheehan S, Hendricks-Sturrup R. Point-of-Care Clinical Trials: Integrating Research and Care Delivery (2022). https://healthpolicy.duke.edu/sites/default/files/2022-05/Point-of-Care%20Clinical%20Trials%20Intergrating%20Research%20Care%20Delivery.pdf
55. .
EFPIA. An EFPIA position paper on randomised pragmatic trials to generate high-quality real-world evidence for regulatory decisions (2023). https://www.efpia.eu/media/guokpw0b/an-efpia-position-paper-on-randomised-pragmatic-trials-to-generate-high-quality-real-world-evidence-for-regulatory-decisions.pdf
56.
PoklepovićPeričić T, Tanveer S. Why systematic reviews matter. Elsevier Connect (2019). https://impact.economist.com/projects/rwe-in-hta/ and https://www.elsevier.com/connect/why-systematic-reviews-matter
57.
Higgins JPT, Thomas J, Chandler Jet al. et al. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane (2023).
58.
Dias S, Ades AE, Welton NJ, Jansen JP, Sutton AJ. Network Meta-Analysis for Decision-Making. (2018). https://onlinelibrary.wiley.com/doi/book/10.1002/9781118951651 Print ISBN:9781118647509 |Online ISBN:9781118951651
59.
Sutton A, Ades AE, Cooper N, Abrams K. Use of Indirect and Mixed Treatment Comparisons for Technology Assessment. Pharmacoeconomics 26(9), 753–767 (2008).
60.
Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. Methods for population-adjusted indirect comparisons in health technology appraisal. Méd. Decis. Mak. 38(2), 200–211 (2017).
61.
Phillippo DM, Dias S, Ades AE et al. Multilevel network meta-regression for population-adjusted treatment comparisons. J. R. Stat. Soc.: Ser. A (Stat. Soc.) 183(3), 1189–1210 (2020).
62.
Kawahara T, Fukuda M, Oba K, Sakamoto J, Buyse M. Meta-analysis of randomized clinical trials in the era of individual patient data sharing. Int. J. Clin. Oncol. 23(3), 403–409 (2018).
63.
Ramsey SD, Onar-Thomas A, Wheeler SB. Real-world database studies in oncology: a call for standards. J. Clin. Oncol. 42(9), 977–980 (2024).
64.
Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar. Behav. Res. 46(3), 399–424 (2011).
65. .
Hendricks-Sturrup R, Tian F, Izurieta H et al. Understanding the Use of Negative Controls to Assess the Validity of Non-Interventional Studies of Treatment Using Real-World Evidence (2023). https://healthpolicy.duke.edu/events/understanding-use-negative-controls-assess-validity-non-interventional-studies-treatment
66.
Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 327(7414), 557 (2003).
67.
Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat. Med. 21(11), 1559–1573 (2002).
68.
Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat. Med. 28(25), 3083–3107 (2009).
69.
Pearl J. Causal diagrams for empirical research. Biometrika 82(4), 669–688 (1995).
70.
VanderWeele TJ, Ding P. Sensitivity analysis in observational research: introducing the E-Value. Ann. Intern. Med. 167(4), 268 (2017).
71.
Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. Good practices for quantitative bias analysis. Int. J. Epidemiology 43(6), 1969–1985 (2014).
72.
Degtiar I, Rose S. A review of generalizability and transportability. Ann. Rev. Stat. App. 10, 501–524 (2023).
73.
Bareinboim E, Pearl J. A general algorithm for deciding transportability of experimental results. J. Causal Inference 1(1), 107–134 (2013).
74.
Turner AJ, Sammon C, Latimer N et al. Transporting comparative effectiveness evidence between countries: considerations for health technology assessments. Pharmacoeconomics 42(2), 165–176 (2024).
75.
Chu P, Antoniou M, Bhutani MK, Aziez A, Daigl M. Matching-adjusted indirect comparison: entrectinib versus crizotinib in ROS1 fusion-positive non-small cell lung cancer. J. Comp. Eff. Res. 9(15), 861–876 (2020).
76. .
FDA. FDA approves entrectinib for NTRK solid tumors and ROS-1 NSCLC (2019). https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-entrectinib-ntrk-solid-tumors-and-ros-1-nsclc
77.
EMA. Rozlytrek EPAR (2020). https://www.ema.europa.eu/en/medicines/human/EPAR/rozlytrek
78.
CADTH. Entrectinib (Rozlytrek) for ROS1-positive Non-Small Cell Lung Cancer (2021). https://www.cadth.ca/entrectinib-rozlytrek-ros1-positive-non-small-cell-lung-cancer
79.
NICE. Entrectinib for treating ROS1-positive advanced non-small-cell lung cancer (2020). https://www.nice.org.uk/guidance/ta643/chapter/1-Recommendations
80.
Seale JP, Gebski VJ, Keech AC. Generalising the results of trials to clinical practice. Méd. J. Aust. 181(10), 558–560 (2004).
81.
Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J. Clin. Epidemiol. 50(6), 683–691 (1997).
82.
Ribeiro TB, Bennett CL, Colunga-Lozano LE, Araujo APV, Hozo I, Djulbegovic B. Increasing FDA-accelerated approval of single-arm trials in oncology (1992 to 2020). J. Clin. Epidemiol. 159, 151–158 (2023).
83.
Subbiah V. The next generation of evidence-based medicine. Nat. Med. 29(1), 49–58 (2023).
84. .
Walker A, PatrickAR, LauerMS et al. A tool for assessing the feasibility of comparative effectiveness research. Comp. Eff. Res. 3, 11 –20 (2013).
85.
Marko NF, Weil RJ. The role of observational investigations in comparative effectiveness research. Value Health 13(8), 989–997 (2010).
86.
Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part I. Value Health 12(8), 1044–1052 (2009).
87.
Andre EB, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs using real-world data: the changing landscape of the regulatory approval process. Pharmacoepidemiol. Drug Saf. 29(10), 1201–1212 (2020).
88.
Abogunrin S, Ashaye AO, Cappelleri JC et al. Safety and effectiveness of classical and alternative sunitinib dosing schedules for metastatic renal cell carcinoma: a meta-analysis. Futur. Oncol. 15(18), 2175–2190 (2019).
89.
Hamza T, Chalkou K, Pellegrini F et al. Synthesizing cross-design evidence and cross-format data using network meta-regression. Res. Synth. Methods 14(2), 283–300 (2023).
90.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183(8), 758–764 (2016).
91.
Verde PE, Ohmann C. Combining randomized and non-randomized evidence in clinical research: a review of methods and applications. Res. Synth. Methods 6(1), 45–62 (2015).
92.
Li G, Taljaard M, Heuvel ERV den et al. An introduction to multiplicity issues in clinical trials: the what, why, when and how. Int. J. Epidemiology 46(2), 746–755 (2017).
93.
PCORI. The Value of Engagement in Research (2024). https://www.pcori.org/engagement/value-engagement-research#:∼:text=The%20meaningful%20involvement%20of%20patients,study%2C%20and%20disseminating%20study%20results
Information & Authors
Information
Published In
Copyright
© 2024 The authors. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 2 July 2024
Accepted: 23 September 2024
Published online: 11 October 2024
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Advancing the role of real-world evidence in comparative effectiveness research. (2024) Journal of Comparative Effectiveness Research. DOI: 10.57264/cer-2024-0101
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Grammati Sarri, Bengt Liljas, Keith R. Abrams, Stephen J. Duffield, Murtuza Bharmal, Mapping the Use of Real-World Evidence Across the EU Health Technology Assessment Regulation: Methodological Considerations, Challenges, and Opportunities for Harmonization, Journal of Market Access & Health Policy, 10.3390/jmahp14020020, 14, 2, (20), (2026).
- Lyndsay A. Nelson, Erin M. Bergner, Amos Adler, Charles Yoon, Kathleen Mieras, Ed Simeone, Jeremy I. Lema-Driscoll, Brendan M. Everett, Deborah J. Wexler, Lindsay S. Mayberry, Development and Value of a Centralized Text Message-based Tracking and Support Program for the PRECIDENTD Study, Medical Care, 10.1097/MLR.0000000000002246, 64, 3S, (S242-S250), (2026).
- Antal T. Zemplenyi, Nai-Chia Chen, Kelly E. Anderson, Blythe Adamson, Michael J. DiStefano, Kavita V. Nair, Robert B. McQueen, Key Considerations for Assessing Real-World Comparative Effectiveness in the Context of the Drug Price Negotiation Program: A Case Study of Pembrolizumab, PharmacoEconomics, 10.1007/s40273-025-01514-y, 43, 9, (1147-1160), (2025).