Skip to main content
Search
Search
Advanced Search
Skip main navigation
Open access
Systematic Review
4 October 2024

Use of transportability methods for real-world evidence generation: a review of current applications

Authors: Natalie S Levy https://orcid.org/0000-0002-6828-8237, Patrick J Arena https://orcid.org/0000-0003-3024-771X, Thomas Jemielita https://orcid.org/0000-0001-7788-1828, Shahrul Mt-Isa https://orcid.org/0000-0002-7163-8402, Shane McElwee https://orcid.org/0000-0003-3694-441X, David Lenis https://orcid.org/0000-0001-8697-1737, Ulka B Campbell https://orcid.org/0000-0002-3503-8178, Ashley Jaksa https://orcid.org/0000-0003-3571-3345, and Gleicy M Hair https://orcid.org/0000-0002-9106-7424 [email protected]Author Info & Affiliations
Volume 13, Number 11
https://doi.org/10.57264/cer-2024-0064
PDFFormats

Abstract

Aim: To evaluate how transportability methods are currently used for real-world evidence (RWE) generation to inform good practices and support adoption and acceptance of these methods in the RWE context. Methods: We conducted a targeted literature review to identify studies that transported an effect estimate of the clinical effectiveness or safety of a biomedical exposure to a target real-world population. Records were identified from PubMed-indexed articles published any time before 25 July 2023 (inclusive). Two reviewers screened abstracts/titles and reviewed the full text of candidate studies to identify the final set of articles. Data on the therapeutic area, exposure(s), outcome(s), original and target populations and details of the transportability analysis (e.g., analytic method used, estimate transported, stated assumptions) were abstracted from each article. Results: Of 458 unique records identified, six were retained in the final review. Articles were published during 2021–2023, focused on the US/Canada context, and covered a range of therapeutic areas. Four studies transported an RCT effect estimate, while two transported effect estimates derived from real-world data. Almost all articles used weighting methods to transport estimates. Two studies discussed all transportability assumptions, and one evaluated the likelihood of meeting all assumptions and the impact of potential violations. Conclusion: The use of transportability methods for RWE generation is an emerging and promising area of research to address evidence gaps in settings with limited data and infrastructure. More transparent and rigorous reporting of methods, assumptions and limitations may increase the use and acceptability of transportability for producing robust evidence on treatment effectiveness and safety.

Plain language summary

What is this article about?

In this article, we investigated whether and how statistical methods known as ‘transportability methods’ have been applied in published studies using data collected during routine healthcare, known as real-world evidence (RWE) studies. Transportability methods use a result based on data from one population to estimate the result for another population by adjusting for relevant differences in demographic, clinical, and/or other factors between the two populations. These methods may help decision-makers evaluate whether evidence from another location can inform assessments of the safety and effectiveness of medical products in their jurisdiction. We conducted a targeted review of the published literature to understand if and how transportability methods have been applied to RWE studies.

What were the results?

After reviewing 458 potential studies identified in our literature search, we found that only six used transportability methods to generate an estimate of the clinical effectiveness or safety of a biomedical exposure for a target real-world population. These studies were all published during 2021–2023 using data from the US or Canada and used similar statistical methods. Two studies discussed all the statistical assumptions and limitations of transportability methods.

What do the results mean?

Transportability methods are just beginning to be used for RWE generation but may help fill evidence gaps in places or situations where relevant data are not available. Additionally, clear and thorough reporting of assumptions and limitations may facilitate the use and acceptability of transportability methods for producing robust RWE on treatment effectiveness and safety.

Supplementary Material

File (supplementary materials.docx)

References

1.
O'Rourke B, Oortwijn W, Schuller T. International Joint Task Group. The new definition of health technology assessment: a milestone in international collaboration. Int. J. Technol. Assess Health Care 36(3), 187–190 (2020).
PubMed
Google Scholar
2.
Goodman C. HTA 101 - Introduction to Health Technology Assessment. United States National Library of Medicine, MD, USA (2014).
Google Scholar
3.
O'Donnell JC, Pham SV, Pashos CL, Miller DW, Smith MD. Health Technology Assessment: Lessons learned from around the world – an overview. Value Health 12, S1–S5 (2009).
PubMed
Google Scholar
4.
Hogervorst MA, Pontén J, Vreman RA, Mantel-Teeuwisse AK, Goettsch WG. Real World Data in Health Technology Assessment of Complex Health Technologies. Front. Pharmacol. 13, (2022).
PubMed
Google Scholar
5.
De Meulemeester J, Fedyk M, Jurkovic L et al. Many randomized clinical trials may not be justified: a cross-sectional analysis of the ethics and science of randomized clinical trials. J. Clin. Epidemiol. 97, 20–25 (2018).
PubMed
Google Scholar
6.
Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trials unnecessary? Picking signal from noise. BMJ 334(7589), 349–351 (2007).
PubMed
Google Scholar
7.
Simpson A, Ramagopalan SV. R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 10. J. Comp. Eff. Res. 12(1), e220194 (2023).
PubMed
Google Scholar
8.
Center for Drug Evaluation and Research, Center for Biologics Evaluation, Oncology Center of Excellence. Considerations for the Use of Real-World Data and Real-World Evidence to Support Regulatory Decision-Making for Drug and Biological Products, U.S. Food and Drug Administration. (2023). https://www.fda.gov/media/171667/download.
Google Scholar
9.
National Institute for Health and Care Excellence. NICE real-world evidence framework, National Institute for Health and Care Excellence. (2022). https://www.nice.org.uk/corporate/ecd9/chapter/methods-for-real-world-studies-of-comparative-effects.
Google Scholar
10.
Gatto NM, Campbell UB, Rubinstein E et al. The structured process to identify fit-for-purpose data: a data feasibility assessment framework. Clin. Pharmacol. Ther. 111(1), 122–134 (2022).
PubMed
Google Scholar
11.
European Medicines Agency, European Medicines Regulatory Network. “DARWIN EU” (2023). https://www.darwin-eu.org/.
Google Scholar
12.
Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. E5 Ethnic Factors in the Acceptability of Foreign Clinical Data., U.S. Food and Drug Administration. (1998). https://www.fda.gov/media/71287/download.
Google Scholar
13.
Drummond M, Barbieri M, Cook J et al. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force Report. Value Health 12(4), 409–418 (2009).
PubMed
Google Scholar
14.
Goeree R, He J, O'Reilly D et al. Transferability of health technology assessments and economic evaluations: a systematic review of approaches for assessment and application. Clin. Outcomes Res. CEOR 3, 89–104 (2011).
PubMed
Google Scholar
15.
Jaksa A, Arena PJ, Chan KKW, Ben-Joseph RH, Jónsson P, Campbell UB. Transferability of real-world data across borders for regulatory and health technology assessment decision-making. Front. Med. 9 (2022).
Google Scholar
16.
Turner AJ, Sammon C, Latimer N et al. Transporting comparative effectiveness evidence between countries: considerations for health technology assessments. Pharmacoeconomics (2023).
PubMed
Google Scholar
17.
Degtiar I, Rose S. A review of generalizability and transportability. Annu. Rev. Stat. Its Appl. 10(1), 501–524 (2023).
Google Scholar
18.
Pearl J, Bareinboim E. Transportability of causal and statistical relations: a formal approach. Presented at: Proceedings of the 25th AAAI Conference on Artificial Intelligence. AAAI Press, CA, USA (August, 2011).
Google Scholar
19.
Westreich D, Edwards JK, Lesko CR, Stuart E, Cole SR. Transportability of trial results using inverse odds of sampling weights. Am. J. Epidemiol. 186(8), 1010–1014 (2017).
PubMed
Google Scholar
20.
Ling AY, Montez-Rath ME, Carita P et al. An overview of current methods for real-world applications to generalize or transport clinical trial findings to target populations of interest. Epidemiol. Camb. Mass 34(5), 627–636 (2023).
PubMed
Google Scholar
21.
Dahabreh IJ, Robertson SE, Steingrimsson JA, Stuart EA, Hernán MA. Extending inferences from a randomized trial to a new target population. Stat. Med. 39(14), 1999–2014 (2020).
PubMed
Google Scholar
22.
Pearl J, Bareinboim E. External validity: from do-calculus to transportability across populations. Stat. Sci. 29(4), 579–595 (2014).
Google Scholar
23.
Stuart EA, Bradshaw CP, Leaf PJ. Assessing the generalizability of randomized trial results to target populations. Prev. Sci. Off. J. Soc. Prev. Res. 16(3), 475–485 (2015).
PubMed
Google Scholar
24.
Kern HL, Stuart EA, Hill J, Green DP. Assessing methods for generalizing experimental impact estimates to target populations. J. Res. Educ. Eff. 9(1), 103–127 (2016).
PubMed
Google Scholar
25.
Rudolph KE, van der Laan MJ. Robust estimation of encouragement-design intervention effects transported across sites. J. R. Stat. Soc. Ser. B Stat. Methodol. 79(5), 1509–1525 (2017).
PubMed
Google Scholar
26.
Dong N, Stuart EA, Lenis D, Quynh Nguyen T. Using propensity score analysis of survey data to estimate population average treatment effects: a case study comparing different methods. Eval. Rev. 44(1), 84–108 (2020).
PubMed
Google Scholar
27.
Lu H, Cole SR, Howe CJ, Westreich D. Toward a clearer definition of selection bias when estimating causal effects. Epidemiol. Camb. Mass 33(5), 699–706 (2022).
PubMed
Google Scholar
28.
Hernán MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC, FL, USA (2020).
Google Scholar
29.
Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensity scores to assess the generalizability of results from randomized trials. J. R. Stat. Soc. Ser. A Stat. Soc. 174(2), 369–386 (2001).
PubMed
Google Scholar
30.
Inoue K, Hsu W, Arah OA, Prosper AE, Aberle DR, Bui AAT. Generalizability and transportability of the national lung screening trial data: extending trial results to different populations. Cancer Epidemiol. Biomark. Prev. Publ. Am. Assoc. Cancer Res. Cosponsored Am. Soc. Prev. Oncol. 30(12), 2227–2234 (2021).
PubMed
Google Scholar
31.
Effective Health Care Program. Research report: developing a protocol for observational comparative effectiveness research: a user's guide. Agency for Healthcare Research and Quality, MD, USA (2019). https://effectivehealthcare.ahrq.gov/products/observational-cer-protocol/research.
Google Scholar
32.
Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data. US Food and Drug Administration (2013). https://www.fda.gov/media/79922/download.
Google Scholar
33.
Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products. US Food and Drug Administration (2021). https://www.fda.gov/media/152503/download.
Google Scholar
34.
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. [A19-43] Development of Scientific Concepts for the Generation of Routine Practice Data and Their Analysis for the Benefit Assessment of Drugs According to §35a Social Code Book V—rapid report, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. (2020). https://www.iqwig.de/en/projects/a19-43.html.
Google Scholar
35.
Jaksa A, Wu J, Jónsson P, Eichler H-G, Vititoe S, Gatto NM. Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance. J. Comp. Eff. Res. 10(9), 711–731 (2021).
PubMed
Google Scholar
36.
Atkins D, Best D, Briss PA et al. Grading quality of evidence and strength of recommendations. BMJ 328(7454), 1490 (2004).
PubMed
Google Scholar
37.
Sterne JA, Hernán MA, Reeves BC et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355, i4919 (2016).
PubMed
Google Scholar
38.
Webster-Clark M, Toh S, Arnold J, McTigue KM, Carton T, Platt R. External validity in distributed data networks. Pharmacoepidemiol. Drug Saf. 32(12), 1360–1367 (2023).
PubMed
Google Scholar
39.
Westreich D, Edwards JK. Invited commentary: every good randomization deserves observation. Am. J. Epidemiol. 182(10), 857–860 (2015).
PubMed
Google Scholar
40.
Dahabreh IJ, Haneuse SJ-PA, Robins JM et al. Study designs for extending causal inferences from a randomized trial to a target population. Am. J. Epidemiol. 190(8), 1632–1642 (2021).
PubMed
Google Scholar
41.
Gershman B, Guo DP, Dahabreh IJ. Using observational data for personalized medicine when clinical trial evidence is limited. Fertil. Steril. 109(6), 946–951 (2018).
PubMed
Google Scholar
42.
Dahabreh IJ, Matthews A, Steingrimsson JA, Scharfstein DO, Stuart EA. Using trial and observational data to assess effectiveness: trial emulation, transportability, benchmarking, and joint analysis. Epidemiol. Rev. mxac011 (2023).
PubMed
Google Scholar
43.
Josey KP, Yang F, Ghosh D, Raghavan S. A calibration approach to transportability and data-fusion with observational data. Stat. Med. 41(23), 4511–4531 (2022).
PubMed
Google Scholar
44.
Wu Y, Hui J, Deng Q. Empirical profile Bayesian estimation for extrapolation of historical adult data to pediatric drug development. Pharm. Stat. 19(6), 787–802 (2020).
PubMed
Google Scholar
45.
Lee D, Yang S, Dong L, Wang X, Zeng D, Cai J. Improving trial generalizability using observational studies. Biometrics 79(2), 1213–1225 (2023).
PubMed
Google Scholar
46.
Montez-Rath ME, Lubwama R, Kapphahn K et al. Characterizing real world safety profile of oral Janus kinase inhibitors among adult atopic dermatitis patients: evidence transporting from the rheumatoid arthritis population. Curr. Med. Res. Opin. 38(8), 1431–1437 (2022).
PubMed
Google Scholar
47.
Cook RR, Foot C, Arah OA et al. Estimating the impact of stimulant use on initiation of buprenorphine and extended-release naltrexone in two clinical trials and real-world populations. Addict. Sci. Clin. Pract. 18(1), 11 (2023).
PubMed
Google Scholar
48.
Basu A, Patel C, Fu AZ, Brown B, Mavros P, Benson C. Real-world calibration and transportability of the Disease Recovery Evaluation and Modification (DREaM) randomized clinical trial in adult Medicaid beneficiaries with recent-onset schizophrenia. J. Manag. Care Spec. Pharm. 29(3), 293–302 (2023).
PubMed
Google Scholar
49.
Mollan KR, Pence BW, Xu S et al. Transportability from randomized trials to clinical care: on initial HIV treatment with efavirenz and suicidal thoughts or behaviors. Am. J. Epidemiol. 190(10), 2075–2084 (2021).
PubMed
Google Scholar
50.
Ramagopalan SV, Popat S, Gupta A et al. Transportability of overall survival estimates from US to Canadian patients with advanced non-small-cell lung cancer with implications for regulatory and health technology assessment. JAMA Netw. Open 5(11), e2239874 (2022).
PubMed
Google Scholar
|