Unmeasured confounding in nonrandomized studies: quantitative bias analysis in health technology assessment
Publication: Journal of Comparative Effectiveness Research
Abstract
Evidence generated from nonrandomized studies (NRS) is increasingly submitted to health technology assessment (HTA) agencies. Unmeasured confounding is a primary concern with this type of evidence, as it may result in biased treatment effect estimates, which has led to much criticism of NRS by HTA agencies. Quantitative bias analyses are a group of methods that have been developed in the epidemiological literature to quantify the impact of unmeasured confounding and adjust effect estimates from NRS. Key considerations for application in HTA proposed in this article reflect the need to balance methodological complexity with ease of application and interpretation, and the need to ensure the methods fit within the existing frameworks used to assess nonrandomized evidence by HTA bodies.
Supplementary Material
File (supplementary materials (3).docx)
- Download
- 53.29 KB
References
1.
Ayling K, Brierley S, Johnson B, Heller S, Eiser C. How standard is standard care? Exploring control group outcomes in behaviour change interventions for young people with type 1 diabetes. Psychol. Health 30(1), 85–103 (2015).
2.
Hatswell AJ, Baio G, Berlin JA, Irs A, Freemantle N. Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999–2014. BMJ Open 6(6), e011666 (2016).
3.
Griffiths EA, Macaulay R, Vadlamudi NK, Uddin J, Samuels ER. The role of noncomparative evidence in health technology assessment decisions. Value Health 20(10), 1245–1251 (2017).
4.
Patel D, Grimson F, Mihaylova E et al. Use of external comparators for health technology assessment submissions based on single-arm trials. Value Health 24(8), 1118–1125 (2021).
5.
Vanderweele TJ, Arah OA. Unmeasured confounding for general outcomes, treatments, and confounders: bias formulas for sensitivity analysis. Epidemiology (Cambridge, Mass.) 22(1), 42 (2011).
6.
Sammon CJ, Leahy TP, Gsteiger S, Ramagopalan S. Real-world evidence and nonrandomized data in health technology assessment: using existing methods to address unmeasured confounding? J. Comp. Eff. Res. 9(14), 969–972 (2020).
7.
Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am. J. Epidemiol. 183(8), 758–764 (2016).
8.
Faria R, Alava MH, Manca A, Wailoo AJ. NICE DSU Technical Support Document 17: The Use of Observational Data to Inform Estimates of Treatment Effectiveness in Technology Appraisal: Methods for Comparative Individual Patient Data. NICE Decision Support Unit, Sheffield, UK (2015).
9.
Phillippo DM, Ades AE, Dias S. NICE DSU Technical Support Document 18: Methods for Population-Adjusted Indirect Comparisons in Submissions to NICE. NICE Decision Support Unit, Sheffield, UK (2016).
10.
Kreif N, Grieve R, Sadique MZ. Statistical methods for cost-effectiveness analyses that use observational data: a critical appraisal tool and review of current practice. Health Econ. 22(4), 486–500 (2013).
11.
Lash TL, Fox MP, Maclehose RF, Maldonado G, Mccandless LC, Greenland S. Good practices for quantitative bias analysis. Int. J. Epidemiol. 43(6), 1969–1985 (2014).
12.
Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB, Wynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. J. Nat. Cancer Inst. 22(1), 173–203 (1959).
13.
Lash TL, Fox MP, Cooney D, Lu Y, Forshee RA. Quantitative bias analysis in regulatory settings. Am. J. Public Health 106(7), 1227–1230 (2016).
14.
National Institute for Health Care and Excellence. Appendix I: Real World Evidence Framework. NICE. Sheffield, UK (2021).
15.
Brumback BA, Hernán MA, Haneuse SJ, Robins JM. Sensitivity analyses for unmeasured confounding assuming a marginal structural model for repeated measures. Stat. Med. 23(5), 749–767 (2004).
16.
Ertefaie A, Small DS, Flory JH, Hennessy S. A tutorial on the use of instrumental variables in pharmacoepidemiology. Pharmacoepidemiol. Drug Saf. 26(4), 357–367 (2017).
17.
Rosenbaum PR. Sensitivity to hidden bias. In: Observational Studies. Springer, NY, USA, 105–170 (2002).
18.
Nattino G, Lu B. Model assisted sensitivity analyses for hidden bias with binary outcomes. Biometrics 74(4), 1141–1149 (2018).
19.
Lu B, Cai D, Tong X. Testing causal effects in observational survival data using propensity score matching design. Stat. Med. 37(11), 1846–1858 (2018).
20.
Hasegawa R, Small D. Sensitivity analysis for matched pair analysis of binary data: from worst case to average case analysis. Biometrics 73(4), 1424–1432 (2017).
21.
Rosenbaum PR. Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika 74(1), 13–26 (1987).
22.
Lin NX, Logan S, Henley WE. Bias and sensitivity analysis when estimating treatment effects from the Cox model with omitted covariates. Biometrics 69(4), 850–860 (2013).
23.
McCandless LC, Gustafson P, Levy AR, Richardson S. Hierarchical priors for bias parameters in Bayesian sensitivity analysis for unmeasured confounding. Stat Med 31(4), 383–396 (2012).
24.
Rosenbaum PR, Rubin DB. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J R Stat Soc Series B Stat Methodol 45(2), 212–218 (1983).
25.
Gustafson P, McCandless LC, Levy AR, Richardson S. Simplified Bayesian sensitivity analysis for mismeasured and unobserved confounders. Biometrics 66(4), 1129–1137 (2010).
26.
McCandless LC, Gustafson P. A comparison of Bayesian and Monte Carlo sensitivity analysis for unmeasured confounding. Stat. Med. 36(18), 2887–2901 (2017).
27.
Huang R, Xu R, Dulai PS. Sensitivity analysis of treatment effect to unmeasured confounding in observational studies with survival and competing risks outcomes. Stat. Med. 39(24), 3397–3411 (2020).
28.
McCandless LC, Gustafson P, Levy A. Bayesian sensitivity analysis for unmeasured confounding in observational studies. Stat Med 26(11), 2331–2347 (2007).
29.
McCandless LC, Gustafson P, Levy AR. A sensitivity analysis using information about measured confounders yielded improved uncertainty assessments for unmeasured confounding. J Clin Epidemiol 61(3), 247–255 (2008).
30.
Zhang X, Faries DE, Boytsov N, Stamey JD, Seaman JW. A Bayesian sensitivity analysis to evaluate the impact of unmeasured confounding with external data: a real world comparative effectiveness study in osteoporosis. Pharmacoepidemiol Drug Saf 25(9), 982–992 (2016).
31.
Dorie V, Harada M, Carnegie NB, Hill J. A flexible, interpretable framework for assessing sensitivity to unmeasured confounding. Stat. Med. 35(20), 3453–3470 (2016).
32.
Groenwold RHH, Sterne JAC, Lawlor DA, Moons KGM, Hoes AW, Tilling K. Sensitivity analysis for the effects of multiple unmeasured confounders. Ann. Epidemiol. 26(9), 605–611 (2016).
33.
Barrowman MA, Peek N, Lambie M, Martin GP, Sperrin M. How unmeasured confounding in a competing risks setting can affect treatment effect estimates in observational studies. BMC Med. Res. Methodol. 19(1), 166 (2019).
34.
Corrao G, Nicotra F, Parodi A et al. External adjustment for unmeasured confounders improved drug-outcome association estimates based on health care utilization data. J. Clin. Epidemiol. 65(11), 1190–1199 (2012).
35.
Ding P, Vanderweele TJ. Sensitivity analysis without assumptions. Epidemiology 27(3), 368–377 (2016).
36.
Vanderweele TJ, Arah OA. Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders. Epidemiology 22(1), 42–52 (2011).
37.
Vanderweele TJ, Ding P. Sensitivity analysis in observational research: introducing the E-value. Ann. Intern. Med. 167(4), 268–274 (2017).
38.
Greenland S. Basic methods for sensitivity analysis of biases. Int. J. Epidemiol. 25(6), 1107–1116 (1996).
39.
Vanderweele TJ. Unmeasured confounding and hazard scales: sensitivity analysis for total, direct, and indirect effects. Eur. J. Epidemiol. 28(2), 113–117 (2013).
40.
Arah OA, Chiba Y, Greenland S. Bias formulas for external adjustment and sensitivity analysis of unmeasured confounders. Ann Epidemiol 18(8), 637–646 (2008).
41.
Cusson A, Infante-Rivard C. Bias factor, maximum bias and the E-value: insight and extended applications. Int J Epidemiol 49(5), 1509–1516 (2020).
42.
Mathur MB, Vanderweele TJ. Robust metrics and sensitivity analyses for meta-analyses of heterogeneous effects. Epidemiology 31(3), 356–358 (2020).
43.
Mittinty MN. Estimating bias due to unmeasured confounding in oral health epidemiology. Community Dent Health 37(1), 84–89 (2020).
44.
Schneeweiss S. Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiol Drug Saf 15(5), 291–303 (2006).
45.
Groenwold RHH, Nelson David BDB, Nichol Kristin LKL, Hoes AW, Hak E. Sensitivity analyses to estimate the potential impact of unmeasured confounding in causal research. Int. J. Epidemiol. 39(1), 107–117 (2010).
46.
Gong CL, Song AY, Horak R et al. Impact of confounding on cost, survival, and length-of-stay outcomes for neonates with hypoplastic left heart syndrome undergoing stage 1 palliation surgery. Pediatr. Cardiol. 41(5), 996–1011 (2020).
47.
Hay JW, Gong CL, Jiao X et al. A US population health survey on the impact of COVID-19 using the EQ-5D-5L. J. Gen. Intern. Med. 36(5), 1292–1301 (2021).
48.
Schultze A, Walker AJ, Mackenna B et al. Risk of COVID-19-related death among patients with chronic obstructive pulmonary disease or asthma prescribed inhaled corticosteroids: an observational cohort study using the OpenSAFELY platform. Lancet Respir. Med. 8(11), 1106–1120 (2020).
49.
Barberio J, Ahern TP, MacLehose RF et al. Assessing techniques for quantifying the impact of bias due to an unmeasured confounder: an applied example. Clin. Epidemiol. 13, 627 (2021).
50.
Klungsoyr O, Sexton J, Sandanger I, Nygard JF. Sensitivity analysis for unmeasured confounding in a marginal structural Cox proportional hazards model. Lifetime Data Anal. 15(2), 278–294 (2009).
51.
Tennant PW, Murray EJ, Arnold KF et al. Use of directed acyclic graphs (DAGs) to identify confounders in applied health research: review and recommendations. Int. J. Epidemiol. 50(2), 620–632 (2021).
52.
Blum MR, Tan YJ, Ioannidis JP. Use of E-values for addressing confounding in observational studies – an empirical assessment of the literature. Int. J. Epidemiol. 49(5), 1482–1494 (2020).
53.
Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG General Methods: Version 6.0 IQWiG, Nordrhein-Westfalen, Germany (2020).
54.
Leahy TP, Ramagopalan S, Sammon C. The use of UK primary care databases in health technology assessments carried out by the National Institute for Health and Care Excellence (NICE). BMC Health Serv. Res. 20(1), 1–9 (2020).
55.
Abrams K. CHTE2020 Sources and Synthesis of Evidence; Update to Evidence Synthesis Methods. NICE Decision Support Unit, Sheffield, UK (2020).
Information & Authors
Information
Published In
Pages: 851 - 859
PubMed: 35678151
Copyright
© 2022 The Authors. This work is licensed under the Creative Commons Attribution 4.0 License
History
Received: 14 February 2022
Accepted: 20 April 2022
Published online: 9 June 2022
Keywords:
Topics
Authors
Funding Information
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Unmeasured confounding in nonrandomized studies: quantitative bias analysis in health technology assessment. (2022) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2022-0029
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Eva Kagenaar, David Gibran Lugo-Palacios, Hannah Bromley, Ajay Aggarwal, Andrew Hutchings, Stephen O’Neill, Bernard Rachet, Jo Cresswell, Ananya Choudhury, Richard Grieve, Surgery or radiotherapy for early-stage cancer study protocol for an emulated target trial of radical radiotherapy versus radical cystectomy, with either following neoadjuvant chemotherapy, for organ-confined muscle-invasive bladder cancer, BMJ Open, 10.1136/bmjopen-2026-119596, 16, 6, (e119596), (2026).
- Thomas P. Leahy, Craig Keenan, Emily Robertshaw, Megan McStravick, Sylvaine Barbier, Stephen Duffield, Seamus Kent, From Framework to Practice: Evaluating the Uptake of Sensitivity Analyses in The National Institute for Health and Care Excellence Real-World External Control Arm Evidence, Value in Health Regional Issues, 10.1016/j.vhri.2026.101651, (101651), (2026).
- Wenshu Zhang, Yu Wang, Yuli Wu, Feiyu Chen, Xiaoping Liu, Masticatory function, inflammation, and mortality after stroke: A prospective cohort study, Medicine, 10.1097/MD.0000000000049038, 105, 22, (e49038), (2026).
- Paul Arora, Sreeram V Ramagopalan, R WE ready for reimbursement? A round-up of developments in real-world evidence relating to health technology assessment: part 23, Journal of Comparative Effectiveness Research, 10.57264/cer-2025-0196, 15, 1, (2025).
- Alison Antoine, Katia Desroziers, Julien Dupin, David Pérol, Rémy Choquet, Enhancing confidence in complex health technology assessments by using real-world evidence: highlighting existing strategies for effective drug evaluation, BMC Medical Research Methodology, 10.1186/s12874-025-02683-2, 25, 1, (2025).
- Codie J. C. Gerlach-Wood, Kate Tilling, Jonathan W. Bartlett, Rachael A. Hughes, Quantitative bias analysis for mismeasured variables in health research: a review of software tools, BMC Medical Research Methodology, 10.1186/s12874-025-02635-w, 25, 1, (2025).
- K. Jack Ishak, Conor Chandler, Fei Fei Liu, Sven Klijn, A Framework for Reliable, Transparent, and Reproducible Population-Adjusted Indirect Comparisons, PharmacoEconomics, 10.1007/s40273-025-01503-1, 43, 7, (691-710), (2025).
- Sebastián Cofre, Camila Sanchez, Gladys Quezada-Figueroa, Xaviera A. López-Cortés, Validity and accuracy of artificial intelligence-based dietary intake assessment methods: a systematic review, British Journal of Nutrition, 10.1017/S0007114525000522, 133, 9, (1241-1253), (2025).
- Shijie Ren, Sa Ren, Nicky J. Welton, Mark Strong, Quantitative bias analysis for unmeasured confounding in unanchored population-adjusted indirect comparisons, Research Synthesis Methods, 10.1017/rsm.2025.13, 16, 3, (509-527), (2025).
- Cyril Esnault, Louise Baschet, Vanessa Barbet, Gaëlle Chenuc, Maurice Pérol, Katia Thokagevistk, David Pau, Matthias Monnereau, Lise Bosquet, Thomas Filleron, Addressing challenges with Matching-Adjusted Indirect Comparisons to demonstrate the comparative effectiveness of entrectinib in metastatic ROS-1 positive Non-Small Cell Lung Cancer, BMC Medical Research Methodology, 10.1186/s12874-025-02500-w, 25, 1, (2025).
- See more