Causal inference and adjustment for reference-arm risk in indirect treatment comparison meta-analysis
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: To illustrate that bias associated with indirect treatment comparison and network meta-analyses can be reduced by adjusting for outcomes on common reference arms. Materials & methods: Approaches to adjusting for reference-arm effects are presented within a causal inference framework. Bayesian and Frequentist approaches are applied to three real data examples. Results: Reference-arm adjustment can significantly impact estimated treatment differences, improve model fit and align indirectly estimated treatment effects with those observed in randomized trials. Reference-arm adjustment can possibly reverse the direction of estimated treatment effects. Conclusion: Accumulating theoretical and empirical evidence underscores the importance of adjusting for reference-arm outcomes in indirect treatment comparison and network meta-analyses to make full use of data and reduce the risk of bias in estimated treatments effects.
Lay abstract
Indirect treatment comparisons (ITCs) and network meta-analyses (NMAs) can help decision makers compare therapies that lack head-to-head randomized trials. However, these estimates are vulnerable to biases due to cross-trial differences in patient characteristics and other factors. In this study, we outline methods to reduce biases associated with ITC/NMA and apply them to three real-world examples (antiretroviral therapy for human immunodeficiency virus, treatments for Type 2 diabetes and biological treatments for psoriasis). Our results show that reference-arm adjustment can have a significant impact on indirectly estimated treatment effects and can improve consistency between indirect evidence and gold-standard evidence from randomized trials. ITC and NMA without reference-arm adjustment present an avoidable risk of misleading or biased treatment effects. We argue that reference-arm adjustment should always be considered and reported when feasible in ITC and NMA.
Formats available
You can view the full content in the following formats:
Supplementary Material
File (suppl_file.docx)
- Download
- 33.62 KB
References
Papers of special note have been highlighted as: • of interest
1.
Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J. Clin. Epidemiol. 50(6), 683–691 (1997).
2.
Lumley T. Network meta-analysis for indirect treatment comparisons. Stat. Med. 21(16), 2313–2324 (2002).
3.
Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat. Med. 23(20), 3105–3124 (2004).
4.
Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ 326(7387), 472 (2003).
5.
Glenny AM, Altman DG, Song F et al. Indirect comparisons of competing interventions. Health Technol. Assess. 9(26), 1–134 iii–iv (2005).
6.
Sutton A, Ades AE, Cooper N, Abrams K. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics 26(9), 753–767 (2008).
7.
Hoaglin DC, Hawkins N, Jansen JP et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health 14(4), 429–437 (2011).
8.
Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU technical support document 2: a generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. National Institute for Health and Care Excellence (NICE), London, UK, (2020). www.ncbi.nlm.nih.gov/books/NBK310366/pdf/Bookshelf_NBK310366.pdf
• A technical support document from the National Institute for Health and Clinical Excellence (NICE) that lays out the methodological foundations of network meta-analyses.
9.
Jansen JP, Fleurence R, Devine B et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health 14(4), 417–428 (2011).
• A report from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) that lays out the methodological foundations of network meta-analyses.
10.
Australian Government Department of Health. Pharmaceutical Benefits Advisory Committee (PBAC) technical working group reports: Indirect Comparisons Working Group (ICWG) report. Australian Government Department of Health, Canberra, Australia, (2020). www.pbs.gov.au/info/industry/useful-resources/pbac-feedback
11.
Schisterman EF, Cole SR, Platt RW. Overadjustment bias and unnecessary adjustment in epidemiologic studies. Epidemiology 20(4), 488–495 (2009).
12.
Neyman J. [Sur les applications de la theorie des probabilites aux experiences agricoles: essai des principes]. Roczniki Nauk Rolniczych 10, 1–51 (1923).
13.
Rubin DB. Estimating casual effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66(5), 688–701 (1974).
• Introduces the causal inference framework.
14.
Holland PW. Statistics and causal inference. J. Am. Stat. Assoc. 81(396), 945–960 (1986).
15.
Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU technical support document 3: heterogeneity: subgroups, meta-regression, bias and bias-adjustment. National Institute for Health and Care Excellence (NICE), London, UK, (2020). http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD3-Heterogeneity.final-report.08.05.12.pdf
• A guidance document from the National Institute for Health and Care Excellence with methods for reference-arm adjustment.
16.
Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat. Med. 16(23), 2741–2758 (1997).
17.
Chou R, Fu R, Huffman LH, Korthuis PT. Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses. Lancet 368(9546), 1503–1515 (2006).
• A direct meta-analysis of antiretroviral therapy for virological suppression of the HIV.
18.
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control. Clin. Trials 7(3), 177–188 (1986).
19.
Smith TC, Spiegelhalter DJ, Thomas A. Bayesian approaches to random-effects meta-analysis: a comparative study. Stat. Med. 14(24), 2685–2699 (1995).
20.
Rubin DB. Bayesian inference for causal effects: the role of randomization. Ann. Stat. 6(1), 34–58 (1978).
21.
Rosenbaum PR, Rubin DB. Comment: estimating the effects caused by treatments. J. Am. Stat. Assoc. 79(385), 26–28 (1984).
22.
Winship C, Morgan SL. The estimation of casual effects from observational data. Annu. Rev. Sociol. 25, 659–706 (1999).
23.
Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for casual effects. Biometrika 70(1), 41–55 (1983).
24.
Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology 10(1), 37–48 (1999).
25.
Pearl J, Glymour M, Jewell NP. Causal Inference in Statistics: A Primer. John Wiley and Sons, Inc, NJ, USA (2016).
26.
Pearl J. Direct and indirect effects [technical report]. Presented at: 17th Conference on Uncertainty in Artificial Intelligence. WA, USA (2001).
27.
Laird NM, Ware JH. Random-effects models for longitudinal data. Biometrics 38(4), 963–974 (1982).
28.
McCulloch CE, Searle SR. Generalized, linear and mixed models. John Wiley and Sons, Inc, NJ, USA, Hoboken (2000).
29.
Ayyagari R. The importance of adjustment for placebo-arm effects in indirect comparisons of glycemic control in Type 2 diabetes mellitus. Presented at: 1st American Diabetes Association Middle East Congress. Dubai, United Arab Emirates (2012).
• An indirect comparison of hemoglobin A1C outcomes in Type 2 diabetes.
30.
Lamel SA, Myer KA, Younes N, Zhou JA, Maibach H, Maibach HI. Placebo response in relation to clinical trial design: a systematic review and meta-analysis of randomized controlled trials for determining biologic efficacy in psoriasis treatment. Arch. Dermatol. Res. 304(9), 707–717 (2012).
31.
Signorovitch JE, Betts KA, Yan YS et al. Comparative efficacy of biological treatments for moderate-to-severe psoriasis: a meta-analysis adjusting for cross-trial differences in reference arm response. Br. J. Dermatol. 172(2), 504–512 (2015).
• A network meta-analysis that assesses the short-term efficacy of biological treatments for moderate-to-severe psoriasis.
32.
Gordon KB, Langley RG, Leonardi C et al. Clinical response to adalimumab treatment in patients with moderate to severe psoriasis: double-blind, randomized controlled trial and open-label extension study. J. Am. Acad. Dermatol. 55(4), 598–606 (2006).
33.
Menter A, Feldman SR, Weinstein GD et al. A randomized comparison of continuous vs. intermittent infliximab maintenance regimens over 1 year in the treatment of moderate-to-severe plaque psoriasis. J. Am. Acad. Dermatol. 56(1), 31.e1–15 (2007).
34.
Saurat JH, Stingl G, Dubertret L et al. Efficacy and safety results from the randomized controlled comparative study of adalimumab vs. methotrexate vs. placebo in patients with psoriasis (CHAMPION). Br. J. Dermatol. 158(3), 558–566 (2008).
35.
Gottlieb AB, Matheson RT, Lowe N et al. A randomized trial of etanercept as monotherapy for psoriasis. Arch. Dermatol. 139(12), 1627–1632
36.
Gottlieb AB, Evans R, Li S et al. Infliximab induction therapy for patients with severe plaque-type psoriasis: a randomized, double-blind, placebo-controlled trial. J. Am. Acad. Dermatol. 51(4), 534–542 (2004).
37.
Leonardi CL, Powers JL, Matheson RT et al. Etanercept as monotherapy in patients with psoriasis. N. Engl. J. Med. 349(21), 2014–2022 (2003).
38.
Leonardi CL, Kimball AB, Papp KA et al. Efficacy and safety of ustekinumab, a human interleukin-12/23 monoclonal antibody, in patients with psoriasis: 76-week results from a randomised, double-blind, placebo-controlled trial (PHOENIX 1). Lancet 371(9625), 1665–1674 (2008).
39.
Papp KA, Tyring S, Lahfa M et al. A global phase III randomized controlled trial of etanercept in psoriasis: safety, efficacy and effect of dose reduction. Br. J. Dermatol. 152(6), 1304–1312 (2005).
40.
Papp KA, Langley RG, Lebwohl M et al. Efficacy and safety of ustekinumab, a human interleukin-12/23 monoclonal antibody, in patients with psoriasis: 52-week results from a randomised, double-blind, placebo-controlled trial (PHOENIX 2). Lancet 371(9625), 1675–1684 (2008).
41.
Tyring S, Gordon KB, Poulin Y et al. Long-term safety and efficacy of 50 mg of etanercept twice weekly in patients with psoriasis. Arch. Dermatol. 143(6), 719–726 (2007).
42.
van de Kerkhof PC, Segaert S, Lahfa M et al. Once weekly administration of etanercept 50 mg is efficacious and well tolerated in patients with moderate-to-severe plaque psoriasis: a randomized controlled trial with open-label extension. Br. J. Dermatol. 159(5), 1177–1185 (2008).
43.
Chaudhari U, Romano P, Mulcahy LD, Dooley LT, Baker DG, Gottlieb AB. Efficacy and safety of infliximab monotherapy for plaque-type psoriasis: a randomised trial. Lancet 357(9271), 1842–1847 (2001).
44.
Reich K, Nestle FO, Papp K et al. Infliximab induction and maintenance therapy for moderate-to-severe psoriasis: a Phase III, multicentre, double-blind trial. Lancet 366(9494), 1367–1374 (2005).
45.
Griffiths CE, Strober BE, van de Kerkhof P et al. Comparison of ustekinumab and etanercept for moderate-to-severe psoriasis. N. Engl. J. Med. 362(2), 118–128 (2010).
46.
Menter A, Tyring SK, Gordon K et al. Adalimumab therapy for moderate to severe psoriasis: a randomized, controlled Phase III trial. J. Am. Acad. Dermatol. 58(1), 106–115 (2008).
47.
Armstrong AW, Puig L, Joshi A et al. Comparison of biologics and oral treatments for plaque psoriasis: a meta-analysis. JAMA Dermatol. 156(3), 258–269 (2020).
48.
Armstrong AW, Betts KA, Signorovitch JE et al. Number needed to treat and costs per responder among biologic treatments for moderate-to-severe psoriasis: a network meta-analysis. Curr. Med. Res. Opin. 34(7), 1325–1333 (2018).
49.
Institute for Clinical and Economic Review. Targeted immunomodulators for the treatment of moderate-to-severe plaque psoriasis: effectiveness and value. Condition update [final evidence report] (2020). https://icer-review.org/wp-content/uploads/2017/11/ICER_Psoriasis_Update_Draft_Report_04272018.pdf
50.
Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 327(7414), 557–560 (2003).
51.
Cochran WG. The combination of estimates from different experiments. Biometrics 10(1), 101–129 (1954).
52.
VanderWeele TJ. Principles of confounder selection [essay]. Eur. J. Epidemiol. 34(3), 211–219 (2019).
53.
Piantadosi S, Byar DP, Green SB. The ecological fallacy. Am. J. Epidemiol. 127(5), 893–904 (1988).
54.
Greenland S, Robins J. Invited commentary: ecologic studies–biases, misconceptions and counterexamples. Am. J. Epidemiol. 139(8), 747–760 (1994).
55.
Signorovitch JE, Sikirica V, Erder MH et al. Matching-adjusted indirect comparisons: a new tool for timely comparative effectiveness research. Value Health 15(6), 940–947 (2012).
Information & Authors
Information
Published In
Pages: 737 - 750
PubMed: 32490682
Copyright
© 2020 Elyse Swallow. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 10 March 2020
Accepted: 7 May 2020
Published online: 3 June 2020
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Causal inference and adjustment for reference-arm risk in indirect treatment comparison meta-analysis. (2020) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2020-0042
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Yu-Kang Tu, James S. Hodges, The Exchangeability Assumption in Network Meta-Analysis: Its Meaning and Evaluation, Value in Health, 10.1016/j.jval.2026.04.013, (2026).
- Carole Lunny, Areti-angeliki Veroniki, Julian P. T. Higgins, Sofia Dias, Brian Hutton, James M. Wright, Ian R. White, Penny Whiting, Andrea C. Tricco, Methodological review of NMA bias concepts provides groundwork for the development of a list of concepts for potential inclusion in a new risk of bias tool for network meta-analysis (RoB NMA Tool), Systematic Reviews, 10.1186/s13643-023-02388-x, 13, 1, (2024).
- Stefanie T. Jost, Marie‐Ann Kaldenbach, Angelo Antonini, Pablo Martinez‐Martin, Lars Timmermann, Per Odin, Regina Katzenschlager, Rupam Borgohain, Alfonso Fasano, Fabrizio Stocchi, Nobutaka Hattori, Prashanth Lingappa Kukkle, Mayela Rodríguez‐Violante, Cristian Falup‐Pecurariu, Sebastian Schade, Jan Niklas Petry‐Schmelzer, Vinod Metta, Daniel Weintraub, Guenther Deuschl, Alberto J. Espay, Eng‐King Tan, Roongroj Bhidayasiri, Victor S.C. Fung, Francisco Cardoso, Claudia Trenkwalder, Peter Jenner, K. Ray Chaudhuri, Haidar S. Dafsari, Levodopa Dose Equivalency in Parkinson's Disease: Updated Systematic Review and Proposals, Movement Disorders, 10.1002/mds.29410, 38, 7, (1236-1252), (2023).
- Karen L. Reckamp, Huamao M. Lin, Holly Cranmer, Yanyu Wu, Pingkuan Zhang, Stephen Kay, Laura J. Walton, Junwu Shen, Sanjay Popat, D. Ross Camidge, Overall survival indirect treatment comparison between brigatinib and alectinib for the treatment of front-line anaplastic lymphoma kinase–positive non–small cell lung cancer using data from ALEX and final results from ALTA-1L, Current Medical Research and Opinion, 10.1080/03007995.2022.2100653, 38, 9, (1587-1593), (2022).