Skip to main content
Open access
Systematic Review
5 October 2017

Handling missing data in propensity score estimation in comparative effectiveness evaluations: a systematic review

Abstract

Aim: Even though systematic reviews have examined how aspects of propensity score methods are used, none has reviewed how the challenge of missing data is addressed with these methods. This review therefore describes how missing data are addressed with propensity score methods in observational comparative effectiveness studies. Methods: Published articles on observational comparative effectiveness studies were extracted from MEDLINE and EMBASE databases. Results: Our search yielded 167 eligible articles. Majority of these studies (114; 68%) conducted complete case analysis with only 53 of them stating this in the methods. Only 16 articles reported use of multiple imputation. Conclusion: Few researchers use correct methods for handling missing data or reported missing data methodology which may lead to reporting biased findings.

Supplementary Material

File (suppl_appendix_1.csv)
File (suppl_appendix_2.doc)
File (suppl_appendix_3.docx)

References

Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Ye C, Beyene J, Browne G, Thabane L. Estimating treatment effects in randomized controlled trials with noncompliance: a simulation study. BMJ Open 4(6), e005362 (2014).
2.
Kausto J, Solovieva S, Virta LJ, Viikari-Juntura E. Partial sick leave associated with disability pension: propensity score approach in a register-based cohort study. BMJ Open 2(6), e001752 (2012).
3.
West SG, Duan N, Pequegnat W et al. Alternatives to the randomized controlled trial. Am. J. Public Health 98(8), 1359–1366 (2008).
4.
Berger ML, Mamdani M, Atkins D, Johnson ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report–Part I. Value Health 12(8), 1044–1052 (2009).
5.
Rosenbaum PR, Silber JH. Matching and thick description in an observational study of mortality after surgery. Biostatistics 2(2), 217–232 (2001).
6.
Stuart EA. Matching methods for causal inference: a review and a look forward. Stat Sci. 25(1), 1–21 (2010).
7.
Thoemmes FJ, Kim ES. A systematic review of propensity score methods in the social sciences. Multivariate Behav. Res. 46(1), 90–118 (2011).
• Provides a current description of how propensity score (PS) methodological aspects are used.
8.
Zakrison TL, Austin PC, Mccredie VA. A systematic review of propensity score methods in the acute care surgery literature: avoiding the pitfalls and proposing a set of reporting guidelines. Eur. J. Trauma Emerg. Surg. (2017) (Epub ahead of print).
• Provides a current description of how PS methodological aspects are used.
9.
Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. JSTOR 70, 41–55 (1983).
• Describes how PS approach was developed.
10.
Toh S, Garcia Rodriguez LA, Hernan MA. Analyzing partially missing confounder information in comparative effectiveness and safety research of therapeutics. Pharmacoepidemiol. Drug Saf. 21(Suppl. 2), 13–20 (2012).
11.
Lee KJ, Simpson JA. Introduction to multiple imputation for dealing with missing data. Respirol. 19(2), 162–167 (2014).
12.
Carpenter JR, Kenward MG, White IR. Sensitivity analysis after multiple imputation under missing at random: a weighting approach. Stat. Methods Med. Res. 16(3), 259–275 (2007).
13.
Mitra R, Reiter JP. A comparison of two methods of estimating propensity scores after multiple imputation. Stat. Methods Med. Res. 25(1), 188–204 (2016).
• Compares various methods of estimating PS after multiple imputation.
14.
Rubin DB, Schenker N. Multiple imputation in healthcare databases: an overview and some applications. Stat. Med. 10(4), 585–598 (1991).
15.
Mayer B, Puschner B. Propensity score adjustment of a treatment effect with missing data in psychiatric health services research. Epidemiol. Biostatistics Public Health 12(1), (2015) (Epub ahead of print).
16.
Nakai M, Ke W. Review of the methods for handling missing data in longitudinal data analysis. Int. J. Math 5(1), 1–13 (2011).
17.
Yao XI, Wang X, Speicher PJ et al. Reporting and guidelines in propensity score analysis: a systematic review of cancer and cancer surgical studies. J. Natl Cancer Inst. 109(8), (2017) (Epub ahead of print).
18.
Kang H. The prevention and handling of the missing data. Korean J. Anesthesiol. 64(5), 402–406 (2013).
19.
Fielding S, Maclennan G, Cook JA, Ramsay CR. A review of RCTs in four medical journals to assess the use of imputation to overcome missing data in quality of life outcomes. Trials 9, 51 (2008).
20.
Bell ML, Fiero M, Horton NJ, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med. Res. Methodol. 14, 118 (2014).
• Reviews how missing data are addressed, though not with respect to the use of PSs.
21.
Karahalios A, Baglietto L, Carlin JB, English DR, Simpson JA. A review of the reporting and handling of missing data in cohort studies with repeated assessment of exposure measures. BMC Med. Res. Methodol. 12, 96 (2012).
• Reviews how missing data are addressed, though not with respect to the use of PSs.
22.
Eekhout I, De Boer RM, Twisk JW, De Vet HC, Heymans MW. Missing data: a systematic review of how they are reported and handled. Epidemiology 23(5), 729–732 (2012).
• Reviews how missing data are addressed, though not with respect to the use of PSs.
23.
Vandenbroucke JP, Von Elm E, Altman DG et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Epidemiology 18(6), 805–835 (2007).
24.
Mchugh ML. Interrater reliability: the kappa statistic. Biochem. Med. (Zagreb) 22(3), 276–282 (2012).
25.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J. Clin. Epidemiol. 62(10), 1006–1012 (2009).
26.
Molenberghs G, Beunckens C, Sotto C. Every missing not at random model has a missingness at random counterpart with equal fit. J. R. Stat. Soc. 70(2), 371–388 (2008).
27.
Molenberghs G, Thijs H, Jansen I et al. Analyzing incomplete longitudinal clinical trial data. Biostatistics 5(3), 445–464 (2004).