What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: To analyze descriptors/definitions of meta-epidemiological studies as well as study design of articles that were self-described as meta-epidemiological studies. Methods: We searched MEDLINE and Embase on 6 August 2019. We extracted definitions of meta-epidemiological studies, as well as study designs, statistics and units of analysis that were reported in studies self-labelled as meta-epidemiological studies. Results: We included 175 information sources in the analysis. Definitions of meta-epidemiological studies varied and some studies used the term meta-epidemiological study to describe methodological research-on-research studies. Less than a half of the studies (n = 54; 42.9%) used the two-step meta-epidemiological approach in data analysis. Among studies self-labelled as meta-epidemiological, 9.4% reported registration in PROSPERO and 11% indicated they reported the study in line with PRISMA. Conclusion: Research community would benefit from consensus about definition of meta-epidemiological study.
Supplementary Material
File (suppl_file.zip)
- Download
- 490.45 KB
References
1.
Naylor CD. Meta-analysis and the meta-epidemiology of clinical research. BMJ 315(7109), 617–619 (1997).
2.
Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Statistics in Medicine 21(11), 1513–1524 (2002).
3.
Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J. Clin. Epidemiol. 77, 24–37 (2016).
4.
Luo X, Wan M, Chang C et al. The reporting of meta-epidemiological studies: A cross-sectional study. Conference abstract. Cochrane Colloquium, Cape Town, South Africa (2017).
5.
Ge L, Tian JH, Li YN et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J. Clin. Epidemiol. 93, 45–55 (2018).
6.
Umberham B, Hedin R, Detweiler B, Kollmorgen L, Hicks C, Vassar M. Heterogeneity of studies in anesthesiology systematic reviews: a meta-epidemiological review and proposal for evidence mapping. British Journal of Anaesthesia 119(5), 874–884 (2017).
7.
Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: A meta-epidemiological study. Eur J Pain 22(10), 1844–1849 (2018).
8.
Gu AP, Gu CN, Ahmed AT et al. Sham surgical procedures for pain intervention result in significant improvements in pain: systematic-review and meta-analysis: Metaepidemiologic research requires reporting requirements. J. Clin. Epidemiol. 87, 108 (2017).
9.
Wood L, Egger M, Gluud LL et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ 336(7644), 601–605 (2008).
10.
Deeks JJ, Dinnes J, D'amico R et al. Evaluating non-randomised intervention studies. Health Technology Assessment 7(27), (2003).
11.
Doig GS, Simpson F, Sweetman EA. Evidence-based nutrition support in the intensive care unit: An update on reported trial quality. Curr. Opin. Clin. Nutr. Metab. Care 12(2), 201–206 (2009).
12.
Koletsi D, Spineli LM, Lempesi E, Pandis N. Risk of bias and magnitude of effect in orthodontic randomized controlled trials: a meta-epidemiological review. Eur. J. Orthod. 38(3), 308–312 (2016).
13.
Bijle MNA, Ekambaram M, Lo ECM, Yiu CKY. A meta-epidemiological review of meta-analysis on anti-caries effect of arginine-containing formulations. J. Evid. Based Dent. Pract. 19(1), 28–33 (2019).
14.
Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: Meta-epidemiological survey. BMJ (Online) 352, i493 (2016).
15.
Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey. CMAJ Open 4(2), E132–140 (2016).
16.
Kovic B, Zoratti M, Michalopoulos S, Silvestre C, Thorlund K, Thabane L. A meta-epidemiological survey of the reporting of effect modification in network meta-analyses. Value in Health 20(5), A330 (2017).
17.
Wallac JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Internal Medicine 177(4), 554–560 (2017).
18.
Katerndahl DA, Lawler WR. Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. Am. J. Epidemiol. 149(5), 429–441 (1999).
19.
Cleophas TJ, Zwinderman AH. Modern meta-analysis. Springer (2017).
20.
Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: Meta-epidemiological study. BMJ (Online) 344(7846), e813 (2012).
21.
Ewald H, Speich B, Ladanie A, Bucher HC, Ioannidis JPA, Hemkens LG. Marginal structural models and other analyses allow multiple estimates of treatment effects in randomized clinical trials: Meta-epidemiological analysis. J. Clin. Epidemiol. 107, 12–26 (2019).
22.
Crossley NA, Sena E, Goehler J et al. Empirical evidence of bias in the design of experimental stroke studies: A metaepidemiologic approach. Stroke 39(3), 929–934 (2008).
23.
Janiaud P, Glais F, Grenet G, Lajoinie A, Cornu C, Kassai-Koupai B. Use of available clinical evidence to extrapolate drug effects from adults to children. Therapie 73(2), 119–125 (2018).
24.
Schuit E, Li AH, Ioannidis JPA. How often can meta-analyses of individual-level data individualize treatment? A meta-epidemiologic study. Int. J. Epidemiol. 48(2), 596–608 (2019).
25.
Tedesco D, Farid-Kapadia M, Offringa M et al. Comparative evidence on harms in pediatric randomized clinical trials from less developed versus more developed countries is limited. J. Clin. Epidemiol. 95, 63–72 (2018).
26.
Vandermeer B, Van Der Tweel I, Jansen-Van Der Weide MC et al. Comparison of nuisance parameters in pediatric versus adult randomized trials: a meta-epidemiologic empirical evaluation. BMC Med. Res. Methodol. 18(1), 7 (2018).
27.
Contopoulos-Ioannidis D, Tseretopoulou X, Ancker M et al. Comparative rates of harms in randomized trials from more developed versus less developed countries may be different. J. Clin. Epidemiol. 78, 10–21 (2016).
28.
Ladanie A, Speich B, Briel M et al. Single pivotal trials with few corroborating characteristics were used for FDA approval of cancer therapies. J. Clin. Epidemiol. 114, 49–59 (2019).
29.
Boef AGC, Dekkers OM, Le Cessie S. Mendelian randomization studies: A review of the approaches used and the quality of reporting. Int. J. Epidemiol. 44(2), 496–511 (2015).
30.
Janiaud P, Cristea IA, Ioannidis JPA. Industry-funded versus non-profit-funded critical care research: a meta-epidemiological overview. Intensive Care Med. 44(10), 1613–1627 (2018).
31.
Francke AL, Smit MC, De Veer AJ, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med. Inform. Decis. Mak. 8, 38 (2008).
32.
Egan M, Tannahill C, Petticrew M, Thomas S. Psychosocial risk factors in home and community settings and their associations with population health and health inequalities: a systematic meta-review. BMC Public Health 8, 239 (2008).
33.
Pandis N, Fleming PS, Worthington H, Salanti G. The Quality of the Evidence According to GRADE Is Predominantly Low or Very Low in Oral Health Systematic Reviews. PLoS ONE 10(7), e0131644 (2015).
34.
Chaimani A, Vasiliadis HS, Pandis N, Schmid CH, Welton NJ, Salanti G. Effects of study precision and risk of bias in networks of interventions: A network meta-epidemiological study. Int. J. Epidemiol. 42(4), 1120–1131 (2013).
35.
Trinquart L, Dechartres A, Ravaud P. Commentary: Meta-epidemiology, meta-metaepidemiology or network meta-epidemiology? Int. J. Epidemiol. 42(4), 1131–1133 (2013).
36.
Ban JW, Emparanza JI, Urreta I, Burls A. Design characteristics influence performance of clinical prediction rules in validation: A meta-epidemiological study. PLoS ONE 11(1), e0145779 (2016).
37.
Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Assessing small study effects and publication bias in orthodontic meta-analyses: a meta-epidemiological study. Clin Oral Investig. 18(4), 1031–1044 (2014).
38.
Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, Van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174(4), 469–476 (2006).
39.
Shinohara K, Suganuma AM, Imai H, Takeshima N, Hayasaka Y, Furukawa TA. Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation. PLoS ONE 12(9), e0184786 (2017).
40.
Storz-Pfennig P. Potentially unnecessary and wasteful clinical trial research detected in cumulative meta-epidemiological and trial sequential analysis. J. Clin. Epidemiol. 82, 61–70 (2017).
41.
Yerokhin VV, Carr BK, Sneed G, Vassar M. Clinical trials registries are underused in the pregnancy and childbirth literature: a systematic review of the top 20 journals. BMC Res. Notes 9(1), 475 (2016).
42.
Trone JC, Ollier E, Chapelle C et al. Statistical controversies in clinical research: Limitations of open-label studies assessing antiangiogenic therapies with regard to evaluation of vascular adverse drug events-A meta-analysis. Annals of Oncology 29(4), 803–811 (2018).
43.
Henriksen M, Hansen JB, Klokker L, Bliddal H, Christensen R. Comparable effects of exercise and analgesics for pain secondary to knee osteoarthritis: A meta-analysis of trials included in Cochrane systematic reviews. J. Comp. Eff. Res. 5(4), 417–431 (2016).
44.
Watzlawick R, Rind J, Sena ES et al. Olfactory Ensheathing Cell Transplantation in Experimental Spinal Cord Injury: Effect size and Reporting Bias of 62 Experimental Treatments: A Systematic Review and Meta-Analysis. PLoS Biology 14(5), e1002468 (2016).
45.
Tzoulaki I, Siontis KCM, Ioannidis JPA. Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: Meta-epidemiology study. BMJ (Online) 343(7834), 1137 (2011).
46.
Dragioti E, Bjork M, Larsson B, Gerdle B. A meta-epidemiological appraisal of the effects of interdisciplinary multimodal pain therapy dosing for chronic low back pain. J. Clin. Med. 8(6), 871 (2019).
47.
Janiaud P, Cornu C, Lajoinie A, Djemli A, Cucherat M, Kassai B. Is the perceived placebo effect comparable between adults and children? A meta-regression analysis. Pediatric Research 81(1), 11–17 (2017).
48.
Oliver S, Bagnall AM, Thomas J et al. Randomised controlled trials for policy interventions: A review of reviews and meta-regression. Health Technology Assessment 14(16), 1–192 (2010).
49.
Papageorgiou SN, Hochli D, Eliades T. Outcomes of comprehensive fixed appliance orthodontic treatment: A systematic review with meta-analysis and methodological overview. Korean J. Orthod. 47(6), 401–413 (2017).
50.
Mickenautsch S, Yengopal V. Do Laboratory Results Concerning High-Viscosity Glass-Ionomers versus Amalgam for Tooth Restorations Indicate Similar Effect Direction and Magnitude than that of Controlled Clinical Trials? - A Meta-Epidemiological Study. PLoS ONE [Electronic Resource] 10(7), e0132246 (2015).
51.
Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence-Based Medicine 22(4), 139–142 (2017).
52.
Zhang Z. Meta-epidemiological study: a step-by-step approach by using R. J. Evid. Based Med. 9(2), 91–97 (2016).
53.
Roberts I, Schierhout G. Personal views - The private life of systematic reviews. Brit. Med. J. 315(7109), 686–687 (1997).
54.
Bae J-M. Meta-epidemiology. Epidemiol. Health 36, e2014019 (2014).
55.
Biondi-Zoccai G. Umbrella Reviews. Evidence Synthesis with Overviews of Reviews and Meta-Epidemiologic Studies. Springer (2016).
56.
Slade SC, Dionne CE, Underwood M et al. Consensus on Exercise Reporting Template (CERT): Modified Delphi Study. Phys. Ther. 96(10), 1514–1524 (2016).
57.
Slade SC, Keating JL. Exercise prescription: a case for standardised reporting. Br. J. Sports Med. 46(16), 1110–1113 (2012).
58.
Gandhi GY, Murad MH, Fujiyoshi A et al. Patient-important outcomes in registered diabetes trials. JAMA 299(21), 2543–2549 (2008).
59.
Lawson DO, Thabane L, Mbuagbaw L. A call for consensus guidelines on classification and reporting of methodological studies. J. Clin. Epidemiol. (2020).
60.
Puljak L, Pieper D. Registration of methodological studies, i.e. ‘research on research’ studies- should it be mandatory? J. Clin. Epidemiol. 35–36 (2019).
61.
Puljak L. Methodological studies evaluating evidence are not systematic reviews. J. Clin. Epidemiol. 110, 98–99 (2019).
62.
U. S. National Library of Medicine. Publication Characteristics (Publication Types) with Scope Notes (2019). www.nlm.nih.gov/mesh/pubtypes.html
63.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273(5), 408–412 (1995).
64.
Puljak L. Methodological research: open questions, the need for ‘research on research’ and its implications for evidence-based healthcare and reducing research waste. Int. J. Evid. Based Healthc. 17(3), 145–146 (2019).
65.
Riado Minguez D, Kowalski M, Vallve Odena M et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth. Analg. 125(4), 1348–1354 (2017).
66.
Savovic J, Jones H, Altman D et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol. Assess. 16(35), 1–82 (2012).
Information & Authors
Information
Published In
Pages: 497 - 508
PubMed: 32372658
Copyright
© 2020 Future Medicine Ltd.
History
Received: 20 December 2019
Accepted: 20 February 2020
Published online: 6 May 2020
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. (2020) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2019-0201
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Hans Lund, Karen A Robinson, History of evidence-based research, Journal of the Royal Society of Medicine, 10.1177/01410768261433038, 119, 4, (107-113), (2026).
- Zhen Peng, Shiqi Fan, Yuan Tian, Yingxia Wang, Zongshi Qin, Suhail Doi, Chang Xu, Comparing the accuracy of AI-assisted data extraction versus human double extraction in evidence synthesis: a randomised controlled trial protocol, BMJ Open, 10.1136/bmjopen-2025-106546, 15, 11, (e106546), (2025).
- Mathew J. Wingerson, Katherine L. Smulligan, Ellie Svoboda, Christine M. Baugh, Samantha N. Magliato, Julie C. Wilson, David R. Howell, Who Are the Participants in Pediatric Concussion Randomized Controlled Trials? A Meta-epidemiology Study, JOSPT Open, 10.2519/josptopen.2025.0112, 3, 4, (395-408), (2025).
- Natasja D. Vijfschagt, Huibert Burger, Marjolein Y. Berger, Thomas R. Fanshawe, Ann van den Bruel, Mariska M.G. Leeflang, Michiel R. de Boer, Gea A. Holtman, Variation in sensitivity and specificity of diverse diagnostic tests across health-care settings: a meta-epidemiological study, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2025.111816, 184, (111816), (2025).
- Silvia Gianola, Stefania Guida, Gaia Ravot, Carole Lunny, Silvia Bargeri, Greta Castellini, Gaps in completeness of reporting and methodological quality: a metaresearch study of 139 network meta-analyses published in January 2023 using PRISMA-NMA and AMSTAR-2, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2025.111783, 183, (111783), (2025).
- Masoud Mohammadi, Meta‐Epidemiology: A Combination of Epidemiology and Meta‐Analysis in Response to the Challenges of Systematic Reviews and Meta‐Analysis, Journal of Evidence-Based Medicine, 10.1111/jebm.70036, 18, 2, (2025).
- Jingyu Zhong, Xianwei Liu, Junjie Lu, Jiarui Yang, Guangcheng Zhang, Shiqi Mao, Haoda Chen, Qian Yin, Qingqing Cen, Run Jiang, Yang Song, Minda Lu, Jingshen Chu, Yue Xing, Yangfan Hu, Defang Ding, Xiang Ge, Huan Zhang, Weiwu Yao, Overlooked and underpowered: a meta-research addressing sample size in radiomics prediction models for binary outcomes, European Radiology, 10.1007/s00330-024-11331-0, 35, 3, (1146-1156), (2025).
- Youlin Long, Yurong Zheng, Xinyao Wang, Qiong Guo, Na Zhang, Ya Deng, Ruixian Tang, Zhengchi Li, Liang Du, The reporting quality of meta-epidemiological studies needs substantial improvement: a research on research study, Systematic Reviews, 10.1186/s13643-024-02661-7, 13, 1, (2024).
- Silvia Bargeri, Giacomo Basso, Ignazio Geraci, Greta Castellini, Alessandro Chiarotto, Silvia Gianola, Raymond Ostelo, Marco Testa, Tiziano Innocenti, Substantial discrepancies exist between registered protocol and published manuscript in trials on exercise interventions for chronic low back pain: a metaresearch study, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2024.111465, 173, (111465), (2024).
- Jingyu Zhong, Yue Xing, Yangfan Hu, Junjie Lu, Jiarui Yang, Guangcheng Zhang, Shiqi Mao, Haoda Chen, Qian Yin, Qingqing Cen, Run Jiang, Jingshen Chu, Yang Song, Minda Lu, Defang Ding, Xiang Ge, Huan Zhang, Weiwu Yao, The policies on the use of large language models in radiological journals are lacking: a meta-research study, Insights into Imaging, 10.1186/s13244-024-01769-7, 15, 1, (2024).
- See more