What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: To analyze descriptors/definitions of meta-epidemiological studies as well as study design of articles that were self-described as meta-epidemiological studies. Methods: We searched MEDLINE and Embase on 6 August 2019. We extracted definitions of meta-epidemiological studies, as well as study designs, statistics and units of analysis that were reported in studies self-labelled as meta-epidemiological studies. Results: We included 175 information sources in the analysis. Definitions of meta-epidemiological studies varied and some studies used the term meta-epidemiological study to describe methodological research-on-research studies. Less than a half of the studies (n = 54; 42.9%) used the two-step meta-epidemiological approach in data analysis. Among studies self-labelled as meta-epidemiological, 9.4% reported registration in PROSPERO and 11% indicated they reported the study in line with PRISMA. Conclusion: Research community would benefit from consensus about definition of meta-epidemiological study.
The term ‘meta-epidemiology’ is relatively recent; it first appeared in literature in 1997, in D Naylor’s editorial titled ‘Meta-analysis and the meta-epidemiology of clinical research’ [1] and in 2002, Sterne et al. published methodological guidance for ‘meta-epidemiological’ studies that evaluate effect of trial characteristics on effect sizes [2].
However, it has been reported that methodology of ‘meta-epidemiological’ studies is not standardized [3] and that terminology used for such studies varies [4]. Additionally, it appears that in recent years, the term ‘meta-epidemiological’ study is also used as a synonym for methodological ‘research-on-research’ studies that explore characteristics of various research-related reports (not only trials), without any considerations of the treatment effect estimates [5–7]. Therefore, it appears that there is an ambiguity in the understanding and use of the term meta-epidemiological study in the research community.
The aim of this study was to analyze descriptors and definitions of meta-epidemiological studies in published literature as well as study design of articles that were self-described as meta-epidemiological studies.
Methods
Study design
This was a methodological (research-on-research) study in which units of analysis were published manuscripts. Protocol for this study was defined a priori; it is presented in the Supplementary Material 1.
Ethics
This study included only analysis of literature, in other words, published manuscripts; therefore, approval of a study protocol by a research ethics committee was not necessary.
Inclusion criteria
We included information sources (manuscripts and research-related reports) published in scholarly journals and indexed in Google Scholar, which mentioned meta-epidemiological studies, defined meta-epidemiological studies and/or were self-described as meta-epidemiological studies. We aimed to include only manuscripts published in English language, but we did not even find any potentially eligible manuscripts in languages other than English. We excluded manuscripts reporting study protocols and conference abstracts. In case when we found multiple reports about the same study, we included the most comprehensive report.
Search
On 6 August 2019, we searched MEDLINE and Embase via OVID, using advanced search, to retrieve studies that either discuss meta-epidemiological studies or were self-described as meta-epidemiological studies. To retrieve such manuscripts, we used the following search strategy:
•
metaepidemiolog*.mp.
•
meta-epidemiolog*.mp.
•
1 or 2
Retrieved records were exported into EndNote X5 reference management software (Clarivate Analytics, PA, USA). We removed duplicates via software and then manually if any duplicates remained.
We also searched Google Scholar on 25 August 2019 for the following phrases ‘definition of meta-epidemiological’, ‘definition of meta-epidemiologic’, ‘definition of metaepidemiological’, ‘definition of metaepidemiologic’, ‘meta-epidemiological study was defined’, ‘metaepidemiological study was defined’, ‘metaepidemiological study was defined’, ‘metaepidemiologic study was defined’, ‘meta-epidemiological study is’, ‘meta-epidemiologic study is’, ‘metaepidemiological study is’ and ‘metaepidemiologic study is’.
Screening
One author retrieved full texts of all manuscripts retrieved by the search. Two authors independently screened the retrieved records and classified them into those that were self-described as meta-epidemiological and those that mentioned/discussed meta-epidemiological studies.
Data extraction & analysis
From included information sources we extracted verbatim: all mentions of expression ‘meta-epidemiological’ (and its wording variations such as metaepidemiological, meta-epidemiologic, etc.), year of publication, journal where the article was published, definitions and descriptions of a meta-epidemiological study, references used by authors to support definition of a meta-epidemiological study and keywords mentioning ‘meta-epidemiological/meta-epidemiology (or similar wording variations).
From studies self-identified as meta-epidemiological, we extracted their aim, description of the study designs, statistical methods used, unit of analysis, whether they had made their study protocol publicly available (and where) and whether they mentioned that they used any reporting guideline/checklist to report their study. If there were several analyses conducted, we focused on the main analysis when extracting statistical methods and units of analysis. From PubMed we extracted information on publication type that included studies were indexed with.
We categorized all data and tabulated them descriptively. For all extracted data items and categorizations, one author extracted data and the second author verified data extraction. We analyzed data with descriptive statistics; data were shown as frequencies and percentages.
Results
Included information sources
In total, 175 information sources were included in analysis (Flow chart in Figure 1). Table of excluded records, with reasons, is shown in Supplementary Material 2. Included information sources were published from 1997 to 2019; the majority was published in the past 5 years, from 2014 to 2019 (n = 132; 75%). The majority of these information sources (n = 151; 86%) were full-text articles published in peer-reviewed journals describing original research; there were also 15 various other types of journal articles such as commentaries, letters, perspectives. Most of those journal articles were published in the J. Clin. Epidemiol. (n = 41/151; 27%), Brit. Med. J. (n = 17/151; 11%) and PLoS ONE (n = 13/151; 9%).
Five information sources were Health Technology Assessment and Agency for Healthcare Research and Quality reports, two were books and two were PhD theses. The list of included information sources is available in Supplementary File 3.
Studies self-identified as meta-epidemiological
Among 175 included information sources, there were 127 (73%) full-text journal articles that authors self-identified with an expression indicating that this was ‘meta-epidemiological’ study or that ‘meta-epidemiological’ analysis was performed, although with varied terminology. These 127 articles were published between 2006 and 2019; the majority was published in the past 5 years, from 2014 to 2019 (n = 99/127; 78%).
A variety of terms related to ‘meta-epidemiological’ study designs, used by authors, are shown in Table 1, with their frequencies and references to the manuscripts that used such term. Table 1 also shows that many authors have used combinations of descriptors for study design of their manuscript, such as ‘systematic review and meta-analysis and meta-epidemiological study’.
There were three full-text journal articles that did not self-report any study design and remaining 45 (26%) information sources only mentioned/referenced meta-epidemiological studies within the manuscript.
Definitions of meta-epidemiological study
Definitions and descriptions of meta-epidemiological studies were found in 40 (25%) records. Some of them defined them broadly as research that examines influence of trial/study characteristics on effect estimates, while for some it appeared that the definition described completely different designs, for example: ‘the study we conducted is a methodological survey (also called meta-epidemiologic research) that aims to evaluate trends and patterns in the literature with the overarching goal of improving the design, methods and conduct of future research.’ [8]. These definitions and descriptions (extracted verbatim) are shown in Supplementary File 4.
In 29 of the 40 records (7%) we found one or more references that supported definitions of meta-epidemiological studies. Details about these references are also shown in Supplementary File 5. Three most commonly used references in definitions of meta-epidemiological study were those of Sterne et al. from 2002 [2] (cited in 20 definitions), Naylor from 1997 [1] (eight definitions) and Wood et al. from 2008 [9] (four definitions).
Statistics & unit of analysis used in studies self-reported as meta-epidemiological
Less than a half of the studies used the two-step approach in data analysis (n = 54, 43%) that was described by Sterne et al. in 2002 [2]. This is, performing a metaregression for each included meta-analysis (first step) and combining these estimates (e.g., ratio of odds ratios for binary outcomes) in a meta-analysis (second step). The other categories of statistics used in these studies are shown in Table 2. The most common units of analysis used in 127 ‘meta-epidemiological’ studies were meta-analysis (n = 60; 47%), randomized controlled trials (n = 32; 25%) and systematic reviews (SR) (n = 12; 9.4%); all categories of units of analysis are shown in Table 2.
| Term | N | References to study/studies that used the term | Ref. |
|---|---|---|---|
| Meta-epidemiological review | 5 | Bijle et al. (2018) Deeks et al. (2003) Doig et al. (2009) Koletsi et al. (2016) Umberham et al. (2017) | [6,10–13] |
| Meta-epidemiological survey | 4 | Hemkens et al. (2016a) Hemkens et al. (2016b) Kovic et al. (2017) Wallach et al. (2017) | [14–17] |
| Meta-meta-analysis | 3 | Katerndahl et al. (1999) Cleophas et al. (2017) Sterne et al. (2002) | [2,18,19] |
| Meta-epidemiological analysis | 2 | Bafeta et al. (2012) Ewald et al. (2019) | [20,21] |
| Metaepidemiologic approach | 2 | Crossley et al. (2008) Janiaud et al. (2018) | [22,23] |
| Meta-epidemiological assessment | 2 | Bijle et al. (2018) Schuit et al. (2018) | [13,24] |
| Meta-epidemiologic empirical evaluation | 2 | Tedesco et al. (2018) Vandermeer et al. (2018) | [25,26] |
| Meta-epidemiologic evaluation | 2 | Contopoulos-Ioannidis et al. (2016) Ladanie et al. (2019) | [27,28] |
| Meta-epidemiological overview | 2 | Boef et al. (2015) Janiaud et al. (2018) | [29,30] |
| Systematic metareview | 2 | Francke et al. (2008) Egan et al. (2008) | [31,32] |
| Cross-sectional meta-epidemiological study | 1 | Pandis et al. (2015) | [33] |
| Meta-assessment approach | 1 | Bijle et al. (2018) | [13] |
| Network meta-epidemiological study | 1 | Chaimani et al. (2013) | [34] |
| Network meta-epidemiology | 1 | Trinquart et al. (2013) | [35] |
| Meta-meta-analytic approach | 1 | Ban et al. (2016) | [36] |
| Meta-meta-epidemiology | 1 | Trinquart et al. (2013) | [35] |
| Meta-analytical review | 1 | Papageorgiou et al. (2014) | [37] |
| Meta-epidemiologic regression approach | 1 | Rutjes et al. (2006) | [38] |
| Meta-epidemiological investigation | 1 | Shinohara et al. (2017) | [39] |
| Cumulative meta-epidemiological analysis | 1 | Storz-Pfenning et al. (2017) | [40] |
| Meta-epidemiologic systematic review | 1 | Yerokhin et al. (2016) | [41] |
| Meta-epidemiological meta-analysis | 1 | Cleophas and Zwinderman (2017) | [19] |
| Multiple expressions about study designs | |||
| ‘Meta analysis’ and ‘meta-epidemiological study’ | 2 | Trone et al. (2018) Henriksen et al. (2016) | [42,43] |
| ‘Systematic review and meta-analysis and meta-epidemiological approach’ | 1 | Watzlawick et al. (2016) | [44] |
| ‘Cumulative meta-epidemiological and trial sequential analysis’ and ‘cumulative meta-analysis and trial sequential analysis’ | 1 | Storz-Pfenning et al. (2017) | [40] |
| ‘Review of meta-analyses’ and ‘meta-epidemiology study’ | 1 | Tzoulaki et al. (2011) | [45] |
| ‘Systematic review’ conducted ‘by using a meta-epidemiology approach’ | 1 | Dragioti et al. (2019) | [46] |
| ‘Systematic review and meta-regression’ conducted ‘by using a meta-epidemiology approach’ | 1 | Janiaud et al. (2017) | [47] |
| ‘Review of reviews and meta-regression’ that used ‘meta-epidemiology approach’ | 1 | Oliver et al. (2010) | [48] |
| ‘Systematic review with meta-analysis and methodological overview’ and ‘meta-epidemiological assessment’ | 1 | Papageorgiou et al. (2017) | [49] |
| Systematic reviews and new empirical investigations (using meta-epidemiological techniques) were conducted | 1 | Deeks et al. (2003) | [10] |
| Characteristic | N (%) |
|---|---|
| Type of statistics | |
| Descriptive statistics | 37 (29.4) |
| Meta-analysis | 17 (13.5) |
| Meta regression | 6 (4.8) |
| One-study independent group comparison | 4 (3.1) |
| Logistic regression | 2 (1.6) |
| Bayesian meta-analysis | 1 (0.8) |
| Cox regression | 1 (0.8) |
| Discriminatory analysis | 1 (0.8) |
| Network meta-analysis | 1 (0.8) |
| Pooled analysis | 1 (0.8) |
| Regression analysis | 1 (0.8) |
| Unit of analysis | |
| Meta-analysis | 60 (47) |
| Randomized controlled trials | 32 (25) |
| Systematic reviews | 12 (9.4) |
| Network meta-analysis | 4 (3.1) |
| Overviews | 1 (0.8) |
| Other† | 19 (15) |
†
The ‘other’ was a group of heterogeneous units of analysis, including various other primary studies that were not RCTs or a combination of trials and other types of information sources (such as ‘journals’ instructions of authors and RCTs’, ‘validation studies included in systematic reviews’ or ‘bodies of evidence).
RCT: Randomized controlled trial.
Protocol registration & availability
Among 127 analyzed studies that self-identified as ‘meta-epidemiological’, 12 (9.4%) reported that they registered their study in PROSPERO, 8% reported that their protocol was published in a peer-reviewed journal, 4% reported that their protocol was both registered in PROSPERO and published in a peer-reviewed journal. We did not find any studies that have published their protocol as a Supplementary File. Two studies wrote that their protocol was not eligible for registration in PROSPERO [41,50]. One reported that the study protocol was published online and this was posted on the website of CAMARADES (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies), in the format and headings of a PROSPERO protocol [44]. Complete information reported about registration and availability of protocols in the studies are shown in Table 3.
| Reported information about protocol availability/registration | N (%) |
|---|---|
| Not reported | 102 (80) |
| Registered in PROSPERO | 8 (6.3) |
| Both registered in PROSPERO and published in a peer-reviewed journal | 4 (3.1) |
| Protocol published in a peer-reviewed journal | 4 (3.1) |
| Both published in a peer-reviewed journal and registered in a clinical trial registry | 1 (0.8) |
| No registered protocol exists | 1 (0.8) |
| Published on Campbell Collaboration web site | 1 (0.8) |
| There was no a priori protocol for this study. | 1 (0.8) |
| No formal protocol exists for this study. | 1 (0.8) |
| Protocol published on the institutional web site | 1 (0.8) |
| Protocol published on CAMARADES web site in the form of PROSPERO record | 1 (0.8) |
| Available on request and not registered in PROSPERO | 1 (0.8) |
| Not eligible for registration in PROSPERO | 1 (0.8) |
| There was no a priori protocol for this study | 1 (0.8) |
Using reporting checklists
Among 127 studies self-identified as ‘meta-epidemiological’, there were 14 (11%) that indicated that the study was reported in line with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). Seven (6%) articles indicated that their article was reported in line with the ‘Guidelines for reporting meta-epidemiological methodology research’ of Murad and Wang [51]. One wrote that a limitation of the study was that there is no specific guideline for meta-epidemiological studies and that Murad and Wang guideline was not EQUATOR-endorsed. Several articles reported that they used multiple reporting guidance, including that of Cochrane and other journal manuscripts. One study indicated that they used STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) checklist for reporting (Table 4).
| Reporting checklist/guidelines used | N (%) | Ref. |
|---|---|---|
| Not reported | 105 (83%) | |
| PRISMA | 9 (7) | |
| Murad and Wang guidelines | 6 (4.7) | [51] |
| Cochrane and PRISMA | 3 (2.4) | |
| There are no reporting guidelines for this type of research. Although a reporting guideline for meta-epidemiological research was proposed, a standardized guideline endorsed by EQUATOR is much needed | 1 (0.8) | [51] |
| STROBE | 1 (0.8) | |
| Loosely conducted and reported in line with Cochrane, PRISMA and Deschartres article | 1 (0.8) | [3] |
| PRISMA, Murad and Wang guidelines, Zhang article | 1 (0.8) | [51,52] |
EQUATOR: Enhancing the Quality and Transparency of Health Research; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; STROBE: STrengthening the Reporting of OBservational studies in Epidemiology.
Keyword ‘meta-epidemiology’
Among the 151 journal articles in our sample, 34 (23%) had a keyword meta-epidemiology or variations of this term. Six variations were used: meta-epidemiology (n = 21/34; 62%), meta-epidemiological study (n = 9/34; 26%), meta-epidemiologic study (n = 1/34; 3%), meta-epidemiological studies (n = 1/34; 3%), meta-epidemiological survey (n = 1/34; 3%) and meta-epidemiological dataset (n = 1/34; 3%).
In 28 of these articles, authors self-reported their study design as a ‘meta-epidemiological study’. Among the remaining six articles, study design was not reported in one and the remaining five were self-reported as scoping review, review, SR, consensus and commentary (one of each).
Indexing ‘meta-epidemiological’ studies on PubMed
Among 127 studies self-identified as ‘meta-epidemiological’ studies, 84 (67%) had information about ‘publication type’ on PubMed. There were 41 studies with one descriptor, 34 with two descriptors and nine with three descriptors. Eight ‘publication types’ were used to describe those publications: meta-analysis (n = 51/84; 61%), review (n = 46; 55%); SR (n = 20; 24%), comparative study (n = 10; 12%), multicenter study (n = 2; 2.4%), letter (n = 2; 2.4%), evaluation studies (n = 2; 2.4%) and observational study (n = 1; 1.2%).
Discussion
The main finding of our study is that there are different perceptions of the meaning of the term ‘meta-epidemiological’ study and that various study designs have been labeled with one or another version of this descriptor. There are also various versions of the names of ‘meta-epidemiological’ study design and it is unclear whether the authors should use all these terms as synonyms. For example, it is currently unclear whether there is any difference between, meta-epidemiological survey, meta-epidemiological overview and meta-epidemiological review.
Two reports are usually cited as the originators of terms ‘meta-epidemiology’ and ‘meta-epidemiological’ study; a 1997 editorial by CD Naylor [1] used term ‘meta-epidemiology’ in the context of the prior report by Roberts and Schierhout, who had published a paper describing difficulties obtaining information about unpublished clinical trials [53] for conducting SRs.
In 2002, Sterne et al. used the term ‘meta-epidemiological’ research/studies and referred to it as a statistical method for assessing influence of trial characteristics on treatment effect estimates [2]. In 2013, Trinquart et al. addressed methodological differences between three terms: meta-epidemiology, meta-meta-epidemiology and network meta-epidemiology and offered detailed comparison of methodological features of these three approaches [35].
Other definitions are moving away from the original one. Bae wrote that the concept of meta-epidemiology has been initially introduced for dealing with methodological limitations of SRs for interventional trials and that the paradigm of meta-epidemiology “has shifted from a statistical method into a new methodology to close gaps between evidence and practice.” [54].
Another source defines meta-epidemiological study as a tertiary research level and puts it in the same group of studies such as umbrella reviews or overviews of SRs, indicating that meta-epidemiological studies usually include secondary research studies but also primary studies, and that such studies could also include only editorials and thus completely disregard primary or secondary type of research [55].
In our study, we found various definitions of meta-epidemiological study; manuscripts of Naylor [1] and Sterne et al. [2] were most commonly referenced by authors in the context of those definitions. Multiple authors have referenced the Sterne et al. [2] manuscript when describing their statistical methods. However, as our categorization of statistical methods indicates, various types of analyses are conducted in studies called meta-epidemiological studies and less than half of studies that were self-reported as meta-epidemiological have used two-step analysis described by Sterne et al. [2].
Additionally, several studies that were not self-labelled as meta-epidemiological have used keywords ‘meta-epidemiology’ or ‘meta-epidemiological study’. Some authors provided examples of ‘meta-epidemiological studies’, but those examples did not have meta-epidemiological analysis described by Sterne et al. For example, a manuscript from our sample [56] referenced a study by Slade and Keating as a meta-epidemiological study [57], but this study had descriptive statistics only – frequencies and percentages – about reporting of certain trial characteristics. Murad and Whang [51] published reporting guideline for meta-epidemiological methodology research and described manuscript of Gandhi et al. [58] as an example of a meta-epidemiological study, even though this study [58] had descriptive statistics and logistic regression.
Some authors used word meta-analysis as a synonym for meta-epidemiological studies, describing their study as both meta-analysis and a meta-epidemiological study [42,43]. One SR with meta-analysis of animal studies indicated that they used ‘meta-epidemiological approach’ [44]. We also reported about other cases of using multiple descriptors about their study type interchangeably, further adding to the confusion about this study type.
Almost 10% of the studies self-described as ‘meta-epidemiological’ reported that their study protocol was registered in PROSPERO, a free-of-charge online registry for protocols of SRs. This may indicate that authors consider that their meta-epidemiological studies are a type of SR. One group of authors wrote that due to nature of their study, their protocol was not eligible for registration in PROSPERO [50]. However, these are not isolated cases. An informal search of PROSPERO conducted on 12 September 2019 indicated that there were 20 records with terms ‘meta-epidemiological’ or ‘meta-epidemiologic’ when the search is restricted to title; when these terms are searched anywhere in the record, there were 50 such records. The first such record was registered in 2013. This adds to the confusion about difference between a SR and a meta-epidemiological study.
It has to be emphasized that PROSPERO indicates in its inclusion criteria that they will consider reviews that have at least one outcome of direct patient or clinical relevance. Therefore, depending on the definition of such outcome, it is likely that some studies labelled as ‘meta-epidemiological’ were denied registration in PROSPERO.
Some of the studies in our sample, in a description of their study design, indicated that they conducted both SR and meta-epidemiological study (analysis/assessment). For example, Deeks et al. published in 2003 ‘Three systematic reviews and new empirical investigations (using meta-epidemiological techniques) were conducted.’ [10]. The authors of meta-epidemiological studies usually employ search in their methods to retrieve records for analysis. This search indeed can be systematic, for example, include a complex and structured search strategy and multiple information sources. However, there is still no consensus regarding differences between a SR, a study design in which there is a clinical question answered and that includes many more methodological steps beyond searching and many types of methodological studies. A call for consensus guidelines on classification and reporting of methodological studies has been published recently [59] and we hope that international research community interested in this topic will reach consensus in due time.
Some of the expressions related to protocol registration might have been prompted by editors and reviewers. However, currently, there are no formal expectations about registration of protocols of methodological studies [60].
About 10% of meta-epidemiological studies indicated that they planned/reported their studies in line with PRISMA, which is a reporting guideline for SRs. This could be due to several reasons. First, the authors might consider that PRISMA is relevant for this kind of studies too. Second, the editors or peer-reviewers might have requested it [61]. One meta-epidemiological study indicated that it was reported in line with STROBE, a reporting checklist for observational research. Few studies cited guidelines for reporting meta-epidemiological methodology research, proposed by Murad and Wang [51]; these guidelines were not endorsed by EQUATOR.
However, we would like to acknowledge that some of the studies in our sample were published before the existence of PROSPERO and before publication of some of the reporting checklists used in these studies.
Keyword ‘meta-epidemiology’ was used in few studies, but these were heterogeneous study designs, based on the authors’ self-report. This also indicates that the term meta-epidemiology is indiscriminately used to denote various study designs, from reports described as SRs or meta-analysis to those that are simply described as meta-epidemiological studies.
Studies self-identified as meta-epidemiological were indexed in PubMed with eight different descriptors for ‘publication types’; more than half were indexed as meta-analysis or review and the next most common indexed publication type was SR. Not a single one was identified in PubMed ‘publication type’ as a meta-epidemiological study. PubMed does not have ‘meta-epidemiological study’ (or any variation of the term) as a publication type [62].
Without reporting guideline and clarity regarding terminology in this particular field, the authors are using various terms that may not be synonymous. If someone would like to analyze such studies, they may be difficult to find without uniform terminology if the authors did not use descriptor ‘meta-epidemiological’ in the title/abstract/keyword. For example, Deschartres et al. [3] wrote that the first meta-epidemiological study was conducted by Schulz et al. in 1995 [63]; however, in the manuscript of Schulz et al., the term ‘meta-epidemiological’, or any variation of that term, was not used. Therefore, the manuscript of Schulz et al. was not retrieved via search that we used for this study.
We would like to invite research community interested in this field of research to recommend uniform terminology for studies. Currently, studies described as meta-epidemiological are also described as ‘meta-research’, methodological studies, research-on-research and in all these cases the terms are ambiguous and could likely be considered as including systematic reviews/meta-analyses that address specific clinical questions [64].
Uniform labeling of types of studies will enhance understanding about the content of those manuscripts and searching for certain types of studies. Heterogeneous terminology is contributing to misunderstanding about the study types.
Even though we have cited specific examples, we would like to emphasize that the purpose of this study was not to name and shame authors who have used label ‘meta-epidemiological’ for various study designs. Some of us have contributed to that heterogeneity as well [7,65].
A limitation of this study is that sometimes it was difficult to categorize a study as ‘meta-epidemiological’, for example, in a manuscript that described study design as ‘combined analysis of metaepidemiological studies’ [66]. Also, one information source was very comprehensive and had multiple descriptions of a meta-epidemiological study, which we did not extract [55]. However, we have transparently reported included information sources in our Supplementary File, so readers can browse them as well.
Furthermore, we have analyzed only information sources indexed in MEDLINE and Embase. Also, we conducted our search in August 2019 and more potentially eligible studies may have been published after that date. However, conducting such studies is always a moving target; after updating the search, screening, data extraction and analysis takes place. By the time all this is completed and manuscript updated, the search is outdated again. Therefore, our findings may not be generalizable to studies described as ‘meta-epidemiological’ that were indexed in other databases and studies published after our search date.
In conclusion, authors of published literature use heterogeneous definitions and descriptors for meta-epidemiological studies. Methodological research-on-research studies are also labeled as meta-epidemiological. Research community would benefit from consensus about definition of a meta-epidemiological study.
•
The term ‘meta-epidemiological’ study is relatively recent but there is no consensus definition about this type of study.
•
Hereby we analyzed various information sources that either have provided definition of a meta-epidemiological study or that their authors have labelled as meta-epidemiological studies.
•
We found that there are different perceptions of the meaning of the term ‘meta-epidemiological’ study.
•
Various study designs have been labeled with one or another version of this descriptor.
•
There are various versions of the names of ‘meta-epidemiological’ study design and it is unclear whether the authors should use all these terms as synonyms.
•
It is unclear whether there is any difference between studies labelled as meta-epidemiological survey, meta-epidemiological overview, meta-epidemiological review.
•
Further work is needed in this field, in order to reach global consensus about nomenclature and definition of studies labelled as meta-epidemiological.
Supplementary data
To view the supplementary data that accompany this paper please visit the journal website at: Supplementary Material
Author contributions
L Puljak and D Pieper designed the study. L Puljak, ZL Makaric, I Buljan and D Pieper performed data acquisition, analysis and interpretation. L Puljak and D Pieper wrote the first draft. L Puljak, ZL Makaric, I Buljan and D Pieper revised the first draft for important intellectual content. L Puljak, ZL Makaric, I Buljan and D Pieper approved the final version and agreed to be accountable for the work.
Financial & competing interests disclosure
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
No writing assistance was utilized in the production of this manuscript.
Supplementary Material
File (suppl_file.zip)
- Download
- 490.45 KB
References
1.
Naylor CD. Meta-analysis and the meta-epidemiology of clinical research. BMJ 315(7109), 617–619 (1997).
2.
Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Statistics in Medicine 21(11), 1513–1524 (2002).
3.
Dechartres A, Trinquart L, Faber T, Ravaud P. Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J. Clin. Epidemiol. 77, 24–37 (2016).
4.
Luo X, Wan M, Chang C et al. The reporting of meta-epidemiological studies: A cross-sectional study. Conference abstract. Cochrane Colloquium, Cape Town, South Africa (2017).
5.
Ge L, Tian JH, Li YN et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J. Clin. Epidemiol. 93, 45–55 (2018).
6.
Umberham B, Hedin R, Detweiler B, Kollmorgen L, Hicks C, Vassar M. Heterogeneity of studies in anesthesiology systematic reviews: a meta-epidemiological review and proposal for evidence mapping. British Journal of Anaesthesia 119(5), 874–884 (2017).
7.
Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: A meta-epidemiological study. Eur J Pain 22(10), 1844–1849 (2018).
8.
Gu AP, Gu CN, Ahmed AT et al. Sham surgical procedures for pain intervention result in significant improvements in pain: systematic-review and meta-analysis: Metaepidemiologic research requires reporting requirements. J. Clin. Epidemiol. 87, 108 (2017).
9.
Wood L, Egger M, Gluud LL et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ 336(7644), 601–605 (2008).
10.
Deeks JJ, Dinnes J, D'amico R et al. Evaluating non-randomised intervention studies. Health Technology Assessment 7(27), (2003).
11.
Doig GS, Simpson F, Sweetman EA. Evidence-based nutrition support in the intensive care unit: An update on reported trial quality. Curr. Opin. Clin. Nutr. Metab. Care 12(2), 201–206 (2009).
12.
Koletsi D, Spineli LM, Lempesi E, Pandis N. Risk of bias and magnitude of effect in orthodontic randomized controlled trials: a meta-epidemiological review. Eur. J. Orthod. 38(3), 308–312 (2016).
13.
Bijle MNA, Ekambaram M, Lo ECM, Yiu CKY. A meta-epidemiological review of meta-analysis on anti-caries effect of arginine-containing formulations. J. Evid. Based Dent. Pract. 19(1), 28–33 (2019).
14.
Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: Meta-epidemiological survey. BMJ (Online) 352, i493 (2016).
15.
Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey. CMAJ Open 4(2), E132–140 (2016).
16.
Kovic B, Zoratti M, Michalopoulos S, Silvestre C, Thorlund K, Thabane L. A meta-epidemiological survey of the reporting of effect modification in network meta-analyses. Value in Health 20(5), A330 (2017).
17.
Wallac JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Internal Medicine 177(4), 554–560 (2017).
18.
Katerndahl DA, Lawler WR. Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. Am. J. Epidemiol. 149(5), 429–441 (1999).
19.
Cleophas TJ, Zwinderman AH. Modern meta-analysis. Springer (2017).
20.
Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: Meta-epidemiological study. BMJ (Online) 344(7846), e813 (2012).
21.
Ewald H, Speich B, Ladanie A, Bucher HC, Ioannidis JPA, Hemkens LG. Marginal structural models and other analyses allow multiple estimates of treatment effects in randomized clinical trials: Meta-epidemiological analysis. J. Clin. Epidemiol. 107, 12–26 (2019).
22.
Crossley NA, Sena E, Goehler J et al. Empirical evidence of bias in the design of experimental stroke studies: A metaepidemiologic approach. Stroke 39(3), 929–934 (2008).
23.
Janiaud P, Glais F, Grenet G, Lajoinie A, Cornu C, Kassai-Koupai B. Use of available clinical evidence to extrapolate drug effects from adults to children. Therapie 73(2), 119–125 (2018).
24.
Schuit E, Li AH, Ioannidis JPA. How often can meta-analyses of individual-level data individualize treatment? A meta-epidemiologic study. Int. J. Epidemiol. 48(2), 596–608 (2019).
25.
Tedesco D, Farid-Kapadia M, Offringa M et al. Comparative evidence on harms in pediatric randomized clinical trials from less developed versus more developed countries is limited. J. Clin. Epidemiol. 95, 63–72 (2018).
26.
Vandermeer B, Van Der Tweel I, Jansen-Van Der Weide MC et al. Comparison of nuisance parameters in pediatric versus adult randomized trials: a meta-epidemiologic empirical evaluation. BMC Med. Res. Methodol. 18(1), 7 (2018).
27.
Contopoulos-Ioannidis D, Tseretopoulou X, Ancker M et al. Comparative rates of harms in randomized trials from more developed versus less developed countries may be different. J. Clin. Epidemiol. 78, 10–21 (2016).
28.
Ladanie A, Speich B, Briel M et al. Single pivotal trials with few corroborating characteristics were used for FDA approval of cancer therapies. J. Clin. Epidemiol. 114, 49–59 (2019).
29.
Boef AGC, Dekkers OM, Le Cessie S. Mendelian randomization studies: A review of the approaches used and the quality of reporting. Int. J. Epidemiol. 44(2), 496–511 (2015).
30.
Janiaud P, Cristea IA, Ioannidis JPA. Industry-funded versus non-profit-funded critical care research: a meta-epidemiological overview. Intensive Care Med. 44(10), 1613–1627 (2018).
31.
Francke AL, Smit MC, De Veer AJ, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med. Inform. Decis. Mak. 8, 38 (2008).
32.
Egan M, Tannahill C, Petticrew M, Thomas S. Psychosocial risk factors in home and community settings and their associations with population health and health inequalities: a systematic meta-review. BMC Public Health 8, 239 (2008).
33.
Pandis N, Fleming PS, Worthington H, Salanti G. The Quality of the Evidence According to GRADE Is Predominantly Low or Very Low in Oral Health Systematic Reviews. PLoS ONE 10(7), e0131644 (2015).
34.
Chaimani A, Vasiliadis HS, Pandis N, Schmid CH, Welton NJ, Salanti G. Effects of study precision and risk of bias in networks of interventions: A network meta-epidemiological study. Int. J. Epidemiol. 42(4), 1120–1131 (2013).
35.
Trinquart L, Dechartres A, Ravaud P. Commentary: Meta-epidemiology, meta-metaepidemiology or network meta-epidemiology? Int. J. Epidemiol. 42(4), 1131–1133 (2013).
36.
Ban JW, Emparanza JI, Urreta I, Burls A. Design characteristics influence performance of clinical prediction rules in validation: A meta-epidemiological study. PLoS ONE 11(1), e0145779 (2016).
37.
Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Assessing small study effects and publication bias in orthodontic meta-analyses: a meta-epidemiological study. Clin Oral Investig. 18(4), 1031–1044 (2014).
38.
Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, Van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174(4), 469–476 (2006).
39.
Shinohara K, Suganuma AM, Imai H, Takeshima N, Hayasaka Y, Furukawa TA. Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation. PLoS ONE 12(9), e0184786 (2017).
40.
Storz-Pfennig P. Potentially unnecessary and wasteful clinical trial research detected in cumulative meta-epidemiological and trial sequential analysis. J. Clin. Epidemiol. 82, 61–70 (2017).
41.
Yerokhin VV, Carr BK, Sneed G, Vassar M. Clinical trials registries are underused in the pregnancy and childbirth literature: a systematic review of the top 20 journals. BMC Res. Notes 9(1), 475 (2016).
42.
Trone JC, Ollier E, Chapelle C et al. Statistical controversies in clinical research: Limitations of open-label studies assessing antiangiogenic therapies with regard to evaluation of vascular adverse drug events-A meta-analysis. Annals of Oncology 29(4), 803–811 (2018).
43.
Henriksen M, Hansen JB, Klokker L, Bliddal H, Christensen R. Comparable effects of exercise and analgesics for pain secondary to knee osteoarthritis: A meta-analysis of trials included in Cochrane systematic reviews. J. Comp. Eff. Res. 5(4), 417–431 (2016).
44.
Watzlawick R, Rind J, Sena ES et al. Olfactory Ensheathing Cell Transplantation in Experimental Spinal Cord Injury: Effect size and Reporting Bias of 62 Experimental Treatments: A Systematic Review and Meta-Analysis. PLoS Biology 14(5), e1002468 (2016).
45.
Tzoulaki I, Siontis KCM, Ioannidis JPA. Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: Meta-epidemiology study. BMJ (Online) 343(7834), 1137 (2011).
46.
Dragioti E, Bjork M, Larsson B, Gerdle B. A meta-epidemiological appraisal of the effects of interdisciplinary multimodal pain therapy dosing for chronic low back pain. J. Clin. Med. 8(6), 871 (2019).
47.
Janiaud P, Cornu C, Lajoinie A, Djemli A, Cucherat M, Kassai B. Is the perceived placebo effect comparable between adults and children? A meta-regression analysis. Pediatric Research 81(1), 11–17 (2017).
48.
Oliver S, Bagnall AM, Thomas J et al. Randomised controlled trials for policy interventions: A review of reviews and meta-regression. Health Technology Assessment 14(16), 1–192 (2010).
49.
Papageorgiou SN, Hochli D, Eliades T. Outcomes of comprehensive fixed appliance orthodontic treatment: A systematic review with meta-analysis and methodological overview. Korean J. Orthod. 47(6), 401–413 (2017).
50.
Mickenautsch S, Yengopal V. Do Laboratory Results Concerning High-Viscosity Glass-Ionomers versus Amalgam for Tooth Restorations Indicate Similar Effect Direction and Magnitude than that of Controlled Clinical Trials? - A Meta-Epidemiological Study. PLoS ONE [Electronic Resource] 10(7), e0132246 (2015).
51.
Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence-Based Medicine 22(4), 139–142 (2017).
52.
Zhang Z. Meta-epidemiological study: a step-by-step approach by using R. J. Evid. Based Med. 9(2), 91–97 (2016).
53.
Roberts I, Schierhout G. Personal views - The private life of systematic reviews. Brit. Med. J. 315(7109), 686–687 (1997).
54.
Bae J-M. Meta-epidemiology. Epidemiol. Health 36, e2014019 (2014).
55.
Biondi-Zoccai G. Umbrella Reviews. Evidence Synthesis with Overviews of Reviews and Meta-Epidemiologic Studies. Springer (2016).
56.
Slade SC, Dionne CE, Underwood M et al. Consensus on Exercise Reporting Template (CERT): Modified Delphi Study. Phys. Ther. 96(10), 1514–1524 (2016).
57.
Slade SC, Keating JL. Exercise prescription: a case for standardised reporting. Br. J. Sports Med. 46(16), 1110–1113 (2012).
58.
Gandhi GY, Murad MH, Fujiyoshi A et al. Patient-important outcomes in registered diabetes trials. JAMA 299(21), 2543–2549 (2008).
59.
Lawson DO, Thabane L, Mbuagbaw L. A call for consensus guidelines on classification and reporting of methodological studies. J. Clin. Epidemiol. (2020).
60.
Puljak L, Pieper D. Registration of methodological studies, i.e. ‘research on research’ studies- should it be mandatory? J. Clin. Epidemiol. 35–36 (2019).
61.
Puljak L. Methodological studies evaluating evidence are not systematic reviews. J. Clin. Epidemiol. 110, 98–99 (2019).
62.
U. S. National Library of Medicine. Publication Characteristics (Publication Types) with Scope Notes (2019). www.nlm.nih.gov/mesh/pubtypes.html
63.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273(5), 408–412 (1995).
64.
Puljak L. Methodological research: open questions, the need for ‘research on research’ and its implications for evidence-based healthcare and reducing research waste. Int. J. Evid. Based Healthc. 17(3), 145–146 (2019).
65.
Riado Minguez D, Kowalski M, Vallve Odena M et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth. Analg. 125(4), 1348–1354 (2017).
66.
Savovic J, Jones H, Altman D et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol. Assess. 16(35), 1–82 (2012).
Information & Authors
Information
Published In
Pages: 497 - 508
PubMed: 32372658
Copyright
© 2020 Future Medicine Ltd.
History
Received: 20 December 2019
Accepted: 20 February 2020
Published online: 6 May 2020
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. (2020) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2019-0201
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Hans Lund, Karen A Robinson, History of evidence-based research, Journal of the Royal Society of Medicine, 10.1177/01410768261433038, 119, 4, (107-113), (2026).
- Zhen Peng, Shiqi Fan, Yuan Tian, Yingxia Wang, Zongshi Qin, Suhail Doi, Chang Xu, Comparing the accuracy of AI-assisted data extraction versus human double extraction in evidence synthesis: a randomised controlled trial protocol, BMJ Open, 10.1136/bmjopen-2025-106546, 15, 11, (e106546), (2025).
- Mathew J. Wingerson, Katherine L. Smulligan, Ellie Svoboda, Christine M. Baugh, Samantha N. Magliato, Julie C. Wilson, David R. Howell, Who Are the Participants in Pediatric Concussion Randomized Controlled Trials? A Meta-epidemiology Study, JOSPT Open, 10.2519/josptopen.2025.0112, 3, 4, (395-408), (2025).
- Natasja D. Vijfschagt, Huibert Burger, Marjolein Y. Berger, Thomas R. Fanshawe, Ann van den Bruel, Mariska M.G. Leeflang, Michiel R. de Boer, Gea A. Holtman, Variation in sensitivity and specificity of diverse diagnostic tests across health-care settings: a meta-epidemiological study, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2025.111816, 184, (111816), (2025).
- Silvia Gianola, Stefania Guida, Gaia Ravot, Carole Lunny, Silvia Bargeri, Greta Castellini, Gaps in completeness of reporting and methodological quality: a metaresearch study of 139 network meta-analyses published in January 2023 using PRISMA-NMA and AMSTAR-2, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2025.111783, 183, (111783), (2025).
- Masoud Mohammadi, Meta‐Epidemiology: A Combination of Epidemiology and Meta‐Analysis in Response to the Challenges of Systematic Reviews and Meta‐Analysis, Journal of Evidence-Based Medicine, 10.1111/jebm.70036, 18, 2, (2025).
- Jingyu Zhong, Xianwei Liu, Junjie Lu, Jiarui Yang, Guangcheng Zhang, Shiqi Mao, Haoda Chen, Qian Yin, Qingqing Cen, Run Jiang, Yang Song, Minda Lu, Jingshen Chu, Yue Xing, Yangfan Hu, Defang Ding, Xiang Ge, Huan Zhang, Weiwu Yao, Overlooked and underpowered: a meta-research addressing sample size in radiomics prediction models for binary outcomes, European Radiology, 10.1007/s00330-024-11331-0, 35, 3, (1146-1156), (2025).
- Youlin Long, Yurong Zheng, Xinyao Wang, Qiong Guo, Na Zhang, Ya Deng, Ruixian Tang, Zhengchi Li, Liang Du, The reporting quality of meta-epidemiological studies needs substantial improvement: a research on research study, Systematic Reviews, 10.1186/s13643-024-02661-7, 13, 1, (2024).
- Silvia Bargeri, Giacomo Basso, Ignazio Geraci, Greta Castellini, Alessandro Chiarotto, Silvia Gianola, Raymond Ostelo, Marco Testa, Tiziano Innocenti, Substantial discrepancies exist between registered protocol and published manuscript in trials on exercise interventions for chronic low back pain: a metaresearch study, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2024.111465, 173, (111465), (2024).
- Jingyu Zhong, Yue Xing, Yangfan Hu, Junjie Lu, Jiarui Yang, Guangcheng Zhang, Shiqi Mao, Haoda Chen, Qian Yin, Qingqing Cen, Run Jiang, Jingshen Chu, Yang Song, Minda Lu, Defang Ding, Xiang Ge, Huan Zhang, Weiwu Yao, The policies on the use of large language models in radiological journals are lacking: a meta-research study, Insights into Imaging, 10.1186/s13244-024-01769-7, 15, 1, (2024).
- See more