Skip to main content

Abstract

Background: Cochrane systematic reviews (CSRs) have a section ‘Implications for research’ where authors make suggestions for improving future research. The authors of the present study assessed the prevalence and time dynamics of different recommendations in the CSRs about anesthesia and pain. Methods: The authors included all CSRs published by the Cochrane Anaesthesia Group and Cochrane Pain and Palliative Care Group before 17 July 2020. The authors analyzed recommendations for improving future research listed in the ‘Implications for research’ section of these CSRs and categorized recommendations for improvements. Results: They analyzed 370 reviews. Four categories of recommendations were present in more than 40% of the reviews. Most reviews recommended a larger sample size and better outcome choice, study design and choice of future intervention. These recommendations gradually increased in frequency in the Cochrane Pain and Palliative Care Group and mainly decreased in the Cochrane Anaesthesia Group. Conclusion: Recommendations from CSRs offer useful advice for trialists designing new trials.

Background

Systematic reviews are considered the ‘gold standard’ in the evidence hierarchy because they attempt to identify, appraise and synthesize all empirical evidence to answer a specific research question [1,2]. Cochrane systematic reviews are reviews published in the Cochrane Database of Systematic Reviews (CDSR). They use explicit, systematic methods that aim to minimize bias and produce more trustworthy findings [3]. Cochrane reviews are prepared by Cochrane Review Groups, which provide authors with methodological and editorial support. Cochrane is organized into more than 50 Cochrane Review Groups, each focusing on a specific area of health.
All Cochrane reviews have a section called ‘Implications for research’, where authors can present their ideas on how future trials relevant to the topic for the review might be improved [4]. Previous research has shown that this section can be useful for identifying research gaps, providing guidance to trialists [5–7]. However, such a study is not available for the fields of anesthesia and pain, although multiple studies have explored clinical trials and systematic reviews of clinical trials in anesthesia and pain, indicating deficiencies in trial design, reporting and the awareness of authors regarding relevant methodological aspects [8–18].
This study examined the ‘Implications for research’ section of reviews published by two Cochrane Review Groups with an aim to analyze recommendations for improvements to future trials in anesthesia and pain. The authors also aimed to analyze changes over time in the most frequent categories of recommendations, as well as their frequency for pharmaceutical and non-pharmaceutical trials.

Methods

Study design

The authors conducted a meta-research study. They analyzed recommendations for improving future research listed in the ‘Implications for research’ section of Cochrane reviews from the fields of anesthesia and pain. This was a replication study that used methods similar to those of the earlier work of Pirosca et al., which focused on two other Cochrane Review Groups [7].

Eligibility criteria

The authors assessed all Cochrane reviews published before mid 2020 by two Cochrane Review Groups: Cochrane Anaesthesia Group (CA Group) and Cochrane Pain, Palliative and Supportive Care Group (PaPaS Group). They limited inclusion criteria to reviews from the fields of anesthesia and pain, as PaPaS reviews also include topics such as palliative and supportive care. However, they did include reviews in the field of palliative care that addressed the treatment of pain in cancer patients. They included only the latest published version of each review, and the full text of eligible reviews was retrieved from the Cochrane Library.

Search

On 17 July 2020 the authors searched the Cochrane Library using the advanced search and ‘Cochrane Group’ as a filter.

Screening

One author (R Runjic) screened all retrieved reviews for eligibility, and another author (L Puljak) verified her screening assessments.

Data extraction

From each eligible Cochrane review, the authors extracted verbatim the content of the ‘Implications for research’ section in the authors' conclusions, as well as the year when the current version of the review was published and the type of intervention (pharmaceutical, other).

Data categorization

Recommendations for improvements to future trials were categorized, starting with the 22 categories defined by Pirosca et al. [7]. The authors of the present study added more categories when they found recommendations that were not covered by the categories previously published by Pirosca et al. [7]. Two investigators (R Runjic and M Plenkovic) independently categorized the recommendations for each included Cochrane review. Disagreements were resolved via discussion or involvement of the other co-authors.

Outcomes

The primary outcome was the frequency of each category of recommendation for improvements to future trials, together and separately for the anesthesia and pain reviews. The secondary outcomes were the frequency of each category of recommendation in different time periods and for the two different types of interventions.

Data analysis

The authors reported data using descriptive analysis, as frequencies and percentages.

Results

Eligibility

The authors retrieved 140 reviews published by the CA Group from 2003 to 2019 and 320 reviews published by the PaPaS Group from 1999 to 2020. After screening these 460 reviews, the authors found 370 reviews eligible for the study.
Out of 90 excluded reviews, 45 had been withdrawn (five from the CA Group and 40 from the PaPaS Group) and 45 investigated topics outside the topic of interest, such as aspects of palliative and supportive care that are not related to pain. The list of excluded reviews, with reasons, is available in Supplementary File 1.
Among the 370 included reviews, 135 (36%) were from the CA Group and 235 (64%) were from the PaPaS Group. The list of included reviews is available in Supplementary File 2.

Frequency of different categories of recommendations

Recommendations for research were categorized into 32 categories. Table 1 shows the frequency of the most common recommendations. The frequency of all recommendations is reported in Supplementary File 3.
Table 1. Frequency of categories of recommendations for future research identified in the ‘Implications for research’ section of reviews published in Cochrane Anaesthesia Group and Cochrane Pain and Palliative Care Group.
Categories of recommendationsOverall use of the category
n = 370
n (%)
Use of the category in Cochrane Anaesthesia Group
n = 135
n (%)
Use of the category in Pain, Palliative and Supportive Care Group
n = 235
n (%)
Larger sample size166 (45)64 (47)102 (43)
Better outcome choice160 (43)63 (46)97 (41)
Better design155 (41)43 (31)112 (47)
Better choice of future intervention or comparator151 (40)47 (34)104 (44)
Better reporting116 (31)27 (20)89 (37)
Adverse events/safety116 (31)46 (34)70 (29)
Additional trials needed92 (24)34 (25)58 (24)
Better choice of eligibility criteria89 (24)31 (23)58 (24)
Use of validated rating scales74 (20)16 (11)58 (24)
Ensure blinding71 (19)28 (20)43 (18)
Standardized/clearly defined interventions70 (18)30 (22)40 (17)
Longer follow-up68 (18)15 (11)53 (22)
Higher-quality trials needed61 (16)25 (18)36 (15)
Better methodology54 (14)23 (17)31 (13)
Better link to current clinical pathways52 (14)20 (14)32 (13)
Use of cost–effectiveness analysis43 (11)18 (13)25 (10)
Use of a set of standardized outcomes43 (11)10 (7,4)33 (14)
Trials of this type are not a priority or are unlikely38 (10)3 (2,2)35 (15)
Other142 (38)33 (24)109 (46)
A total of 32 categories were identified; shown here are categories of recommendations that were reported in more than 10% of reviews.
The most frequent categories of recommendations, present in more than 40% of reviews in both Cochrane groups were larger sample size, better outcome choice, better design and better choice of future intervention (Table 1). More than 20% of all reviews recommended better reporting, the need for additional research, more investigation of adverse events and safety and better choice of eligibility criteria.
15% (35/235) of the eligible reviews from the PaPaS Group and 2.2% (3/135) of the reviews from the CA Group stated that further research into the topic of the review is not a priority or is unlikely to happen. This recommendation was most commonly explained by noting that good alternative interventions were already available (n = 15) or that sufficient evidence was available about the studied intervention (n = 9). The details of these recommendations are shown in Supplementary File 4.
The most common categories of recommendations that were more frequent in the PaPaS Group than in the CA Group were better design, better reporting and use of validated rating scales. The categories of recommendations that were more frequent in the CA Group than in the PaPaS Group were related to adverse events and safety and the need for standardized or clearly defined interventions (Table 1).

Frequency of categories of recommendations throughout different time periods

The authors analyzed changes over time for the five most frequent categories of recommendations in each group in 2-year periods, as shown in Figures 1 & 2. The data before 2013 were analyzed together due to the small number of available reviews.
Figure 1. The five most frequently used ‘Implications for research’ recommendations for the Cochrane Anaesthesia Group published from 2003 to 2019.
The percentages shown are the averages over a 2-year period, except for the data from 2003–2013, which were analyzed together due to a smaller amount of conducted research.
Figure 2. The five most frequently used ‘Implications for research’ recommendations for the Cochrane Pain and Palliative Care Group published from 1999 to 2019.
The percentages shown are the averages over a 2-year period, except for the data from 1999–2013, which were analyzed together due to a smaller amount of conducted research.
The numbers of analyzed reviews published by the CA Group over time were 20 reviews in 2003–2013 (15%), 43 reviews in 2014–2015 (32%), 39 reviews in 2016–2017 (29%) and 33 reviews in 2018–2019 (24%). The need for trials to have larger sample sizes is the only category of recommendation that increased in frequency throughout the years (from 35% of reviews in the first time period to 55% of reviews in the last time period). The frequencies for the other most common categories of recommendations decreased (Figure 1).
The numbers of analyzed reviews from the PaPaS Group published over time were 85 reviews in 1999–2013 (36%), 61 reviews in 2014–2015 (26%), 65 reviews in 2016–2017 (28%) and 24 reviews in 2018–2019 (10%). An increase in frequency was noted in all of the five most common categories of recommendations. Categories for better reporting and better outcome choice increased almost threefold, while the other most common categories had an almost twofold increase in frequency (Figure 2).

Frequency of categories of recommendations with regard to a different type of intervention

The five most common categories of recommendations were also analyzed with regard to whether the applied intervention was pharmaceutical or non-pharmaceutical (other). As shown in Figure 3, among the reviews published by the CA Group, 71 reviews (53%) had a pharmaceutical intervention and 64 reviews (47%) had a non-pharmaceutical intervention. Compared with reviews with a non-pharmaceutical intervention, reviews with a pharmaceutical intervention had a more frequent recommendation for a larger sample size (51 vs 44%) and better choice of future intervention (39 vs 30%; Figure 3).
Figure 3. The five most frequently used ‘Implications for research’ recommendations for the Cochrane Anaesthesia Group with regard to the type of intervention.
Among the reviews published by the PaPaS Group, 180 reviews (77%) had a pharmaceutical intervention and 55 reviews (23%) had a non-pharmaceutical intervention (Figure 4). Almost all of the most common categories of recommendations were more frequent in reviews with non-pharmaceutical interventions, with categories for better reporting (60 vs 31%), better design (56 vs 45%) and larger sample size (55 vs 40%) showing the largest difference in frequency (Figure 4).
Figure 4. The five most frequently used ‘Implications for research' recommendations for the Cochrane Pain, Palliative and Supportive Care Group with regard to the type of intervention.
All raw data collected in this study, with all categorizations, are available in Supplementary File 5 (available online at Open Science Framework: https://osf.io/6pj4m/).

Discussion

This analysis of the implications for research in Cochrane reviews from the fields of anesthesia and pain showed that four categories of recommendations stood out, having been mentioned in more than 40% of the analyzed reviews. Those recommendations call for larger sample size, better outcome choice, better design and better choice of future intervention.
The size of the sample affects the quality of the research [19]. Turner et al. investigated the effect of size on the results of meta-analyses and defined ‘adequate power’ as ≥50% power to detect a 30% relative risk reduction [20]. They concluded that underpowered studies do not affect meta-analyses that include at least two adequately powered studies. However, they also noticed that underpowered studies made up all of the evidence in most Cochrane reviews, with all studies being underpowered in 70% of the meta-analyses [20]. This is reflected in the present findings, with the need for future trials to have larger sample sizes being recommended in 45% of the analyzed reviews.
Good choice of outcomes is a crucial factor for a high-quality study. The best design and the most rigorous methods cannot make up for a poor choice of outcomes [21]. There are many guidelines for choosing the right outcomes [22] and an increasing number of research fields have a recommended set of outcomes, known as a core outcome set (COS) [23]. A COS is a pre-agreed minimum of outcomes that should be measured and reported in research in a specific field, so that similar trials can be compared and their results included in a meta-analysis as necessary [23]. It has been previously shown that trials and systematic reviews from the field of anesthesiology are suboptimal when it comes to using existing COSs [11–13,24,25] and that researchers may not be aware of the existence of a COS [11,13].
The results of the present study show that the need for trials to improve their choice of outcomes is very high; 43% of the reviews recommended a better choice of outcomes, while 11% of the reviews recommended the use of a standardized set of outcomes. A similar result was noted by Pirosca et al. [7], who found that more than half of the analyzed Cochrane reviews recommended a better choice of outcomes. Additionally, the frequency of that category of recommendation has increased throughout the years in the PaPaS Group. In the CA Group, an increase in the frequency of recommendations for a larger sample size was observed, while other categories of recommendations became less common over time. In the PaPaS Group, however, an increase in frequency was observed in all five most common categories, with an almost threefold increase in the proportion of reviews recommending better reporting and better choice of outcomes.
Recommendations from systematic reviews should, ideally, improve future research. However, the question is whether trialists read and adhere to the recommendations from Cochrane reviews when designing their new studies. It has, for example, been noted previously that trialists frequently do not mention the use of systematic reviews when reporting the results of their trials in the field of anesthesiology [18].
In the analyzed Cochrane reviews, the calls for better design and better reporting reflect the findings of multiple studies from the fields of anesthesiology and pain, where deficiencies have been reported and recommendations for improvement advised [10,14–16,26–28].
It has been reported that studies of non-pharmaceutical interventions score lower on assessment of methodology than studies of pharmaceutical interventions [29]. Some support for this was found in the PaPaS Group, but not in the CA Group.
The ‘Implications for research’ section in Cochrane reviews has been analyzed in previous studies [7,21]. The findings from a study conducted by Pirosca et al., which investigated two other Cochrane Review Groups, are similar in part to the current results, with better outcome choice, better choice of future intervention and larger sample size among the most frequent recommendations in both studies [7].
On the other hand, a recommendation for longer follow-up, which was found in more than a third of the reviews analyzed in the study of Pirosca et al. [7], was present in only 23% of the PaPaS Group's reviews and 11% of the CA Group's reviews in the present study. This difference may be due to differences in individual areas of research, since the fields of interest for Pirosca et al. [7] were schizophrenia, multiple sclerosis and other rare diseases of the central nervous system, all of which are lifelong diseases.
The present findings regarding differences in the frequency of recommendations for improvement in reviews of pharmaceutical versus non-pharmaceutical interventions were not consistent. In the CA Group, reviews with a pharmaceutical intervention had a more frequent recommendation for larger sample size and better choice of future intervention. On the contrary, in the PaPaS Group, the most common categories of recommendations were more frequent in reviews with non-pharmaceutical interventions. Further research on a larger sample of reviews may be warranted to explore any systematic differences in pharmaceutical versus non-pharmaceutical reviews in this respect.
In 2007, Clarke et al. described their analysis of the ‘Implications for research’ section of Cochrane reviews published in 2005 [4]. Their primary aim was to determine the frequency of recommendations for future research. A total of 2535 reviews were included and, among other results, 82% of reviews mentioned the need for a better choice of future intervention, which is twice the proportion of reviews in the present study (41%). This does not necessarily mean that trials have improved on this aspect over time because there are major differences in the quality and rigor of the conduct of primary research in different areas of medicine.

Limitations

Even though the categorization was done by two authors independently, who then resolved all their disagreements, the authors cannot exclude the possibility that other authors would have categorized the data in this study differently. For this reason, the full raw data, with all categorizations, are provided in Supplementary File 5 (available at: https://osf.io/6pj4m/). In this way, readers can assess the text of the implications for research of each analyzed review and how the authors decided to categorize that text.
It is also important to note that the authors cannot extrapolate the results from this study to reviews in other Cochrane Review Groups due to a lack of diversity of the collected data. Cochrane includes more than 50 different Review Groups, each specialized in a certain field, which by itself can be a confounding factor.

Conclusion

Cochrane reviews are not only useful sources of high-quality evidence syntheses but also sources of advice for trialists regarding improvements needed in the design and conduct of future trials. The ‘Implications for research’ section in Cochrane reviews offers insight into the shortcomings of the available trials. Thus, trialists should use these recommendations to design better studies in the future. Funders of research could use this information in at least these two fields to encourage authors to respond to particular design challenges highlighted in relevant systematic reviews.
Summary points
Four categories of recommendations were brought up in more than 40% of reviews: larger sample size, better outcome choice, better design and better choice of future intervention.
The most frequent recommendation for future trials overall was a larger sample size (45% of reviews).
The ‘Implications for research’ section in Cochrane reviews has been analyzed in previous studies, and two other Cochrane Review Groups were investigated. The results were similar in part to the present study's results, with better outcome choice, better choice of future intervention and larger sample size among the most frequent recommendations in both studies.
In the Cochrane Anaesthesia Group, an increase in the frequency of recommendations for a larger sample size was observed, while other categories of recommendations became less common over time.
In the Cochrane Pain, Palliative and Supportive Care Group, an increase in frequency of recommendations was observed in all five most common categories, with an almost threefold increase in the proportion of reviews recommending better reporting and better choice of outcomes.

Open access

This work is licensed under the Creative Commons Attribution 4.0 License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Author contributions

R Runjic: study design, data analysis, writing of the first draft of the manuscript, review of the manuscript, approval of the final version of the manuscript. M Plenkovic: study design, data analysis, review of the manuscript, approval of the final version of the manuscript. S Pirosca: study design, data and analysis review, review of the manuscript, approval of the final version of the manuscript. M Clarke: study design, data and analysis review, review of the manuscript, approval of the final version of the manuscript. S Treweek: study design, data and analysis review, review of the manuscript, approval of the final version of the manuscript. L Puljak: study design, data and analysis review, review of the manuscript, approval of the final version of the manuscript.

Financial & competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending or royalties.
No writing assistance was utilized in the production of this manuscript.

Data sharing statement

All raw data collected within the study are reported in Supplementary File 5 (available online at Open Science Framework: https://osf.io/6pj4m/).

Supplementary Material

File (supplementary file 1.docx)
File (supplementary file 2.docx)
File (supplementary file 3.docx)
File (supplementary file 4.docx)

References

1.
Puljak L. Evidence synthesis and methodological research on evidence in medicine – why it really is research and it really is medicine. J. Evid. Based Med. 13(4), 253–254 (2020).
2.
Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance for undertaking reviews in health care (2009). (Accessed 22 September 2021). www.york.ac.uk/inst/crd
3.
Higgins JPT, Lasserson T, Chandler J et al. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Cochrane, London, UK (2021).
4.
González U, Williams H. Implications for research: getting the most out of Cochrane reviews. Cochrane Database Syst. Rev. 1, ED000037 (2011).
5.
Clarke L, Clarke M, Clarke T. How useful are Cochrane reviews in identifying research needs? J. Health Serv. Res. Policy 12(2), 101–103 (2007).
6.
Chapman E, Reveiz L, Chambliss A, Sangalang S, Bonfill X. Cochrane systematic reviews are useful to map research gaps for decreasing maternal mortality. J. Clin. Epidemiol. 66(1), 105–112 (2013).
7.
Pirosca S, Clarke M, Treweek S. Learning from Cochrane systematic reviews: what improvements do these suggest for the design of trials? F1000Res 9, 178 (2020).
8.
Vucemilovic AS, Puljak L. Outcome domains and measures for pain in psoriasis used in registered trials: analysis of studies on ClinicalTrials.gov. J. Comp. Eff. Res. 10(11), 909–915 (2021).
9.
Saric L, Vucic K, Dragicevic K et al. Comparison of conference abstracts and full-text publications of randomized controlled trials presented at four consecutive World Congresses of Pain: reporting quality and agreement of results. Eur. J. Pain 23(1), 107–116 (2019).
10.
Biocic M, Fidahic M, Cikes K, Puljak L. Comparison of information sources used in Cochrane and non-Cochrane systematic reviews: a case study in the field of anesthesiology and pain. Res. Synth. Methods 10(4), 597–605 (2019).
11.
Dosenovic S, Nikolic Z, Ivancev B, Jelicic Kadic A, Puljak L. Awareness and acceptability of initiative on methods, measurement, and pain assessment in clinical trials core outcome set for chronic pain among surveyed neuropathic pain authors. J. Comp. Eff. Res. 8(9), 671–683 (2019).
12.
Boric K, Jelicic Kadic A, Boric M et al. Outcome domains and pain outcome measures in randomized controlled trials of interventions for postoperative pain in children and adolescents. Eur. J. Pain 23(2), 389–396 (2019).
13.
Boric K, Boric M, Dosenovic S et al. Authors' lack of awareness and use of core outcome set on postoperative pain in children is hindering comparative effectiveness research. J. Comp. Eff. Res. 7(5), 463–470 (2018).
14.
Penic A, Begic D, Balajic K, Kowalski M, Marusic A, Puljak L. Definitions of blinding in randomised controlled trials of interventions published in high-impact anaesthesiology journals: a methodological study and survey of authors. BMJ Open 10(4), e035168 (2020).
15.
Begic D, Janda-Martinac C, Vrdoljak M, Puljak L. Reporting and analyses of sex/gender and race/ethnicity in randomized controlled trials of interventions published in the highest-ranking anesthesiology journals. J. Comp. Eff. Res. 8(16), 1417–1423 (2019).
16.
Jellison S, Nolan J, Vo N, Thai M, Puljak L, Vassar M. Reporting of interventions used in anesthesiology trials: analysis using the Template for Intervention Description and Replication (TIDieR) checklist. J. Clin. Epidemiol. 118, 115–116 (2020).
17.
Gabelica M, Cavar J, Puljak L. Authors of trials from high-ranking anesthesiology journals were not willing to share raw data. J. Clin. Epidemiol. 109, 111–116 (2019).
18.
Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: a meta-epidemiological study. Eur. J. Pain 22(10), 1844–1849 (2018).
19.
Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J. Clin. Epidemiol. 53(11), 1119–1129 (2000).
20.
Turner RM, Bird SM, Higgins JP. The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS ONE 8(3), e59202 (2013).
21.
Jerosch-Herold C. An evidence-based approach to choosing outcome measures: a checklist for the critical appraisal of validity, reliability and responsiveness studies. Br. J. Occup. Ther. 68, 347–353 (2016).
22.
Coster WJ. Making the best match: selecting outcome measures for clinical trials and outcome studies. Am. J. Occup. Ther. 67(2), 162–170 (2013).
23.
COMET Initiative. Core outcome measures in effectiveness trials. www.comet-initiative.org/
24.
Dosenovic S, Jelicic Kadic A, Jeric M et al. Efficacy and safety outcome domains and outcome measures in systematic reviews of neuropathic pain conditions. Clin. J. Pain 34(7), 674–684 (2018).
25.
Boric K, Dosenovic S, Jelicic Kadic A, Boric M, Jeric M, Puljak L. Efficacy and safety outcomes in systematic reviews of interventions for postoperative pain in children: comparison against the recommended core outcome set. Pain Med. 19(11), 2316–2321 (2018).
26.
Marusic MF, Fidahic M, Cepeha CM, Farcas LG, Tseke A, Puljak L. Methodological tools and sensitivity analysis for assessing quality or risk of bias used in systematic reviews published in the high-impact anesthesiology journals. BMC Med. Res. Methodol. 20(1), 121 (2020).
27.
Biocic M, Fidahic M, Puljak L. Reproducibility of search strategies of non-Cochrane systematic reviews published in anaesthesiology journals is suboptimal: primary methodological study. Br. J. Anaesth. 122(6), e79–e81 (2019).
28.
Janackovic K, Puljak L. Reporting quality of randomized controlled trial abstracts in the seven highest-ranking anesthesiology journals. Trials 19(1), 591 (2018).
29.
Boutron I, Tubach F, Giraudeau B, Ravaud P. Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. JAMA 290(8), 1062–1070 (2003).