Assessing risk of bias judgments for blinding of outcome assessors in Cochrane reviews

This was a primary methodological study, in which we analyzed methodology of Cochrane reviews published in the Cochrane Database of Systematic Reviews.

Cochrane Database of Systematic Reviews was searched for reviews of RCTs (or both RCTs and nonrandomized studies; but we analyzed RoB assessments only for RCTs) of interventions published from July 2015 to June 2016. Advanced search option was used to limit results to content type and publication date. We excluded diagnostic Cochrane reviews, overviews of systematic reviews, empty or withdrawn reviews and other Cochrane reviews containing no RCTs about interventions.

Titles and abstracts of Cochrane reviews were assessed by first author (O Barcot) who established inclusion eligibility. These assessments were verified by the second author (S Dosenovic).

The first author (O Barcot) wrote series of macroinstructions in Visual Basic for Applications (VBA, Microsoft, WA, USA) to automate data scraping from The Cochrane Library webpage to Microsoft Excel 2010 (Microsoft) workbook. Automatic extraction of RoB tables for every eligible Cochrane review was done offline with new set of coded instructions, as described earlier [6]. Errors during data extraction were logged and checked manually.

The first author (O Barcot) developed a special user interface (MS Excel VBA User Form) to facilitate parsing. In this case, natural language text (comments, citations) was transformed to ordinal or nominal variables, as described earlier [6]. The second author (S Dosenovic) analyzed 500 random trials in order to pilot test and adjust the tool. This analysis was verified by the first author (O Barcot). Prior to usage of the tool, other authors were instructed to follow specific rules established in pilot testing.

In the developed user interface, we made new assessment of detection bias for RCTs in which Cochrane authors provided full detection bias assessment, in other words, both judgment (RoB is low, high or unclear) and an accompanying comment. We followed instructions for rating detection bias from the Cochrane Handbook (Section 8.12.2) [1] and defined that two main questions need to be correctly answered to adequately assess the detection bias. The question #1 was: who is assessing the outcome? – because different outcome assessors can be used, and it has to be assessed whether they were blinded or not. The question #2 was: is there a possibility of RoB in the outcome assessment? – because not all outcomes are equally prone to detection bias. For example, for an outcome such as death, lack of blinding of an outcome assessor may not influence the outcome. To appraise whether Cochrane authors mentioned type of outcome that was assessed within analyzed detection bias domain, we categorized every outcome into one of six predefined categories: objectively measured/subject-independent outcomes, clinician-rated/reported/related outcomes, patient/self-reported/rated outcomes, subjective outcomes, all outcomes, not specified, based on consideration how subjective or objective the outcome was [10]. The latter two categories do not relate to specific type of outcome in the same way as the first four do. ‘All outcomes’ mean that the authors judged RoB for all outcomes together, in other words, they did not specify certain subgroups of outcomes in the name of the RoB domain. We categorized that outcomes were ‘not specified’ when authors did not mention any particular type outcome with the name of a domain in the RoB table. Not specifying outcome that a domain is assessing is the default setting of the RevMan [11], software that is used for writing/analyses of Cochrane reviews. Lastly, we compared our new assessments with the assessments made by the Cochrane authors.

Judgments for detection bias assigned by Cochrane authors were analyzed by number, type and adequacy. Definition standard in our assessment was the Cochrane Handbook. We considered that judgment from Cochrane authors was inadequate if it did not completely adhere to the Cochrane Handbook guidance.

We analyzed prevalence of splitting of the detection bias domain (whether Cochrane authors split a detection bias domain into two or more sub-domains), types of outcomes for which splitting was used, and adequacy of detection bias judgments in different outcome categories.

We presented all descriptive data as frequencies and percentages. For all statistical tests we used type I error α = 0.05, and type II error β = 0.2. Statistical analyses were performed using MedCalc for Windows, version 12.5.0.0 (MedCalc Software, Ostend, Belgium). All datasets were tested for normality by the Kolmogorov–Smirnof test. The Mann-Whitney test was utilized for comparison of independent samples of nonparametric data, and the Wilcoxon test was used for paired samples. Difference in proportions was tested with Chi-squared test. Hypotheses, outcome measures, statistical tests used and its results are logged in Supplementary File 1.

Out of 955 retrieved Cochrane reviews, 227 were not eligible. In the remaining 728 reviews there were 10,523 trials. Additional 2897 trials were excluded: all of 2480 trials from 141 reviews in which performance and detection bias domains were merged; all of 379 trials from 11 reviews and additional 28 trials without detection bias domain stated in the RoB table; one review with four trials without RoB table and additional four trials without RoB table as well; two trials were duplicate entries due to computational error (Figure 1).

Finally, we included in our study RoB tables from 575 Cochrane reviews (listed in Supplementary File 2), which included a total of 7626 trials (Figure 1). In those 7626 trials there were 8656 domains (judgments) for detection bias, because in some Cochrane reviews this domain was split (had multiple assessments for various types of outcomes). In 720 out of 7626 (9.4%) trials, RoB domain for detection bias was split into multiple subdomains based on different outcomes (ranging from 2 to 8). In those 720 trials, there were 1750 judgments for specific outcomes.

In the whole sample of 8656 detection bias judgments, for the majority (7110/8656; 82%) Cochrane authors specified in the RoB table that the judgment referred either to ‘all outcomes’ (6072/8656; 70%) or did not specify to which outcomes the domain was referring to (1038/8656; 12%). For the remaining 18% of judgments, Cochrane authors specified to which outcomes the domain (or subdomain) was referring to (Table 1).

Distribution of six categories of outcomes in the whole sample (8656 judgments) and a subsample of trials with split domain (N = 720 trials, N = 1750 judgments) was significantly different (p < 0.05; Supplementary File 1), as in this subsample only 4% judgments were specified for ‘all outcomes’ and 14% were not specified (Table 2 and Supplementary File 3). In 231 trials with split domain (accounting for 540 judgments), risk of detection bias judgment was identical within all of their split outcomes (in a single trial, all of the RoB judgments were of the same level: all high, all low or all unclear).

In the main analysis, among 8656 judgments for detection bias, there were 1679 (19%) that Cochrane authors judged with high risk, 3374 (39%) with low risk and 3603 (42%) with unclear RoB (Table 3).

Out of 8656 detection bias judgments, 6747 (78%) were judged adequately by Cochrane authors. Our assessment of adequacy for those judgments indicated that the highest prevalence of inadequate judgments was found for trials judged with low risk of detection bias (1287 of 3374; 38%), followed by those judged with high risk (239 of 1679; 14%) and those judged with unclear risk (383 of 3603; 11%; Table 3). For transparency purposes and ease of visualization we provided a table (Supplementary File 4) that presents a few detailed examples for each type of disagreement in decisions (between Cochrane authors and our team) for different types of outcome categories.

Adequacy of judgments for ‘all outcomes’ and judgments where no outcomes were specified was 77 and 78%, respectively (Table 2 – right column). Among the remaining 18% of judgments, higher adequacy versus 78% adequacy in whole sample (all p < 0.05; Supplementary File 1) was observed among outcomes described as clinician-rated (84%), patient-rated (87%) and subjective outcomes (93%). These three groups of outcomes are considered subjective and we noticed significantly higher accuracy of judgments in merged group of outcomes compared with whole sample (86.9 vs 77.9%,;p < 0.05; Supplementary Files 1 & 5) as well as in comparison of this merged subgroup to subsample with split domain (85.7 vs 80.6%; p < 0.05). Finally, judgments of objective outcomes had lower adequacy (72%) compared with the whole sample (78%; p < 0.05; Supplementary File 1).

Subsample of trials with split domain where all judgments in a trial were the same (i.e., all judgments were low RoB; Table 2) showed higher adequacy of judgments (92 vs 78% in whole sample p < 0.05). On the other hand, judgments in a subsample of trials that split domain in various outcomes and judged the risk of detection bias differently were as adequate as the whole sample (76 vs 78%; p = 0.0553).

Length of comment (LOC) supporting the judgment has an impact on assigned judgment, achievement of blinding and calculated judgment. Shortest comments with a median of 45 characters were found in RoB tables of trials with unclear risk of detection bias assigned (Table 4). The longer the comment (LOC over 91) – the higher the chance for successful blinding was described, along with low RoB judgment for detection bias assigned by Cochrane authors or calculated according to the Cochrane Handbook.

We noticed significantly higher accuracy of judgments for outcomes that were described as subjective, compared with other categories of outcomes, both in trials with split domains and in the entire sample of trials. We also found an inversion in proportions of adequate judgments depending on whether a domain for detection bias was split into subdomains or not. In trials where the domain was not split, proportions of adequate judgments for specified outcomes versus those that indicated they judged all outcomes or did not specify types of outcomes were 18 versus 82%, respectively. Among trials that had split domain for various outcomes, the proportion of adequacy was inverse – 82% when subcategories of outcomes were specified versus 18% for all outcomes or when outcomes were not specified. Furthermore, in a subgroup of trials with split domains, there was higher prevalence of inadequate judgments for ‘objective outcomes’ than for other groups of outcomes. We believe this is due to erroneous categorization of outcomes due to lack of specific instructions in the Cochrane Handbook about which outcomes should be considered objective.

Splitting detection bias judgment into multiple subdomains seems to have some specifics. First, this subgroup seemed to have higher accuracy of judgments, compared with the whole sample, but this is due to distribution of outcomes toward subjective ones. Subjective outcomes had higher accuracy of judgments but larger number of cases diminishes the effect of lower adequacy of misjudged objective outcomes. Second, the authors frequently assigned the same RoB judgment for all outcome types within a single trial, regardless of the outcome category. Third, although this was rare, in 4% of reviews with split outcomes, the authors also used subdomain ‘all outcomes’ together with one or more specific outcomes as well. If the authors decided to split domain into multiple subdomains based on specific outcomes, it is unclear why they would then, additionally, use subdomain for ‘all outcomes.’

While splitting of outcomes may be associated with errors due to erroneous categorization of certain outcomes, it provides more information to the readers than overall judgment for all outcomes, and therefore we can conclude that assessment of overall risk of detection bias for all outcomes should not be used. At least, the authors should define the category and clarify that all outcomes belong to the same category, and that this was the reason why they were all judged together.

Cochrane has recently published new version of its RoB tool [14]. At the time of the submission of this manuscript (December 2019), the new tool has not been implemented yet in Cochrane reviews. Even the authors of developing Cochrane protocols that were not published yet were not obliged to use the RoB 2.0 version. The RoB 2.0 has changed content, structure and type of applicable judgments. The current (old) RoB tool has seven domains, and each domain can be judged with three types of judgments – risk is considered to be high, low or unclear. The RoB 2.0 tool has five domains; each domain has from three to seven signaling questions; signaling questions are responded with five potential answers (yes, probably yes, no, probably no, no information). Based on the answers on signaling questions each domain is scored as low RoB, high RoB or having some concerns. Overall RoB assessment is also provided. Domain #4 is ‘Bias in measurement of the outcome’; this domain has three signaling questions referring to blinding of outcome assessors, asking whether outcome assessors were blinded to the intervention, whether assessment of an outcome could be influenced by lack of blinding of outcome assessor and whether it is likely that the assessment was actually influenced by the lack of blinding of outcome assessor. These three signaling questions represent major change from the current RoB tool. Based on the results of this study, we expect that these three signaling questions should facilitate detection bias judgments, and should help in making more transparent, adequately explained decisions.

Results of our study can help improve adequacy of RoB assessment in systematic reviews. Even though Cochrane has announced RoB 2.0 tool, the ‘old’ tool is still in use. Additionally, the ‘old’ tool is also used by the majority of non-Cochrane reviews [15]. With the availability of our data, presented in this study, once the RoB 2.0 tool will be used in all Cochrane reviews, we will be able to compare in future whether adequacy of judgments of Cochrane authors for this particular domain has improved with the new tool. Therefore, our results can be useful to authors who will continue to use the current RoB tool, as well as from the research methodology perspective for measuring adequacy of methods in the new RoB 2.0 tool, and comparing it with the results for the old tool. New research methods continue to be developed, but it is important to make sure that they are better than previous methods.

Our study is important because it is yet another confirmation that more attention should be paid to Cochrane methods used by Cochrane authors. Erroneous judgments of RoB will lead to erroneous conclusions, which ultimately can send misleading messages to consumers, healthcare workers and decision makers. Editors and peer-reviewers could help ensure adequate use of systematic review methods.

In this study, we have used software for data extraction, but it is possible that we have made errors in data interpretation. We tried to avoid this possibility by double-checking all decisions by two independent authors. We also need to state that in our study we did not try to detect possible bias in original trials, for example using the Berger-Exner test [16–18], because we did not have proper input for it to be executed. Even in our previous works on bias arising from inadequate random sequence generation [4] and allocation concealment [5] this was the case. Our dataset of Cochrane authors’ comments and judgments, on which we performed analysis, did not include analysis of full texts of original trials. Instead, we only relied on the content of RoB tables from Cochrane reviews, and these tables only include information that the Cochrane authors have reported in them. This could have helped verifying if the judgment of Cochrane authors was adequate, when authors have failed to provide adequate explanation in their supporting comment.