Authors’ lack of awareness and use of core outcome set on postoperative pain in children is hindering comparative effectiveness research

Based on a previous overview of SRs, we identified 48 SRs with data from a total of 576 unique RCTs [5,6]. These SRs were published between 2003 and 2017 and analyzed interventions for postoperative pain in children. Analyzed interventions were both pharmacological and nonpharmacological; any intervention and comparator were eligible for inclusion [5,6].

We downloaded full text articles of all those studies, extracted email addresses of corresponding authors of those SRs and RCTs and sent them an email with an invitation to participate in our survey. Since there were only 48 SRs, we also searched the internet to find email addresses of other SR authors in order to increase the number of potential participants. For RCTs we contacted only corresponding authors; if the email address was not indicated we tried to find it online, and if we could not find it, we were not able to contact those authors in this study.

Only a third of surveyed authors of SRs and RCTs about postoperative pain in children had heard about the PedIMMPACT COS for acute pediatric pain. Most of them showed lack of knowledge about six domains of that COS, and the majority did not use the COS while preparing their studies. Problems indicated as preventing them from using the COS were lack of awareness, difficulties with implementation and lack of resources. When asked to indicate which outcome domains should be part of the COS for acute pediatric pain, they chose domains that only partly overlap with outcome domains of the PedIMMPACT for acute pain.

A limitation of our study is low response rate. Even after five email messages, only 16% of SR authors and 10% of RCT authors participated in the survey. A minority of emails bounced back as undelivered, so lack of willingness of authors to participate in such survey may also be an indication about insufficient awareness about the COS.

To our best knowledge, this is the first study that analyzed the knowledge, utilization and opinion of SR and RCT authors about the PedIMMPACT COS for acute pediatric pain. It has been reported that adherence to the IMMPACT COS for pain in adults is also insufficient [7]. In 2015, Mulla et al. published analysis of 156 trials about opioids for chronic noncancer pain. They found that reporting of the IMMPACT recommended COS was very variable, ranging from 99% for ‘pain’ to 7% for ‘interpersonal functioning’. They also conducted regression analysis to study factors that could be associated with using recommended IMMPACT outcome domains. Identified factors were associated with reporting of certain individual outcome domains, not the COS [7].

The PedIMMPACT was published in 2008, and it is the only published recommended COS for acute pediatric pain [4]. Ten years later, we found out that SR and RCT authors have not been using this COS since it was published [6]. We hypothesized that there could be two explanations; either the authors do not find the PedIMMPACT COS appropriate, or they are not aware of it. Therefore, we decided to test these hypotheses, and we found that both explanations are part of the story. One of the study participants indicated that the PedIMMPACT COS was complicated and difficult to implement. The participant did not elaborate this further. It is possible that the participant is referring to the fact that the PedIMMPACT COS recommended some outcome domains which did not have validated measures [4].

The consensus required was not reported in the manuscript that described the PedIMMPACT COS. We agree that any consensus involves the possibility that not all of the authors agree, and that also not all the authors will agree with the suggested COS. For this reason we studied whether they find it appropriate.

Developing a COS is a complex process that involves multiple steps [8]. However, developing a COS is only the first step. It has been recommended that the COS should be reviewed periodically, as a very important part of validation of a COS. Such periodic evaluations can ensure that the outcome domains in a COS are still relevant, to review the possibility of adding new outcomes, to analyze whether implementation of the COS was successful and to make sure that new stakeholders are engaged when appropriate [8]. Our study can be considered as part of this periodic evaluation of the PedIMMPACT COS for acute pediatric pain. Based on our findings, implementation of that COS was not successful, and this situation calls for different actions and interventions.

New studies in this field could focus on other stakeholders as well, not only published authors, to see whether there is a need for revision of the COS. We plan to submit these results to the developers of the PedIMMPACT COS so that they can be used for audit and potential update of the COS, as recommended by the experts in the field of COS development and the COMET handbook [9].

In 2013, Kirkham et al. published analysis of outcome measures used in trials about rheumatoid arthritis (RA) over the period of 50 years, and within the study they contacted authors of trials published after publication of the RA COS in 1994. Their study had different aims than ours. They asked trialists whether they were aware of COS during designing of their study and choosing outcome domains, and whether they would consider using the RA COS in their future trials. They also included a small sample of authors. Among the 38 authors that responded, 13 did not use the full RA COS, and the majority of them indicated that they were not aware of it at the time they designed their study [10]. In this study we did not intend to ask the same questions; instead, we mainly wanted to know whether authors of SRs and trials in a particular field were aware of the relevant COS and did they find it appropriate; only one question was about whether they used the COS for their study.

Multiple surveyed authors indicated that they are not aware of the PedIMMPACT COS. This lack of knowledge could be addressed at several levels, but only before the study begins. Once the authors submit their manuscript to a journal is already too late because the study was completed and the authors cannot analyze new outcomes now from the beginning. First, education about the importance of COS can be added to research methodology courses, particularly in graduate schools. Second, members of ethics committees, or internal review boards should be aware of the COS and reject protocols for trials that do not use COS if they exist in a given research area. Third, research funders, both from industry and academia, should not fund trials or SRs that do not use relevant COS. As a commendable example, the UK's National Institute for Health Research has the following guidance in their guidance notes for applicants of full proposals: “Where established Core Outcomes exist they should be included amongst the list of outcomes unless there is good reason to do otherwise” [11].

Fourth, public registries of clinical trials such as Clinicaltrials.gov and PROSPERO for SRs should introduce relevant sections that ask whether the protocol uses relevant COS. Already in 2015, Clarke and Williamson proposed that trial registries “should encourage researchers to note their use of the core outcome set and to specify each of the outcomes from the core set, as well as any additional outcomes, that they will measure” [12].

Resources available via COMET initiative website should be used as a part of the solution [13]. The COMET is an initiative that brings together individuals that are interested in developing and applying agreed COSs. The initiative's website enables search of the COMET's database where studies about COS that are planned, ongoing or completed are collated [13]. Registries can provide links to the COMET website [13], where authors can check whether there are COS for any given research field, if they are not aware of it already. It is encouraging that the ISRCTN registry has already done this; on their website with instructions about primary outcome measures authors are instructed that they should refer to the COMET about the core outcome measures that should be used [14].

Furthermore, as one surveyed author suggested, interventions for raising awareness about the relevant COS in the academic community are necessary. This is the part of action defined as engaging new stakeholders when appropriate after development of COS [8]. All relevant stakeholders should be identified and included in discussion and action that will aim to improve awareness about and acceptance of the relevant COS. Industry is particularly important stakeholder in this respect because many trials are funded by producers of drugs and medical devices.

Although the number of authors participating in the study was low, this number is comparable with the overall number of authors that participated in the study of Kirkham et al., which had 38 authors that responded to their email message about their usage of RA COS [10].

Furthermore, due to low number of SRs identified, we invited more than one author per SR to participate in the survey. It is likely that responses of authors that participated in the same SR were not the same, and therefore they would represent their personal opinion, and not representing the SR as the unit of analysis.

Despite the low response rate, our study still brings important message to the research community and developers of the PedIMMPACT COS. Since the authors do not use the COS, the studies are not comparable. Trials will use heterogeneous outcome domains, and SRs will have trouble synthesizing evidence if there are many outcome domains measured. Using standardized COS is in the interest of patients and entire research community to ensure that evidence can be justly compared and synthesized in SRs.

In conclusion, interventions are needed that will increase authors’ awareness about the relevant COS. Additionally, further discussion about the most relevant outcome domains for pediatric pain may be warranted, considering that the majority of surveyed authors gave preference to outcome domains that do not completely overlap with PedIMMPACT COS. Insufficient use of the COS is hindering comparative effectiveness research because it is difficult to assess relative performance among competing interventions.

We are grateful to I Buljan for assistance with the SurveyMonkey interface. We would also like to thank the anonymous peer reviewers, who provided particularly constructive feedback, for contributing their time, effort and engagement to make this manuscript better.

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.

The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.