Querying stakeholders to inform comparative effectiveness research

In the initial stages of the CER process, stakeholders can identify CER topics that are important to them. Patients can propose topics important to them, based on their personal experiences living with the illness as well as what they have heard from other patients [22]. Providers can help identify key topics related to their clinical practice, where there is a gap in evidence, or topics related to improving the quality of healthcare services they provide. Payers may bring a health technology assessment perspective to the table and propose topics pertaining to value assessments or the financial burden posed by costly diseases and treatments. Policymakers can help identify topics of public health significance as well as CER questions that impact priority populations. Through stakeholder engagement, all stakeholders, including patients, clinicians, payers and policymakers have a greater chance of having their questions answered, and together, the team of researchers and stakeholders can determine the most relevant and useful topic to be studied [7,23].

Once potential research topics are identified, stakeholders can engage in the process of determining which topic is most relevant and urgent, and help researchers decide on the topics that should be given priority.

In Step 3 of the research process, framing the question, stakeholders can participate in the generation of specific aims of the study and specific questions that the study will answer, while researchers can determine which information is already available and assess what new information this study will add to the field. The key role that stakeholders play in this step is ensuring that the research question is framed in a manner which will provide useful information that each stakeholder can use [22].

Once the research question is appropriately framed, stakeholders can engage in the process of determining patient-centered outcomes and comparators. Researchers can inquire about outcomes that are of greatest interest to stakeholders, and together, select the most relevant comparators which will have an impact on decision-making. Patients can bring a unique perspective and suggest comparators and outcomes stemming from their personal experiences that will likely have the biggest impact on patients’ quality of lives. Providers may suggest comparators based on current treatment patterns, whereas payers may propose comparators that could potentially lead to cost savings while optimizing patient outcomes. Policymakers may provide input by proposing comparators and outcomes that can be used to evaluate current policy and direct future policy. The range of perspectives provided by stakeholders can enrich the comparator and outcome selection process.

Stakeholders can also play an important role in determining and refining the conceptual framework, by providing input on the formulation of hypotheses and supplementing the framework with additional factors based on their own personal, clinical, regulatory and payer experiences [24].

In the development of the analysis plan, stakeholders can provide valuable input on aspects such as the most appropriate outcome measures and time frame that should be used. Stakeholders can help ensure that the analysis plan adequately accounts for subgroups of interest, while ensuring that the proposed analysis plan does not compromise the generalizability of the study results to the overall population.

At the stage of data collection, stakeholders can suggest the most appropriate data source(s) for capturing variables and populations of interest and propose ways of collecting data that can produce believable results. This may involve participating in defining data elements, or reviewing questionnaires that will be used for data collection [25].

Once the data have been collected and analyzed, and the results become available, these results can be presented to the stakeholders for their review and interpretation. Stakeholders’ roles may include, but not be limited to assessing the believability of results and providing possible explanations for the results seen, which may be from a cost perspective in the case of payers, or based on clinical expertise with providers. Patients can assess if the results reflect their experiences living with the medical condition and describe the implications of the results for patient quality of life [24]. Researchers may help stakeholders as necessary in their review of the study findings; however, it is important for researchers to allow stakeholders to develop their own interpretations freely.

Stakeholders can assess if the conclusions drawn make sense, and work together with researchers to translate the results into a comprehensible message for dissemination that is appropriate for the target population. For instance, the use of medical jargon would be appropriate if healthcare providers were the target audience, but not appropriate for the patient population.

Stakeholders can suggest venues and means for disseminating results and recommendations for various audiences. Stakeholders can help tailor communication strategies to increase awareness and understanding of study findings by target audiences [22,24]. For example, key messages may be conveyed via local organizations, news outlets or professional, or lay publications depending on the target audience. Providers can help formulate messages for healthcare providers, whereas patients can help develop messages for the lay public and suggest potential avenues for dissemination. Also, patients, providers, payers, and policymakers can all provide input on how to encourage the uptake of the evidence generated, which is our ultimate goal.

Stakeholder engagement in CER enhances the applicability of research in providing practical information that improves patients’ decision-making and maneuver through healthcare delivery systems.

Each stakeholder has unique views to contribute to the research process. Patients are the end receivers of healthcare. The 10-step approach identifies and incorporates patients’ views that are based on their personal experiences of the illnesses and healthcare interventions. Furthermore, engaging diverse patient groups would help produce CER results that are applicable to specific patient subgroups, and generate findings that go beyond ‘average treatment effects [26]’. Providers are also encouraged at each step of engagement to bring their unique perspectives based on their professional expertise and experience as practitioners. The 10-step approach further affords payers an opportunity to bring their financial concerns and interests to the table so that they can be equipped with the ability to make sound financial decisions that are mutually beneficial to all stakeholders. The richness of the perspectives and ideas that stakeholders bring to the table in each of the ten steps proves the importance of partnering with stakeholders in CER so that researchers may incorporate diverse views and ultimately provide value to all stakeholders involved.

Previous research has shown that providers, policymakers, and payers are not as frequently engaged in research relative to patients, and that stakeholder engagement occurs much less frequently at later stages of the research process [6]. It may be challenging for researchers to effectively engage and elicit all stakeholder groups’ views throughout the entire research process. However, there are meaningful roles for all stakeholder groups to play throughout a research study and value they can add. With a clearer understanding of the value and perspectives each stakeholder is able to bring, researchers may be able to more easily create questions they can ask stakeholders that ultimately lead to more meaningful engagement.

The value of incorporating stakeholder views in CER is well described, but there is limited practical guidance on how specifically to elicit stakeholder views. We have provided practical suggestions on how to elicit stakeholder views throughout the research process by outlining potential stakeholder roles and providing examples of questions that researchers may ask in order to enable impactful collaboration between stakeholders and researchers. It is our hope that the guidance we have provided will assist researchers initiate meaningful engagement of stakeholders in CER.

With the growing interest in research that engages stakeholders as research partners [27], it is our hope that the next 10 years will see a shift toward stakeholders being readily engaged throughout the entire CER process. The need for evaluation of stakeholder engagement in research has been identified in the literature [28–32]. Moving forward, there will likely be a growth in research efforts aimed at evaluating the outcomes associated with stakeholder engagement in CER, which will further inform and support stakeholder engagement by identifying the most effective modes of engagement.

Research reported in this article was partially funded through a Patient-Centered Outcomes Research Institute Award ME-1409–20792. The statements presented in this article are solely the responsibility of the authors and do not necessarily represent the views of Patient-Centered Outcomes Research Institute, its Board of Governors or Methodology Committee. This project was supported in part by grant number R24HS022135 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.