An evaluation of stakeholder engagement in comparative effectiveness research: lessons learned from SWOG S1415CD

TrACER's ESAG is comprised of volunteer community members with an interest and expertise in the project subject matter and outcomes. Many members were identified through the research team's individual networks, informed by years of working closely with payer, patient and provider communities on issues related to colony stimulating factor use for oncology, cancer care delivery research and cancer patient advocacy. Additional patient partner members were recruited from the SWOG Cancer Research Network Patient Advocate Committee [12]. ESAG members were reimbursed for travel expenses and compensated based on hours of contribution to the project.

An engagement plan for the study was developed and informed by PCORI's Engagement Principles [13]. In preparation for the study, an original group of 16 stakeholders were recruited in 2014 to serve on the ESAG representing the following areas: medical ethics, national guidelines, clinical oncology, nursing, patient advocacy, insurance providers and pharmacy. This early iteration of the ESAG was engaged in development of the research proposal for PCORI, including assisting with the formulation of research questions, reviewing the intervention, study design, setting, patient populations, recruitment procedures, feasibility and impact of the intervention and study end points [11]. The ESAG was convened regularly and provided feedback to refine the study design in two separate webinars.

The final ESAG membership at the time of study activation was expanded to 21 individuals: ten patient partners, two payers, two pharmacists, two guideline experts, four providers and one medical ethicist. ESAG member demographics were not routinely collected by the study team. All patient partners had prior experiences working with cancer care professionals and/or researchers and navigating the healthcare system in their roles as patient advocates, research partners, patients/survivors and caregivers. Membership was consistent over the 6-year research period with only two people, a patient partner and a payer, eventually leaving the project and being replaced with new members recruited through existing stakeholder networks.

The ESAG was convened annually through one in-person meeting, two patient partner webinars and two full ESAG webinars. Additionally, study staff maintained communication with ESAG members through targeted emails, a monthly dashboard with study specific updates, and an annual report detailing major activities, study updates and the impact of the ESAG on that year's project activities. Over 7 years of engagement, ESAG members contributed to many different areas of the project, including study design and implementation, site recruitment and retention strategies, patient accrual strategies, and dissemination planning (Figure 1) [11]. The research period concluded in August 2021.

The original evaluation plan for TrACER was informed by PCORI's engagement principles, which include reciprocal relationships, co-learning, partnership, trust, transparency and honesty [13]. To determine an appropriate framework for evaluating stakeholder engagement, the research team performed a literature review, prioritizing frameworks that overlapped with PCORI's engagement principles and that could be applied to a multi-stakeholder setting and would generate actionable metrics to guide future practice. We determined that the meta-criteria for effective engagement provided by Lavallee et al. was the most comprehensive framework to evaluate the stakeholder engagement processes that were implemented through TrACER. The framework considers multi-stakeholder engagement as opposed to solely considering the public sector [7], and is based on a conceptual model developed by the authors to facilitate the design, implementation and evaluation of stakeholder engagement activities in CER that includes both process and outcome criteria that can be measured to determine the effectiveness of stakeholder engagement [14]. Additionally, the framework incorporates PCORI's engagement principles (i.e. co-learning is considered a part of respect) and expands on the principles in areas the study team felt were important, such as legitimacy and accountability. The six meta-criteria for evaluation defined by Lavallee et al., and used for this evaluation, are rooted in the normative theory of public participation and deliberation [15–19].

The study team partnered with the American Institutes for Research (AIR) [20] to develop and conduct semi-structured interviews with the ESAG with the goal of learning about the ESAG experience working on the TrACER study and the successes and challenges of the engagement processes implemented by the study team. Starting with the meta-criteria outlined and defined by Lavallee et al., a semi-structured interview guide was developed by the research team with questions that mapped back to each of the six criteria. The interview guide and criteria were then reviewed by staff at AIR and refined for flow, priority setting and to ensure questions were reflective of the interview goals. The final interview guide is included as Appendix 1.

ESAG members were recruited over email and asked to participate in one hour semi-structured interviews that took place between May–July 2021. To keep ESAG interviews anonymous to the study team, we worked with AIR [20] to complete the interview process. Interviews were conducted by AIR staff, transcribed verbatim, deidentified and returned to the study team. All identifying information was removed from interviews and transcriptions were labeled as either PP (patient partner) or S (non-patient stakeholder). The study team held regular meetings with AIR staff during this time to ensure the interview guide was sufficient to obtain the desired information. The evaluation was reviewed and approved as exempt by the Fred Hutchinson Cancer Research Center Institutional Review Board.

The six meta-criteria (respect, trust, legitimacy, fairness, competence and accountability) were used for thematic analysis of the interviews. Interviews were coded in a deductive approach, with the meta-criteria and their definitions used as a codebook. Quotes throughout the interviews that related to the themes of the meta-criteria were grouped and analyzed to determine the ESAG's perceptions on the process and outcomes of the engagement process.

Themes relating to respect were identified in 10 (76.9%) of the ESAG interviews. ESAG members largely felt that their ideas were considered and respected by the study team, as well as other members of the group and that they were asked to provide input related to their areas of expertise. The patient partners participating in interviews recognized that the relationship for the study team was mutually beneficial and allowed the researchers to learn about the study from a patient or caregiver perspective and the patients to learn more about some of the scientific and regulatory processes that go into a large clinical trial. ESAG members felt that the theme of respect showed up consistently through the project.

To achieve feelings of respect, the study investigators and staff spent time at the outset of the study to learn how to communicate with (i.e. avoid jargon and acronyms), elicit input from and listen to stakeholders as well as how to address patient and other stakeholder concerns during meetings. This created an environment that allowed for partnership and co-learning between stakeholders and study team members. In addition, study team members recognized the expertise of each stakeholder group and elicited feedback from relevant stakeholders through the course of the study. For example, pharmacy and guideline experts were asked to help develop a list of appropriate chemotherapy regimens for the intervention while patient partners were asked to provide input on patient-facing materials such as consents and study brochures (see Figure 1).

Themes relating to trust were brought up in 8 (61.5%) ESAG interviews. ESAG members discussed feeling like there was adequate time in meetings to discuss opinions and come to consensus. Patient partners also mentioned the value that separate webinars were set up for patient partners early in the study prior to full ESAG webinars. ESAG members also discussed the importance of relationship building and providing time around in-person meetings to connect on a more personal level.

Trust was built throughout the project by allowing ample time for feedback in a variety of settings. Over the course of the project, webinars were held bi-annually and one in-person meeting was held each year. ESAG members were also able to reach out via email directly to the study team at any time. Patient partners were provided with a separate webinar before the full ESAG convened to orient patient partners to more technical topics, provide an opportunity for patient partners to raise questions or concerns and discuss any needed refinements to the agenda in order to better address patient-important topics. In addition, the study team provided time outside of meetings, such as group lunches, for ESAG and study team members to socialize and get to know each other as individuals.

Themes relating to legitimacy were mentioned in 7 (53.8%) of the ESAG interviews. Stakeholders felt that there was a wide variety of backgrounds, including patient advocates from different cancer disease groups, represented on the ESAG. However, there was also commentary that the group could benefit from more racial and ethnic diversity.

The study team invited stakeholders to participate in the advisory group through individual networks, informed by years of working closely with payer, patient and provider communities on issues related to colony stimulating factor use for oncology, cancer care delivery research and cancer patient advocacy. Pulling on existing networks with wide expertise in the area of research allowed for a diverse and informed stakeholder group to provide input throughout the study.

Fairness, as defined by Lavallee et al., means that stakeholders understand their role, have equal access to information and ample time to provide input [7]. Themes relating to fairness were mentioned in 7 (53.8%) of the ESAG interviews. ESAG members felt that meeting facilitators made sure all opinions were considered and that that they had ample time to give feedback and voice any concerns. When asked to define their roles on the study, both patient partners and non-patient stakeholders were able to clearly define their purpose on the project.

To ensure a fair process, the study team provided clear definitions of stakeholder roles and expectations at the outset of the study. Meetings were structured in a way that provided ample time for stakeholder feedback and discussion. Additionally, an annual survey was developed and distributed to stakeholders that included questions around whether members felt they had adequate access to information and opportunity to provide input to ensure that the steps the study team were taking to keep stakeholders informed were appropriate.

Themes relating to competence were brought up in 8 (61.5%) ESAG interviews. ESAG members felt they received meeting materials in advance and had adequate time to review materials before meeting, allowing for robust discussion during meetings. ESAG members also discussed being kept abreast of study updates and having the ability to be involved in several study activities, including the opportunity to be co-authors on publications.

The study team developed and distributed agendas at least one week prior to any meetings to ensure ESAG members had time to review materials and come to meetings prepared for discussion. Between meetings, the study team provided regular monthly updates to the ESAG via email to keep members abreast of study progress. During the accrual period of the study, the study team developed a monthly accrual dashboard that was sent out to ESAG members to keep them in the loop on how accrual was progressing, which ESAG members highlighted as a tool that kept them well informed in real-time.

A key component of accountability is a feedback loop to stakeholders, ensuring that their input is received, considered and they are aware and understand the decision to ultimately incorporate or not incorporate their specific feedback [7]. Themes relating to accountability were mentioned in 7 (53.8%) of the ESAG interviews. Stakeholders highlighted that they were consistently able to provide input and that study team members followed up on that feedback, whether their suggestions were incorporated or not.

At the outset of the study, the study team developed a spreadsheet to track ESAG input. The study coordinator tracked any ideas provided by ESAG members and ensured that all suggestions were discussed and considered by investigators. The tracking sheet provided a way to track input and outcomes of that input, which allowed for a feedback loop to ESAG members about how their suggestions were incorporated or reasons why suggestions could not be incorporated.

We acknowledge some limitations to our approach. There is not one accepted framework for the evaluation of stakeholder engagement in CER, thus researchers may choose to evaluate stakeholder engagement in a variety of ways. Another limitation is that we have not routinely collected demographic information from our ESAG members and thus cannot evaluate the representativeness of the ESAG in that regard or the role of factors such as age, gender, or race in our engagement practices and outcomes, particularly in relation to the study participants. ESAG members were selected as a convenience sample through formed connections with a focus on professional and lived cancer experience. We also acknowledge that not all ESAG members chose to participate in these interviews and the feedback is reflective of the subset of members that chose to participate. However, we were able to obtain feedback from a majority of representatives with six patient partners and seven non patient partners participating and 4 representatives from each group choosing not to participate. This evaluation allowed us to distill our successes and lessons learned into a list of best practices for stakeholder engagement in CER. Future CER studies can use these best practices to develop their engagement plan.