Stakeholder engagement in research: a scoping review of current evaluation methods

We conducted a scoping review using an established methodological framework developed by Arksey and O’Malley to ensure rigor and reproducibility [19]. The use of a scoping review allowed us to address a broader topic and review evidence generated by a range of study designs and quality [19]. Additionally, it supported our goal of describing the range of evidence available on the evaluation of stakeholder engagement and identifying gaps in this knowledge base for future investigation.

The search yielded 2880 articles, which were screened and selected for analysis as summarized in Figure 1 [22]. A total of 93 articles were reviewed at the full-text phase; 76 of those articles were excluded as documented in Figure 1. The final sample for analysis consisted of 17 articles (Table 2), of which 16 had been conducted in the USA and 1 in Hong Kong.

In accordance with previous evidence, we encountered inconsistent reports of who stakeholders are and what constitutes engagement in research [7,8,14]. For example, many articles in our initial sample (n = 2880) that reported collaborating with stakeholders included community members as study participants only, a role that is inconsistent with ongoing partnership and shared decision-making. The 17 articles in our final sample described a collaboration with stakeholders and evaluated their engagement within an existing study. Further, although stakeholder groups to consider for engagement have been previously described [10,23], there is no common framework for reporting the type of stakeholders engaged and what perspectives contributed to the evaluation of engagement. In our final sample, 53% of studies provided detailed descriptions of the stakeholder groups who participated in the evaluation of engagement. The remaining 47% of studies did not describe the stakeholder groups who participated in the evaluation of engagement or named partners as a collective (e.g., advisory board, stakeholder panel, research network, etc.).

In response to our guiding question, we identified four approaches to evaluating stakeholder engagement: ‘qualitative approaches, surveys, engagement logs and other data collection approaches’. These four approaches were not mutually exclusive. Across our final sample of 17 articles, 7 studies described using one approach to evaluate stakeholder engagement [13,24–29], 6 discussed using two [14,30–34] and 4 studies used three approaches for measuring engagement [35–38]. We discuss each of the four approaches in detail below.

This scoping review described multiple approaches for evaluating stakeholder engagement in research. All studies collected data from a range of stakeholders, such as researchers, advisory or steering committee members and community members. Evaluation approaches most frequently sought to capture the perspective of stakeholders on their involvement as research collaborators. We did not evaluate the quality of each evaluation approach or of the stakeholder engagement partnership itself.

In accordance with the concept of collaborative stakeholder engagement [21], our final sample of articles described a partnership with stakeholders and evaluated engagement within an existing study. One of the articles in our final sample also sought to develop a community engagement measure from the data collected given the lack of evidence-based approaches or existing guidance for valid and reliable measurements of stakeholder engagement [13]. No systematic reviews met the inclusion criteria.

A key finding of our descriptive scoping review study was that approaches for capturing stakeholder engagement varied widely and their descriptions were often inconsistent or incomplete. Evaluation details and depth of description differed between studies or was missing for topics including: evaluation facilitators, how it was conducted (e.g., in-person or online), when and how frequently it was administered, and the content addressed. What specific stakeholder groups participated in the evaluation process (e.g., caregivers, healthcare providers, policy-makers, etc.) were not always specified. Additionally, the evaluation approaches (and combinations of approaches) used were different for each study. In our final sample (n = 17), nine studies used one evaluation approach, five used two approaches and three used a combination of three approaches. Surveys were by far the most common approach for evaluating stakeholder engagement and were used in 11 studies. Most articles did not describe how their evaluations were developed, and what role if any, community partners had in their design. Exceptions included Goodman and colleagues who indicated that their survey was developed following a review of literature on the principles of community-engaged research, and Nowell et al. who drew from their study funders’ stakeholder engagement rubric [13,33].

The purpose of capturing stakeholder engagement and its fit with the content addressed by each evaluation approach was not clearly stated for each study. Some evaluation approaches directly engaged stakeholders as respondents (e.g., surveys, interviews), others interacted with project data to characterize engagement (e.g., meeting minutes, tracking log-ins to an online service), and some also described plans for evaluation in a future study (e.g., creating activity logs). We identified the most common topics addressed through evaluation approaches which included: stakeholders’ personal perspective on the collaboration; stakeholders’ perceived impact on the research process and study outcomes; the barriers and facilitators stakeholders encountered to engagement; stakeholder satisfaction with the collaboration and feelings of being valued by the study team and recommendations for improving future collaborations.

We found limited information on evaluation design, accessibility, and the role that stakeholders themselves have had in evaluation conceptualization, dissemination and analysis of results. Given the need for collaborative practices in stakeholder-engaged research and a focus on equity and trustworthiness, partnering with stakeholders to design and implement inclusive evaluation approaches is an important area of future research.

Recent literature supports a continued interest in stakeholder engagement practices. For example, recent articles describe research team member-led reflections of engagement and a focus on enrollment or retention as indicators of the impact of engagement [39,40]. Reports of funder-led evaluations of stakeholder engagement of funded projects have also been published [9,41]. However, there is still a need to address evaluation approaches from a variety of perspectives. Existing evaluation approaches may prioritize researcher or organizational experiences instead of diverse stakeholder partners and may not account for power hierarchies and other factors (e.g., familiarity with research, socioeconomic status, access to resources, educational level, language proficiency).

We identified that the results of stakeholder engagement evaluations were limited or absent, as was a description of how the research team utilized or responded to evaluation findings. This gap in reporting further compromises transparency, equitable partnerships, and the potential for future research to produce findings that improve the health and healthcare of vulnerable populations. There is an urgent need for common practices for evaluating stakeholder engagement and systematic tools for doing so that capture the plurality of stakeholder backgrounds and perspectives.

Challenges to measuring stakeholder engagement include the lack of: a unifying conceptual model of stakeholder-engaged research; consensus on how to define meaningful engagement; a common understanding of who stakeholders are and what groups they may represent and unifying definition of what constitutes an engaged stakeholder [8,14]. Moreover, study results and evidence for impact of stakeholder engagement may not be available at the time of manuscript preparation, or authors may be limited by space limitations in publications [14]. However, previous literature has begun to address the need for assessment, recommendations, and models to capture stakeholder engagement [7,9,13,14,41]. Future work refining and appraising evaluation approaches, as well as investigations into other ways to capture stakeholder engagement is necessary to advance research methodologies on the topic.

Clear guidelines for identifying stakeholders and reporting data in future publications can add to our knowledge of current strategies for engagement and its evaluation. As a result, investigators of stakeholder-engaged studies may be better able to incorporate regular and valid evaluations of engagement in their initial study designs, demonstrate the impact that stakeholders have on research, and systematically document their results in research reports. Through such high quality evaluations of engagement, investigators may be better equipped to develop inclusive partnerships with stakeholders from all backgrounds, respond to power hierarchies or other factors impacting collaborative engagement (e.g., familiarity with research, socioeconomic status), and increase representation from vulnerable populations experiencing healthcare disparities.

There are multiple limitations to our study. Since we conducted a scoping review, we did not evaluate the quality or the methodology of the included studies. As the volume of evidence examining stakeholder engagement methods evolves, future systematic reviews on the topic will need to critically appraise the quality of the methods. We included studies published between January 2013 and June 2018, which would have excluded more recent publications. In addition, relevant literature may have been inaccurately excluded during the title and abstract screening phase. Thus, there may be additional citations that could have resulted from utilizing other databases or alternative search terms. Our search also utilized terms common to stakeholder-engaged research. Given the lack of consensus on terminology, it is possible that relevant studies where stakeholder engagement occurred but was termed something else could have been excluded. Finally, given the recent and emergent evidence on this topic, a systematic review is not appropriate at this time.

As a growing number of studies employ stakeholder-engaged research, investigators must be prepared to collaborate with nonresearcher partners, describe how this occurred throughout the study and discuss a systematic evaluation approach that addresses stakeholder contributions and experiences [9,11]. Although the importance of engaging stakeholders and end-users of research in the scientific process is widely supported [42,43], evaluation approaches and reports of methods, results or response to evaluation findings varied widely. As a result, the extent of stakeholder engagement, its role in the study, and the perspectives of stakeholder partners were often unclear. As the number of projects engaging stakeholders and their publication grows, our findings highlight the urgent need to use validated tools at regular intervals that use a common language, framework or criteria for engagement to reflect the priorities of end-users but also meet funders’ reporting requirements. Findings from these evaluation approaches can help more clearly articulate the value and role of stakeholder engagement in the research investigation and describe effective strategies for collaborating with the public. As a result, common strategies that embed meaningful partnerships with stakeholders in the scientific process and research findings that enhance health outcomes for vulnerable populations can be facilitated.

Given the increased interest in stakeholder engagement in research and its prioritization by funders, it is important to identify the role that stakeholders play, how they are engaged in a study and capture their perspective research partners using valid and reliable methods. To do so, a rigorous plan for ongoing evaluation of stakeholder engagement at regular intervals is a key component of stakeholder-engaged research that can address this gap in knowledge. To support future investigators leading stakeholder-engaged research, a consensus on stakeholder engagement practices, its goals and guidance for evaluating engagement is needed and represents an important area of future investigation.

Research reported in this publication was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (IHS-1608-35732). The statements in this publication are solely the responsibility if the authors and no not necessarily represent the views of the PCORI, its Board of Governors or Methodology Committee. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

J Martinez, C Wong, and NE Leland conceived the presented idea and designed the study. J Martinez and C Wong preformed the scoping review with the guidance of NE Leland who was also available to resolve any questions about the inclusion of articles in the sample. J Martinez, C Wong, and NE Leland wrote the manuscript with support and review from all authors. DC Bieber and BL Perry provided a review of the manuscript to ensure that research stakeholders’ perspectives were represented. CV Piersol provided extensive writing support and mentorship for the article resubmission.

We thank K Saric for assistance with formulating and implementing our search strategy, as well as L Terhorst for support with technical editing of the manuscript.

Research reported in this publication was funded through a Patient-Centered Outcomes Research institute (PCORI) Award (IHS-1608-35732). The statements in this publication are solely the responsibility if the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.