Evaluating patient and stakeholder engagement in research: moving from theory to practice
Abstract
Despite the growing demand for research that engages stakeholders, there is limited evidence in the literature to demonstrate its value – or return on investment. This gap indicates a general lack of evaluation of engagement activities. To adequately inform engagement activities, we need to further investigate the dividends of engaged research, and how to evaluate these effects. This paper synthesizes the literature on hypothesized impacts of engagement, shares what has been evaluated and identifies steps needed to reduce the gap between engagement's promises and the underlying evidence supporting its practice. This assessment provides explicit guidance for better alignment of engagement's promised benefits with evaluation efforts and identifies specific areas for development of evaluative measures and better reporting processes.
Background
Engagement of patients and other healthcare stakeholders (herein, stakeholder partners) is increasingly recognized as essential to patient-centered comparative effectiveness research (CER), which is intended to answer questions of importance to patients and their caregivers [1]. Often described as researchers doing research with patients, rather than for, at or to them [2–4], patient-engaged research implies a level of involvement that extends beyond the role of research subject [5]. This shift has been fueled in part by the 2010 creation of the Patient-Centered Outcomes Research Institute (PCORI) by Congress with the passage of the Patient Protection and Affordable Care Act. PCORI is committed to producing and promoting high-integrity CER that is ‘guided by patients, caregivers, and the broader healthcare community’. [6].
The PCORI Methodology Report states that patient engagement can include: defining topics and formulating study questions, identifying a study population and choosing interventions, comparators and outcomes, developing optimal strategies for recruitment and retention of study participants, conducting a study and analyzing results and disseminating research findings into clinical practice [7]. Generally, the value of engaging stakeholder partners in research is that it can help reorient and improve the research enterprise, reduce clinical uncertainty and speed adoption of meaningful findings that hold great promise with the ultimate goal of improving patients’ care experience, decision-making and health outcomes [6].
Despite the growing interest in and demand for research that engages stakeholder partners – or, ‘research done differently’, [8] there is limited evidence in the published literature to demonstrate the value – or return on investment – of this engagement. This evidence gap is reflective of a general lack of evaluation of research engagement activities. In fact, the majority of published research with a stakeholder partner engagement dimension does not include an evaluation component [9–11]. To adequately inform and support engagement activities, we need to know more about the dividends of engaged research, and how to evaluate these effects.
The purpose of this paper is to synthesize what the literature proposes as the hypothesized impacts of engagement (i.e., the benefits), share what has actually been evaluated and assessed and propose the steps needed to reduce the gap between research engagement's promises and the underlying evidence base supporting its practice. By mapping the hypothesized impacts of research engagement to what has actually been assessed, we build upon Workman's call to action for greater evaluation of engagement processes [12]. Our assessment provides guidance to researchers for future evaluation metrics, and calls for the field to plan in advance their purposes of engagement and move beyond surface level analyses of stakeholder partner/researcher experiences and satisfaction.
Methods
This manuscript is directly informed by a scan of the peer-reviewed literature using PubMed and PsychInfo to identify articles published from January 2005 to May 2013 focusing on patient, public and stakeholder engagement in the healthcare field. We selected search terms through an iterative review and revision process with PCORI staff. We searched article titles for a combination of terms covering ‘who’ (e.g., patient, consumer, stakeholder, etc.), ‘action’ (engagement, participation, involvement, etc.) and ‘purpose’ (research, prioritization, resource allocation, etc.) (Supplementary Material; see online at: Supplementary Material).
Five research team members selected articles for abstraction by reviewing titles and abstracts and applying inclusion and exclusion criteria (after calibration amongst team members). Inclusion criteria were as follows: research studies that involved or described the engagement of patients or others stakeholders in the planning and/or conduct of research (i.e., beyond the role of research subjects); or articles that presented an engagement framework with an evaluation component; or articles that synthesized relevant aspects of the literature (e.g., review articles) or seminal articles prior to 2005 that were frequently referenced. Articles were excluded if they did not provide enough level of detail to ascertain whether patients or other healthcare stakeholders were engaged beyond the role of research subjects, or if they did not provide any information on possible evaluation measures or outcomes associated with engagement.
Expert advisors and staff at PCORI provided guidance throughout the design and implementation of the work. In addition, members of the Consumer Patient Researcher Roundtable suggested relevant articles to include in the scan. The Consumer Patient Researcher Roundtable is a convening activity of AcademyHealth that recognizes the shared interests of both patients and researchers (i.e., the desire for better health and healthcare), and facilitates dialogue on topics of mutual interest and collaborative work.
To supplement our scan, we retrospectively incorporated a selection of articles from a bibliography of papers published between 1995 and 2011 focused on patient and public involvement in research [13]. We selected articles from this scan meeting the inclusion criteria based upon our review of the titles, as well as a determination of whether they would add new information to our scan – specifically regarding evaluation of engagement. A selection of gray literature products was also included, though not identified through an exhaustive search. In total, the literature scan considered over 1000 articles from various sources. After the application of our criteria, we selected 108 for abstraction, most of which were reviews and case studies published in peer-reviewed journals.
We assigned the articles amongst five researchers who abstracted information from each article into an MSExcel format. Abstracted fields included information such as author, date, primary objective, methods, group being engaged, main intent behind engaging the specified group, methods of evaluation for engagement in research and suggested or measured outcomes of engagement. As we proceeded with article abstraction, we eliminated those that (upon scanning the full manuscript) did not present new information compared with what we already captured. After abstraction was complete, two researchers divided the data and conducted a content analysis of the abstracted information using NVivo software looking for common themes and then summarized the data.
Hypothesized impacts of engagement
The literature proposes several hypothesized impacts of stakeholder partner engagement. Identifying and understanding the intended purposes of engagement is key to structuring the activity of stakeholder partner engagement, and also provides a goal post against which engagement can be assessed. The following purported reasons are identified by the literature, some of which may only apply to patients (as indicated below).
Better quality research
One of the major hypothesized reasons for engaging stakeholder partners in research is that they provide a unique perspective, sometimes with direct knowledge and experience, which has the potential to promote more useful evidence that is more relevant and responsive to patients’ and stakeholders’ needs [7,14–19].
Expanded applicability of research, particularly in ‘hard-to-reach’ populations
Some literature, particularly in the area of community-based participatory research (CBPR), argues that patient engagement in research is key to reaching hard-to-reach or otherwise marginalized populations [20,21]. By more actively engaging these community members as partners in the research process, these authors argue that it improves a population's participation, and ultimate ‘ownership’ of the research process and outcomes. These authors also suggest that engagement may address disparities in inclusiveness in medical research study samples, and consequently facilitate research findings that are more applicable and acceptable to a wider range of patients.
Empowering patients
Many authors argue that research engagement is essential to empowering patients in their healthcare, which is a fundamental premise of the CBPR approach [20–22]. While the precise relationship of CBPR to PCOR has yet to be determined, incorporating patients into the research process is also a central component of PCOR. Empowering patients is also perceived as having valuable downstream effects. By taking greater ownership of research, authors hypothesize that engaged patients may play a more active role in their healthcare [21].
Increased translation, dissemination & uptake of results
Authors also argue that stakeholder partner engagement in early stages of the research process aids in the translation and interpretation of the findings, which ultimately increases the ‘actionability’ of research results [3,19]. Authors also argue that by incorporating patients and other stakeholders as partners throughout the research process, they can effectively serve as early ambassadors of research efforts and subsequent findings, which may help to expand to audiences beyond peer-reviewed journals, and may facilitate increased uptake of results into the community and healthcare setting thereby accelerating its adoption into practice [15,23–25].
Democracy & accountability
From a political perspective, engaging patients serves the democratic ideals of accountability and transparency [26]. It can increase the likelihood that individual researchers, research organizations and funders of research (public or private) are accountable and responsive to users of medical knowledge and the broader community, providing some assurance that resources are being allocated in ways that serve the public interest [27]. In this way, engaging patients in research can contribute to the legitimacy and credibility of research [19,26].
Moral obligation
The moral argument underlying patient engagement in research is that patients and other stakeholders have a fundamental right to be involved in the process. Related to the notion of social justice, the idea is that, as ultimate end-users of the research results, they should have the opportunity to be involved in shaping its conduct [17,28,29]. This premise is embodied in the well-known quote ‘nothing about us, without us’. [30]. Given these arguments, researchers would have a moral obligation to involve patients in research beyond the role of subjects.
Measurement & evaluation of engagement
The research literature also proposes and applies a range of measures to evaluate stakeholder partner engagement, which we categorize according to three key dimensions: context evaluation (the environment and conditions that underpin and support engagement); process evaluation (how engagement is implemented) and impact evaluation (the intended effects of engagement) [31,32]. Context refers to the conditions required for engagement to have an impact while process refers more to how the involvement is done [31]. Examples of each are listed below.
Context & process
With respect to evaluating engagement context and process, the literature theorized several suggested measures; however, many authors recognize a lack of clarity of these concepts and how to effectively measure them [31–33].
In our scan of the literature, we did not identify any research reporting findings of these concepts or dimensions (Table 1).
Evaluation of the impacts of engagement
While many papers theorize or propose intended impacts, very few empirical studies have applied or tested hypothesized impacts. Tables 2 & 3 map the intended purposes of engagement (hypothesized impacts) against those indicators that have actually been developed and measured (suggested and assessed measures). We also have distinguished between assessments that are primarily qualitative versus quantitative. It is worth noting that the categorization presented here assumes that ‘hypothesized impacts’ of engagement are the ‘end goals’ of patient and stakeholder engagement in research; it is possible, however, that many of the indicators listed preceding the longer-term measures could reasonably serve as intermediate outcomes of these intents.
As depicted in Tables 2 & 3, the limited evaluation measures that we found in the literature primarily measured the near-term effects of engagement (e.g., better quality research, patient empowerment). We did not identify any suggested or assessed measures that evaluate the longer-term, downstream effects of engagement, such as improved decision-making or health outcomes.
Discussion
The most striking observation taken from this exercise and reflected in both tables is how few studies actually assess or formally evaluate any measures of engagement. Of the studies that formally evaluated impact, most were qualitative, involving retrospective accounts of engagement experience and relying on self-report through focus groups, one-on-one semistructured interviews, informal observation and/or written surveys with open-ended text responses. This finding is in line with other authors’ conclusions [9,38,51,52,69]. We identified only two studies with quantitative results, one of which used an unvalidated Likert-scale survey to assess patient and researcher perceptions about engagement and one systematic review that reported the results of 19 CBPR clinical trials [46]. Of the assessed measures, the literature most often reported the impact of engagement on those patients engaged, and on the research itself. Of the research impacts evaluated, there appeared to be more evidence on engagement in the following stages of the research process: topic solicitation, agenda setting, research question selection, study design and methods development, a finding in line with other authors’ assessments. [9,69] This is not surprising because engagement in these stages is more often reported in the literature generally (in comparison to other stages of research); with more research in these stages overall, it is natural that there is more evaluation and/or reported impact found by our scan.
The level of detail reported on the assessed outcomes also varied greatly. Research whose primary objective was to measure patient engagement in research often included more detailed results; case studies including patient engagement measures reported them as a subsection of their results. However, even in studies focused on reporting the impact of research engagement exclusively, the level of detail provided regarding the evaluation design, strategy and results was very limited, and generally inadequate for our purposes. For example, in many studies it was not clear whether the evaluation was planned in advance of study execution, or conducted as an informal post hoc exercise. Furthermore, authors often did not explicitly report the extent to which the engagement exercise made a demonstrable impact on the measures reported.
Collectively these findings support other researchers’ accounts of the lack of rigorous evidence underpinning patient and stakeholder engagement efforts internationally, as well as the lack of robust tools available for evaluation of engaged research [9,51,70]. What this exercise contributes is a mapping of the hypothesized impacts of engagement with what has actually been evaluated and assessed in the literature.
In the absence of a rigorous evidence base to guide engagement and subsequent evaluation efforts, qualitative findings and empirical reviews point toward some key considerations that may prove useful to those involved in or contemplating PCOR studies. To evaluate stakeholder partner engagement in PCOR, researchers should consider:
• The need to establish or select an evaluative framework or set of criteria for patient or stakeholder engagement in research prior to the execution of any research activity (in conjunction with the patients or other stakeholders engaged for collaborative and user-led types of engagement) [26,50,61];
• Documenting the context and process of engagement as fundamental components of the evaluation (i.e., context refers to funding, policy, physical environment or attitudes of those involved; process refers to the level of involvement of users, how they are involved, when, and what procedures are in place to improve likelihood of success) [51].
To our knowledge, this paper is the first to compare the intended purposes of engagement (hypothesized impacts) with actual applied measures (suggested and assessed measures). By mapping the intended outcomes to the evaluation measures, a clearer picture of whether engagement has fulfilled its intended purposes should emerge. However, our effort reveals a fundamental lack of evidence underlying much of the impetus behind the practice of stakeholder partner engagement in research. Conceptually, there appears to be little clarity or consensus on the major purpose(s) of engagement and, empirically, there is little evidence to support its potential impact. Only a few studies have conducted any kind of formal evaluation, and the quality of these studies is highly variable. Given the state of the evidence base, research engagement appears to be primarily driven by the promise that – if done well, it will improve research generally, and moral and political grounds [10]. While these are important and justifiable aims, more research is needed to truly understand how to pursue engagement strategies under specific conditions.
The lack of congruence between the stated aims for patient engagement (hypothesized impacts) and suggested measures should guide future evaluation efforts. In fact, studies often did not make clear their goals for engaging stakeholder partners at all. In a recently published Delphi study, Snape et al. found significant disagreement between stakeholders on the purpose of engagement in research as well as its justification on ethical and patient empowerment grounds [71]. Without agreement and clarity on the purposes for stakeholder partner engagement in research, it is virtually impossible to assess the extent to which its objectives have been met, if at all.
To build an evidence base that is coherent, generalizable and facilitates comparison across studies and jurisdictions, the field needs to develop more conceptual guidance and consensus to drive evaluation efforts. While we may never be able to adequately control for every contextual factor, a more solid understanding of the goals of patient engagement, some level of consensus on measurable impacts and development and validation of measures is clearly needed. Furthermore, we suggest that future evaluations of engagement research be designed a priori as an embedded component of the research process.
Of the existing guidance on evaluation, the literature tended to emphasize the process side of engagement as opposed to its impact. And, of the evaluations conducted, studies focused primarily on immediate or near-term outcomes as opposed to longer-term health outcomes. These findings are in line with those of Workman, who found that the near-term effects of research were most explored and included impact on patients and researchers (e.g., satisfaction, increased skills or knowledge) [12]. These two observations point to the challenges associated with developing indicators that are considered feasible to measure [72]. Longer-term outcomes typically have complex causes that are difficult to trace back to one research study, let alone the engagement of patients or other stakeholders in the research enterprise [27]. Beyond clarifying the goals for engagement, the field would benefit from the development of corresponding indicators to facilitate and promote a stronger, more coherent evidence base behind engagement efforts.
While process measures of engagement were emphasized more than outcomes, there is still a need to further understand who should be engaged, when should this engagement occur (i.e., at what points in the research process), and how this engagement should be done (i.e., what are the approaches to engagement that yield the desired results). Further research in these areas has been called for by others in the field, notably by Workman, who suggested the development of measurable impacts and tools beyond those focused on changes in attitudes and satisfaction among those involved in the research [12].
Numerous research teams, many supported by the PCORI are devising, developing, applying and testing strategies for engaging stakeholder partners in the research process [73]. To understand the value of these processes, and also drive toward improvement of engagement process (and theoretically improve the results of the engagement), context and process measures need to be further developed and validated.
Despite these challenges, our review surfaced promising efforts in reporting and evaluating outcomes. Researchers in the field of CBPR have established measures for the evaluation of some aspects of engagement, which can help inform the development of this effort in PCOR [74]. Furthermore, researchers are beginning to develop evaluative frameworks for public participation in research, which may help provide more guidance to those looking to integrate evaluation into their engagement activities [75–77]. Based upon engagement efforts in the UK, researchers have developed a checklist to improve the quality, content, detail, consistency, transparency and completeness of reporting of engagement activities [31]. Though it has not been widely used by the research community since its publication, increased uptake of this tool and subsequent improved reporting could help strengthen the evidence base and allow comparisons across similar studies [31]. Building on these efforts, CONSORT-style guidance will be released to assist researchers in their reporting [51]. While traditional peer-reviewed journals should encourage the reporting of the evaluation in their methodology and results, researchers should also look to other dissemination mechanisms for sharing lessons learned in engagement, as well as evaluation methodology and results.
The current evidence base may also be a product of research incentives and existing infrastructure. A commitment from research institutions and funders to support stakeholder partner engaged research is needed to adequately resource and support rigorous evaluation of research engagement activities. While there are increased political and funding imperatives to conduct PCOR in the United States, most significantly in the establishment of PCORI, other funders and institutions should assist to support and fund the development of measures and methods for such evaluation.
Limitations of this paper
This paper was informed by a literature scan that draws upon multiple sources of relevant evidence. It is not a systematic review, therefore we may not have captured all of the potential studies that evaluated engagement activities. Still, the findings in this paper are in line with others’ findings in the field who implemented systematic reviews [9,51]. Furthermore, we triangulated multiple data sources to ensure the inclusion of seminal papers and provide a snapshot of the field, which is quickly evolving. Our review is also different in that it emphasized literature that focused on more active, collaborative approaches to patient and stakeholder engagement. Articles solely focused on engaging patients as research subjects were not included. While other recent reviews in the field included articles assuming a broader definition of engagement [19], therefore covering a larger number of articles, our paper builds upon the state of the evidence suggested by these reviews with a deeper and more thorough assessment of various approaches to patient and stakeholder engagement. Furthermore, this is the only assessment of the literature known to date that attempts to consider both suggested/intended impact as well as evaluated measures. This paper demonstrates the distance the field needs to travel in order to substantiate claims and provide evidence-based guidance for the practice of stakeholder partner engaged research.
Conclusion
This paper maps and compares the intended purposes of engagement with actual applied measures to determine whether engagement has fulfilled its promised goals. Our assessment confirms the lack of evidence underlying much of the impetus behind the practice of patient and stakeholder engagement in research. Conceptually, there appears to be little clarity or consensus on what the major purpose(s) of engagement is and, empirically, there is little evidence to support its potential impact. Only a few studies have conducted any kind of formal evaluation, and the quality of these studies is highly variable. Given the state of the evidence base, research engagement appears to be primarily driven by the promise that – if done well, it will improve research generally, and moral and political grounds. This paper provides explicit guidance for better alignment of engagement's promised benefits with evaluation efforts and identifies specific areas for development of evaluative measures and better reporting of engagement processes.
Future perspective
To build an evidence base that is coherent, generalizable and allows comparison across studies and jurisdictions, the field needs to develop more conceptual and practical guidance to drive toward rigorous evaluation of research engagement. While we may never be able to adequately control for every contextual nuance, a more solid understanding of the goals of patient engagement, some level of consensus on measurable impacts and development and validation of measures is clearly needed. We suggest that future evaluations of engagement research be designed a priori as an embedded component of the research process. While traditional peer-reviewed journals should encourage the reporting of the evaluation in their methodology and results, researchers should also look to other dissemination mechanisms for sharing lessons learned in engagement, as well as evaluation methodology and results. There should also be a commitment from institutions and funders to support patient-centered research by building adequate resources into their funding opportunities to support rigorous evaluation of research engagement activities.
| Category | Suggested dimensions to measure | Assessed dimensions |
|---|---|---|
| Context | Appropriate funding, budget and/or resources available for engagement (e.g., remuneration, travel subsidies, staff time and honoraria) [5,31,32,34] | None reported |
| Attributes of the sponsoring organization (e.g., level of commitment to, attitudes toward, and/or values about engagement) [31,32] | ||
| Attributes of the engaged community/individuals (e.g., organizational attributes and attitudes toward engagement) [32] | ||
| Availability of training to patients/other stakeholders and researchers [5,19,31,35,36] | ||
| Characteristics of the issue being deliberated (e.g., degree of scientific uncertainty, type of decision, decision timeline [32] | ||
| Researcher–decision-maker relationships [32] | ||
| Appropriate time allocation [31] | ||
| Process | Diversity and representativeness of patients engaged; inclusivity, participation rate [37] | None reported |
| Access to sufficient/appropriate resources to facilitate engagement [37] | ||
| Patient/stakeholder satisfaction [32,37] | ||
| Timing, frequency and/or stages of engagement [31,32,37–40] | ||
| Process fairness, flexibility and transparency [32,37] | ||
| Use of an explicit process [26], task/role definition and structured decision-making [5,32,37] | ||
| The extent to which the mechanisms for patient engagement are adequate and appropriate (continuum from less engaged (i.e., passive, tokenistic) to more engaged (i.e., active, collaborative) [31,35,41,42] |
| Suggested dimensions to measure | Qualitative assessments | Quantitative assessments |
|---|---|---|
| Research quality and research process [28,43,44] | Improved quality and validity from engaging patients [45] | 85% of patients and researchers (n = 59) agreed that the quality of research improved [41] |
| Enhanced credibility and improved the research [46] | ||
| New funding and funding opportunities [41,47] | Patients’ contributions on research proposal, lay language summary and encouragement for reapplying [48] | None reported |
| Research topic, priorities and/or resource allocation become more appropriate/relevant to patients [38,49–52] | Patients suggested research ideas and provided feedback on usefulness of research to the benefit of the research process [48] | None reported |
| Improved relevance of research ideas and topics [45] | ||
| Identification of study topics by patients was an impetus for research [53] | ||
| Research questions, hypotheses, interventions and medical technologies become more relevant/usable for patients [1,54,55] | Patients’ experiential knowledge helped shape research question [23] | None reported |
| Research design, methods and study procedures become more appropriate, sensitive and ethically acceptable [9,17,21,34,38,41,51,55–57] | Patient research partners provided insight/access into social networks, patient organizations and the healthcare field [23] | None reported |
| Patients helped translate scientific jargon into plain language materials [23] | ||
| Recruitment, accrual rates and retention improves [9,28,34,38,54] | Patients contributed ideas to increase recruitment (e.g., feedback on patient advertisements) and modifications to interview schedules [48] | Majority of clinical trials using CBPR from systematic review demonstrated effective recruitment of minority populations and high retention rates [20] |
| Representativeness/diversity of research subjects (i.e., inclusion of more hard-to-reach patients) [34,36,41] | ||
| Data collection procedures and data quality changes [34,36,38,51,57,58] | None reported | None reported |
| Intervention and/or survey implementation by patients [17] | Data collected from peer interviewers (i.e., patients engaged) increased ‘frankness’ of responses [45] | None reported |
| Research respondents appreciated presence of a patient research partner in interviews or focus groups [23] | ||
| Data analysis and/or results interpretation by patients [34,38,41,51] | Patients co-analyzed qualitative data and suggested additional themes [48] | None reported |
| Joint data analysis helped acknowledge patient perspective and minimized risks of misinterpretation [23] | ||
| Researchers’ knowledge and capacity increases [38,44,59] | Researchers reported increased learning [38] | None reported |
| Changes in researchers’ attitudes about the value of patient perspective [5] | Perception of value added from patient engagement and allaying of researcher concerns [60] | None reported |
| Researchers perceived improved research and value added from engaging patients [45] | ||
| More useful evidence for clinical and health policy decision making [26]† | None reported | None reported |
| More relevant evidence for patients [61] Results more applicable to entire spectrum of Americans (including minorities) [21] and changes to health disparities [41]† | None reported | None reported |
| Changes to health outcomes, including overall population health, morbidity and mortality [26,36,44,59]† | None reported | None reported |
†Long-term outcomes.
CBPR: Community-based participatory research.
| Hypothesized impacts | Suggested dimensions to measure | Qualitative assessments | Quantitative assessments |
|---|---|---|---|
| Patient empowerment | Knowledge of community needs (empathy), services available, motivation to help community (citizenship literacy) [28] | Patients motivated to take interest in community, developed greater empathy and understanding of others in community [62] Improved ‘citizenship’ literacy [45] | None reported |
| Patient or community research knowledge, skills and capacity [10,18,41,44,47,49,59] | Increased learning of new skills (including research skills) and knowledge on topic area [18] Increased learning by patients [38] Increased knowledge and skills, putting learning/training into practice and raised awareness of issues and services [25] Enhanced teamwork skills [45] | None reported | |
| Self-empowerment, increased self-confidence, feeling of being valued and making a contribution [10,18,21,24,31,34,35,38,39,44,51,54,55,63,64] | Increased confidence/sense of personal achievement [45] Equal status on project team, ownership of services being received, social support, increased confidence and self-esteem [25] | None reported | |
| Impact on mental health of patients [62] | Improved emotional health feel valued and involved and having activities to look forward to [25] Enhanced outlook on life [45] Development of patients’ pride and ownership of their work [62] | None reported | |
| Patient and public satisfaction (as indicated by continued willingness to participate, buy-in, attitudes and perceptions) [44,49,60,63]† | Future engagement in research by patients previously engaged [45] Participation in new research and educational activities [48] Patients’ increased recognition of research as legitimate and useful [62] | None reported | |
| Future opportunities for patients to participate in research or in job market, etc [34]† | Forming links with professionals and increased employment skills [25] Employability increased as patient-researchers were sought after for their experience [45] | None reported | |
| Increased translation, dissemination, and uptake of results | Highlighting findings most relevant to patients [38,41,51] | None reported | None reported |
| Changes in translation and dissemination of research into practice and community [7,25,34,38,41,44,55] | Patient partners attended events and presented at conferences and served as champions for the research [48] Improved dissemination (including presentations at community events and conferences and authorship on papers) [45] | None reported | |
| Improved dissemination to patients and the community [5,51]† | None reported | None reported | |
| Faster uptake of new evidence [38]† Changes healthcare practice [44]† Changes in health regulation and policy [28,36,44,49]† Shifted priorities and changes in use of healthcare resources [26]† Changes in the cost of healthcare [65]† | None reported | None reported | |
| Democracy and accountability | Transparency [3,26,38,61,66,67] Legitimacy [26,27] and credibility [27] Accountability [26,27,38,61,66] Public trust in public institutions [26,27] | Majority of patients and researchers felt that involvement improved credibility [46] | None reported |
| Moral obligation | Fairness [26–28,50,66] Respect and trust between researchers and engaged stakeholders [9,26,27,68] | Development of trust between researchers and service users [38] | None reported |
| More ethically acceptable research [52]† | None reported | None reported |
†Long-term outcomes.
Background
• The literature proposes several benefits and hypothesized impacts of patient and stakeholder engagement in research yet there is an absence of evidence underlying these claims.
• This paper is the first to our knowledge to compare the hypothesized impacts of engagement with what has actually been evaluated and assessed.
Hypothesized impacts
• The proposed benefits of research engagement are: better quality research (includes expanded applicability of research), empowering patients, increased translation, dissemination and uptake of results, democracy and accountability, and moral obligation.
Measurement & evaluation of engagement
• With respect to evaluating the context and process of engagement (who, when and how to engage), the literature theorizes several suggested measures but we did not identify any research reporting findings along these dimensions.
• The limited number of studies that evaluate the impacts of engagement focus exclusively on the near-term effects of engagement.
• Of the studies that formally evaluated impact, most were qualitative, involving retrospective accounts and relied on self-report, informal observation or surveys.
Discussion
• This assessment confirms the lack of evidence underlying much of the impetus behind the practice of patient and stakeholder engagement in research.
• Conceptually, there appears to be little clarity or consensus on what the major purpose(s) of engagement is and, empirically, there is little evidence to support its potential impact.
• Given the state of the evidence base, research engagement appears to be primarily driven by the promise that – if done well, it will improve research generally, and moral and political grounds.
• This assessment provides explicit guidance for better alignment of engagement's promised benefits with evaluation efforts.
• It also identifies specific areas for development of evaluative measures and better reporting of engagement processes.
• This culture shift will require investment and dedication from both researchers and funding institutions.
Acknowledgements
The authors would like to acknowledge L Ellis, L Forsythe and L Frank for their advice, guidance and other contributions to this paper throughout its development; the authors specifically appreciate their input on the literature scan, and their thoughtful insights during the evidence gathering and synthesis process. The authors would also like to thank our colleagues at AcademyHealth for their assistance and support with the literature scan that informed this work including: V Thomas, R Sabharwal, L Edmundson and A Brodt. Finally, the authors would also like to acknowledge participating individuals of the Consumer Patient Researcher Roundtable, B Dowd, University of Minnesota; J Goldwater, Clinovations; M Gorman, Patient Advocate; Alice Leiter, Center for Democracy and Technology; M Mayer, AdvancedBC.org; E Powell, National Partnership for Women and Families; J Santa, Consumers Union, S Sofaer, Baruch College and M Stoto, Georgetown University, who provided insight and recommendations for relevant sources for the literature scan.
Financial & competing interests disclosure
This work was supported through a Patient-Centered Outcomes Research Institute (PCORI) Contract Award (2000-20-10-10 CMO-0001). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the PCORI, its Board of Governors or Methodology Committee. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Supplementary Material
File (suppl.docx)
- Download
- 15.86 KB
References
Papers of special note have been highlighted as: • of interest
1.
Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Affairs 29, 1834–1841 (2010).
2.
INVOLVE. What is public involvement in research? www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/.
3.
Concannon TW, Meissner P, Grunbaum JA et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 27, 985–991 (2012).
4.
Deverka PA, Lavallee DC, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 1, 181–194 (2012).
• Seminal and early article in the field of stakeholder engagement in comparative effectiveness research (CER) that aims to define stakeholder engagement and presents a conceptual model for involving stakeholders in CER.
5.
Telford R, Boote JD, Cooper CL. What does it mean to involve consumers successfully in NHS research? A consensus study. Health Expect. 7, 209–220 (2004).
6.
Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA 307, 1583–1584 (2012).
7.
PCORI Methodology Committee. The PCORI Methodology Report. Patient-Centered Outcomes Research Institute (PCORI), Washington, DC, USA (2013).
8.
Krumholz H. National priorities and research agenda. In: Patient-Centered Outcomes Research Institute National Patient and Stakeholder Dialogue. Patient-Centered Outcomes Research Institute, Washington, DC, USA (2012).
9.
Garces JPD, Lopez GJP, Wang Z et al. Eliciting Patient Perspective in Patient-Centered Outcomes Research: a Meta Narrative Systematic Review. Patient-Centered Outcomes Research Institute (PCORI), Washington, DC, USA, 1–163 (2012).
• Systematic review of various dimensions of patient engagement in research.
10.
Tritter JQ. Revolution or evolution: the challenges of conceptualizing patient and public involvement in a consumerist world. Health Expect. 12, 275–287 (2009).
11.
Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: a scoping review. Health Policy 91, 219–228 (2009).
12.
Workman T, Maurer M, Carman K. Unresolved tensions in consumer engagement in CER: a US research perspective. J. Comp. Eff. Res. 2, 127–134 (2013).
• Makes the case for developing the evidence base behind patient and stakeholder engagement in research.
13.
Boote J. Patient and Public Involvement in Health and Social Care Research: A Bibliography. NIHR Research Design Service for Yorkshire and the Humber (2011). www.clahrc-ndl.nihr.ac.uk/documents/ppi/08-boote-2011-ppi-bibliography.pdf.
14.
Watson V, Carnon A, Ryan M, Cox D. Involving the public in priority setting: a case study using discrete choice experiments. J. Public Health (Oxf) 34(2), 253–260 (2011).
15.
Saunders C, Crossing S, Girgis A, Butow P, Penman A. Operationalising a model framework for consumer and community participation in health and medical research. Aust. New Zealand Health Policy 4, 13 (2007).
16.
O'Haire C, McPhetters M, Nakamoto E et al. Engaging stakeholders to identify and prioritize future research needs. Methods Future Research Needs Reports, No. 4. Agency for Healthcare Research and Quality (US), MD, USA (2011). www.ncbi.nlm.nih.gov/books/NBK62565/#!po=12.5000.
17.
Boote J, Baird W, Sutton A. Public involvement in the design and conduct of clinical trials: a narrative review of case examples. Trials 12, A82 (2011).
18.
Barber R, Boote JD, Parry GD, Cooper CL, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect. 15, 229–241 (2012).
19.
Shippee ND, Domecq Garces JP, Prutsky Lopez GJ et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. (2013) (Epub ahead of print).
• Summarizes the findings from the systematic review above and offers a general framework for engaging patients in research.
20.
Las Nueces D, Hacker K, DiGirolamo A, Hicks LS. A systematic review of community‐based participatory research to enhance clinical trials in racial and ethnic minority groups. Health Serv. Res. 47, 1363–1386 (2012).
21.
Mullins CD. Integrating Patients’ Voices in Study Design Elements with a Focus on Hard-to-reach Populations. Patient-Centered Outcomes Research Institute (PCORI), Washington, DC, USA (2012).
22.
Pickersgill MD. Research, engagement and public bioethics: promoting socially robust science. J. Med. Ethics 37(11), 698–701 (2011).
23.
Abma TA, Nierse CJ, Widdershoven GA. Patients as partners in responsive research: methodological notions for collaborations in mixed research teams. Qual. Health Res. 19, 401–415 (2009).
24.
Barreteau O, Bots P, Daniell K. A framework for clarifying participation in participatory research to prevent its rejection for the wrong reasons. Ecol. Soc. 15(2), 1 (2010).
25.
Minogue V, Boness J, Brown A, Girdlestone J. The impact of service user involvement in research. Int. J. Health Care Qual. Assur. Inc. Leadersh. Health Serv. 18, 103–112 (2005).
26.
Lavallee DC, Williams CJ, Tambor ES, Deverka PA. Stakeholder engagement in comparative effectiveness research: how will we measure success? J. Comp. Eff. Res. 1, 397–407 (2012).
• Provides an overview of the theory underlying stakeholder engagement in research and then proposes six meta-criterion framework for evaluating stakeholder engagement in research.
27.
Abelson J, Forest P-G, Eyles J, Smith P, Martin E, Gauvin F-P. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Soc. Sci. Med. 57, 239–251 (2003).
28.
Ahmed SM, Palermo A-GS. Community engagement in research: frameworks for education and peer review. Am. J. Public Health 100, 1380–1387 (2010).
29.
Robinson L, Newton J, Dawson P. Professionals and the public: power or partnership in health research? J. Eval. Clin. Pract. 18, 276–282 (2012).
30.
Charlton JI. Nothing About Us Without Us: Disability Oppression and Empowerment. University of California Press, CA, USA (1998).
31.
Staniszewska S, Brett J, Mockford C, Barber R. The GRIPP checklist: strengthening the quality of patient and public involvement reporting in research. Int. J. Technol. Assess. Health Care 27, 391–399 (2011).
32.
Abelson J, Gauvin F-P. Assessing the Impacts of Public Participation: Concepts, Evidence and Policy Implications. Canadian Policy Research Networks, Ottawa, Canada (2006).
• Provides an overview of the theory, evidence base, and challenges of the evaluation of public participation along with a general evaluative framework.
33.
Rowe G, Frewer LJ. Evaluating public-participation exercises: a research agenda. Sci. Technol. Human Values 29, 512–556 (2004).
34.
McLaughlin H. Keeping service user involvement in research honest. Br. J. Social Work 40, 1591–1608 (2010).
35.
Wilson C, Fothergill A, Rees H. A potential model for the first all Wales mental health service user and carer‐led research group. J. Psychiatr. Ment. Health Nurs. 17, 31–38 (2010).
36.
Anderson M, Manganiello M. Back to Basics – HIV/AIDS Advocacy as a Model for Catalyzing Change. Faster Cures, HCM Strategists, 1–32 (2011). www.fastercures.org/reports/back-to-basics.
37.
Rowe G, Frewer LJ. A typology of public engagement mechanisms. Sci. Technol. Human Values 30, 251–290 (2005).
38.
Barber R, Beresford P, Boote J, Cooper C, Faulkner A. Evaluating the impact of service user involvement on research: a prospective case study. Int. J. Consumer Studies 35, 609–615 (2011).
39.
Oliver SR, Rees RW, Clarke‐Jones L et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 11, 72–84 (2008).
40.
Greenwood M. Stakeholder engagement: beyond the myth of corporate responsibility. J. Business Ethics 74, 315–327 (2007).
41.
Weglicki LS, Rice VH, Hammad A. Science and the community: a collaborative model for integration of research within the community. Ethn. Dis. 15(Suppl. 1), S135–S138 (2005).
42.
Cook WK. Integrating research and action: a systematic review of community-based participatory research to address health disparities in environmental and occupational health in the USA. J. Epidemiol. Community Health 62, 668–676 (2008).
43.
Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy 61, 213–236 (2002).
44.
Allen-Meares P, Hudgins CA, Engberg ME, Lessnau B. Using a collaboratory model to translate social work research into practice and policy. Res. Social Work Practice 15, 29–40 (2005).
45.
Williamson T, Brogden J, Jones E, Ryan J. Impact of public involvement in research on quality of life and society: a case study of research career trajectories. Int. J. Consumer Studies 34, 551–557 (2010).
46.
Wyatt K, Carter M, Mahtani V, Barnard A, Hawton A, Britten N. The impact of consumer involvement in research: an evaluation of consumer involvement in the London Primary Care Studies Programme. Family Practice 25, 154–161 (2008).
47.
Suarez-Balcazar Y, Harper GW, Lewis R. An interactive and contextual model of community-university collaborations for research and action. Health Educ. Behav. 32, 84–101 (2005).
48.
Lindenmeyer A, Hearnshaw H, Sturt J, Ormerod R, Aitchison G. Assessment of the benefits of user involvement in health research from the Warwick Diabetes Care Research User Group: a qualitative case study. Health Expect. 10, 268–277 (2007).
49.
Sibbald SL, Singer PA, Upshur R, Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv. Res. 9, 43 (2009).
50.
Happell B, Roper C. Consumer participation in mental health research: articulating a model to guide practice. Australas Psychiatry 15, 237–241 (2007).
51.
Brett J, Staniszewska S, Mockford C et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 17(5), 637–650 (2014).
• Systematic review of patient and public involvement in research that aims to assess and describe its impact.
52.
Staley K, Minogue V. User involvement leads to more ethically sound research. Clin. Ethics 1, 95–100 (2006).
53.
McKevitt C, Fudge N, Wolfe C. What is involvement in research and what does it achieve? Reflections on a pilot study of the personal costs of stroke. Health Expect. 13, 86–94 (2010).
54.
Callard F, Rose D, Wykes T. Close to the bench as well as at the bedside: involving service users in all phases of translational research. Health Expect. 15, 389–400 (2012).
55.
Staniszewska S, Jones N, Newburn M, Marshall S. User involvement in the development of a research bid: barriers, enablers and impacts1. Health Expect. 10, 173–183 (2007).
56.
Gliklich RE, Leavy MB, Velentgas P et al. Incorporating stakeholder perspectives in developing a translation table framework for comparative effectiveness research. J. Comp. Eff. Res. 1, 281–292 (2012).
57.
Beresford P. The role of service user research in generating knowledge-based health and social care: from conflict to contribution. Evidence Policy 3, 329–341 (2007).
58.
Savory C. Patient and public involvement in translative healthcare research. Clin. Governance 15, 191–199 (2010).
59.
Viswanathan M, Ammerman A, Eng E et al. Community-based participatory research: assessing the evidence. Agency of Healthcare Research and Quality (AHRQ) (2004). http://archive.ahrq.gov/downloads/pub/evidence/pdf/cbpr/cbpr.pdf.
60.
Howe A, MacDonald H, Barrett B, Little B. Ensuring public and patient participation in research: a case study in infrastructure development in one UK Research and Development consortium. Primary Health Care Res. Develop. 7, 60–67 (2006).
61.
Nass PL, Levine S, Yancy C. Methods for Involving Patients in Topic Generation for Patient-Centered Comparative Effectiveness Research – an International Perspective. Patient-Centered Outcomes Research Institute (PCORI), Washington, DC, USA (2012).
62.
Newell CJ, South J. Participating in community research: exploring the experiences of lay researchers in Bradford. Community Work Family 12, 75–89 (2009).
63.
Johnston B, Forbat L, Hubbard G. Involving and engaging patients in cancer and palliative care research: workshop presentation. Int. J. Palliat. Nurs. 14, 554–557 (2008).
64.
Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H. The PIRICOM study: a systematic review of the conceptualisation, measurement, impact and outcomes of patients and public involvement in health and social care research. United Kingdom Clinical Research Collaboration, UK (2010). www.ukcrc.org/wp-content/uploads/2014/03/Piricom+Review+Final+2010.pdf.
65.
Carman KL, Dardess P, Maurer M et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Affairs 32, 223–231 (2013).
66.
Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy 102, 105–116 (2011).
67.
Face K, Boivin A, Gracia J et al. Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int. J. Technol. Assess. Health Care 26, 224–240 (2010).
68.
Hamilton Lopez M, Holve E, Rein A, Winkler J. Involving Patients and Consumers in Research: New Opportunities for Meaningful Engagement in Research and Quality Improvement. EDM Forum Community. Issue Briefs and Reports. EDM Forum Products and Events. 6-1-2012 (2012). http://repository.academyhealth.org/cgi/viewcontent.cgi?article=1001&context=edm_briefs.
69.
Boote J, Wong R, Booth A. ‘Talking the talk or walking the walk’? A bibliometric review of the literature on public involvement in health research published between 1995 and 2009. Health Expect. 18(1), 44–57 (2012).
70.
Concannon TW, Fuster M, Saunders T et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J. Gen. Intern. Med. 1–10 (2014).
71.
Snape D, Kirkham J, Preston J et al. Exploring areas of consensus and conflict around values underpinning public involvement in health and social care research: a modified Delphi study. BMJ Open 4, e004217 (2014).
72.
Boote J, Barber R, Cooper C. Principles and indicators of successful consumer involvement in NHS research: results of a Delphi study and subgroup analysis. Health Policy 75, 280–297 (2006).
73.
Eugene Washington PCORI Engagement Awards. www.pcori.org/funding-opportunities/eugene-washington-pcori-engagement-awards.
74.
Sandoval JA, Lucero J, Oetzel J et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Education Res. 27, 680–690 (2012).
75.
The Public Involvement Impact Assessment Framework. www.piiaf.org.uk.
76.
May CR, Mair F, Finch T et al. Development of a theory of implementation and integration: normalization process theory. Implement Sci. 4, 29 (2009).
77.
Rein A, Esmail LC, Moore E, Sabharwal R, Thomas V. Evaluative framework for patient engagement in research. Presented at: AcademyHealth Annual Research Meeting. Baltimore, MD, USA (2013).
Information & Authors
Information
Published In
Copyright
© Future Medicine Ltd.
History
Published online: 31 March 2015
Keywords:
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Downloaded 2,789 timesCitations
How to Cite
Evaluating patient and stakeholder engagement in research: moving from theory to practice. (2015) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer.14.79
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Sidney Yap, Rashell R. Allen, Katherine S. Bright, Matthew R. G. Brown, Lisa Burback, Jake Hayward, Olga Winkler, Kristopher Wells, Chelsea Jones, Phillip R. Sevigny, Megan McElheran, Keith Zukiwski, Andrew J. Greenshaw, Suzette Brémault-Phillips, Exploring the Perspectives of Canadian Clinicians Regarding Digitally Delivered Psychotherapies Utilized for Trauma-Affected Populations, International Journal of Environmental Research and Public Health, 10.3390/ijerph22010081, 22, 1, (81), (2025).
- Lindsay M Martin, Christine D McKinney, Lia Escobar Acosta, Janelle W Coughlin, Noelene K Jeffers, Alexandra Solano-Umaña, Kathryn A Carson, Nae-Yuh Wang, Wendy L Bennett, Kelly M Bower, Remote Lifestyle Intervention to Reduce Postpartum Weight Retention: Protocol for a Community-Engaged Hybrid Type I Effectiveness-Implementation Randomized Controlled Trial, JMIR Research Protocols, 10.2196/62847, 14, (e62847), (2025).
- Julia Kittscha, Valerie Wilson, Greg Fairbrother, Vida Bliokas, How the expert nursing role was used to facilitate the co-design of a patient interview study, Nurse Researcher, 10.7748/nr.2024.e1918, 32, 2, (14-21), (2024).
- N. Abhishek, Neethu Suraj, M. S. Divyashree, Quality of ESG Reporting and Disclosure Practices of Indian Companies, ESG and Ecosystem Services for Sustainability, 10.4018/979-8-3693-3771-4.ch008, (197-234), (2024).
- Bronwen Merner, Sophie Hill, Freya Saich, Ariane Virgona, Defeng Jin, Alisa Pedrana, Coral Keren, Rachel Kar Yee Chung, Deborah Osborne, Anna Lee Wilkinson, Alison Coelho, Lisa Gibbs, Katherine B. Gibney, Margaret Hellard, Dean Lusher, Rebecca Ryan, What Lies Beneath? The Role of Community Engagement in Translating COVID-19 Research Findings to Policy-Makers, International Journal of Health Policy and Management, 10.34172/ijhpm.2024.8249, 13, (8249), (2024).
- Chelsea Jones, Shaylee Spencer, Elly O’Greysik, Lorraine Smith-MacDonald, Katherine S. Bright, Amy J. Beck, R. Nicholas Carleton, Lisa Burback, Andrew Greenshaw, Yanbo Zhang, Phillip R. Sevigny, Jake Hayward, Bo Cao, Suzette Brémault-Phillips, Perspectives and Experiences of Public Safety Personnel Engaged in a Peer-Led Workplace Reintegration Program Post Critical Incident or Operational Stress Injury: A Qualitative Thematic Analysis, International Journal of Environmental Research and Public Health, 10.3390/ijerph21070949, 21, 7, (949), (2024).
- Sidney Yap, Rashell R. Allen, Carley R. Aquin, Katherine S. Bright, Matthew R. G. Brown, Lisa Burback, Olga Winkler, Chelsea Jones, Jake Hayward, Kristopher Wells, Eric Vermetten, Andrew J. Greenshaw, Suzette Bremault-Phillips, Current and Future Implementation of Digitally Delivered Psychotherapies: An Exploratory Mixed-Methods Investigation of Client, Clinician, and Community Partner Perspectives, Healthcare, 10.3390/healthcare12191971, 12, 19, (1971), (2024).
- Maëlle Corcuff, Marie-Eve Lamontagne, François Routhier, Ernesto Morales, Co-design knowledge mobilization tools for universal accessibility in municipalities, Frontiers in Rehabilitation Sciences, 10.3389/fresc.2024.1331728, 5, (2024).
- Sandra Algarin Perneth, Gilberto Perez Rodriguez Garcia, Juan P. Brito, Tejal Gandhi, Carma L. Bylund, Ian G. Hargraves, Naykky Singh Ospina, Developing a framework for understanding diagnostic reconciliation based on evidence review, stakeholder engagement, and practice evaluation, Diagnosis, 10.1515/dx-2024-0132, (2024).
- Louise E. Mew, Vanessa Heaslip, Tikki Immins, Arul Ramasamy, Thomas W. Wainwright, Bridging the gap: enhancing orthopaedic outcomes through qualitative research integration, Bone & Joint Open, 10.1302/2633-1462.511.BJO-2024-0145.R1, 5, 11, (953-961), (2024).
- See more