Engaging patients and stakeholders in preresearch: findings from the Pipeline to Proposal Awards Initiative
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: The Patient-Centered Outcomes Research Institute Pipeline to Proposal (P2P) Awards Initiative funded 177 awardees to engage patients and stakeholder partners in preresearch. Based on P2P, we described engagement strategies; outcomes; facilitators; and challenges to inform research funders and stakeholders participating in preresearch. Materials & methods: We used a qualitative approach based on content analysis of program data and interviews with P2P awardees and partners. Results: Awardees developed partnership infrastructure by recruiting patients and stakeholders, establishing clear roles, and providing training. Building trust was key to engaging patients and stakeholders in preresearch. Awardees reported partners were more likely to engage in PCOR in the future. Conclusion: P2P awardees increased capacity of patient and stakeholder partnerships to conduct PCOR.
Research operates in a paradigm where clinical and translational research is initiated and conducted by researchers. Recently, research funders have moved toward requiring patient and stakeholder engagement in the research process, despite the inexperience of many researchers in conducting engagement activities [1–4]. There is a need for a new paradigm where researchers work directly with patients and other stakeholders through a process of capacity building that begins prior to receiving research funding. This early research phase, referred to herein as preresearch, but also called initiation in other literature [5], is a distinct phase of the research cycle that includes idea and topic generation, topic selection, prioritization and framing the research question [6]. Studies focusing on patient and stakeholder engagement have acknowledged important activities during preresearch [1,7,8] although, more recently, studies have explored the preresearch phase independently [5].
Evidence suggests that patient or community contributions increase research question relevance, study recruitment and retention, and face and construct validity of measures [9–12]. There have been multiple strategies identified to engage patients and stakeholders in research, generally, such as providing clear information about stakeholder roles and expectations (e.g., time commitment and responsibilities) [12–14]; valuing nonresearch perspectives and encouraging the participation of nonresearch trained stakeholders and patients [5,15,16]; and developing and maintaining ongoing relationships [12,15].
The strategies and tools for promoting successful engagement in preresearch, specifically, are only beginning to emerge in the literature. For example, a study by Natafgi and colleagues identified different levels of patient engagement in preresearch activities [17]. The authors note that ‘authentic early engagement’ is characterized by multiple opportunities for patient involvement in all activities, high investment in all engagement activities, high willingness of patients to participate, and high satisfaction of researchers and patients. On the other end of the spectrum is ‘passive early engagement,’ in which there are limited opportunities for patient involvement in all activities, moderate investment, moderate willingness and low satisfaction of researchers and patients. In another study, Zych et al. found that, in order to build researcher–patient or researcher–stakeholder partnerships, the preresearch phase may last between 6 months and 2 years for pre-existing partnerships or 6 years for newly formed partnerships [5]. Other key findings from their study included that the formation of new partnerships requires funding and multiple points of contact between the researcher and the patient or stakeholder, and it be challenging for partnerships to maintain continuity and enthusiasm during the preresearch phase.
There is a need to develop, implement and evaluate patient and stakeholder engagement approaches that encourage preresearch activities [5]. However, there is a lack of tools to assist researchers in engaging patients and stakeholders in preresearch, such as partner matching tools and preresearch engagement toolkits, and also a lack of funding for nonresearch activities (such as partnership building) that are critical for preresearch [1].
Among the existing literature on collaborative research, there are numerous known barriers to establishing and sustaining researcher–patient–stakeholder partnerships to research. Common barriers include insufficient time to develop partnerships [10]; concerns regarding power imbalances and distrust between different stakeholder communities [10,18,19]; and insufficient transparency on roles and responsibilities regarding decision making [12,13]. Transparency and accountability are two of the most important factors in fostering strong partnerships with stakeholders [20].
Funders are investing in programs that facilitate the development of partnerships of researchers, patients and other stakeholders to build capacity for preresearch. In the USA, the Patient-Centered Outcomes Research Institute (PCORI), a nonprofit organization created through the 2010 Patient Protection and Affordable Care Act, funds PCOR, which is ‘the evaluation of questions and outcomes meaningful and important to patients and caregivers’ [11]. PCORI is at the forefront of PCOR and, through its initiatives, is advancing the science of patient and stakeholder engagement in the research process. Other funders that have supported partnerships for preresearch include the Canadian Institutes of Health, which has brought together researchers, patients, and other stakeholders to build capacity for research through its Research Strategy for patient-oriented research [4]. In the UK, the National Institute for Health Research established the INVOLVE program to support public involvement in public health and social care research [3]. These efforts are paving the way to a better understanding of the optimal strategies for engaging patients and other stakeholders as equitable partners in preresearch.
The purpose of this study was to learn from the experiences of a PCORI program called the Pipeline to Proposal (P2P) Awards Initiative, which funded partnerships of patients and other stakeholders to conduct preresearch activities and build capacity for comparative effectiveness research (CER) studies. CER compares the effectiveness of two or more interventions or approaches to healthcare, examining their risks and benefits. Based on the experiences of P2P awardees and their partnerships, this paper describes strategies for engaging patients and other stakeholders in partnerships for preresearch, the outcomes of those strategies, and facilitators and challenges to inform funders of preresearch and stakeholders participating in preresearch.
Materials & methods
P2P program context
In 2013, PCORI launched P2P, which provided funding and technical assistance to build partnerships of patients and other stakeholders to conduct preresearch activities. We define partnership as collaboration between partners as part of P2P. Partners are members of the research team involved in preresearch activities. In P2P, partners included individuals with lived experience with a particular health condition (patients, caregivers, family members) and members of constituencies whose perspectives are based on professional rather than personal experience (policymakers, clinicians, researchers, hospital staff and others). These categories are not mutually exclusive. In this program, when describing P2P partnerships, the terms partner and stakeholder are used interchangeably. The goals of P2P were fourfold: strengthen researcher–patient–stakeholder relationships; build partnership capacity for PCOR funding; engage partnerships in the research process; and successfully establish infrastructure for patients, caregivers and other stakeholders to increase knowledge about PCOR and engagement in research, dissemination and implementation. The P2P logic model is provided in Supplementary Material 1.
P2P was a multitier, multicycle program where distinct P2P teams (referred to as awardees) were funded between 2013 and 2018 over one, two or three periods of time, each called a tier. Awardees applied for PCORI’s approval to advance to subsequent tiers. Throughout P2P, PCORI made changes to the program structure, including streamlining P2P tiers into one tier comprising 50 awardees as well as offering an independent, one-time funding for four awardees to develop scientifically rigorous PCOR proposals. In total, 177 awardees were awarded US$10 million to conduct 353 projects, a project being composed of a leadership structure and activities within a single tier.
P2P awardees worked with five technical assistance organizations, which helped with partnership and infrastructure development, strategic planning, reporting progress and sharing lessons learned. P2P awardees aligned their projects with PCORI’s Engagement Rubric, a ‘blueprint’ for implementing engagement strategies in the research process. The Rubric describes six principles important for engagement: transparency, honesty and trust; reciprocal relationships; valued partnerships; and co-learning [21]. The Engagement Rubric was developed based on the promising practices identified in studies from the PCORI portfolio.
Research design
This study used a grounded theory approach [22] based on program data collected by PCORI from P2P awardees and qualitative interviews with P2P awardees and partners. This research was conducted between June 2018 and June 2019 as part of an independent evaluation of P2P funded by PCORI. We conducted content analysis of program data from PCORI, and concurrently collected and analyzed qualitative interview data. Data were analyzed separately and then coded and discussed. Interview data were used to confirm, validate and expand upon the findings in the program data. Findings were integrated into a final report for PCORI, of which findings in this paper are based upon. Our reporting of the qualitative research findings is consistent with the Consolidated Criteria for Reporting Qualitative research. The NORC Institutional Review Board determined this project was not human subjects research and granted a certificate of exemption.
Data sources
Interviews
Sample & recruitment
We used purposive sampling [23] to identify and recruit P2P awardees (specifically, P2P project leads, i.e., the principal investigator [PI] or co-PI of the P2P project) and partners for interviews. Criteria for sampling were tier, project geography (city and region), target population, project health topic, type of awardee organization and technical assistance organization. This helped us to select information-rich cases and learn about the experiences of different P2P partnerships. The awardees selected for interviews from each tier roughly matched the proportion of awardees in each tier in the P2P portfolio. We applied this sampling strategy to identify P2P project leads. Upon recruiting project leads, we asked project leads to identify one partner associated with their projects for interviews. In cases where project leads provided multiple recommendations for project partners, we prioritized patient partners to ensure we gathered perspectives of nontraditional research partners. Individuals received an email invitation to participate that stated the reason for this research (i.e., PCORI contracted with an independent organization to conduct this study). Participants were scheduled for interviews by the research team. The research team did not have a prior relationship with participants.
Data collection
AI Bayne, P Dullabh, M Skillman, S Zainulbhai, and KJ Devers, with prior training and experience conducting qualitative research and other credentials (MPA, MD, MPA, MPP, PhD), led interviews. Interviewers’ occupation was social science research at the time of the study. A semistructured interview protocol was developed from a literature review on patient and stakeholder in engagement in research, and in consultation with PCORI, and focused on engagement strategies, challenges and outcomes. Interview topics are listed in Box 1. In addition to these topics, participants were prompted to elaborate upon their responses.
•
Awardee and partner roles on the project
•
Stakeholder engagement strategies (most and least successful)
•
Experiences with technical assistance (most and least valuable aspects)
•
Experiences pursuing funding for Patient-Centered Outcomes Research project
•
Challenges and facilitators
•
Sustainability of the partnership
•
Outcomes, including unexpected positive or negative outcomes
•
Suggested improvements to the program
Interviews were conducted between 29 October 2018 and 9 January 2019. Interviews were 1 h in length, conducted by telephone and audio recorded with permission from all participants. To prepare for interviews, we reviewed the program data (described below) available for each awardee in advance. A total of 66 participants completed 64 interviews: 38 project leads and 28 project partners representing 35 awardees. A total of 15 people declined the invitation to participate. We set out to complete interviews with leads of a 20% sample of the 177 awardees and achieved this aim. We aimed to conduct a similar number of partner interviews, but concluded early because we reached thematic saturation, as determined by AI Bayne, P Dullabh, M Skillman and KJ Devers, upon review and discussion of the partner transcripts. Interviews were approximately 60 min in length and audio recorded with participant consent. A research analyst took notes during the interviews and prepared nonverbatim transcripts, referencing the recording afterward as needed.
Program data
Program data were from three different sources: P2P awardee applications, final reports and PCORI administrative data. These data were extracted by PCORI into a database, and provided to the research team for analysis. Final reports were used to determine the types of stakeholders participating in each P2P project and to understand awardee engagement strategies, challenges, facilitators and outcomes in P2P. We reviewed 348 awardee final reports submitted to PCORI. Four awardees did not submit final reports for their last tier due to early project termination; instead, we reviewed their last submitted report. One awardee’s final report was missing. Thirty awardees were restricted by the number of partners they could list in their final report.
To generate descriptive information about awardees, we reviewed data from 177 awardee applications. We report awardee region, defined as the Census region corresponding with each awardee lead organization’s mailing address state; awardee lead organization, which is the organization that the P2P PI was affiliated with; and health topic for projects (with the exception of 27 awardees where we drew information from final reports because it was not available in applications).
PCORI also provided administrative data including projects that did not advance to subsequent tiers, location of awardee partnerships, technical assistance offices assigned to awardees, and awardees that were funded by other PCORI programs post-P2P.
Data analysis
Data were analyzed and tabulated using MS Word, MS Excel and NVivo software (QSR International Pty Ltd, Melbourne, Australia). NVivo was used to inductively and deductively code and analyze data from both program data and qualitative interviews [24]. We deductively developed a codebook for final reports comprised of 36 codes based on salient analytic dimensions (e.g., contextual features, PCORI Engagement Rubric principles, outcomes and impacts).
Initial codes were developed based on the PCORI Engagement Rubric principles, an existing taxonomy provided by PCORI, literature on patient and stakeholder engagement, and topics in the interview protocol. We then inductively refined the code list based on a coding pilot during coder training, described below. As a result, we developed additional codes for implementation, engagement and outcomes. For example, during the inductive period, we incorporated new codes related to engagement strategies including ‘time and contribution of partnerships valued,’ ‘roles and decision making authority of partners,’ and ‘transparency, honest, and trust.’ Building off of the final report codebook, we added additional codes exclusively for interviews and used them to explore patient and stakeholder engagement and sustainability in depth, adding new codes such as training, effective engagement strategies, sustainability and measurement of engagement strategies. The final codebook contained 55 codes and 19 interview-specific codes. Box 2 presents topics in the codebook.
•
Pipeline to Proposal (P2P) awardee characteristics (e.g., location and health issue of focus)
•
Project context (e.g., organizational features and local environment)
•
P2P implementation and engagement (e.g., convening meetings, creating advisory boards and developing research questions)
•
Engagement principles (e.g., reciprocal relationships; honesty, transparency and trust)
•
Outcomes and impacts (e.g., sustainability, increasing awareness and strengthened capacity to engage in research)
To train coders, MS used well-established techniques (e.g., Kappa scores of 0.61–0.80) to assess interrater reliability and monitored outputs over time to ensure coding consistency [25]. Four coders piloted the codebook and practiced on a 5% sample (15 reports) of the final report dataset and a 6% sample of interviews (four transcripts), revising codes as needed. Using NVivo’s Compare query tool, MS calculated interrater reliability between the first and second coder on coded text segments until coders reached at least 0.7 kappa for final report data and 0.6 kappa for interview data with at least two other coders on the team. We followed a quality assurance protocol wherein coders flagged coding ambiguities and sent questions to a central coding manager (MS) who acted as an arbiter and communicated codebook changes to coders.
Data were analyzed by AI Bayne, M Skillman, P Ubri, C Rotondo, and S Zainulbhai using code query, key word searches, or a combination within NVivo. For instance, when reviewing data about ways in which awardees established partnerships, we conducted key word searches of data containing phrases such as “word of mouth,” “phone calls,” or “website” within the “establishing partnerships” code. We also extracted data corresponding with certain final report questions to compare awardee responses. We met to discuss findings. All study findings were derived from the data. We present exemplar quotes and participant identifiers (‘L’ for awardee project leads and ‘P’ for project partners) to demonstrate the diversity of experiences reflected in this paper.
Strengths & limitations
A key strength of this study is that we created a new dataset that is novel, systematically collected, and rich in information. The dataset combines primary and secondary data, which captures experiences of P2P awardees and partners participating in P2P. We used this dataset to triangulate primary data from the qualitative interviews) and secondary program data. The analyses in this report have several limitations:
•
Data collection and analysis activities were conducted simultaneously. The study design precluded an iterative approach where information gained from reviewing the program data determined the primary data collection and analysis approach. Also, data were not available from participants at baseline.
•
Questions asked of awardees in the final reports significantly evolved over the course of P2P, making consistently aggregating some data points across P2P difficult or, in some cases, impossible. There was variation in the structure, level of detail and content of the awardee final reports, which limited our ability to study certain topics across awardees. Some questions were not asked of all awardees. Some awardees used general language and few examples, whereas other awardees reported specific information. Twenty-one final reports were minimally completed by awardees.
•
The final reports primarily asked awardees about accomplishments, experiences with technical assistance and project activities. Therefore, it is possible that awardees did not articulate all challenges associated with implementing P2P.
Results
Participant characteristics
P2P awardee characteristics
Geographic region
The 177 P2P awardees represented geographic regions across the USA and Puerto Rico, with most located in the West or South – 37 and 25%, respectively; 20% were located in the Midwest and 18% were located in the Northeast.
Organizational affiliation
Most P2P project leads and co-leads reported affiliations with some type of entity. A total of 42% were affiliated with a university, followed by healthcare providers or organizations (19%), and community-based or advocacy organizations (18%). 5% of awardees did not report an organizational affiliation.
Health topic
P2P awardees focused on different health issues with the aim of increasing and advancing PCOR in their communities. The four most common health issues of focus reported by awardees were chronic conditions (66 awardees, 37% of 177 awardees), mental and/or behavioral health conditions (48 awardees, 27%), diabetes (26 awardees, 15%) and cancer (23 awardees, 13%). Twenty awardees (12%) focused P2P projects on health issues related to rare diseases (classified based on the National Organization for Rare Disorders' rare disease list). A majority of awardees focused their projects on the adult population (97 awardees, 55%) or older adults, specifically (82 awardees, 46%).
Types of partners
A total of 127 P2P awardees representing 214 projects found that the vast majority of P2P partnerships included researchers (88%, 112 awardees) and patients and consumers (85%, 108 awardees) (Figure 1). P2P awardees also reported engaging caregivers or patient family members; clinicians; clinics, hospitals or health systems; and advocacy organizations acting on behalf of patients, consumers or caregivers. The 127 awardees collectively reported engaging 1906 stakeholders, ranging from seven to ten stakeholders per project.
Figure 1. Types of stakeholders in Pipeline to Proposal partnerships.
Source: Awardee final reports. Notes: Thirty awardees in one tier could not report more than ten partners in final reports. Percentages are rounded, >0.5 percentage point rounded up to the nearest whole percentage point. Out of 177 total awardees, 127 (72 percent) described their P2P partnership size (1906 partners reported).
P2P: Pipeline to Proposal.
Interview participant characteristics
A total of 81 individuals were invited to participate in interviews. Two people declined to participate, one person agreed to participate but did not attend the interview, and twelve people did not respond to our request for an interview. Sixty-six individuals were interviewed – 38 leads and 28 partners. Leads and partners had projects covered a diverse range of health topics and were based in 23 US states in the Midwest, Northeast, South, West, the District of Columbia and noncontinental USA .Table 1 describes the type of stakeholder groups represented by partners and leads. To prevent identification, we have not presented further information on participant characteristics.
| Stakeholder type | Leads (n = 38) | Partners (n = 28) | All (n = 66) |
|---|---|---|---|
| Researcher | 27 | 8 | 35 |
| Patient/caregiver | 2 | 11 | 13 |
| Patient advocate | 7 | 0 | 7 |
| Patient, consumer or caregiver advocacy organization/community-based organization | 2 | 3 | 5 |
| Clinic/hospital/health system/provider | 0 | 5 | 5 |
| Government stakeholder | 0 | 1 | 1 |
| Region | |||
| District of Columbia, noncontinental USA | 3 | 3 | 6 |
| Midwest | 3 | 4 | 7 |
| Northeast | 9 | 6 | 15 |
| South | 9 | 5 | 14 |
| West | 14 | 10 | 24 |
Strategies to engage patients & other stakeholders in partnerships for preresearch
P2P awardees used three key strategies to engage patients and other stakeholders in partnerships for preresearch. They built infrastructure for partnerships by recruiting patients and other stakeholders to participate in their projects, established clear roles for patients and stakeholders, and trained patients and stakeholders.
Building infrastructure for partnerships
Partnership development is a critical step in building capacity for preresearch. P2P provided funding to build infrastructure for partnerships of patients and other stakeholders. We first explored how awardees connected with and recruited patients and other stakeholders to participate in the P2P partnerships. Establishing partnerships was not a static activity but continued throughout the course of an awardee’s project to fill gaps in expertise and add new perspectives. In final reports and interviews, P2P awardees noted the importance of personalizing outreach to patients and other stakeholders. Awardees tailored recruitment to prospective partners by phone, email and/or in-person meetings. In-person communications were reported as the most effective approach to recruiting patients and caregivers and was also useful for recruiting clinicians and researchers.
Recruitment approaches
According to final reports and reinforced in interviews, P2P awardees leveraged their personal and professional networks to identify and recruit partners to participate in the project. Awardees identified three methods for recruiting stakeholders, which were not mutually exclusive: leveraging existing relationships, conducting peer-to-peer outreach and recruiting via intermediary organizations. Existing relationships lent awardees credibility, especially when engaging hard-to-reach and underserved patient populations like veterans, survivors of intimate partner violence and populations with low health literacy. Awardees with community trust – often formed through existing connections to the project’s target population, disease of focus and/or geographic community – reported being better equipped to form partnerships than awardees who were not as well connected.
In both final reports and interviews, awardees noted that peer-to-peer recruitment was another important strategy for recruiting patients and caregivers as it helped remove power differentials, particularly when reaching out to underserved or hard-to-reach populations. As awardees noted in final reports, using peer outreach such as community health workers, cultural brokers and leveraging social capital from existing relationships (e.g., word-of-mouth referrals) resulted in greater community trust. During an interview, one awardee described the success of peer-to-peer recruitment:
We had mostly parent partners because there are a lot of parents with infants born withdrawn from opioids….[we] asked them to help us recruit. Next, we went door-to-door and hung flyers for our community conversations and we were able to get a lot more parent involvement by having them do the recruiting. (L1)
In interviews, awardees noted that it was easier to engage with patients when outreach occurred through a trusted organization with existing connections to the patient or stakeholder community or a specific disease or health condition (e.g., hospital networks, community-based nonprofits, faith-based organizations, patient advocacy organizations and national associations). An added benefit of this strategy is that intermediary organizations were often in a position to identify patients and stakeholders who were likely to be highly engaged in P2P projects.
Establishing roles for patients & stakeholders on the project
Once partnerships were established, P2P awardees focused on establishing clear roles for patients and other stakeholders. According to final reports, most awardees (83%, 147 awardees) used advisory boards to organize their partnerships. Advisory boards were composed of researchers, patients and other stakeholders such as caregivers, representatives from advocacy organizations, clinicians and policymakers, among others. Advisory boards were local, statewide or national in scope, typically reflecting the project’s scope and target population. Stakeholders on the advisory board convened in-person as well as by phone on a regular basis. P2P awardees also described the importance of using culturally appropriate strategies to engage partners in projects (e.g., holding advisory board meetings in the community’s first language).
A key first task of the advisory board was to identify roles for all stakeholders on the project. During advisory board meetings, and through informal communications, awardees asked patients and stakeholders how they were interested in participating in the project. Patients and other stakeholders played different roles, such as facilitating meetings and leading recruitment activities to engage new partners. Some awardees formed smaller committees and work groups for patients and other stakeholders to take on responsibilities that aligned with their interests. During an interview, one awardee explained this process:
Once we had our core team…the focus was more internal to our partnership team. We wanted everyone to remain engaged…We did that through regularly scheduled meetings. I communicated with the group very frequently, even if it's just sharing an article. We wanted to keep in touch so people felt like they had membership on a team. And once we had actual work, we then formed smaller committees where we asked people to take on additional responsibility…they were able to allow them to pick and choose things they were interested in and get involved. (L2)
In another example, a P2P awardee laid out all of the tasks that needed to be accomplished on the project – outreach and recruitment, communication, a literature review, and data collection – and asked partners, “Where do you see yourself in this?” In a third example, a P2P awardee with a geographically dispersed partnership requested volunteers through a project-specific newsletter. Even if partners were not interested in leading an activity, they could express their preference for how they wanted to be involved. These approaches created an environment in which all partners could use their skills in ways that were meaningful to them and would advance the project.
One P2P patient partner described the role that they played on the project:
My expertise and presence were both always respected…I went to a number of community meetings and interfaced with [community stakeholders]. One of the gifts I have, and I call it a gift because I don't know too many other people who could do it, but because I grew up poverty-stricken as a Black man and because I had been in leadership positions…I am comfortable regardless of whether people are poor, wealthy, black, white…It doesn't matter to me who is in the room. I appreciate different cultures…And so [the project lead] would always have me in the community meetings. (P1)
P2P awardees expressed in final reports and interviews the importance of having patient representatives on the advisory board, and some awardees created patient membership thresholds on advisory boards to ensure adequate patient representation. To ensure that patient and caregiver experiences were represented – and that they and other stakeholders were comfortable sharing perspectives – P2P awardees established voting rules (e.g., weighted patient votes or equal vote values among stakeholders); used first names rather than titles during meetings; convened in a neutral setting (i.e., outside of a medical setting); and used breakout groups during meetings to foster more intimate conversations. These practices helped partnerships to establish a flat partnership structure in which patients and other stakeholders were placed on equal footing with researchers and clinicians. One P2P partner explained this during an interview:
I began to understand my role…that I would be a contributor and all perspectives were going to be equally respected and recorded. We wouldn't worry about who is in charge and who we would have to pass the opinions through. My involvement from early on was you are as important as the physicians, please raise your voice and contribute. (P2)
P2P awardees also engaged other patients and stakeholders who were not on their advisory boards. Approximately 66% of awardees (117) reported in final reports that they also sought the perspectives of patients and other stakeholders outside of those directly involved in their advisory boards. Many awardees used surveys (e.g., via Facebook, in-person) to reach patients, providers and other stakeholders (e.g., hospitals, policymakers) to gather feedback on potential research questions. Some awardees convened and facilitated forums and town halls designed to gather input. These were different from advisory groups in that they were open to patients and stakeholders beyond the project’s core partnership.
Training for patients & stakeholders
P2P awardees received training and technical assistance from PCORI and technical assistance offices to build partnership capacity. In interviews, both P2P awardees and partners described training topics including, but not limited to, education on the target health issue, disease or target community; project orientations to introduce new partners to P2P, share project goals and provide education on topics such as cultural competency, unconscious bias, community-based participatory research and partnership evaluation; and research methods and CER to answer questions, such as, ‘What is CER?’ and ‘How do you participate in research?’ In particular, awardees shared in interviews that it was important that all partners had a strong understanding of the health issue or disease so that they were able to engage in preresearch activities. For example, one project lead noted:
One of the things we discovered…[patients] have extremely low knowledge about diabetes. So, in order for them to actively engage in the research, it's necessary for them to understand their disease better. (L3)
As described in interviews, P2P awardees, partners and representatives of external organizations also led trainings. Awardees leveraged their connections with institutions or community groups and invited knowledgeable individuals to provide culturally competent trainings or present information about CER and research methods. Additionally, some awardees created their own trainings, using resources from PCORI’s website (e.g., webinars). One P2P awardee explained the training process they used:
Once we started our programing the very first thing we did was we created some slides based on PCORI information. We used that to educate the patients and caregivers about PCORI and also about the PCOR and CER. (L4)
It was important to recognize that the goal of education and training was to build a common understanding and respect for each stakeholder’s unique expertise – not to bring all partners to the same level of knowledge on all issues. As one awardee said:
[We're] not trying to turn people into research[ers], but to give them the basics of quantitative and qualitative research … to tell us what [they think], and give us input. (L5)
Awardees also noted that training should align with community values and abilities (e.g., by providing training in Spanish if applicable to the target community). One awardee also recommended that future programs compile training resources for partnerships, noting that there was variability in the level of training that partnerships received and the comfort of partnerships with PCOR and CER:
We created a lot of materials internally of what is research, and typical research processes, and what it means to seek funding, and everything. And so, if there were more resources that we could have pulled from…a library would have been nice that explained these basic concepts. (L6)
Facilitators & challenges
Facilitators
Funding
P2P awardees described the P2P funding as a key facilitator for building partnerships of patients and other stakeholders for preresearch. One P2P awardee shared in their final report that funding was critical to building the type of trust that facilitates capacity building in PCOR:
The funding allowed us to have face-to-face stakeholder meetings that enhanced co-learning, building trust and sharing of information and experiences that benefitted the capacity building. This [led] to continued interest and excitement for future engagement to purse CER questions. Funding also allowed us to provide incentives, engage a facilitator and hire two consultants (experts) to serve on the leadership team. (L7)
Understanding of research
Another facilitator of awardees' success was an understanding of CER and PCOR. One awardee noted in a final report:
The project lead and co-lead have experience conducting community-based participatory research.… Their prior experience designing and implementing patient-centered research projects provided an environment conducive to supporting [technical assistance] efforts focused on infrastructure building. (L8)
Social media
In final reports, awardees reported that social media (e.g., Facebook and Twitter) helped facilitate engagement. Convenient and easy to scale up, outreach via social media was appropriate when in-person outreach was not a practical recruitment strategy (e.g., for partnerships spread across a large geographic area) or, as mentioned above, when awardees sought to elicit research ideas from population subgroups impacted by a specific health issue. For instance, a project focused on reducing death from lung cancer developed a project hashtag and conducted a Twitter chat and Google Hangout on treatment. The project’s lead said during an interview:
We have developed a social media group…[on lung cancer]. …The founders were myself and other community stakeholders including a family member of a lung cancer patient, medical oncologist, a patient with stage 4 lung cancer, and another family member of an individual who died of lung cancer. That group was loosely affiliated. We would hold tweet chats twice a month. (L9)
In addition, when convening advisory boards, awardees made efforts to ensure stakeholders could actively and regularly participate in virtual meetings using video chats or teleconferencing:
To help the caregiver feel more involved in the group, a space was set for her at the table, and a name card was made to display in front of the iPad to replicate name cards used for all other participants who were physically present. We encouraged her participation the same as all of the in-person participants by speaking directly with and to her, acknowledging her contributions, and even responding to her emotions during discussions. (L10)
Challenges
Convening stakeholders & sustaining engagement
In final reports, P2P awardees reported challenges convening patients and other stakeholders due to competing priorities; transportation barriers; and health issues that affected participation. For example, one awardee expressed during an interview that their greatest challenge was engaging patients:
I think the biggest challenge was that we didn't anticipate how much manpower was going to go into it because we needed so much personalization. And because the patient required so much one-on-one attention, it was important for the researchers to have folks that would be responsible for attention to the patients. By the end of the partnership, we had one person who was designated just to work with patients to make sure their needs were met – including babysitting and transportation. Patient stipends. We made sure the patients could come, which meant we had to pay attention to which barriers they had to attending. (L11)
Similarly, during an interview, another awardee described challenges maintaining patient involvement in their project due to health issues that prevented patients from participating:
Our patient involvement was really challenging. We did have a patient come to one or two meetings, but then they had health issues and had to pull out. Because of our population, many are elderly and home bound - also getting people interested - that was a big issue for us…We wish we had more patient participation. (L2)
Another key challenge was engaging stakeholders who had very busy schedules and sustaining their engagement over time. Awardees described challenges finding times for meetings that were convenient for clinical stakeholders, in particular. One partner shared their experience during an interview:
Another big challenge was getting our clinical partners there. We did some teleconferences, sometimes we held separate meetings with clinical people. We got input however we could if everyone couldn't be at the meeting at the same time. With a clinical partner's schedule we set the meetings up to accommodate that but it didn't always work. We used technology through the video conference to try and get around that. (P3)
Additionally, P2P awardees noted that maintaining engagement over multiple years is challenging, particularly when working with healthcare providers. One awardee explained this during an interview:
Something we struggled with was that the P2P awards are stretched out over the course of 3 years - different yearly awards. Keeping someone engaged over a 3 year period is challenging. I get why they did this: initially to build capacity and relationships and trust, it takes time. However, I think providers aren't used to that time frame. There were surgeons and people who normally have a much more narrow project time span. (L12)
P2P awardees addressed these challenges by expecting and preparing for stakeholder turnover in their partnership. One awardee noted in a final report that it was important to “learn to accept that change is normal; people will enter and leave your partnership for a number of different reasons. Don't take it personally.” (L13) P2P awardees prepared for turnover by recruiting multiple people from one organization to ensure that ties remained if one person left the partnership. One awardee also noted that it was helpful to periodically recruit new members and revisit the composition of the partnership as project needs evolved.
Familiarity with PCOR & CER
A final key challenge was a lack of familiarity with PCOR and CER. During interviews, we asked P2P partners if they were familiar with PCOR and CER prior to this study and many explained that they were unfamiliar with these concepts at the outset of the project. In response to this question, one P2P stakeholder said, during an interview:
No. [P2P project lead] knew we were supposed to submit a grant–[that] was the goal – to apply for a three year grant about CER, so we started exploring it and doing our self-study so in the end we were familiar with it. But, I still don't know if everyone fully understood it. I still don't fully understand it. (P4)
Given that PCOR and CER were new concepts, awardees described in their final reports that there was a learning curve. One specific challenge that some awardees faced was that patients and other stakeholders were interested in pursuing topics and research questions that were not appropriate for CER, for example, because they lacked comparators (i.e., different interventions that patients can consider in the real world) or a robust evidence base – both which are needed to conduct CER. Finally, while funding was a facilitator of building partnership capacity, P2P awardees also noted that increased funds were needed; some awardees shared during interviews that they had to carry out project activities on their own time.
Outcomes
Development of trusted partnerships
Awardees shared in final reports and interviews that an important outcome of P2P was that they were successful in building trust in their partnerships. Awardees expressed that trust was essential to engaging patients and stakeholders in partnerships, and also foundational for conducting preresearch activities. The P2P funding gave partnerships the resources to conduct face-to-face meetings, which was critical for building trust among patients and other stakeholders. In a final report, one P2P awardee described trust as an important outcome of their project:
We continued to develop a greater understanding of research and the process for developing a research question among the parent/caregivers who participate in the partnership. This project provided the resources needed to convene regularly and spend the time it takes to not only build shared understanding around the complexities of research, but also to translate the content into Spanish. Over the twelve month period we had open discussions across partners about this topic, each bringing their own expertise, which gave everyone a chance to have their voice heard and to hear varying perspectives. Through continued ongoing meetings, we had the opportunity to build trust and appreciation among members, which will be a critical foundation as we move into the next phase, including more intensive work and more resources at stake. (L14)
Trust within the P2P partnerships helped to facilitate co-learning and sharing of information and experiences, and build respect among stakeholders.
Increased capacity of partnerships to engage in PCOR
In final reports, 70% of all awardees (124 awardees) reported increased capacity for their partnership to engage in and conduct research. Research capacity included strengthening and/or expanding partnerships; creating partnerships that did not previously exist; and establishing reputations as credible researchers and experts on their community’s needs. As stated by an awardee in a final report:
We did not anticipate…that our key project ethnic partners would become sought-after speakers and consultants both re[garding] using the method and re[garding] participatory patient engagement generally. (L15)
Looking forward, 96% (152 of 158 awardees) indicated in final reports that their partners were more likely to engage in PCOR in the future as a result of P2P participation.
In final reports, some awardees reported building collective knowledge on how to conduct PCOR in the future; applying learnings from P2P to other patient-centered projects; and increasing both partners’ knowledge about the health issue of focus. Patient partners who had participated in clinical research studies in the past noted in interviews that they had never before helped define problems and identify potential solutions in a study. During interviews, some awardees and partners, particularly those who did not have a research background, reported initially lacking an understanding of CER and PCOR and developing new skills and knowledge through P2P. While it was not possible to systematically identify awardees and partners who were new to research, we learned through the interviews that P2P helped young researchers develop new skills. One awardee shared:
The program is meeting [PCORI's] mission of having the community and stakeholders determine what the research should be – especially for new researchers. I was less than 10 years out from getting my PhD and the funding I received and the training I got was very helpful in getting my feet on the ground [and] helping me feel confident in the work I was doing. (L16)
Funded proposals to conduct PCOR
Based on a review of 170 awardee final reports to PCORI as well as administrative information from PCORI following the end of the P2P, 20% of all awardees (36) received funding from PCORI or non-PCORI funders. These included one awardee funded for CER and 27 awardees who received a PCORI Engagement Award (a program which supports patient and stakeholder engagement in research), and 12 awardees funded by non-PCORI sources – four of which had also received a PCORI Engagement Award for capacity building among other awards. 70% of these awardees (19 awardees) were universities or community-based or advocacy organizations.
Awardees also pursued opportunities related to stakeholder engagement, research and advocacy or policy work. At the time of their final report to PCORI, 16% (28 of 170 reporting awardees) reported submitting a proposal to non-PCORI funders, including foundations (e.g., Robert Wood Johnson Foundation), government entities (e.g., NIH) and universities or university-affiliated hospitals. Of these, 12 awardees knew at the time of final reports that their project had been awarded by non-PCORI sources (e.g., hospital or university, foundation, government and private donor).
Plans to sustain partnerships
Regardless of whether awardees were funded, they reported a desire to sustain their partnerships and pursue PCOR after P2P. Of 108 awardees reporting information about sustainability in final reports, most had a positive outlook about the relationships and partnerships created through P2P and reported plans to sustain partnerships. Some awardees planned to continue convening their advisory boards with interested partners. Others described plans to work on preresearch projects; seek input from patients during preresearch; and share findings from P2P.
Discussion
There is growing effort to integrate patients and other stakeholders as partners in clinical research from conceptualization and design to implementation and dissemination. As demonstrated through its investment in P2P, PCORI is at the helm of efforts to build partnership capacity for PCOR. In December 2019, Congress reauthorized PCORI’s funding for 10 years and the legislation was signed into law. PCORI’s reauthorization is a critical step in the advancement of PCOR, including the engagement of patients and stakeholders in preresearch. The purpose of this study was to describe strategies used by 177 awardees funded by P2P to engage patients and stakeholder in preresearch, identify facilitators and challenges and describe outcomes. To our knowledge, P2P is the largest program in the USA that has funded patient and stakeholder engagement in preresearch.
A key finding from our study was that P2P awardees reported that they achieved true patient and stakeholder engagement in their projects – moving beyond ‘passive engagement,’ where patients are human subjects in clinical research trials and the engagement is researcher-driven and ‘tokenistic engagement’ where patients and stakeholders provide input but are not true partners in the research [17]. P2P partnerships achieved what Natafgi and colleagues refer to as ‘authentic engagement,’ characterized by ‘long-term partnerships with patients and their communities, including capacity building and enabling and facilitating shared leadership.’ Awardees accomplished this by building infrastructure for partnerships – recruiting patients and other stakeholders to participate in P2P projects; establishing defined roles for patients and stakeholders; and training patients and other stakeholders in how to conduct PCOR and CER.
Based on the authors’ analysis of the findings, we gleaned three key learnings, which offer guidance for funders of patient and stakeholder partnerships for preresearch as well as researchers interested in engaging patients and other stakeholders in preresearch. First, consistent with other studies, we found that foundational elements of patient and stakeholder engagement in preresearch are mutual respect, equitable power and trust [9,10,15,16]. The P2P awardees implemented strategies to achieve these aims, according to awardee accounts in reports and interviews.
The second lesson learned is that there is an opportunity to strengthen the voice of patients and stakeholders in partnerships for preresearch through education and training. Studies have noted the need for practice guidance, resources, tools and training to ensure effective patient and stakeholder engagement in research [26]. Recognizing this need, a key aspect of the P2P program was to provide education and training to awardees and partnerships about PCOR and engagement best practices through technical assistance. The technical assistance offices provided training and education to awardees who, in turn, then provided training to stakeholders involved in the partnership to increase their understanding of PCOR and CER. While the partnerships benefited from this training and technical assistance, our findings suggest a need for additional training on PCOR and/or research methods and a need for ongoing technical assistance.
First, in order for patients and other stakeholders to contribute as ‘fully fledged members of research teams,’ they must have – as noted by Carroll and colleagues – “the appropriate ‘readiness’ [27].” In the future, funders should continue to invest in the development of resources and tools to help partnerships to build capacity for PCOR, including tools tailored to support teaching the concepts of PCOR and CER to nonscientific audiences. Resources and trainings must be easy to understand, accessible and free of jargon. One P2P awardee suggested that a library of materials would be helpful to support patient and other stakeholder engagement in preresearch, an idea that was also suggested in Zych et al.’s study which identified a need for tool kits that could offer guidance and templates to support researcher and research user partnerships [5].
Second, there is a need for education for researchers about how to best engage with patients and other stakeholders in preresearch. There is awareness of the need for patient and stakeholder engagement in research. What is missing is an understanding of how to best accomplish and implement this [28]. Based on P2P, we learned there is an important role for ongoing technical assistance, in addition to training, throughout the preresearch process to help partnerships build capacity to conduct PCOR.
A third lesson learned is that patient and stakeholder engagement in preresearch takes time, and requires funding. Engagement activities are not often accounted for in project budgets and timelines. It takes time to build trust. Based on the P2P awardees’ experiences, it is clear that it can take years for partnerships to advance from a place of convening and building trust among a group of diverse stakeholders to coalescing around a research topic and submitting a PCOR or CER proposal for funding.
Finally, as has been noted in other studies, it is critical for funders to use consistent measures of patient and stakeholder engagement to capture input, ideally over time to allow for comparison with assess growth or changes in partnerships. There is not consensus in the literature on optimal measures of patient and stakeholder engagement [15,29,30]. While observational engagement measures are easy to obtain (e.g., counts of attendance at an event) they do not capture attitudes directly, yet self-reported measures – while reflecting attitudes of patients and stakeholders – lack universally agreed upon constructs of engagement and responses may not be comparable across communities [15,31]. Further, measuring engagement entails considering its complexity, such as the types of stakeholders involved, areas of expertise, involvement in research activities (e.g., choosing research questions) and intensity of engagement (e.g., symbolic engagement and truly engaged participation) [15,20,21,31]. There is a need for future research to create engagement measures that clearly and consistently define the components being measured and how they map to engagement, particularly for preresearch activities.
Conclusion
P2P was a landmark program that provided partnerships composed of patients and other stakeholders with funding and technical assistance to build capacity for preresearch. The strategies used by P2P awardees offer valuable information for research funders and stakeholders engaging in PCOR. The outcomes of P2P highlight the need for continued investments in programs that focus on preresearch and patient and stakeholder engagement to build upon what was learned in this study. Developing capacity to conduct preresearch requires dedicated funding and time. Currently, there are no other PCORI awards programs to fund projects focused exclusively on engagement for preresearch. In the future, funders should consider how to support preresearch programs like P2P, which have potential to promote the development of rigorous, high-quality research proposals that reflect the unique insights, experiences, and priorities of patients and other individuals impacted by treatments for specific diseases or conditions. Overall, P2P enabled patients and other stakeholders to engage in PCOR – a shift that will undoubtedly improve health and healthcare.
Background
•
Preresearch is an early phase of the research cycle that includes idea and topic generation, topic selection, prioritization and framing the research question.
•
There is a need to develop, implement and evaluate patient and stakeholder engagement approaches that encourage preresearch activities.
Materials & methods
•
This grounded theory study used program data collected by Patient Centered Outcomes Research Institute from Pipeline to Proposal (P2P) awardees, and qualitative interviews with P2P awardees and partners.
Results
•
P2P awardees used three key strategies to engage patients and other stakeholders in partnerships for preresearch.
•
Facilitators were funding, understanding of research, and technology and social media.
•
Challenges were convening stakeholders and sustaining stakeholder engagement and a lack of familiarity with PCOR.
•
P2P awardees achieved important outcomes including the development of trusted partnerships, increased the capacity of partnerships to participate in PCOR, funded proposals and planned to sustain partnerships.
Discussion
•
P2P awardees achieved authentic patient and stakeholder engagement in their preresearch projects.
•
Foundational elements of P2P awardees’ patient and stakeholder engagement strategies were to build mutual respect and trust.
•
There is a need for education for researchers about how to best engage with patients and other stakeholders in preresearch.
•
Patient and stakeholder engagement in preresearch takes time and requires funding.
•
There is a need for future research to create engagement measures that clearly and consistently define the components being measured and how they map to engagement, particularly for preresearch activities.
Conclusion
•
P2P was a landmark program that provided partnerships composed of patients and other stakeholders with funding and technical assistance to build capacity for preresearch. The strategies used by P2P awardees offer valuable information for researchers and other stakeholders engaging in PCOR.
Supplementary data
To view the supplementary data that accompany this paper please visit the journal website at: Supplementary Material
Author contributions
All listed authors listed above participated sufficiently in the work to take responsibility for the content. All author contributions include: substantial contributions to the conception or design of the work; or the acquisition, analysis or interpretation of data for the work; AND drafting the work or revising it critically for important intellectual content; AND final approval of the version to be published; AND agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Acknowledgments
This paper presents the findings of a study entitled, ‘Pipeline to Proposal (P2P) Awards Initiative Evaluation: Patient and Stakeholder Engagement Report,’ sponsored by the Patient-Centered Outcomes Research Institute (PCORI) under IDIQ Task Order #9-NORC-AOSEPP-ENG to NORC at the University of Chicago. The authors gratefully acknowledge PCORI for funding this work and providing leadership, support, and guidance for the study, especially C Clyatt, MA, MPH, Program Officer, Engagement, and L Hotchkiss, MPH, Director, Eugene Washington PCORI Engagement Awards Program. We extend our sincere appreciation to the P2P technical assistance office staff and the many P2P awardees and partners who volunteered their time to speak with us about their experiences. The authors would also like to acknowledge M Cherney; C Cooley; E Moriarty; K Spear; E Tanenbaum at NORC for their support with data collection and analysis.
Financial & competing interests disclosure
NORC conducted the evaluation of P2P for Patient Centered Outcomes Research Institute (PCORI) under IDIQ Task Order #9-NORC-AOSEPP-ENG. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
The authors state that they have obtained appropriate institutional review board review of the evaluation and protocols, which determined that this project was “not human subjects research” and, therefore, received a certificate of exemption.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/
Supplementary Material
File (suppl_file.docx)
- Download
- 349.24 KB
References
Papers of special note have been highlighted as: • of interest
1.
Rolfe DE, Ramsden VR, Banner D et al. Using qualitative health research methods to improve patient and public involvement and engagement in research. Res. Involv. Engagem. 4, 49 (2018).
2.
Patient-Centered Outcomes Research Institute. The value of engagement (2018). www.pcori.org/engagement/value-engagement
3.
INVOLVE. What is public involvement in research? (2020). www.invo.org.uk/
4.
Canadian Institutes of Health Research. Strategy for patient-oriented research (2019). https://cihr-irsc.gc.ca/e/41204.html
5.
Zych MM, Berta WB, Gagliardi AR. Initiation is recognized as a fundamental early phase of integrated knowledge translation (IKT): qualitative interviews with researchers and research users in IKT partnerships. BMC Health Serv. Res. 19, 772 (2019).
• Qualitative research paper characterizes the initiation phase in integrated knowledge translation.
6.
Stallings SC, Boyer AP, Joosten YA et al. A taxonomy of impacts on clinical and translational research from community stakeholder engagement. Health Expect. 22, 731–742 (2019).
• Introduces The Community Stakeholder Impact on Research Taxonomy, which can be used to standardize the reporting and evaluation of engagement activities in research projects.
7.
Gagliardi AR, Berta W, Kothari A et al. Integrated knowledge translation (IKT) in health care: a scoping review. Implement. Sci. 11, 38 (2015).
• Scoping review describes integrated knowledge translation approaches, barriers, facilitators and outcomes.
8.
Kothari A, MacLean L, Edwards N et al. Indicators at the interface: managing policymaker-researcher collaboration. Knowl. Manag. Res. Pract. 9, 203–214 (2011).
9.
Kirwan JR, De Wit M, Frank L et al. Emerging guidelines for patient engagement in research. Value Health 20(3), 481–486 (2017).
10.
Kendall C, Fitzgerald M, Kang RS et al. “Still learning and evolving in our approaches”: patient and stakeholder engagement among Canadian community-based primary health care researchers. Res. Involv. Engagem. 4, 47 (2018).
11.
Frank L, Basch E, Selby JV. The PCORI perspective on patient-centered outcomes research. JAMA 312(15), 1513–1514 (2014).
12.
Guise J-M, O'Haire C, McPheeters M et al. A practice-based tool for engaging stakeholders in future research: a synthesis of current practices. J. Clin. Epidemiol. 66(6), 666–674 (2013).
13.
Boudes M, Robinson P, Bertelsen N et al. What do stakeholders expect from patient engagement: are these expectations being met? Health Expect. 21(6), 1035–1045 (2018).
14.
Cornman DH, White CM. Discerning the Perception and Impact of Patients Involved in Evidence-based Practice Center Key Informant Interviews. Agency for Healthcare Research and Quality, MD, USA (2017). www.effectivehealthcare.ahrq.gov/reports/final.cf
15.
Forsythe L, Heckert A, Margolis MK, Schrandt S, Frank L. Methods and impact of engagement in research, from theory to practice and back again: early findings from the Patient-Centered Outcomes Research Institute. Qual. Life Res. 27(1), 17–31 (2018).
• Seminal paper describes the Patient-Centered Outcomes Research Institute’s evaluation framework for assessing the impacts of engagement.
16.
Hacker KE, Smith AB. Engaging stakeholders and patient partners. Surg. Oncol. Clin. N. Am. 27(4), 665–673 (2018).
17.
Natafgi N, Tsahai Tafari A, Chauhan C, Bekelman JE, Mullins DC. Patients' early engagement in research proposal development (PEER-PD): patients guiding the proposal writing. J. Clin. Epidemiol. 8(6), 441–453 (2019).
• Describes and compares different levels of patient engagement in preresearch.
18.
Khodyakov D, Grant S, Meeker D, Booth M, Pacheco-Santivanez N, Kim KK. Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics. J. Am. Med. Inform. Assoc. 24(3), 537–543 (2016).
19.
Duffett L. Patient engagement: what partnering with patient in research is all about. Thromb. Res. 150, 113–120 (2017).
20.
Carman KL, Workman TA. Engaging patients and consumers in research evidence: applying the conceptual model of patient and family engagement. Patient Educ. Couns. 100(1), 25–29 (2017).
21.
Patient-Centered Outcomes Research Institute. Engagement Rubric for Applicants (2014). www.pcori.org/sites/default/files/Engagement-Rubric.pdf
22.
Chun Tie Y, Birks M, Francis K. Grounded theory research: a design framework for novice researchers. SAGE Open Med. 7, 1-8 (2019).
23.
Patton MQ. Chapter 5: Designing Qualitative Studies In: Qualitative Research and Evaluation Methods (3rd Edition). Sage, CA, USA (2002).
24.
Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv. Res. 42(4), 1758–1772 (2007).
25.
McHugh ML. Interrater reliability: the kappa statistic. Biochem. Medica 22(3), 276–282 (2012).
26.
Bishop AC, Elliott MJ, Cassidy C. Moving patient-oriented research forward: thoughts from the next generation of knowledge translation researchers. Res. Involv. Engagem. 4, 23 (2018).
27.
Carroll SL, Embuldeniya G, Abelson J, McGillion M, Berkesse A, Healey JS. Questioning patient engagement: research scientists' perceptions of the challenges of patient engagement in a cardiovascular research network. Patient Prefer. Adherence 11, 1579 (2017).
28.
Costello W, Dorris E. Laying the groundwork: building relationships for public and patient involvement in pre-clinical paediatric research. Health Expect. 23, 96–105 (2020).
29.
Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J. Comp. Eff. Res. 4(2), 133–145 (2015).
• Synthesizes the literature on the impacts of engagement and the evidence of patient and stakeholder engagement in research.
30.
Concannon TW, Fuster M, Saunders T et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J. Gen. Intern. Med. 29(12), 1692–1701 (2014).
31.
Bowen DJ, Hyams T, Goodman M, West KM, Harris-Wai J, Yu JH. Systematic review of quantitative measures of stakeholder engagement. Clin. Transl. Sci. 10(5), 314 (2017).
Information & Authors
Information
Published In
Pages: 721 - 736
PubMed: 32672474
Copyright
© 2020 Alycia Bayne et al. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 19 December 2019
Accepted: 1 May 2020
Published online: 16 July 2020
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Engaging patients and stakeholders in preresearch: findings from the Pipeline to Proposal Awards Initiative. (2020) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2019-0203
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Precious McCowan, Taya Joseph, Cassandra B. Picataggio, Lindsay Bailey, Sahib Chandi, Amanpreet Singh, Shivani Surati, Laura M. Dember, Jennifer E. Flythe, Meaningful Engagement of Patients in Research, American Journal of Kidney Diseases, 10.1053/j.ajkd.2026.01.020, (2026).
- Scott McDonald, Minh Duong Kim Ngo, Scott McDonald, Minh Duong Kim Ngo, Third-Party Logistics (3PL): Today’s Jack-of-All-Trades, Level Up Your Logistics, 10.1007/978-981-95-0533-3_4, (121-209), (2025).
- Georgia Tobiano, Brigid M. Gillespie, Joan Carlini, Rachel Muir, Jananee Rasiah, Ching Shan Wan, Tamara L. McCarron, Karen Moffat, Sepideh Jahandideh, Wendy Chaboyer, Establishing patient partners’ roles on research teams: a scoping review, Research Involvement and Engagement, 10.1186/s40900-024-00664-1, 10, 1, (2024).
- Henrike J. Westerink, Tom Oirbans, Mirjam M. Garvelink, Cornelia F. van Uden-Kraan, Ouisam Zouitni, Hans A.J. Bart, Philip J. van der Wees, Paul B. van der Nat, Barriers and facilitators of meaningful patient participation at the collective level in healthcare organizations: A systematic review, Health Policy, 10.1016/j.healthpol.2023.104946, 138, (104946), (2023).
- Maureen E. Maurer, Tandrea Hilliard‐Boone, Karen Frazier, Laura Forsythe, Rachel Mosbacher, Kristin L. Carman, Examining how study teams manage different viewpoints and priorities in patient‐centered outcomes research: Results of an embedded multiple case study, Health Expectations, 10.1111/hex.13765, 26, 4, (1606-1617), (2023).
- Emily D. Quinn, Kathleen Cotter, Kim Kurin, Kim Brown, Conducting a Community Engagement Studio to Adapt Enhanced Milieu Teaching, American Journal of Speech-Language Pathology, 10.1044/2021_AJSLP-21-00100, 31, 3, (1095-1113), (2022).
- Jenny Martínez, Catherine Verrier Piersol, Kenneth Lucas, Natalie E. Leland, Operationalizing Stakeholder Engagement Through the Stakeholder-Centric Engagement Charter (SCEC), Journal of General Internal Medicine, 10.1007/s11606-021-07029-4, 37, S1, (105-108), (2022).