What works when: mapping patient and stakeholder engagement methods along the ten-step continuum framework
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: This study provides a recommended ‘patient engagement translation table’ that identifies evidence-based methods for meaningful patient engagement along a ten-step framework for continuous engagement. Materials & methods: We used a mixed methods research design to collect data on preferred engagement methods, including an environmental scan of available literature, interviews and focus groups with patient-centered outcomes research stakeholders to match methods with research steps and a modified Delphi process with subject matter experts to create the final translation table. Results: Evidence-based engagement methods included community partnerships, focus groups, interviews, meetings, sharing print materials, social media, storytelling, surveys and including patients as research team members. Conclusion: Our recommended patient engagement translation table is designed to assist investigators in determining appropriate engagement methods for meaningful interactions with stakeholders.
Patients’ voices are a critical part of designing, conducting, disseminating and evaluating results of comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) studies and for assessing meaningful patient engagement in research [1]. The Patient Protection and Affordable Care Act (ACA) and the creation of the Patient-Centered Outcomes Research Institute (PCORI) created heightened interest to determine how best to achieve the goal of authentic and respectful engagement of patients and other stakeholders such as clinicians, caregivers, health systems administrators, researchers and payers [2].
The lived experiences and perspectives of persons with an illness or injury should be an important input to the design and conduct of research. So is the perspective of each member of the ill or injured person’s support and health systems; in other words, other stakeholders such as family members, caregivers, clinicians, healthcare providers and policy makers. Incorporating these diverse perspectives in the research process can make studies more patient centered, relevant and useful. Patient engagement throughout a study may lead to greater use and uptake of research findings by patients and stakeholders within their community [3]. PCORI has published multiple peer-reviewed articles on the importance and value of patient and stakeholder engagement [4–10].
There is no one-size-fits-all approach to patient engagement. Approaches and methods can – and should – be different depending on the type of study and numerous other factors, such as the target population, geography, healthcare setting and social determinants of health. Understanding these contextual factors is critical to determine which approach to take and which methods to use [3]. The investigator must also consider how to implement the selected methods within the constraints of the study’s budget, timeline, staffing and capabilities. One conclusion from ‘Patient engagement in research: a systematic review’ is that research dedicated to identifying the best methods to achieve engagement is lacking and clearly needed [11]. A review on rare diseases funded by PCORI reiterated that the most effective approaches to engagement have not been well defined [12]. While previous and concurrent studies have advanced PCOR methods for patient and stakeholder engagement, they stop short of mapping engagement methods to specific steps of the PCORI engagement rubric [13]. Tunis et al. also noted that the methodology gap on effective and systematic engagement must be bridged for the ‘translation table’ recommended by PCORI’s methodology committee to function well [14]. The PCORI engagement rubric deliberately did not prescribe what engagement activities should occur at each stage of research [15]. The reasoning for not being prescriptive was to support future exploration of appropriate methods of engagement along the PCOR continuum and across diverse subpopulations and healthcare delivery systems [8]. This omission, however, makes selecting effective engagement methods challenging for novice PCOR researchers. We believe that knowing what engagement methods work in general is insufficient without understanding how methods align with each step of the research process, and potentially vary by patient characteristics.
Although patient engagement is fairly new, community engagement in research is not a new concept. Two models, community-based participatory research and community action research [16–18], precede our approach to provide recommendations for codeveloping patient-centered research. However, these frameworks focus mostly on community-centric change, rather than patient-centered pragmatic research. One early work specific to CER and PCOR is the UK’s National Institute for Health Research Patient and Public Involvement initiative [19–21]. This model provides recommendations for involving both patients and the public throughout the research lifecycle. Similar to the approach taken in the present study, the UK model maps patient involvement to the following areas of research: identifying and prioritizing issues, designing studies, developing grant proposals, undertaking and managing projects, analyzing and interpreting results, disseminating findings, implementing appropriate methods, and monitoring and evaluating projects [22].
Our aims included: develop a patient engagement translation table (PETT) that maps patient engagement methods to the ten-step framework; recommend general guidance on the time, staffing and financial resources required for planning and executing patient engagement activities for each method, as well as any special considerations about additional resources required for under-represented populations; and highlight patient engagement methods that enhance engagement of under-represented patient groups.
Materials & methods
The foundation of our study was the ten-step PCOR continuum framework for patient and stakeholder engagement (‘ten-step framework’) [23]. Our study aimed to support PCOR investigators’ ability to discriminate between and among engagement methods, assess the applicability of methods according to stage of research, and determine the feasibility of methods by providing guidance on resources required for implementation. We hypothesized that any single method was unlikely to be appropriate for all steps or all participants; thus, we mapped patient engagement methods to each stage of the research process (Figure 1).
Figure 1. Alignment of ten-step framework to Patient-Centered Outcomes Research Institute engagement rubric.
PCORI: Patient-Centered Outcomes Research Institute.
Our 3-year study occurred in three phases: Phase I: environmental scan and quantitative analysis (May 2015–September 2016), Phase II: interviews and focus groups with qualitative and quantitative analysis (November 2016–January 2018) and Phase III: Delphi panel (February 2018–April 2018). Findings from Phase I and II informed the first round of the Delphi process in Phase III. The final round of the Delphi process resulted in our final PETT and recommendations. Figure 2 illustrates the process.
Phase I: environmental scan
In Phase I, we followed the PRISMA evidence-based minimum guidelines for reporting in systemic reviews as we identified, screened and abstracted data (Supplementary Figure 1A) [24]. The overall search strategy involved three components: conventional database searches, gray literature searches and citation chaining.
Conventional database searches
Databases searched via conventional methods included the Cumulative Index of Nursing and Allied Health or CINAHL® (n = 290), Embase®/MEDLINE® (n = 4711) and PsycINFO® (n = 3524). Initially, we searched MEDLINE through PubMed®; however, the resulting number of articles included an unduly large number of out-of-scope documents. Because Embase includes MEDLINE content and allows for proximity searching (e.g., search ‘patient’ within two words in either direction of ‘engage’), we were able to calibrate the Embase search to retrieve a more viable number of results for title/abstract screening.
Supplementary Table 1A specifies the search string terms used for the CINAHL database as an example, including controlled vocabulary (when available) and natural language keywords. This process identified peer-reviewed literature, guidance documents and other relevant materials, and it uncovered terminology for methods, engagement and populations.
Gray literature searches
Given the relatively recent and rapidly emerging body of literature on patient engagement in CER and PCOR, a search of gray literature supplemented the conventional search strategy and ensured that literature retrieved was comprehensive. Gray literature includes government, nonprofit, foundation, association and organization reports; meeting proceedings; and commentaries. We used two approaches to retrieve gray literature:
•
Organizational website searches: we conducted searches of organizational websites identified by advisors as key entities with a vested interest in patient engagement (e.g., PCORI, AHRQ and INVOLVE) using the Google ‘site:’ feature. Google enables users to search a single website by typing ‘site:’ in front of a site or domain name. For example, typing ‘site:ahrq.gov’ alongside search terms such as patient engagement in research retrieves results found only on the AHRQ website;
•
Google web search: we conducted a Google query on patient engagement in research.
Citation chaining
We identified supplemental documents by citation chaining (i.e., ‘chaining backward in time’), harvesting references from, and citations to, key publications that our stakeholder advisory board (SAB) or others brought to our attention. We used Scopus® to retrieve references from these documents [11,25–27].
The conventional search of literature focused on publications dated after 1 January 2005, through 31 May 2016. Because the ACA enacted in 2010 prompted a groundswell of growth in CER and PCOR, we selected the year 2005 to include the 5 years preceding the ACA legislation to capture any literature during that period.
Evidence grading
The qualifying 259 articles were ‘graded’ using the University of Maryland Center of Excellence in Regulatory Science and Innovation criteria [28]. The University of Maryland Center of Excellence in Regulatory Science and Innovation criteria are based on a collaborative agreement between the US FDA and the University of Maryland, which focuses on modernizing and improving the ways drugs and medical devices are reviewed and evaluated. Specifically, we used the following questions to screen for eligibility:
•
Was the article original research, review, gray literature or prescriptive (editorial or commentary)?
•
Which of the ten-step framework methods were discussed?
•
Which methods of engagement were used?
•
Were resources for patient engagement discussed (e.g., time, staff and budget)?
•
Which stakeholder group(s) was engaged?
•
Were patient stakeholder characteristics described (e.g., rare disease, veteran, women, minority and low income)?
We also evaluated each engagement method mentioned for the quality of evidence – in other words, the level to which the method was reported to have had a positive impact. We used the following rubric, based on one by Perfetto et al. [28]:
•
Were patients, caregivers or proxies recognized as partners, advisors, reactors or study subjects?
•
Was patient engagement continuous, sporadic, singular or not engaged?
•
Was the patient engagement meaningful, partial, superficial or nonexistent?
•
Was the engagement effort comprehensive, representative, limited or nonexistent?
•
Did engagement occur at appropriate, acceptable, poor or inappropriate times throughout the process?
When grading evidence, we identified a group of articles that articulated either a research step or an engagement method, but did not explicitly state one or the other to fully meet our inclusion criteria. For example, the research engaged patients in a PCOR study, but it did not explain what methods were used or how or when the methods were used. For these articles, we took an extra step and performed additional screening (sensitivity analysis) to see if we could map data to the PETT. The additional screening included a full review of the entire article by at least two coders for eligibility to be included in the full review and database inventory. This sensitivity analysis identified 59 articles that we excluded, leaving 200 (12%) qualifying articles for further analysis. The authors recognize that quality of evidence on the impact of various methods is still emerging, and therefore, the recommendations are beneficial as tools within a toolkit rather than interpreted as gold standard.
We reviewed 200 qualified articles, identifying and retrieving data about the types of patient engagement methods, estimates of resources required for patient engagement and methods to engage under-represented populations. This phase yielded an exhaustive inventory of methods (quantitative data) for meaningful patient engagement. We analyzed and organized this list into engagement methods categories which became the card deck for the activity used in Phase II.
Phase II: focus groups & interviews
In Phase II, we conducted focus groups and interviews to collect stakeholder feedback, digging deeper into the methods identified in Phase I. We used a card-mapping activity to collect quantitative data, enriched by focus groups and interviews to collect qualitative data. We developed the interview guide and card-mapping tools specifically for this study (Appendix A). The card deck included the following methods and descriptions:
•
Community partnership: a collaboration focused on partnering with community members, organizations, and researchers to ensure research is fair and representative;
•
Focus group: a small group of people talking about a specific topic with a facilitator. It may be held in person, by computer or by phone;
•
Interview: a one-on-one meeting where one person asks specific questions about the other person’s opinions, views and experiences. The discussion is conversational, not like a test. Examples include individual or personal, small group, telephone or in-depth interviews, and projective techniques;
•
Meeting: a group of people coming together to answer a question, resolve a problem or provide advice on a particular issue. A leader or speaker(s) may be presenting information. It may be held in person, by computer or by phone. Examples include community forums, advisory boards, online webinars, trainings, workshops, Delphi panels and nominal group techniques;
•
Print material: information presented in writing so that a person can view it and express an opinion or reaction. Examples include reviews, books and textbooks, newspapers, academic journals, press releases, and advertising such as flyers, pamphlets, leaflets and multimedia;
•
Social media: an online interaction among people using applications such as Facebook, Twitter, Instagram and other platforms that invite people to express their opinions and experiences. Examples include blogging, micro blogging, location, social networking, aggregators, online forums, collaboration and multimedia;
•
Storytelling: a method where people share their experiences with others by telling stories orally and through the use of photos, pictures, animation, videos, plays, poems, music, presentations, books and other written methods;
•
Survey: a method using preset questions to learn about a person’s experiences and needs. Surveys can be done in person, over the phone, by mail or online. People can fill out a survey on their own or have someone else fill out the survey on their behalf;
•
Research member: a person who is actively involved in making decisions about the direction and implementation of the research process. Examples include steering committee member, community member and patient research team member.
We based the interactive card-mapping activity on the concept of community resource mapping. Community resource mapping is a process of identifying a community’s assets and resources and plotting how they might interconnect and be combined to meet specific needs or goals. The technique has been used to map the services, supports and programs within a community to identify at least three key issues: existing resources available to a specific population, new or additional resources, and resources that may be used to assist in creating and building capacity. The process can also be used to develop a plan to improve the alignment, coordination and delivery of services [29]. Community resource mapping is increasingly used in the public health sector to collaborate with communities to identify existing resources, stretch the reach of limited resources, implement new or expanded programs and potentially improve health outcomes [30,31].
Community resource mapping has several benefits that aligned with our need for an interesting and interactive activity to open our focus group and interview sessions, build trust and rapport, and collect feedback from each individual [29]. Namely, it is a relationship-driven exercise and focuses on ‘the development of partnerships – a group of equals with a common interest working together over a sustained period of time to accomplish common goals’ [32]. We selected this technique to generate evidence about various patient engagement methods, to understand patients’ and stakeholders’ expectations for the methods, to identify which method(s) work with which research steps, and to evaluate patient engagement overall. The card-mapping activity also facilitated a roleplaying component; participants had to think like a PCOR designer and decide which methods were most appropriate along the ten-step framework.
The card-mapping activity addressed varying degrees of literacy and a variety of learning styles, making it especially effective for engaging a diverse range of stakeholders. For example, we designed this activity to incorporate three forms of learning. Participants were provided with cards using text and images (visual) (see Appendix A), asked to place marks on a worksheet to indicate ‘placement’ (tactile), and encouraged to contribute thoughts about the activity through oral comments and discussion (auditory). The resulting ‘map’ was a good tool to prompt additional insight and feedback, as seeing data in front of you often increases your understanding and sharpens your thought process [32]. See Table 1 for data source, description and utility in the Phase II activities.
| Source | Description | How used in analysis |
|---|---|---|
| Completed game board responses | Completed map of methods plotted across the continuum of research steps used as foundation for focus group and interview discussion | • Game boards were coded and analyzed in Microsoft Excel |
| Facilitator and cofacilitator notes | Notes taken by the interviewer or cofacilitator during focus groups and interviews | • Notes were merged with transcripts and imported into NVivo 11 • Notes provided additional context and insights to coded text |
| Focus group and interview transcripts | Transcription based on detailed notes from focus group and interview recordings | • Transcripts were imported into NVivo 11 • Text was coded per the codebook guidelines |
We identified, selected, and recruited participants representative of our target population and included a wide variety of healthcare roles: patients, patient advocates, researchers, counselors, caregivers, community leaders, providers and other stakeholders. We used a purposive sampling approach to recruit participants with diverse socioeconomic and ethnic backgrounds, ages and disease states. In addition to the diversity of roles, our study was particularly interested in targeting historically under-represented populations, including:
•
African-Americans and other minorities;
•
Speakers of English as a second language;
•
People with physical and sensory impairments (e.g., blind or vision impaired);
•
People with low literacy skills;
•
People with low socioeconomic status;
•
Patients aged 55 and older.
Patients and other stakeholders were recruited primarily in the Baltimore metropolitan area through a community liaison, and providers were recruited from a local hospital. Researchers with experience in PCOR, CER, community-engaged research and/or community-based participatory research were recruited through newsletter advertisements and personal contacts. In the handful of cases in which patients and other stakeholders were unable to attend an in-person focus group, we conducted individual interviews, either in person or by web conferencing, depending on participant preference and needs. We conducted all researcher interviews individually by web conferencing, as this stakeholder group was dispersed across the country. The study groups and related delivery methods (e.g., in person or via web conferencing) were as follows:
•
Patients: five in-person focus groups, nine in-person individual interviews and one web conferencing individual interview;
•
Researchers: 50 individual interviews via web conferencing;
•
Other stakeholders: ten in-person focus groups and five individual interviews via web conferencing.
We used direct questions and took a narrative approach to encourage stakeholder feedback and the exploration of concepts (Table 2). We adapted the level of vocabulary and style of language to ensure that participants fully understood the questions. When discussing engagement methods, we did not ask participants to consider any particular goal of engagement; rather we asked them to focus on methods that would reach and engage members of their community for each step of the research process. The notion of ‘best’ was conceptualized via questions about preferences, associations and appropriateness.
| Sample key topics | Participant groups | |
|---|---|---|
| Knowledge of patient engagement activities | • What trouble, if any, did you have in defining or understanding what each method or research step was? • What may have made it easier to understand the method or step? • Are there any methods that you have experience with? What was that experience? | • Patients • Researchers • Other stakeholders |
| Acceptability of patient engagement activities along ten-step framework | • Are there any methods listed that you think will work well for all the steps? • What methods do you think would work well for all groups of people? • Why do these methods work well across multiple steps? Resources, communication and type of outreach | • Patients • Researchers • Other stakeholders |
| Feasibility of the card-mapping activity | • Does the practice game board (meals and types of food) translate to the main objective of the study? | • Patients • Researchers • Other stakeholders |
| Methods that are appropriate for general populations compared with specific populations | • Are there any methods listed that you would avoid using either for a particular step or across all steps? • Which methods work well for particular groups of people? For example, age, health condition, socioeconomic status, geographic location and underserved population | • Patients • Researchers • Other stakeholders |
| Resources and considerations for researchers when implementing methods | • What resources are important for implementing particular methods? For example, staff, supplies and funding | • Researchers |
Data collection
Coders held meetings on a biweekly basis throughout the data collection period to discuss how to ensure that the codebook, application of codes and coded text were appropriate and accurately reflected what was heard in the focus groups and interviews. The community liaisons also attended these meetings to discuss recruitment challenges and solutions. With the consent of participants, we audio recorded the focus groups and interviews. Facilitators and interviewers also took notes during each session. Using the audio recordings, a transcriber organized and cleaned-up the notes, added detail and relevant quotes, and developed a session transcript. The interviewer or facilitator then reviewed the transcript for accuracy and completeness and added their own notes to provide overall insight about the session, for example, barriers to data collection and facilitators to emergent themes. We used NVivo 11 to collect, organize and conduct a systematic analysis of qualitative data from the focus groups and interviews. For data security, we stored session recordings and notes on a password-protected network at Westat that was accessible only to our team. We removed all identifying personal or affiliation data from transcribed documents and replaced it with generic data, for example, ‘name of facility’ replaced the actual organization name. We collected game board worksheets at the end of each session, limiting the potential for data tampering or missing data. We felt that removing incomplete or inaccurately completed worksheets made more sense than trying to add values; for that reason, we did not examine sensitivity of inferences of missing data.
Phase III: modified Delphi panel
In Phase III, we used a modified Delphi process to evaluate findings from the previous two phases and achieve consensus on a PETT recommendation. Developed by the RAND Corporation, the Delphi panel is a flexible research technique commonly used within the fields of health and social sciences [33]. In reviewing the composition and credentials of our nine-person stakeholder advisory board (SAB), we judged that our SAB members had the desired balance of representation and required level of expertise to serve successfully as Delphi panelists. Thus, we used our SAB as our Delphi panel. Each member met the following criteria: knowledge of health disparities or healthcare delivery systems (or both), skills to participate actively as PCOR advisor and/or ability to guide PCOR methods or patient engagement. In this capacity, panelists brought diversity of roles, comprising patients and patient advocates, patient caregivers, PCOR researchers, community leaders and industry leaders. Panelists also brought diversity of age, race and physical ability, as people of color and people with physical impairments were on the panel.
The technique uses a highly structured, multistage survey process involving interaction between group members (the panel) to reach consensus from multiple opinions. We modified the traditional Delphi process in three ways:
•
We conducted the first two rounds electronically by email instead of by mail or in person;
•
We conducted the third round as a group in person instead of anonymously;
•
Rather than provide feedback to a set of survey questions, we asked the panelists to give us feedback on an initial PETT consolidated from Phase I and II findings.
No standard method exists to calculate an appropriate size for a Delphi panel; rather, the number of panelists is determined by considering what is reasonable and viable based on study goal and resources available [34]. Based on these considerations and our targeted population, we concluded that eight to ten panelists would be an appropriate size for the scope of our study.
Data sources included a consolidated PETT that informed the first round of the process and feedback from each panelist from each of the three rounds. The consolidated PETT was a combination of findings from Phase I and II plus SAB individual input. We created a packet including all four PETTs from the prior two phases (environmental scan PETT from Phase I, and the patient PETT, researcher PETT and other stakeholders PETT from Phase II), the full qualitative results report and a blank game board to make individual recommendations for the final PETT. This packet was given to our SAB for review and evaluation. We asked each board member to complete the game board independently and return it to the research team. After compiling SAB submissions, we pulled the results into a consolidated PETT that was the basis for the first round of the three-round Delphi process.
Our research team followed the Delphi guidelines for independent review of results. To prevent bias, two investigators conferred on the synthesis of feedback from each panelist. Feedback was provided with a statistical representation of the group response. Each panelist responded independently and confidentially, except for the final round, which was conducted as a group in person.
In round 1, we asked panelists to provide feedback on the consolidated PETT, analyzed and summarized their responses, and communicated these findings back to the panel in a revised PETT to be reviewed in the next round. The process was repeated for round 2. We analyzed and summarized responses from round 3 and the collective opinion resulted in the final PETT and our recommendation. Round 1 and 2 of the Delphi process occurred through web conferencing with each panelist individually. Round 3 occurred in person as a group, with all panelists meeting at the institutional campus. Panelists had an equal role on the Delphi panel and feedback from each panelist was valued and weighted equally.
Results
Environmental scan (Phase I) results
We identified 8394 records meeting our search criteria, and through the literature review process illustrated in Figure 3, determined that 200 articles qualified for further analysis. From these articles, we identified data about patient engagement methods, methods to engage under-represented populations and related resources.
Phase I yielded an inventory of methods (quantitative data) for meaningful patient engagement. Descriptive results are further delineated in Tables 3 & 4. Table 4 presents the full list of methods and submethods developed from the methods inventory. We analyzed and organized findings into engagement method categories; these became the card deck for the card-mapping activity used in Phase II. Additionally, we developed an environmental scan PETT (Figure 5), which was considered our baseline PETT. This PETT represents the number (count) of studies from the literature review with the method-step pairing indicated by each cell. Although our study evaluated ‘appropriateness’, not ‘effectiveness’, we did overlay quality of evidence criteria (score) onto the results of the literature review (count).
| Qualifying criterion | Type and articles (n) |
|---|---|
| Which of the ten steps for patient engagement were discussed in the article? | • Topic prioritization: 81 • Data collection: 79 • Topic solicitation: 77 • Dissemination of study findings: 71 • Reviewing and interpreting results: 66 • Framing the question: 60 • Selection of comparators and outcomes: 49 • Analysis plan: 46 • Translation of results: 37 • Creating the conceptual framework: 32 |
| Was the article original research, prescriptive (editorial or commentary), or review (summary or literature review)? | • Original research articles: 178 • Prescriptive articles: 12 • Review articles: 10 |
| What methods of engagement were used to engage the patients/caregivers/stakeholders? | • Focus group: 73 • Advisory board: 54 • Research team member: 47 • Meetings: 44 • Individual interviews: 43 • Surveys/questionnaires: 32 • Small group interviews: 21 • Community partnership: 20 • Delphi process: 15 • Online/web: 12 • Telephone: 11 • Storytelling: 10 • Print materials: 8 |
| Breakout by role (‘are patients/caregivers recognized as partners in the process throughout the research cycle?’) | • Partner: 94 • Advisor: 55 • Reactor: 24 • Unsure/not clear: 23 • Study subject: 4 |
| Breakout by frequency (‘is patient/caregiver engagement continuous, throughout the research process and lifecycle? That is, it is not a one-time or sporadic event.’) | • Continuous: 72 • Sporadic: 60 • One time: 35 • Unsure/not clear: 17 |
| Breakout by evaluated meaningfulness (‘was patient/caregiver engagement meaningful?’) | • Meaningful: 105 • Partial: 55 • Unsure/not clear: 24 • Superficial: 16 |
| Breakout by area of impact (‘the author(s) reported that patient engagement influenced which of the following?’) Responses to area of impact may include multiple answers | • Data collection: 87 • Framing the question: 72 • Topic prioritization: 71 • Dissemination of study findings: 68 • Creating the conceptual framework: 64 • Topic solicitation: 63 • Reviewing and interpreting results: 57 • Selection of comparators and outcomes: 40 • Analysis plan: 40 • Translation of results: 40 |
| Method | Submethod(s) | Method | Submethod(s) |
|---|---|---|---|
| Community partnership | Collaborative research | Print material | Reviews |
| Community-engaged research | Books and textbooks | ||
| Patient and public involvement | Newspapers | ||
| Community action research | Academic journals | ||
| Patient-centered research | Press releases and advertising | ||
| Patient-centered outcomes research | Flyers, pamphlets and leaflets | ||
| Consumer research | Print multimedia | ||
| Participatory action research | Research member | Steering committee member | |
| Health services research | Community member or patient | ||
| Co-production | Story telling | Photo voice | |
| Community-based participatory research | Digital or visual | ||
| Focus group | Not applicable | Story boards | |
| Interview | Individual/personal interview | Scenario generation | |
| Small group interview | Storytelling through videos | ||
| Telephone | Skits and plays | ||
| Depth interview | Animation | ||
| Projective techniques | Text or talk and image | ||
| Key informant interview | Social media | Blogging | |
| Meeting | Community forums | Microblogging | |
| Advisory board | Location | ||
| Online webinars | Social networking | ||
| Trainings | Aggregators | ||
| Delphi process | Collaboration | ||
| Survey | Not applicable | Social multimedia |
Figure 5. Environmental scan patient engagement translation table.
Shading indicates average quality score quartiles from darkest (3.76–4) to progressively lightest (3.61–3.75), (3.43–3.6) and (2.86–3.42).
Average max quartile.
Card mapping & focus group (Phase II) results
Participants completed a demographic survey and the card-mapping activity individually (quantitative data), followed by the focus group or interview discussion (qualitative data). Findings resulted in three PETTs representing feedback from each study group; in other words, a patients’, researchers’ and other stakeholders’ PETT. We collected data from 160 total participants in our three study groups: 67 patients, 50 researchers and 43 other stakeholders. Participant’s ages ranged from 22 to 80 years (mean of 53 years). The majority of participants self-identified as black/African-American (59%), followed by white/Caucasian (43%), Latino/Hispanic (7.5%), Asian American/Pacific Islander (2.5%), American Indian/Alaskan Native (1.25%) and other/multiracial (2.5%).
We asked the researchers’ study group members about their source of funding and type of research. Of the 50 researchers, the majority reported that their funding source was the federal government (72% of members), followed by private entities and nongovernmental organizations (26%), pharmaceutical companies (16%), state or local government (12%) and other (10%). Many had several sources of funding. Common types of research they reported conducting included PCOR (74%); gender studies, sociology and behavioral science (58%); and clinical trials (26%). The majority of researchers (72%) reported that they had served as a principal investigator on a project.
We asked the other stakeholders’ group to describe their involvement with patients and community members. Other stakeholders may have chosen one or more categories. Of the 43 participants, 30% were counselors or social workers, 28% were caregivers, 21% were pastors or community leaders, 21% were healthcare providers, 7% were city planners or urban developers and 14% were other professions.
Supplementary Tables 1B, 1C & D present the game board results by study group; in other words, patients (Supplementary Table 1B), researchers (Supplementary Table 1C) and other stakeholders (Supplementary Table 1D). All participant results are combined in Figure 6. Note that we did not examine joint preferences of patients, researchers and other stakeholders as to what method(s) to use at which step of the research process. The color shading is simply a visual representation of the frequency count, making it easier to identify more quickly the frequency (popularity) of responses – in other words, the darker the shading the higher the frequency count and the lighter the shading the lower the count (less popular the method). Please note that the color shading has a different meaning for the Phase II tables than Phase I: Supplementary Tables 1B, 1C & D & Figure 6 reflect popularity or frequency rather than the results in the Phase I table, Figure 5, which reflects quality of evidence.
General considerations
Supplementary Table 1E presents general considerations expressed by participants, which emerged from our qualitative evaluation by two independent evaluators using NVivo software. Investigators should keep these key themes in mind when planning and implementing a patient engagement method. Supplementary Table 1E shows the number of times a concept was mentioned in a session, broken out by patient, researcher or other stakeholder. Key themes are listed in descending order of frequency, in other words, most frequently mentioned to least frequently mentioned by researchers.
Patients, researchers and other stakeholders expressed general considerations about the time, staff and financial resources that should be in place before patient- and community-engagement methods are implemented. Researchers emphasized the importance for funders and academia to recognize the need to integrate stakeholder engagement into research budgets and infrastructure. They also stressed that investigators need specialized training in patient- and community-engagement methods, as these principles and ethics are not typically incorporated into academic training programs.
Specific to community resources, all three study groups mentioned transportation, childcare, incentives and interpreters when needed. Researchers stated that allotting sufficient time for pre- and continuous engagement was a necessity. Researchers also pointed to the need for dedicated funding to support staff for community engagement activities, as well as the need to hire staff to conduct this engagement work.
Patients and other stakeholders, and to a greater extent, researchers thought that using a variety of methods is the most appropriate approach because of individual differences in participants, communities and research questions. All three study groups stressed the importance of accommodating the community, and its needs, interests and practical limitations.
Researchers in particular (and to a lesser extent, patients and other stakeholders) expressed the need for bidirectional communication and information sharing. Many researchers encouraged the involvement of a community representative to liaise between the community and investigators. Only researchers described the need for institutions and funders to build infrastructural support for engagement work.
Patients, researchers and other stakeholders stressed the importance of accommodating the community and their needs, interests and limitations when selecting engagement methods. Additional considerations included promoting transparency between researchers and the community, including sharing information and findings; being flexible with scheduling events and data collection activities; and selecting locations that are accessible to the community.
Modified Delphi process (Phase III) & final PETT
We used a Delphi panel to achieve consensus on a final PETT after three rounds of review and feedback. A consolidated PETT was the basis for the first round of the process, informed by findings from the previous two phases, in other words, the baseline PETT from Phase I and the three participant subgroup PETTs from Phase II.
After two rounds of review and refinement, the result of the third and final round was our recommended PETT (Figure 4). It should be noted that the color shading for Figure 4 has a different meaning than that of the tables for Phase I and II, representing the level of recommendation for each method(s) that participants identified as appropriate for each step. Specifically, the darker shading demonstrates that we had more evidence to recommend the appropriateness of that method at that particular step. If the shading is white, we had no evidence to recommend appropriateness. We remind readers that these summary recommendations are based on a combination of findings from all three research phases; they are not evidence that these are the most effective methods to use. Figure 4 was informed by the five separate PETTs developed in the prior two phases: environmental scan PETT (Phase I) plus the patient PETT, researcher PETT, other stakeholders PETT and aggregate PETT (Phase II). Figure 4 reflects the perceived appropriateness of a particular method throughout the ten-step framework by the shading gradient: the darker shading can be interpreted as receiving larger numbers of responses during Phase II from researchers, patients and other stakeholders.
Figure 4 and the corresponding legend demonstrate the recommendations based on the shading of the PETT. The recommended PETT suggests that community partnerships are the single most appropriate engagement methods for all research steps, especially in the planning (steps 1–3) and sharing (steps 8–11) stages. Meetings are generally important and equally appropriate for every step (steps 1–11) of the research process. For the research step of describing patients’ journey, every engagement method is generally important and equally appropriate.
Discussion
General considerations
Patients and other stakeholders, and to a greater extent, researchers thought that using a variety of methods is the most appropriate approach because of individual differences in participants, communities and research questions. All study groups stressed the importance of accommodating the community, and its needs, interests and practical limitations.
Researchers in particular (and to a lesser extent patients and other stakeholders) expressed the need for bidirectional communication and information sharing. Many researchers encouraged the involvement of a community representative to liaise between the community and investigators. Only researchers described the need for institutions and funders to build infrastructural support for engagement work.
Patients, researchers and other stakeholders stressed the importance of accommodating the community, and their needs, interests and limitations when selecting engagement methods. Additional considerations included promoting transparency between researchers and the community, including sharing information and findings; being flexible with scheduling events and data collection activities; and selecting locations that are accessible to the community.
General populations
Participants across all study groups reported that all methods can be appropriate for general populations. Participants stated that meetings and focus groups are useful for all populations, given that they are easily administered and can be conducted at participants’ convenience.
Researchers reported that interviews and surveys are appropriate engagement methods for general populations. Both methods can be conducted via various modalities (web and phone), with a variety of participants (regardless of age, mobility and economic status), and at participants’ convenience. Researchers reported that research members and community partnerships provide guidance, input and clarification during all stages of the research process for all populations. Researchers also stated that sharing and hearing personal stories through storytelling methods is universally appealing.
Cross-cutting considerations
Patients, researchers and other stakeholders described the power of bidirectional conversation between researchers and community members, especially face-to-face communication. Many methods were listed as having this element, which encourages trust and engagement in the study. These methods included focus groups, interviews, meetings, storytelling, research members and community partnerships. All three study groups expressed the importance of including the community in all research steps, especially with regard to community partnerships and involvement of the research members.
All three study groups explained the need to tailor each method to the target community’s needs, interests, culture, language and practical considerations. This sentiment was expressed in relation to print materials, social media, storytelling, surveys, research members and community partnerships. All study groups felt that print materials and social media were of limited value for community engagement, and researchers also felt that surveys were limited in how effectively they engage the target population.
Interviews, meetings and community partnerships were listed by all three study groups as useful methods for pretesting study materials to ensure that they reflect the interests and requirements of the community. Researchers also mentioned storytelling as useful for pretesting materials, especially to confirm that researchers have the questions, phrasing and other details correct. Only researchers, however, explained that focus groups, interviews and surveys can be resource intensive and may not be cost-effective, given one’s research goals.
Subpopulations
All three study groups stated that all methods, with modifications, can be used for specific populations. Patients, researchers and other stakeholders did not discuss research members, community partnerships and meetings in the context of specific populations, because they regarded these methods as applicable to general populations.
All study groups reported that social media can be beneficial to engage younger populations, as elderly populations may not be as familiar with this method. Members of established patient or community organizations may be familiar with social media. All study groups also stated that surveys can be used to reach those with lower literacy, as surveys can be read by data collectors or administered through special software. Surveys may also be conducted online, such as through Facebook, which may appeal to younger participants.
Researchers, and to a lesser extent patients and other stakeholders, suggested that interviews can be useful to engage those with lower literacy skills or who are experiencing sensitive and stigmatizing conditions. As interviews can be conducted at locations convenient to participants, this method can be appropriate for hard-to-reach populations and individuals with various impairments or chronic conditions. Researchers, and to a lesser extent patients and other stakeholders, stated that storytelling can be an effective engagement method for patient groups and underserved populations, as this method provides an opportunity for those whose voices are not heard, to share their experiences.
Researchers reported that print materials could be useful for patients with chronic conditions or those who belong to patient organizations, as those populations may be accustomed to reviewing print materials as health communication tools. Researchers also stated that the focus group process and guides can be modified to address the unique needs of those with lower literacy skills or various impairments or chronic conditions.
Applicable methods
All three study groups identified community partnerships, research members, focus groups, interviews and meetings as applicable methods across all research steps. Patients, researchers and other stakeholders stated that focus groups and meetings can be used to ask targeted questions across the steps.
Researchers said that incorporating community partnerships and research members ensures that the research process, plans, instruments and materials are patient and community centered. Patients and other stakeholders reported that surveys can be used across all steps, as they can be easily administered and completed by participants. Patients and other stakeholders stated that social media is appropriate to use throughout the steps, as it provides a platform to readily send or receive information.
Patients and other stakeholders, and researchers to a lesser extent, stated that community events such as bake sales and health fairs are effective ways to advertise projects to the community, recruit participants and disseminate information. Patients and other stakeholders reported that engaging and disseminating information to the community can be done via TV and radio advertisements and announcements. Radio reading and personalized talking print phone services can be used to advertise projects and recruit blind or visually impaired patients. Community door-to-door outreach and word-of-mouth were also cited by patients and other stakeholders as methods to recruit hard-to-reach patients and to build trust with the community.
Resources
Patients, researchers and other stakeholders felt that giving monetary incentives was necessary to engage patients and stakeholders. Only patients and other stakeholders suggested that providing transportation and childcare are important considerations for which researchers should plan. Only researchers described the need for funding to support staff time for community engagement (both pre-engagement and continuous).
Engagement methods
Patients and other stakeholders, and to a greater extent researchers, stated that funding is necessary for conducting focus groups and meetings and for covering other direct costs such as honoraria, meals and transportation reimbursement. Researchers also stated that funding for direct costs was necessary for surveys, community partnerships and interviews.
Researchers stated the need for study staff to have expertise in implementing the engagement method. All three study groups saw meeting facilitation and social media as requiring trained staff. Researchers stated that other engagement methods also required expertise to implement well, including focus groups, interviews, print materials, storytelling, surveys and research members.
All three study groups acknowledged that studies are strengthened when community partnerships are in place before the research seeks to engage stakeholders. As such, study budgets should account for time, services and other resources that are needed for pre-engagement and continuous community engagement. Patients, researchers and other stakeholders stated that pre-engagement and continuous community engagement were important to recruit for interviews and meetings, and in terms of web-based connections, to increase the impact of social media. Researchers also mentioned focus groups and community partnership as methods requiring continued investment in the target community.
Regarding focus groups, all study groups felt that making the method convenient to the participant was important. Researchers also reported that participants’ convenience was important for interviews. Only researchers expressed the need for investigators to give something to the community in return for research participation when implementing community partnerships; ways of giving back to the community include stipends, training and education. Researchers were also the only group that advocated involvement of a patient or other stakeholder as a research member in the study.
Interpreting the PETT
A few examples of how researchers might interpret and apply our recommended PETT (Figure 4) as a tool in their own work:
•
Community partnerships are the single most appropriate engagement method for seven (dark purple) of the 11 research steps – in other words, steps 1–3 (planning) and 9–11 (sharing);
•
Including patients and other stakeholders as ‘research members’ is generally important for all research steps, and especially appropriate for four (dark purple) of the steps – in other words, steps 6–9 (research);
•
Meetings are generally important and equally appropriate (medium purple) for every step of the research process – in other words, steps 1–11 (planning, sharing and research);
•
For the planning stage, interviews are the most appropriate method for step 1 (dark purple) and storytelling is the most appropriate for step 3 (dark purple);
•
Print materials are most appropriate in the research stage for describing the patient journey (step 5 in medium purple), and in the sharing stage for translation and dissemination (steps 9 and 10 in medium purple, respectively). By contrast, print materials were less recommended for other steps of the research process.
Study limitations
Limited evidence & potential bias
We noted that respondents underscored the significant amount of time and financial resources necessary to implement the various engagement methods; however, participants were not able to provide empirical data associated with implementing these different engagement methods. Additionally, evidence about the impact of various methods on research and health outcomes is extremely thin; thus, we were unable to evaluate the impact of these methods in the literature. However, the literature review in Phase I yielded valuable data as we were able to ascertain what methods were used to engage patients and stakeholders in answering a primary research question.
Furthermore, our process for title and abstract screening may have excluded eligible studies meeting our screening criteria in instances where only a title was available or the title/abstract was ambiguous. In these instances, a full text may have been needed to confirm eligibility. Excluding these prospective studies may have introduced publication bias into our selection process. Additionally, potential bias or lack of knowledge about patient engagement or PCOR tools could have influenced data collection, analysis and reporting.
Limited representativeness
As mentioned, one of our intentions was to identify ineffective methods in Phase I; however, this information was not reported at the time of our study. The patients’ and other stakeholders’ study groups were also recruited predominantly in the Baltimore area, which means that our results might therefore not be nationally representative.
We believed strongly that it was important to ensure that other stakeholders’ voices were represented in the focus groups and interviews in Phase II, in addition to the critical patients’ and researchers’ voices. With an unlimited budget, we might have further stratified our other stakeholders’ study group into specific subgroups (e.g., insurance, pharmacy and policy); however, our budget allowed for only a general ‘other stakeholders’ study group. Blue Cross was represented on our Delphi panel in Phase III and we had payer representation in the other stakeholders’ study group in Phase II, but we were unable to get a study group exclusively comprising payers to represent this perspective. This limited representation may affect the feasibility of many PCOR interventions.
Conclusion
To further assist PCOR researchers with selecting evidence-based engagement methods requires more definitive results regarding which methods work best when and for which populations and questions. The ‘appropriate’ methods for advancing trustworthiness in research may be different from any ‘appropriate’ approaches for improving health or decision making. For that reason, we must first decide what metric(s) to use as the criteria for an ‘appropriate engagement method’.
No matter how much new information evolves, it will never produce a single ‘preferred’ engagement method. That is true both in general and for each step along the PCOR continuum. The reasons are that the ‘preferred’ method will depend not only on the question and study type but also on the population being engaged. Some evidence exists already and, even more evidence can be expected in the future, that the most appropriate methods will differ by subgroups such as age, race or ethnicity, and physical and cognitive impairments. The effectiveness of evidence-based methods of patient engagement in research varies among the steps in the research process. Our PETT is a tool that assists investigators in deciding how to successfully engage patients and other stakeholders at different points in the research process.
•
Patient engagement methods can – and should – be different depending on the type of study and numerous other factors such as the target population, geography, healthcare setting and social determinants of health.
•
Literature-cited engagement methods included: community partnership, focus group, interview, meeting, print material, social media, storytelling, survey and research member.
•
Focus group, interview and meeting were the most frequently identified methods in the literature, with community partnership, survey and research member as moderately identified methods in the literature. Print material, social media and storytelling were the least identified methods in the literature.
•
Two methods – community partnerships and including patients and stakeholders as research team members – were highly rated methods for each step. For specific research steps, the following methods received high ratings:
○
Planning research: focus groups, interviews, meetings and surveys;
○
Conducting research: meetings;
○
Sharing results: focus groups, meetings and storytelling;
○
Evaluating patient engagement: focus groups, interviews and surveys.
•
Patients, stakeholders, and to a greater extent, researchers thought that using a variety of methods is the most appropriate approach because of individual differences in participants, communities and research questions. All three study groups stressed the importance of accommodating the community, and its needs, interests and practical limitations.
•
Additional considerations included promoting transparency between researchers and the community, including sharing information and findings; being flexible with scheduling events and data collection activities; and selecting locations that are accessible to the community.
Acknowledgments
The co-authors acknowledge the following contributors: J Levy, T Joseph and K Yang.
Financial & competing interests disclosure
Research reported in this publication was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1409-20792). The statements in this work are solely the responsibility of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
The authors state that they have obtained appropriate institutional review board approval. The University of Maryland Baltimore Institutional Review Board (IRB) determined Phase I of the study (protocol HM-HP-00061232) to be exempt on 28 October 2014. Phase II of the study was approved for expedited review on 8 November 2016. Continuing review was approved on 15 November 2017, and again on 12 October 2018, expiring on 5 October 2019. Informed consent obtained from study participants was verbal, as approved by the IRB, because the research presented no more than the minimal risk of harm and the principal risk was a breach of confidentiality. Informed consent was obtained from the participants involved.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/
Supplementary Material
File (supplemental tables.docx)
- Download
- 978.21 KB
References
Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Forsythe LP, Carman KL, Szydlowski V et al. Patient engagement in research: early findings from the Patient-Centered Outcomes Research Institute. Health Aff. (Millwood) 38(3), 359–367 (2019).
•• Analyzes 126 peer-reviewed articles that include descriptions of Patient-Centered Outcomes Research Institute research teams’ experiences with engagement.
2.
Conway PH. How the Recovery Act’s federal coordinating council paved the way for the Patient-Centered Outcomes Research Institute. Health Aff. (Millwood) 29(11), 2091–2097 (2010).
3.
Carman KL, Dardess P, Maurer M et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff. (Millwood) 32(2), 223–231 (2013).
4.
Frank L, de Wit M, Kirwan JR, Guillemin F, Bartlett SJ. Author response: patient engagement. Value Health 20(10), 1433 (2017).
5.
Frank L. Active patient engagement in research. Health Aff. (Millwood) 32(2), 438–439 (2013).
6.
Kirwan JR, de Wit M, Frank L et al. Emerging guidelines for patient engagement in research. Value Health 20(3), 481–486 (2017).
7.
Clauser SB, Gayer C, Murphy E, Majhail NS, Baker KS. Patient centeredness and engagement in quality-of-care oncology research. J. Oncol. Pract. 11(3), 176–179 (2015).
8.
Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez KA. The PCORI engagement rubric: promising practices for partnering in research. Ann. Fam. Med. 15(2), 165–170 (2017).
•• Defines principles of engagement; definitions of stakeholder types; key considerations for planning, conducting and disseminating engaged research; potential engagement activities; and examples of promising practices from Patient-Centered Outcomes Research Institute-funded projects.
9.
Shelef DQ, Rand C, Streisand R et al. Using stakeholder engagement to develop a patient-centered pediatric asthma intervention. J. Allergy Clin. Immunol. 138(6), 1512–1517 (2016).
10.
Forsythe LP, Frank LB, Workman TA, Hilliard T, Harwell D, Fayish L. Patient, caregiver and clinician views on engagement in comparative effectiveness research. J. Comp. Eff. Res. 6(3), 231–244 (2017).
•• Describes a survey study on patient, caregiver and clinician views toward engagement as partners in health research.
11.
Domecq JP, Prutsky G, Elraiyah T et al. Patient engagement in research: a systematic review. BMC Health Serv. Res. 14, 89 (2014).
12.
Forsythe LP, Szydlowski V, Murad MH et al. A systematic review of approaches for engaging patients for research on rare diseases. J. Gen. Intern. Med. 29(Suppl. 3), S788–S800 (2014).
13.
Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez KA. Advisory Panel on Patient Engagement (2013 inaugural panel). The PCORI engagement rubric: promising practices for partnering in research. Ann. Fam. Med. 15(2), 165–170 (2017).
14.
Tunis SR, Messner DA, Mohr P, Gliklich RE, Dubois RW. A translation table for patient-centered comparative effectiveness research: guidance to improve the value of research for clinical and health policy decision-making. J. Comp. Eff. Res. 1(3), 259–262 (2012).
15.
House Office of the Legislative Counsel. Compilation of Patient Protection and Affordable Care Act: extracted sections concerning patient-centered outcomes research and the authorization of the Patient-Centered Outcomes Research Institute (PCORI) Washington, DC (2015). https://www.pcori.org/assets/PCORI-Authorizing-Legislation-0323101.pdf
16.
Faridi Z, Grunbaum JA, Gray BS, Franks A, Simoes E. Community-based participatory research: necessary next steps. Prev. Chronic Dis. 4(3), A70 (2007).
17.
Cornwall A, Jewkes R. What is participatory research? Soc. Sci. Med. 41(12), 1667–1676 (1995).
18.
Cawston PG, Mercer SW, Barbour RS. Involving deprived communities in improving the quality of primary care services: does participatory action research work? BMC Health Serv. Res. (2007).
19.
Mockford C, Staniszewska S, Griffiths F, Herron-Marx S. The impact of patient and public involvement on UK NHS health care: a systematic review. Int. J. Qual. Heal. Care 24(1), 28–38 (2012).
20.
Crocker JC, Boylan AM, Bostock J, Locock L. Is it worth it? Patient and public views on the impact of their involvement in health research and its assessment: a UK-based qualitative interview study. Heal. Expect. 20(3), 519–528 (2017).
• Explores the views of public and patient involvement contributors involved in health research regarding the impact of public and patient involvement on research, whether and how it should be assessed.
21.
Stewart D, Wilson R, Selby P, Darbyshire J. Patient and public involvement. Ann. Oncol. 22(Suppl. 7), 54–56 (2011).
22.
NHS National Institute for Health Research. Patient and public involvement in health and social care research: a handbook for researchers (2014). https://www.nihr.ac.uk/about-us/CCF/funding/how-we-can-help-you/RDS-PPI-Handbook-2014-v8-FINAL.pdf
23.
Mullins CD, Abdulhalim AM, Lavallee DC. Continuous patient engagement in comparative effectiveness research. JAMA 307(15), 1587–1588 (2012).
•• Provides an overview and ten-step framework for continuous stakeholder engagement in the design, development, dissemination and evaluation stages of comparative effectiveness research.
24.
Liberati A, Altman DG, Tetzlaff J et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6(7), e1000100 (2009).
25.
Mullins CD. Integrating patients’ voices in study design elements with a focus on hard-to-reach populations Washington, DC. https://www.pcori.org/sites/default/files/IntegratingPatientsVoices.pdf
26.
Newhouse R, Goodman S, Aronson N, Patient-Centered Outcomes Research Institute et al . PCORI methodology report (2017). https://www.pcori.org/sites/default/files/PCORI-Methodology-Report.pdf
27.
Patient-Centered Outcomes Research Institute (PCORI). PCORI engagement rubric. Publ. Febr. 4, 2014. Updat. Oct. 12, 2015 (2015). https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf
28.
Perfetto EM, Oehrlein EM, Beal A et al. Assessing meaningful patient engagement in drug development: a definition, framework, and rubric, M-CERSI conference on patient-focused drug development planning committee (2015). https://www.pharmacy.umaryland.edu/media/SOP/wwwpharmacyumarylandedu/centers/cersievents/pfdd/mcersi-pfdd-framework-rubric.pdf
29.
Mooney M, Crane K. Community resource mapping (NCSET essential tools) (2005). https://conservancy.umn.edu/handle/11299/172995
30.
Briggs L, Huang Y. Asset-based community maps: a tool for expanding resources in community health programs. Pedagog. Heal. Promot. 3(3), 195–201 (2017).
31.
Fiechtner L, Puente GC, Sharifi M et al. A community resource map to support clinical–community linkages in a randomized controlled trial of childhood obesity, Eastern Massachusetts, 2014–2016. Prev. Chronic Dis. 14, 160577 (2017).
32.
Skinner B, Crane K. Community resource mapping: a strategy for promoting successful transition for youth with disabilities (NCSET information brief) (2003). https://conservancy.umn.edu/handle/11299/172853
33.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J. Adv. Nurs. 32(4), 1008–1015 (2000).
34.
McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int. J. Clin. Pharm. 38(3), 655–662 (2016).
Information & Authors
Information
Published In
Pages: 999 - 1017
PubMed: 34082571
Copyright
© 2021 Hillary A Edwards. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 19 February 2021
Accepted: 26 April 2021
Published online: 4 June 2021
Keywords:
Topics
Authors
Funding Information
Patient-Centered Outcomes Research Institute: ME-1409-20792
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
What works when: mapping patient and stakeholder engagement methods along the ten-step continuum framework. (2021) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2021-0043
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Krista B. Highland, Janiece L. Taylor, Keri F. Kirk, Lisa M. Harris, Christopher Ryan Phillips, Megan O’Connell, Stephen W. Kay, Nathan Turner, Isabelle Hasty, Julee A. Rendon, A Patient-Level Simulation Tool to Inform Data-Driven Pain Treatment Decisions and Policy in the US Military Health System, Value in Health, 10.1016/j.jval.2026.01.021, 29, 4, (575-582), (2026).
- Marik Moen, Kathena Storm, Donald Young, Rosalind Haines-Jones, Darin Loveland, Sarah Kattakuzhy, Establishing a research advisory committee in addictions services with a focus on social needs and social isolation, Research Involvement and Engagement, 10.1186/s40900-025-00770-8, 11, 1, (2025).
- Abdou Simon Senghor, Michelle Medeiros, Claudia Baquet, Franklin Lance, C. Daniel Mullins, A Proposal for Addressing Bioethical Concerns Along the 10‐Step Framework for Community Engagement, Health Expectations, 10.1111/hex.70345, 28, 4, (2025).
- Meagen Rosenthal, Future Research and Practice Changes, The Pharmacist’s Pocket Guide for Diabetes and Obesity, 10.1007/978-3-031-88835-9_17, (355-371), (2025).
- Richard Boulton, Antonina Semkina, Fiona Jones, Nick Sevdalis, Expanding the pragmatic lens in implementation science: why stakeholder perspectives matter, Implementation Science Communications, 10.1186/s43058-025-00730-z, 6, 1, (2025).
- Karim Lekadir, Alejandro F Frangi, Antonio R Porras, Ben Glocker, Celia Cintas, Curtis P Langlotz, Eva Weicken, Folkert W Asselbergs, Fred Prior, Gary S Collins, Georgios Kaissis, Gianna Tsakou, Irène Buvat, Jayashree Kalpathy-Cramer, John Mongan, Julia A Schnabel, Kaisar Kushibar, Katrine Riklund, Kostas Marias, Lameck M Amugongo, Lauren A Fromont, Lena Maier-Hein, Leonor Cerdá-Alberich, Luis Martí-Bonmatí, M Jorge Cardoso, Maciej Bobowicz, Mahsa Shabani, Manolis Tsiknakis, Maria A Zuluaga, Marie-Christine Fritzsche, Marina Camacho, Marius George Linguraru, Markus Wenzel, Marleen De Bruijne, Martin G Tolsgaard, Melanie Goisauf, Mónica Cano Abadía, Nikolaos Papanikolaou, Noussair Lazrak, Oriol Pujol, Richard Osuala, Sandy Napel, Sara Colantonio, Smriti Joshi, Stefan Klein, Susanna Aussó, Wendy A Rogers, Zohaib Salahuddin, Martijn P A Starmans, FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare, BMJ, 10.1136/bmj-2024-081554, 388, (e081554), (2025).
- Nabil Natafgi, Katie Parris, Erin Walker, Tracey Gartner, Jeanette Coffin, Ariana Mitcham, Luis Sanchez Ferrer, Maushmi K. Patel, Haley Wymbs, Ann Blair Kennedy, Through Their Eyes: Using Photovoice to Capture the Capacity‐Building Journey of Long Covid Patient Experts, Health Expectations, 10.1111/hex.70094, 27, 6, (2024).
- Joe Vandigo, Hillary A. Edwards, Bridget Seritt, Kavita V. Nair, Addressing gaps to strengthen patient-centricity in formulary decision-making: An example applied to Colorado’s prescription drug affordability board implementation, Journal of Managed Care & Specialty Pharmacy, 10.18553/jmcp.2024.30.10.1189, 30, 10, (1189-1190), (2024).
- Ann Blair Kennedy, Ariana Mitcham, Katherine Parris, Faith Albertson, Luis Sanchez Ferrer, Conor O'Boyle, Maushmi K. Patel, Tracey Gartner, Amy M. Broomer, Evan Katzman, Jeanette Coffin, Jennifer T. Grier, Nabil Natafgi, Wonderings to research questions: Engaging patients in long COVID research prioritization within a learning health system , Learning Health Systems, 10.1002/lrh2.10410, 8, S1, (2024).
- T. Joseph Mattingly, A Research Framework to Improve Health Disparity Evidence Gaps in Value Assessments, PharmacoEconomics, 10.1007/s40273-023-01340-0, 42, 3, (253-259), (2023).