Skip to main content
Free access
Research Article
10 July 2019

Engaging parent–child dyad and healthcare provider stakeholders in a patient-centered comparative effectiveness study


Aim: Describe engagement activities in a comparative effectiveness study evaluating two interventions for promoting psychosocial health among youth ages 10–17 who have recently experienced a nonintentional injury. Methods: Institutional, community and patient stakeholders from four children’s hospitals were engaged through consultation meetings, individual interviews and a collaborative meeting. Results: 67 engagement activities were conducted across four hospitals. Feedback to improve recruitment, retention and continuous engagement in the study was obtained. Finally, disseminating study interventions to school and healthcare settings, and adding alternative delivery formats were identified as priority next steps. Conclusion: Results highlight diverse methods of engaging patient and professional stakeholders, critical recommendations for improving study engagement and retention, and future directions for this patient-engaged comparative effectiveness research.
Patient and stakeholder engagement has been increasingly recommended as a strategy to improve the relevance of research questions, transparency of research activities and to accelerate the adoption of evidence into practice [1–3]. While efforts to make comparative effectiveness research (CER) patient centered have emphasized thoughtful consideration of the stakeholders selected, the type of information to be exchanged and the trajectory of engagement throughout the research process, much of the engagement literature has focused on intervention development and formative research [4–6]. Less commonly described are patient and stakeholder engagement activities throughout comparative effectiveness study implementation, dissemination and future direction stages of research in multisite trials. This paper describes the engagement of institutional and community stakeholders, parent–child dyad patient consultants, and clinical experts in the community to gather feedback on study implementation, study interventions sustainability and adaptation, and dissemination of research findings. The purpose of this paper is: to describe how these engagement activities influenced the implementation, dissemination and future directions of the present research study; and to highlight challenges and lessons learned in this process in order to guide patient and professional engagement strategies in future research studies.

Overview of CER study design: evaluating link for kids

The engagement activities described in this paper pertain to a CER study that was designed to evaluate the effectiveness of two programs: Link for Injured Kids program; and a trauma education booklet, ‘So Your Child Has Been in an Accident… A Book for Parents about Dealing with Accidents’. The programs were tested among youth ages 10–17 who had recently experienced an unintentional injury and received medical treatment in a pediatric inpatient unit. Every year, over 9 million children seek emergency care for injuries, and an average of 12,000 children die from their injuries. Of those seeking emergency care, 225,000 children necessitate further hospitalization [7,8]. The injury event, pain and physical symptoms, hospitalization, medical interventions utilized, and a lengthy and socially isolating recovery period can contribute to a number of chronic psychological conditions, of which post-traumatic stress disorder (PTSD) is the most prominent [9,10]. PTSD and other post-traumatic comorbid disorders can severely impact overall health and wellbeing long after the physical injuries have healed [11,12]. If left unaddressed, these psychological wounds follow children into adulthood, placing them at greater risk for mental health and social functioning problems throughout their lives [13,14]. In practice, very little, if any, post-trauma support is provided to parents and children after an injury hospitalization. Parents and providers strongly suggest the provision of interventions in the hospital setting before discharge to support an injured child’s process of emotional recovery [15].
The first intervention, Link for Injured Kids, was developed to promote psychosocial health and prevent traumatic stress among recently injured youth [15]. Link for Injured Kids was developed by modifying the Listen Protect Connect (LPC) intervention, a program of psychological first aid that involves listening skills and referral to advanced care used in school settings [16], for use in acute hospital settings. The empirical question of whether the LPC intervention could be translated to the pediatric unintentional injury patient population was initially posed by a team of researchers who were also involved in school-based LPC intervention [15]. The original LPC program was modified into a parent-provided intervention that included motivational interviewing skills, screening for nonspecific stress and referral to advanced care. The motivational interviewing skills were specifically adapted from a parent-based intervention for safe teen drivers [17]. The second intervention, a trauma education booklet entitled, ‘So Your Child Has Been in an Accident…’, is an early intervention strategy that provides education about trauma, its impact and referral to resources. Given time and other constraints of hectic hospital settings and busy families, educational materials are designed to be simple and cost effective [18,19].
Prior to implementation, the components of Link for Injured Kids were reviewed by a pilot group of parents and nurses, who identified communication and screening as the two most relevant skills for this patient population. After the program was developed, additional focus groups were conducted with caregivers of children with a recent injury and pediatric trauma care providers [15]. This feedback provided valuable insight into overcoming recruitment and training barriers, as well as the quality of the intervention materials themselves. The current study’s expansion to additional institutions was predicated on this feedback and the increasing need for trauma-informed interventions in healthcare settings [20], and in turn continued to generate important perspectives for future intervention iterations and implementations.
Trauma education is composed of a booklet that contains information about injuries, its psychosocial impacts and referral to services. Link for Injured Kids trains parents in specific motivational interviewing skills and monitoring skills to promote their children’s full emotional recovery from injury. Specifically, the intervention aims to help parents effectively affirm their child’s emotions, encourage adaptive problem solving and coping with daily hassles, and identify symptoms of distress that require professional intervention. It also equips parents with medical and mental health resources in their community. The intervention materials are interactive, which allow the parent to rehearse and receive immediate performance feedback from the interventionist regarding their reflective listening skills, as well as to discuss anticipated barriers to implementing these skills. In order to examine the comparative effectiveness of these interventions, participants were randomized to either the Link for Injured Kids intervention or the Trauma Education intervention. Both Link for Injured Kids and Trauma Education participants received a booster session, specific to their intervention condition, by phone at 6 weeks’ follow-up. The purpose of booster sessions was to ask about children’s emotional recovery since leaving the hospital, to determine the level of utilization of study materials and to facilitate continued skill development for parents in the Link arm. Across treatment conditions, both caregiver and child were asked to complete baseline and follow-up measures of behavioral problems, health-related quality of life, symptoms of depression and PTSD, and coping strategies. Follow-up occurred at 6 weeks, 3 months and 6 months following baseline.

Patient & stakeholder partner engagement model

This paper explores the perspectives of parent–child dyad patient partners and community, and clinician stakeholders from four clinical sites regarding the design, implementation and dissemination of this CER study. Our approach to patient and stakeholder engagement draws on frameworks proposed by Deverka et al. [5] and Frank [1] that equally weigh both expert and lay perspectives, as well as quantitative and qualitative approaches, to synthesizing evidence of engagement. These models highlight that the intended product of stakeholder and expert input in the research process is to make decisions about the research design, implementation strategies, as well as dissemination and adaptation. Below are the Patient-Centered Outcomes Research Institute’s (PCORI) definitions for stakeholder and partners [21]:
Stakeholder partners include members of constituencies based on professional, rather than personal, experience. For example these may be clinicians, purchasers, payers, industry, hospitals and health systems, policy makers and training institutions.
Patient partners include:
patients (those with lived experience);
potential patient partners;
family members;
the organizations that are representative of the population of interest in a particular study.
Potential patient partners are children/parents with similar demographics as those who suffered an injury but were not injured per se during the study period. This approach is encouraged by PCORI to broaden the pool of stakeholders who may have perspectives that differ from those with actual lived experiences. It is important that patient partners are not confused with patient subjects; patient partners are members of the research team and involved in the planning, conduct and dissemination of the research, whereas patient subjects are those individuals actually enrolled into the study as participants.


This study was conducted at four children’s hospitals in the Midwest: University of Iowa Stead Family Children’s Hospital (Level I Pediatric Trauma Center), Blank Children’s Hospital (Level II Pediatric Trauma Center), Children’s Mercy Kansas City (Level I Pediatric Trauma Center) and Children’s Minnesota (Level I Pediatric Trauma Center). Professional and patient stakeholders were engaged through: informal and formal meetings with hospital and community trauma-informed care and injury prevention groups; individual interviews with parent–child patient partners, who were recruited after they completed study participation, and clinical experts from community settings; and a collaborative research and community engagement stakeholder meeting comprised of researchers from the four sites, parent–child patient partners, hospital and community partners. All engagement activities were reviewed by the primary site’s Institutional Review Board (IRB) at the University of Iowa, and site-specific IRBs, when applicable. Engagement activities were not considered human subjects research at the sites where activities were conducted. Each of these engagement activities are described below.

Data tracking of engagement activities

All engagement activities were summarized in an engagement-tracking form maintained in a REDCap database developed by the study team. The tracking form contained descriptive variables related to the engagement event (i.e., date, title of event, occurrence during study period, attendance, communication methods, partnership type, topics discussed and lessons learned). Engagement entailed four primary activities: initial consultation meetings with hospital and community stakeholder groups; informal and formal meetings with institutional stakeholders at each clinical site; individual interviews with patient and community partners; and a culminating collaborative stakeholder meeting. Each clinical research site was responsible for entering records of engagement activities specific to their site at the time the engagement event occurred. The results reported in this paper include information from 67 unique engagement activities that were recorded during the study period (1 February 2014 through 31 March 2018). Frequencies of descriptive statistics (e.g., types of activities, occurrence within study period) were calculated using Stata for quantitative data [22] and are presented in Table 1.
Table 1. Stakeholder involvement by topic and study period.
Study componentInstitutional stakeholders (n = 45)Community stakeholders (n = 9)Patient/parent stakeholders (n = 12)Other: all-site engagement meeting (n = 1)
Study topic area involvement, indicated by ‘X’
Survey designXXXX
Intervention booster callsXXX
Additional dataXXX
Reporting resultsXXXX
Manuscript collaborationX
Occurrence within study period
Involvement represents number of activities, not number of individuals.
As a result of these 12 patient partner activities, 27 individual interviews were conducted.

Initial consultation meetings with hospital & community stakeholder groups

Following notice of the study award, the two initial clinical research sites identified relevant institutional and community stakeholders to consult in the development and implementation of the study in their hospital setting. In general, these initial consultation meetings focused on the study procedures, timeline and gathering feedback on how to enhance feasibility at each of the sites. These initial meetings occurred in April 2016, and continued throughout the study in order to further monitor the feasibility and effectiveness of study procedures within each institution.

Informal & formal institutional stakeholder meetings at each clinical site

Both formal and informal meetings between research staff and institutional stakeholders were conducted at each clinical site. Engagement activities that were conducted during formal meetings among the clinical sites include: trauma meetings, surgeon’s meetings, grand rounds, Family Advisory Council meetings, Youth Advisory Council meetings, Trauma-Informed Care Workgroup meetings, Project Collaborative board meetings, and the Project All-site engagement meetings. Engagement activities that were conducted at informal meetings include: meetings with pediatric inpatient unit nurses, hospitalists, medical directors of inpatient specialty units and hospital social workers.

Individual interviews with patient & community partners

We developed discussion guides for interviews with patient and community partners to further explore issues of study design, implementation and dissemination. We identified three categories of patient partners: parent–child dyads who had completed the research study assigned to the Link condition; trauma education condition; and parent–child patient dyads with a history of traumatic injury who were naive to the research study. Each site also identified clinical experts in trauma-informed care in the community, including social workers, school administrators and healthcare providers in a range of settings. Potential patient partners were identified based on their interactions with research staff throughout the duration of the CER. All patient partners identified had completed all study activities and were no longer actively enrolled as research participants in the CER. Community partners were identified based on the visibility of their expertise in their community. Research staff from each site sent an email to each potential partner with information about becoming a research partner. The informational letter emphasized that all activities were voluntary and separate from the obligations of the research study (for parent–child patient dyads). Professional and patient consultants were considered research partners, rather than research participants, and as such were paid as contract employees of the University of Iowa. Ultimately, 22 individuals were identified as research partners.
A total of 27 consultant interviews were conducted, and ranged in duration from 25 to 80 min. Some consultants participated in multiple interviews. All interviews were audio recorded with the partners’ permission and transcribed verbatim with the exception of one interview, which was recorded by taking notes containing direct quotes. Transcripts were entered into NVivo 12 [23], a software that facilitates qualitative analysis. Members of the research team applied a directed content analysis approach [24] using a codebook comprised of some a priori codes focused on the research engagement experience and codes that were generated from the open coding process. A third of the transcripts (nine transcripts) were coded by two members of the research team to allow for intercoder reliability assessment. Intercoder reliability was calculated using the κ statistic throughout the coding phase. When the κ was <0.5, the team met to resolve coding discrepancies. At the conclusion of coding, a final κ was calculated for each code. The final mean κ across all codes was 0.91 (range: 0.51–1.0), demonstrating very strong coding agreement [25].

Collaborative engagement stakeholder meeting

A collaborative engagement stakeholder meeting was conducted following the collection and preliminary analysis of engagement data. Goals of the meeting were to: review and interpret results from the CER trial; identify key research questions to examine in future intervention feasibility and sustainability studies; determine dissemination activities; and identify and prioritize future intervention study research questions (priorities for adaptation). Nominal group technique (NGT) was used to reach consensus about next steps for dissemination and future research studies. This validated technique [26] was selected on the basis of time availability, cost–effectiveness, and to ensure the ability of all stakeholders to contribute to next steps for dissemination and research development. The consensus meeting was standardized and followed several steps. First, a review of the CER study results and relevant literature was presented by the study Principal Investigator (M Ramirez) to all of the stakeholders present at the collaborative engagement stakeholder meeting (N = 30). The presentation was followed by a description of the NGT by one of the study co-investigators (B Woods-Jaeger) who specializes in community-engaged research. During the remainder of the meeting, stakeholders addressed dissemination and future intervention study research questions using NGT. Stakeholders independently completed worksheets to generate ideas based on two specific questions posed: ‘in what settings should we disseminate the Link for Injured Kids intervention?’ and ‘for what types of trauma should this intervention be used? We chose to initially focus on Link for Injured Kids for the in-person meeting due to the greater complexity of the intervention and greater amount of time and resources needed to implement Link for Injured Kids, which pose unique challenges for dissemination and possibilities for adaptation that are best discussed in person. The plan was to send a follow-up survey to participants that posed these same questions for the trauma education intervention. However, the discussion ended up focusing on both interventions, highlighting what components of the two interventions are best suited for different settings and that both interventions should be implemented for a variety of trauma types. After completing their worksheet, each stakeholder shared his or her ideas in response to the questions, one at a time, while the NGT facilitator recorded them on a flip chart viewed by all participants. Stakeholders then produced a list of the most viable dissemination options for each intervention and recorded them on a priority sheet. Individual voting was completed by assigning a ranking to each option on the sheet, with higher rankings indicating greater importance. Voting outcomes were shared via email by the study coordinator along with follow-up questions to clarify specific settings with the top-ranked setting categories (i.e., schools and healthcare settings) and specific ideas regarding future study questions that would inform necessary adaptations and address key implementation questions related to feasibility and sustainability.


Informal & formal institutional stakeholder meetings at each clinical site

Recommendations from initial consultation meetings with hospital and community stakeholder groups that occurred before the CER began at the two initial sites (described above) were used by researchers to determine recruitment methods, develop study materials and determine the frequency and intensity of follow-up activities. Additionally, informal and formal institutional stakeholder meetings that occurred during and after the CER at all four sites allowed for improvements of the study data collection process, provided feedback of additional types of data that would add to study results, provided a platform for collaboration on manuscripts among researchers, and cultivated recommendations for dissemination of results and future adaptations of the research.

Individual interviews with patient & community partners

Three of the sites (Children’s Mercy Kansas City, the University of Iowa Stead Family Children’s Hospital and Blank Children’s Hospital) conducted one-on-one interviews with patient and community partners. One site encountered challenges in conducting these activities due to requirements from their IRB that they did not have capacity to meet. This resulted in a different level of engagement at the various sites and is something the study team discussed and problem solved through bi-weekly engagement calls. Increasingly, single IRB is recommended for multisite research to increase efficiencies in the IRB review process in multisite research without reducing the protection of human subjects [27]. Through our engagement process, the study team learned that reliance on a single IRB is also important for equitable engagement opportunities across study sites. 17 individuals served as patient partners, and included former patient subjects from the Link for Injured Kids study arm (four parents and four children), former patient subjects from the Trauma Education study arm (three parents and three children), and a parent and child who had lived an injury experience similar to research participants of the CER, but were not study participants. Five community partners were interviewed. This group was comprised of a former director of school nurses from a local school district, a trauma-sensitive school clinician from a local school district, a social worker embedded within a local metropolitan police department, a school-age coordinator of a local not-for-profit organization who provides educational and wrap around services for school-aged children, and a mental health practitioner at a metropolitan social service agency.
Key themes from these interviews related to improving study enrollment and participation included recommendations to enhance privacy during study enrollment, reduce the number of surveys and forms to complete at baseline, implement study and intervention activities after hospital discharge, and include additional family members in the interventions (Table 2). Further, consultants provided recommendations to improve engaging patient and community stakeholders as partners in the research including offering electronic options for engagement, providing opportunities for engagement that required less time and tailoring engagement activities to partner’s specific skills, interests and experiences (Table 2).
Table 2. Patient and community partner recommendations for improving comparative effectiveness research study design and procedures.
Project phaseSuggestionExample quoteHow input was utilized
Study enrollmentFewer formsThe more you can do verbally, the better… The forms did have a bit of deterrent, particularly when we were trying to gather data from (child). He is not receptive to paperworkFor study consultants: patient and community partners were interviewed via the phone following study participation rather than emailed surveys/forms
For study participants: as we pursue future translation and dissemination studies we will involve patient and community partners in refining and piloting assessments
 Research staff enroll children and collect baseline child data without parents present after parental consentIt's better if somebody else does it. He (child) would be more open to someone else asking the question and not looking at the person going, “well, they think I'm gonna answer this.” He’s just more free to say whatever he thinks. Whereas the parent, I don’t think they’re as free to answerAttempted to collect baseline data from child without parents present; however, this was difficult due to location of assessment and intervention occurring within hospital room
 Greater privacy during enrollmentI’ll tell you that it was a bit awkward how it was introduced. There wasn’t the privacy. It was an open setting. We did go to an area that was more private, but it was still a public area… It was fine when it was in the patient’s room. That seemed private enoughGreater emphasis was placed on recruitment to occur in patients’ hospital rooms whenever possible
 After recruited, complete enrollment at home after discharge to encourage study comprehensionWhen you get home, you are usually home with your child for quite some time, and you have more time to think about it, and that’s when I was able to sit down and look at things a little bit more. But it would be nice even like a week or two after you’re home…for someone to come outDuring enrollment, potential participants were given the option to consider the study and ask questions for as long as desired before enrolling
Study participationShorter surveysI think there was some repetition of the questions and I understand how they redirect those and whatever, but sometimes I felt like it was the same page being asked over again just in a different way. It would be nice if it was just a little bit shorterAs we pursue future translation and dissemination studies we will involve patient and community partners in refining and piloting assessments
 Email or text as primary mode of communicationI’m probably better with the emailed information. I don’t need the handholding of a call necessarilyFor study consultants: patient and community partners were contacted by email or text if they indicated that was their preferred mode of communication.
For study participants: as we pursue future translation and dissemination studies we will include email and text as methods to communicate with study participants
 Send reminders to use the intervention toolsJust that refresher. We’ll have things we send out to customers to remind them of our services, showing them how the service could be utilized. The same thing is what I’m thinking of for that basic information. Just showing an illustration of the tools in use, helping them with the recovery process, just at periods after the initial introduction to the information as a reminderVideos demonstrating the motivational interviewing skills originally taught to parents during the intervention were shared with parents via email. Following this feedback, if parents were interested in watching the videos again, we resent the video links and/or the skills video
 Open intervention to multiple caregiversWe…went and watched the video and got early information, and my spouse did not, and I’m not sure… Certainly for all of the family members that you may have in the room, there was nothing about the information that was being shared that wouldn’t be beneficial to anybody who wanted to stay that was in the room. They could have the option not to stayAlthough we were not able to include additional caregivers in the intervention during the recruitment phase of the study, spouses or other caregivers were invited to the collaborative engagement stakeholder meeting to learn more about the study and be involved in providing feedback about the study
 Have child patients be trained in the intervention to increase their own emotional awarenessTeenagers… go through a lot of changes and have a lot of emotional energy. And any information we can provide on that helps him get through the teenage years is greatAlthough we were not able to include patients (teenagers) in the intervention during the recruitment phase of the study, patients were invited to the collaborative engagement stakeholder meeting to learn more about the study and be involved in providing feedback about the study
Patient consultingOffer the option for consultants to share their perspectives over email, not just over the phoneI don’t know if people maybe don’t like to talk just openly, but personally if you could do it maybe through email, I don’t know if they would respond better just to a forum through email asking questions, if they would even respond that wayThe feasibility of online forums (email, social media) for consultants to utilize was explored in individual consultant interviews and stakeholder meetings. We are still exploring this as a potential format for continued engagement, but are limited by funding
 Structure consultant opportunities around small increments of timeA good hour… Maybe a half hour? It just depends how busy the kid is on that day, because they’re not gonna be insanely busy every day, there’s gonna be some nights they aren’t and sometimes they are, but at least every day just getting at least a half hour in and dedicating their time to that would be greatEvery effort was made to reduce consultants’ time commitment when possible, and interviews and other consultant opportunities were scheduled around consultant availability
 Individualize consultant participation to correspond to individual skillset, interest and competing demands in their livesIt differs with the individuals. If the individuals are retired or they’re providing care to the family and they’re not working outside the home, the time restraints may be a little bit different. But, again, interest level, some folks may prefer talking to you than doing their day job and have the flexibility that they can talk to you for hours. So, a case by caseWe are presenting a variety of options for continued engagement of varying time commitments (e.g., co-authoring manuscripts and presentations; responding via email to ideas and opportunities for next steps)
 Advertise consultant opportunitiesYou guys could advertise (opportunities) and eventually if they’re not doing anything one day, they could go do it, and they could maybe like it and start doing it more and moreWe are developing a formal communication plan for sharing information about opportunities to engage in dissemination and translation of studies activities; however. this is also limited by funding

Collaborative engagement stakeholder meeting

The collaborative engagement stakeholder meeting was conducted on 17 February 2018 at one of the four sites, with options for in-person and virtual participation. The meeting agenda corresponded to the meeting objectives with 90 min devoted to reviewing and interpreting results from the CER trial; 60 min devoted to identify activities across sites related to intervention feasibility and sustainability through discussion of consultants’ experience of barriers and facilitators of the study and intervention implementation; and 90 min prioritizing dissemination activities and recommendations for future intervention study questions and aims using NGT. Healthcare and school settings were identified as priority settings for dissemination of the study interventions and commitment from institutional stakeholders, staff, time, and resources were identified as key issues to explore related to feasibility and sustainability (Table 3). Priority adaptations included implementing and examining study interventions for a range of early adversities and traumas, including but not limited to unintentional injury, adding peer group/social network building component to study interventions, and including web and social media resources (Table 3).
Table 3. Consultant perspectives on intervention dissemination.
Future intervention settingRationale for adaptation/disseminationFacilitators for disseminationBarriers to dissemination
Schools- Easiest way to reach the most children with a variety of trauma experiences.
- Easy access to children creates high return on investment
- The social and academic environment is complicated, so extra help is needed
- Children spend so much time in school so this is primary area of adjustment after injury
- Seamless way to expand mental health services to rural settings
- Children have strong relationships and histories with adults at schools
- Already have school professionals like counselors and teachers aids who could be trained
- School oversight bodies could implement training policies
- School leadership needs to champion intervention to generate support
- Opportunities to utilize required school in-service trainings
- Child’s relationship with school staff
- Sharing data on program effectiveness for buy-in
- User-friendly intervention design
- Creating web-based trainings to increase accessibility
- Schools can be close-knit communities, which means staff may already be aware of child’s need
- Following up with parents.
- Busy school schedules.
- Limited funding and resources for implementation.
- Not all staff may be motivated.
- School leadership may act as gatekeepers.
- Children may not want to be asked these questions on school settings
Acute healthcare settings- This is the most critical and stressful point in time for family
- Location where recovery first begins
- You can target families who have experienced injury
- Help families prepare for discharge
- Important training for physicians and nurses
- High return on investment due to concentration of injured patients
- Enthusiasm of physicians and nurses who interact with injured patients frequently
- Hospital departments provide trainings for their staff
- User-friendly design intervention design
- Creating web-based trainings to increase accessibility
- Available time to use intervention with patients because they may be in hospital for weeks
- There can be a strong connection between care providers and patients
- Hospitals are large organizations, which can be challenging to create change.
- Patients see a lot of different hospital staff
Outpatient healthcare settings- Patients often work hard to recover in outpatient settings and this can be frustrating
- Emotional trauma can interfere with physical recovery
- Lack of understanding of emotional trauma among physical/occupational therapists
- Lack of mental health resources so families may reach out to other healthcare providers after discharge
- Identified need for outpatient health professional training.
- User-friendly intervention design.
- Creating web-based trainings to increase accessibility
- Difficulty reaching the myriad of outpatient specialists that patients see
Support groups- Participants did not identify specific rationales for support groups in follow-up interviews (though identified as a need through nominal group technique)
However, they did frequently speak to the need for social support networks for parents, families and their children’s friends
- Desire to share experience, receive advice, and not feel alone- Voluntary so may not have high participation rates
- Less return on investment due to participation rates
- Perception that support groups would not address pressing needs
- May need to drive long distances in rural areas
Website and social media resources- Need for information about typical situations, stressors and easily understandable medical information
- Strong need for information about community resources (health, educational, child/family rights, financial)
- Interest in having Link intervention materials on website
- Need to connect with others and find personal connection online
- Children need resources specific to their experiences and vocabulary
- Children want tips to talk to their peers.
- Has the potential to easily expand and be disseminated
- Easily accessible connection through social media platforms.
- Targeted connection to individuals with shared experience
- Resources and websites can easily be shared
- Motivation among children and parents both to find more information
- Online privacy concerns, especially with children


This engagement process, which utilized the PCORI Rubric model [1], yielded new insights about how best to engage patient and healthcare provider stakeholders in intervention study implementation and dissemination. Several methods can be used to collect information from stakeholders using formal qualitative methods as well as informal discussions and consultations. In addition, capturing voices from a variety of stakeholders is essential to informative engagement. Our engagement activities focused on eliciting the unique perspectives of patients/participants themselves, caregivers (in our case, parents), clinicians and community advocates. Findings show the value – as well as the challenges – that come from implementing a stakeholder approach that equally weighs both expert and lay perspectives.
We were able to incorporate selected feedback from stakeholders during implementation of our intervention trial study provided that our research design was not modified. While it is possible to make design changes in adaptive trials, our study did not allow for ongoing modifications to the study design. We were, however, able to use the engagement from diverse stakeholders to improve recruitment and retention rates as well as to suggest next steps in dissemination and adaptation. For example, patient stakeholders suggested utilizing more private spaces in the hospital for recruitment, and private rooms in the in-patient wards were identified by the team for recruitment of future participants. Another suggestion was that surveys be completed at home rather than solely at the hospital to allow for more dedicated, uninterrupted time to answer survey items. For patients and parents, we allowed for completion of surveys after discharge and provided self-addressed-stamped envelopes for submission of baseline surveys. Other feedback from stakeholders was more informative for future multisite patient-centered effectiveness research trials. Having a single IRB, for example, would increase efficiency. In our study, two sites had their own IRB procedures, which posed some challenges. Further, our engagement strategy was complicated by differences in how different IRBs approached stakeholder engagement in the context of an active study, which made it difficult to ensure consistent methodologies across studies. Another suggestion, which came later, was to reduce the length of data collection instruments; this is a key consideration for future studies.
Aside from feedback about the design of the study, we received ideas about methods to efficiently and conveniently engage stakeholders. Notably, as paid consultants, the study stakeholders appreciated value placed upon their time and willingly provided their expertise. However, time was a significant challenge to engagement even with compensation, especially for busy parents, children, clinicians and community advocates. Virtual opportunities such as through online venues were suggested by stakeholders, many of whom came from rural communities. Shorter engagement activities like brief check-ins and emails consultations were also recommended to obtain quick feedback. These lessons learned about how to engage patient stakeholders and how to minimize participant burden in the context of research are incredibly valuable. One of the biggest threats to external validity in research studies is attrition, and unfortunately high rates of drop out are common across research studies [28,29].
Finally, by end of our study, lessons learned from the engagement process informed our future steps. Our large group discussions led to targeted ideas for dissemination of findings through peers and social networks established by parents, children and community members. Further, next steps include taking results from this CER study to scale through a focus on dissemination of research findings. This process will involve our parent, patient and clinical consultants by sharing ownership of dissemination products (e.g., co-authored policy briefs, articles and research papers). Another next step for research is to translate our interventions to address other types of adverse child experiences such as violence or stressful home or community experiences. The school was identified as ideal setting to implement interventions to support traumatized children. In light of this suggestion, the team is pursuing research in school settings to translate and evaluate Link in Schools as an intervention to reduce student violent behaviors. This research agenda is a direct result of involving a range of professional stakeholders and patient consultants in our research study.
Our experience is not unique to this study. Involving patients in research has changed the course not only of how research is being conducted but also what is being studied in a range of areas of mental and physical health. For example, researchers in the field of rheumatoid arthritis (RA) have redefined the core outcomes of the disease on the basis of feedback from patient research partners [2]. After being initially omitted from the core set of outcomes, fatigue was endorsed as a primary part of the assessment of RA on the basis of patient research partners’ feedback about its importance to daily life functioning [30–32]. Critical to the successful collaboration of patient partners and researcher and clinical stakeholders is a sustained effort by organizers of conferences, research funding organizations and other research leaders in ensuring that the voices of patient and professional stakeholders are valued. While there are examples of successful collaborations – such as patients being integrated into the international research organization Outcome Measures in Rheumatology in the case of RA, in which the involvement of patients is ‘so pervasive that it is difficult to separate their contribution from those of researchers and other stakeholders’ [2] – tokenism is also prominent in the involvement of patients in research and efforts to improvement healthcare service delivery [6]. By asking patients, caregivers and clinicians to work together to generate and prioritize research topics for funding, PCORI is helping to ensure patient involvement from the top down as well as the bottom up [33].
As we explore translations of our interventions in new settings, patient engagement methods will continue to be critical in every stage of the development, execution and dissemination of our research. Several issues related to both study participant engagement and study consultant engagement will continue to be examined by our team as we pursue future patient-engaged CER. These issues include better understanding study participants’ motivation to enroll in the study and how this relates to their understanding and expectations of the study consultant role. Further, exploring ethical issues related to informed assent in a study like this with a heavy parent focus is important. For example, what are the benefits and drawbacks of introducing the study during the child’s inpatient stay but following up for informed assent with child participants and study enrollment post discharge? We are eager to pursue these and other questions related to patient engagement in CER research as we continue to engage with our patient partners through the dissemination phase of this study.
Summary points


Patient and stakeholder engagement should be conducted throughout the stages of comparative effectiveness research, but is often challenging in the context of multisite study implementation activities.
This paper describes the engagement of institutional and community stakeholders, parent–child dyad patient consultants, and clinical experts in the community in a comparative effectiveness research study to evaluate two programs for decreasing traumatic stress in children who have experienced unintentional injury.

Key findings

This engagement process shows the value – as well as the challenges – that come from implementing a stakeholder approach that equally weighs both expert and lay perspectives.
Providing multiple different opportunities to engage (e.g., informal and formal meetings, consultant interviews, large multisite meeting) was important to facilitate engagement from a diverse group of stakeholders.
Changes were made to the way potential participants were recruited and approached, and how data were collected, as well as to the future directions for research on the basis of feedback from patient and professional stakeholders.
Hosting a collaborative in-person and virtual engagement stakeholder meeting was useful in facilitating an equitable exchange among researchers, patient consultants and professional stakeholders from all four research sites.

Implications for comparative effectiveness research

Ensuring multisite collaboration in the approach to engaging patient and professional stakeholders is key to an authentic, equitable exchange among researchers and patient and community partners.
Involving patients and community stakeholders in research changes the course of how research is conducted and also what is prioritized in disseminating findings and future research directions.


The opinions in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Financial & competing interests disclosure

Research reported in this publication was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (CER-1306–02918). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.

Ethical conduct of research

The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.


Papers of special note have been highlighted as: • of interest; •• of considerable interest
Frank L, Forsythe L, Ellis L et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual. Life Res. 24, 1033–1041 (2015).
•• Presents an overview of the Patient-Centered Outcomes Research Institute approach to patient-engaged research and efforts to establish promising practices for engagement to guide the field of comparative effectiveness research.
Kirwan JR, de Wit M, Frank L et al. Emerging guidelines for patient engagement in research. Value Health 20(3), 481–486 (2017).
• Discusses experiences engaging patients as research partners from the perspectives of researchers, patients and funders, and suggests guidelines to develop research partnerships with patients.
Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez KA. Advisory panel on patient engagement (2013 inaugural panel). The PCORI engagement rubric: promising practices for partnering in research. Ann. Fam. Med. 15, 165–170 (2017).
Concannon TW, Meissner P, Grunbaum JA et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 27, 985 (2012).
Deverka PA, Lavallee DC, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 1(2), 181–194 (2012).
•• Presents a conceptual model for stakeholder engagement in comparative effectiveness research to provide guidance for researchers, stakeholders and funders interested in comparative effectiveness research.
Ocloo J, Matthews R. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual. Saf. 25(8), 626–632 (2016).
Borse NN, Gilchrist J, Dellinger AM, Rudd RA, Ballesteros MF, Sleet DA. CDC Childhood Injury Report: Patterns of Unintentional Injuries Among 0–19 Year Olds in the United States, 2000–2006. CDC – NCIPC, GA, USA (2008).
Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. National Action Plan for Child Injury Prevention: An agenda to Prevent Injuries and Promote the Safety of Children and Adolescents in the United States. CDC – NCIPC, GA, USA (2012).
Bisson J, Andrew M. Psychological treatment of post-traumatic stress disorder (PTSD). Cochrane Database Syst. Rev. 18 (3), CD003388 (2007).
Bisson JI. Post-traumatic stress disorder. Brit. Med. J. 334(7597), 789–793 (2007).
Michaels AJ, Michaels CE, Moon CH et al. Posttraumatic stress disorder after injury: impact on general health outcome and early risk assessment. J. Trauma 47(3), 460–466; discussion 466–467 (1999).
Schneeberg A, Ishikawa T, Kruse S et al. A longitudinal study on quality of life after injury in children. Health Qual. Life Outcomes 14(1), 120 (2016).
Barzilay R, Calkins ME, Moore TM et al. Association between traumatic stress load, psychopathology, and cognition in the Philadelphia Neurodevelopmental Cohort. Psychol. Med. 49(2), 325–334 (2019).
January AM, Zebracki K, Chlan KM, Vogel LC. Symptoms of depression over time in adults with pediatric-onset spinal cord injury. Arch. Phys. Med. Rehabil. 95(3), 447–454 (2014).
Ramirez M, Toussaint M, Woods-Jaeger B et al. Link for injured kids: a patient-centered program of psychological first aid after trauma. Pediatr. Emerg. Care 33(8), 532 (2017).
Ramirez M, Harland K, Frederick M, Shepherd R, Wong M, Cavanaugh JE. Listen protect connect for traumatized schoolchildren: a pilot study of psychological first aid. BMC Psychol. 1(1), 26 (2013).
Peek-Asa C, Cavanaugh JE, Yang J, Chande V, Young T, Ramirez M. Steering teens safe: a randomized trial of a parent-based intervention to improve safe teen driving. BMC Public Health 14(1), 777 (2014).
Kenardy J, Cobham V, Nixon RD, McDermott B, March S. Protocol for a randomised controlled trial of risk screening and early intervention comparing child- and family-focused cognitive-behavioural therapy for PTSD in children following accidental injury. BMC Psychiatr. 10(1), 92 (2010).
Kenardy J, Thompson K, Le Brocque R, Olsson K. Information–provision intervention for children and their parents following pediatric accidental injury. Eur. Child. Adolesc. Psych. 17(5), 316–325 (2008).
Oral R, Ramirez M, Coohey C et al. Adverse childhood experiences and trauma informed care: the future of health care. Pediatr. Res. 79(1–2), 227 (2015).
PCORI Engagement Rubric. PCORI (Patient-Centered Outcomes Research Institute) website.
StataCorp. Stata Statistical Software: Release 15. StataCorp LLC, TX, USA (2017).
QSR International Pty Ltd. NVivo qualitative data analysis software; version 10 (2014).
Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual. Health Res. 15, 1277–1288 (2005).
Viera AJ, Garrett JM. Understanding inter-observer agreement: the κ statistic. Family Med. 37(5), 360–363 (2005).
McMillan SS, King M, Tully MP. How to use nominal group and Delphi techniques. Int. J. Clin. Pharm. 38, 655–662 (2016).
National Institutes of Health. Final NIH policy on the use of a single institutional review board for multi-site research. Notice number: NOT-OD-16-094.
Fewtrell MS, Kennedy K, Singhal A et al. How much loss to follow-up is acceptable in long-term randomised trials and prospective studies? Arch. Dis. Child. 93(6), 458–461 (2008).
Robinson KA, Dinglas VD, Sukrithan V et al. Updated systematic review identifies substantial number of retention strategies: using more strategies retains more study participants. J. Clin. Epidemiol. 68(12), 1481–1487 (2015).
Kirwan JR, Hewlett S. Patient perspective: reasons and methods for measuring fatigue in rheumatoid arthritis. J. Rheumatol. 34, 1171–1173 (2007).
Kirwan JR, Minnock P, Adebajo A et al. Patient perspective: fatigue as a recommended patient centered outcome measure in rheumatoid arthritis. J. Rheumatol. 34, 1174–1177 (2007).
Aletaha D, Landewe R, Karonitsch T et al. Reporting disease activity in clinical trials of patients with rheumatoid arthritis: EULAR/ACR collaborative recommendations. Ann. Rheum. Dis. 67, 1360–1364 (2008).
Fleurence R, Selby JV, Odom-Walker K et al. How the Patient-Centered Outcomes Research Institute is engaging patients and others in shaping its research agenda. Health Aff. (Millwood) 32(2), 393–400 (2013).
• Describes the approach of the Patient-Centered Outcomes Research Institute in promoting meaningful engagement of patients in the research process and challenges of that approach.

Information & Authors


Published In


Received: 15 November 2018
Accepted: 26 April 2019
Published online: 10 July 2019


  1. comparative effectiveness research
  2. health services research
  3. mental health
  4. pediatrics
  5. stakeholder engagement
  6. trial design



Briana A Woods-Jaeger* [email protected]
Developmental and Behavioral Sciences, Children’s Mercy Hospital, Kansas City, MO 64108, USA
Emory University, GA 30322, USA
Chris C Sexton
Developmental and Behavioral Sciences, Children’s Mercy Hospital, Kansas City, MO 64108, USA
Kansas Center for Anxiety Treatment (KCCAT), KS 66212, USA
Cassidy Branch
University of Iowa, IA 52242, USA
Mallory Bolenbaugh
University of Iowa, IA 52242, USA
Lisa Roth
University of Iowa, IA 52242, USA
Blank Children’s Hospital, IA 50309, USA
Elisabeth A Stelson
University of Iowa, IA 52242, USA
Nikki Braaten
Children’s Minnesota, MN 55404, USA
Marizen Ramirez
University of Iowa, IA 52242, USA
University of Minnesota, MN 55455, USA


Author for correspondence: [email protected]

Metrics & Citations


Article Usage

Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.

Downloaded 121 times


How to Cite

Engaging parent–child dyad and healthcare provider stakeholders in a patient-centered comparative effectiveness study. (2019) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2018-0126

Export citation

Select the citation format you wish to export for this article or chapter.

View Options

View options


View PDF

Get Access

Restore your content access

Enter your email address to restore your content access:

Note: This functionality works only for purchases done as a guest. If you already have an account, log in to access the content to which you are entitled.







Copy the content Link

Share on social media