Show me the money! An analysis of underserved stakeholders’ funding priorities in Patient Centered Outcomes Research domains
Abstract
Aim: Develop an accessible exercise to engage underserved populations about research funding priorities; analyze the criteria they use to prioritize research; contrast these criteria to those currently used by Patient Centered Outcomes Research Institute (PCORI). Materials & methods: Academic and community partners collaborated to develop an Ipad exercise to facilitate group deliberation about PCOR funding priorities. 16 groups (n = 183) of underserved individuals in both urban and rural areas participated. Recordings were qualitatively analyzed for prioritization criteria. Results: Analysis yielded ten codes, many of which were similar to PCORI criteria, but all of which challenged or illuminated these criteria. Conclusion: Directly involving underserved populations in determining funding criteria is both feasible and important, and can better fulfill PCORI’s goal of incorporating patient priorities.
First draft submitted: 2 February 2017; Accepted for publication: 21 April 2017; Published online: 7 July 2017
Patient-centered outcomes research (PCOR) in general, and the Patient Centered Outcomes Research Institute (PCORI) in particular, recognize and implement the important shift to including patients and other stakeholder voices in health research and practice [1,2]. While incorporating patients and stakeholders’ viewpoints in the research process has precedents in community-engaged approaches such as Community-Based Participatory Research, Participatory Action Research or even polling and focus groups, the incorporation of patients and stakeholders in the prioritization of which research to fund poses challenges that are both complex and novel. Exploratory ventures into incorporating stakeholders into the research prioritization process have been conducted in recent years, but rarely in the USA [3–5].
Priority setting in research is a policy-making endeavor that is usually unfamiliar to those who engage in healthcare on-the-ground. As a result, it may seem impossible to acquire general public input on this topic because it requires background knowledge about research funding as well as the incorporation of group judgments for how to allocate public resources (we contrast ‘general public’ with the interested, motivated and educated subsets of the public, who are arguably the ones who probably were involved in PCORI’s committees and public comment periods). For contexts such as these, many have argued that deliberative methods of public engagement are optimal [6,7]. These methods gather nonprofessional members of the public together to learn and deliberate about a topic. This process usually involves some level of education, group consideration and the task of coming to an agreed-upon solution.
In order to ascertain whether deliberative methods could be effective in incorporating public input into priority setting for research and what the outcomes of that input would be, the research team adapted an existing deliberation exercise, called Choosing All Together (CHAT; originally Choosing Health Plans Altogether, now Choosing All Together) to facilitate deliberative priority setting among participants from underserved and underengaged populations, such as low income populations, rural populations and inner city minority populations [6]. CHAT, both in its original and adapted form, is designed to elicit reasoned conversations among participants in an “inclusive, informative and engaging manner” [8,9].
In Goold et al. 2017, we present the quantitative outcomes of the CHAT study. That analysis evaluated both the categories of research funding that these populations selected, whether the exercise of group deliberation affected those selections, and how participants felt about the CHAT exercise itself [7]. In contrast, here we conduct a qualitative analysis of the discussions among participants during the exercise. In this way, we are able to learn why participants chose the priority areas that they did (i.e., the outcomes of the previous publication), in order to learn about the criteria they used to choose them [6].
This is an important question in order to ascertain whether the factors that impact which research the public funds reflect or challenge the factors that current funders, such as PCORI, utilize. To address this question, we analyzed and thematized the factors that our participants expressed to justify their funding choices, and then compare and contrast these with the criteria already outlined and utilized by PCORI.
Materials & methods
As mentioned above, the methods for the larger project of which this analysis is a part was to design and implement a public deliberation exercise focused around a novel Ipad exercise called CHAT. We conducted 16 focus groups of underserved populations playing the CHAT game (see Table 1 for participant characteristics). Most focus groups included between 10 and 12 participants (183 total) from urban or rural areas in two states (Michigan and Missouri, USA). The exercise consisted of four rounds, with participants playing alone during the first and last round of the game, but playing together and discussing decisions in the second (small groups) and third (all together) round. The game board is shown in Figure 1. For more detail about the methods of developing and implementing the exercise, as well as quantitative outcomes of choices, please see Goold et al. [6].
| Participant characteristics | n (%) |
|---|---|
| State of residence (n = 183) | |
| – Michigan | 105 (57.4) |
| – Missouri | 78 (42.6) |
| Female (n = 183) | 112 (61.2) |
| Age in years (n = 178) mean (SD; range) | 46.4 (14.7; 18–83) |
| Race (n = 183): | |
| – White | 63 (34.4) |
| – Black or African–American | 98 (53.6) |
| – Other† | 22 (12.0) |
| – Hispanics (CHAT in Spanish or self-identified Hispanics, n = 183) | 41 (22.4) |
| Education (n = 182): | |
| – High school/GED or less | 72 (39.6) |
| – Some college | 63 (34.6) |
| – Bachelor’s degree | 24 (13.2) |
| – More than bachelor’s degree | 23 (12.6) |
| Rural (vs urban region; n = 172) | 45 (26.2) |
| Income (n = 174): | |
| – Less than US$15,000 | 67 (39.0) |
| – $15,000–34,999 | 56 (32.6) |
| – $35,000 or more | 49 (28.5) |
| No. of people in household (mean [SD; range], n = 177) | 2.9(1.7; 1–11) |
| ≤Federal poverty level (n = 171) | 67 (39.2) |
| Living alone (n = 177) | 43 (24.3) |
| Perceived health status (n = 177): | |
| – Fair or poor | 66 (37.3) |
| – Good | 50 (28.3) |
| – Very good or excellent | 61 (34.5) |
Cell values are n (%) unless otherwise described. n does not add to 183 when some responses are missing.
†Other includes other race and mixed race.
CHAT: Choosing All Together; GED: General Educational Diploma; SD: Standard deviation.
Reproduced with permission from [6].
Figure 1. PCOR-CHAT Gameboard.
CHAT: Choosing All Together; PCOR: Patient-centered outcomes research.
Reproduced with permission from [6].
While the quantitative analysis told us what the participants chose to fund, both individually and as a group, it could not shed light on why they made the choices that they did. The goal of this analysis was to analyze the discussions that took place among focus group participants to elicit patterns of reasoning for these decisions, and then compare those patterns to those utilized by PCORI to make funding decisions. For this qualitative study, we recorded and transcribed the conversations that took place during the second and third rounds of the game and transcribed them for analysis.
Prior to analyzing the transcripts, we did a close examination of the current criteria by which PCORI currently determines which research to fund. The Patient Protection and Affordable Care Act (ACA) which established and set parameters for PCORI, requires that PCORI take into account several criteria, or ‘factors’ to make funding decisions [10]. These criteria are listed in the second column of Table 2.
| PCORI prioritization criteria | ACA prioritization criteria |
|---|---|
| Impact on health of individual and populations | Disease incidence, prevalence, and burden in the USA |
| Address current gaps in knowledge/variation in care | Gaps in evidence in terms of clinical outcomes, practice variations, and health disparities in terms of delivery and outcomes of care |
| Probability of improvability via research | Potential for new evidence to improve patient health, well-being, and the quality of care |
| Impact on healthcare system performance | Effect on national expenditures associated with a healthcare treatment, strategy, or health conditions |
| Patient-centeredness | Patient needs, outcomes, and preferences |
| Potential to influence decision-making | Relevance to patients and clinicians making informed health decisions |
PCORI took up this charge from the ACA and subsequently articulated its own prioritization criteria through a process of: reviewing previous stakeholder engaged prioritization processes; culling those priorities that most reflected prior engagement; aligning those priorities with the PCORI mission; and submitting them for public comment [12]. PCORI’s method of public involvement is notably different from ours given below. These methods, such as public comment periods, most often involve the most interested, educated and motivated members of the public. These do not necessarily reflect the views of those stakeholders most impacted by the decisions. For example, Fischer et al. conducted a stakeholder assessment of comparative effectiveness research needs for Medicaid populations by engaging with Medicaid Medical Directors, not Medicaid recipients [11]. PCORI’s process culminated in a list of prioritization criteria that they claim correspond to and reflect each of the ACA prioritization criteria. We included both of these lists in our analysis because it remains an open question whether the PCORI criteria are an accurate reflection of the ACA prioritization guidelines. They are clearly not equivalent.
Once we had this grasp of the existing PCORI and ACA prioritization criteria, we moved on to examine the criteria that could be derived from the discussions from our research participants. Two of the authors (SS Cargill and LL Baker) conducted iterative content analysis of the transcripts in several stages. First, one author (SS Cargill) analyzed the verbatim responses in the transcripts to identify and extract instances where participants articulated their reasons for making particular choices in the exercise. The most explicit of these were phrases such as “because…” or “The reason why is…” among others. Sometimes reasons were given more implicitly as well. This yielded 235 distinct quotes.
Second, the same author (SS Cargill) then coded these 235 quotes into preliminary themes. These themes were informed by, but not limited to, the prioritization criteria articulated by the ACA and PCORI (Table 2). For example, we analyzed the quotes with an eye to those that expressed reasons that were “patient-centered”, but these ended up manifesting in three different codes, rather than a one-to-one correspondence. Likewise, we found themes in the transcripts that could not comfortably align with any of the PCORI or ACA criteria, and these we kept as codes with no correspondence to PCORI or ACA criteria.
Third, the second author (LL Baker) was given the complete list of extracted quotes and the list of preliminary themes to do an independent pass at coding. The two authors then met to reconcile the coding themes and their application to the quotes. Prior to reconciliation, there was 76.6% agreement between the raters (180/235). After clarifying the codes themselves and a second pass by each rater to reconcile their coding, there was almost complete agreement (94.5%; 222/235) and the codes which we could not reconcile were omitted from the analysis (5.5%; 13/235).
Results
Analysis of transcripts
Analysis of the transcripts yielded 235 quotes that articulated criteria for priorities. From these quotes, the authors developed ten codes for priorities, as well as an 11th “other” category to capture reasons that did not fit clearly into the other categories. These codes are listed and defined in Table 3.
| Code | Definition | Examples |
|---|---|---|
| Patient-centered: personal experience | References to me and mine | ‘My mother had a…; I had a…; if I had it, I would want X’ |
| Patient-centered: community-centered | References to the community at large | ‘It will help us; we have…; it doesn’t affect us’ |
| Patient-centered: generic patient | Generic patient experience; relevant to patients, their caregivers, among others | ‘Patients need; caregivers have to…' |
| Justice | References to fairness, deserving | ‘Everyone should be able to; to be fair; the vulnerable/underserved need…’ |
| Cost | Related to monetary expense | ‘It costs less; most cost effective; gets too much money' |
| Impact | Wide reaching effect (includes prevalence) | ‘Lots of people have xyz…’ |
| Gap in knowledge/not gap in knowledge | References to needing more knowledge or already possessing requisite knowledge | ‘Nothing now works; need new ideas; current treatment is ineffective; what we have already works’ |
| Weighing & distribution | Concerned with ranking, importance or allocation between categories | ‘It’s important, but not as important as…; every category should get something, or not’ |
| Foundational/not foundational | References to root cause/not root cause, fundamental/not fundamental, and good health essential to a good life | ‘If they discovered the cure…; if we knew what caused the disease; it doesn’t matter if…xyz will still be the same; because it will take care of other problems…’ |
| Implementation in practice | Affects people on the ground | ‘I want to see results; putting things into practice’ |
| Other | Unclear or ambiguous reasons outside the scope of the other categories | ‘That sounds important; don’t understand why we need to fund xyz…’ |
Patient-centered: personal experience
Numerous participants voiced their first person individual experiences to justify priority areas that they emphasized. For example, many participants articulated the importance of research into patient–doctor relationships based on their personal experiences. One woman emphasized her experiences as a person from an often misunderstood ethnic group. She stated:
“When you’ve got ethnic diversity in an area and people are not aware…For example, we don’t shake hands with men. How many of you knew that?” (urban, Missouri).
Similarly, another participant expressed frustration with communication between care providers:
“Being my mother’s power of attorney and taking her to doctors and appointments and hospitals and emergency rooms and things like that, miscommunication between an ER and their doctor and if they don’t have an electronic file…They give her an antibiotic and she’s on a blood thinner. This is a problem that’s happened many times. You know, that’s one condition, let alone the other stuff.” (rural, Michigan).
Another participant spoke of the importance of research on families/caregivers by referring to her parents’ struggles qualifying for home healthcare funding:
“On a personal note, I know several people who could use a caregiver. One would be my parents, and because they make (US) $2 over bill money when they worked and so whatever their retirement is above what you’re supposed to have, they can’t get the home healthcare. They can’t receive it because they make $1.50 more and they actually worked, and so they get Blue Cross Plus. If you get Medicaid, you get whatever you want. If I go to work and I retire and I get Blue Cross and whatever that part D is, I don’t qualify for that because I make too much money. But I could use the service.” (urban, Michigan).
Within the personal experience code, we often came across rationales based not on participants’ experiences as patients, but based upon their professional experience with patients. For example, one participant stated:
“I was the Social Service director at a nursing home, and I spent every day talking to people that had already given up because they can’t walk or they can’t go to the bathroom for themselves or they can’t shower or play with their kids, and they had already given up. No matter how much money you put into it, it’s not gonna make them better. All they want is somebody to be there with them, I mean, to kind of ease the going.” (rural, Missouri).
Another participant, when discussing research for those with special needs, also used personal work experience to justify research prioritization. Stating:
“I work with the mentally ill every day, and I know that … I see them being just pushed out on the street, and there’s no services. If they don’t get regional center involved, then they have nothing. Their regional center is being cut, and so they don’t have that much to offer all of them. You’re gonna see them out on the street, and you know this is the cause because the money is not there.” (rural, Missouri).
Patient-centered: community-centered
In addition to their own experiences (both personal and professional), many participants justified their choices by invoking more of a ‘we’ than an ‘I’. Quite a few participants abstracted from their individual perspectives and experiences to focus on their community needs and interests. When justifying funding quality of life research, one participant explicitly voiced that she prioritized community concerns over the concerns of the individual, stating that: “I’m not too concerned about individuals. I’m more concerned with the overall community rather than the quality for individuals.” (rural, Missouri).
And when discussing health inequity, another participant articulated “I’m not disadvantaged, so to spend money for my family didn’t make sense, but for the community, you know, it’s easier to spend those dollars.” (rural, Michigan).
A third participant raised community concerns which extended outside the medical realm:
“So my number one point was that … in promoting health is that we now start the need to promote health in our own community instead of, you know, ending cancer. So, you know, even though we’re affected by these things…with diabetes, hypertension, high blood pressure, heart disease…These are problems that are in our community. We ain’t got no walk. We don’t even have a hike in the neighborhood. That’s why I would put…That’s why I would pick Level 3.” (urban, Michigan).
Patient-centered: generic patient
A third distinct way that participants explained their choices was by speaking of patients in general. There was often explicit mention that these were not their own experiences or even those of their communities, but participants believed that patients at large have these types of needs and preferences. For example, when discussing patient–doctor research, one participant invoked the case of an unknown patient and a generalized conception of the doctor to justify their allocation decision, stating “Well, I think too many times the relationship between the doctor and the patient is not there” (urban, Missouri). Similar generic patients were invoked when discussing funding “promoting health”, access to health services, and “quality of life.”
“Because so many people, you know, their health goes to the wayside, so to speak, where they’re not getting their care because they can’t afford it, or they don’t think it’s as important, and it is. I think promoting health…getting it out there is really important so they’ll know that there’s resources out there to help you get better.” (urban, Missouri).“Because I think sometimes people lack to be able to get to the doctor. There’s not…a lot of it is major barriers. They have children at home that need healthcare and they can’t them to their healthcare, either transportation or other reasons.” (urban, Michigan).
Another participant expressed concern for the generic patient over what she understood as the chain reaction nature of poor health:
“Because you very rarely find a person that only has one thing wrong with them, and once you start treating something…Once you start treating somebody for one thing…Say, for instance, you start treating me for. High blood pressure. Next thing you know, I’ve got diabetes…And that’s exactly where I’m at with that. Or, it also. Giving people. You give a person a medication to treat one thing, and then it causes something else. It’s always. It’s never just I can treat this and you’ll be okay. Because once I start treating this, then this is gonna happen.” (urban, Missouri).
Cost
Our participants manifested a complex understanding of the financial implications of their choices. This was gratifying since one explicit goal of the design of the CHAT game was to enable participants to clearly see how their priorities manifested in financial sacrifices, both in terms of their overall expenditures and in how their expenditures in some areas impacted their expenditures in others. Our analysis yielded themes that showed that participants thought of spending decisions in several distinct ways.
First, participants often directly discussed the cost of a certain choice, and how this cost did or did not reflect what they believed to be the intrinsic value of the choice. For example, during deliberation one patient stated, “You get a big bang for your buck on this category’” (rural, Michigan). Another participant asserted “With limited resources, you said, we can’t ignore it, but let’s not overspend. So I think (level) 1 is right.” (rural, Michigan).
Impact
While participants did not utilize the scientific language of incidence and prevalence to discuss the impact of the research, they frequently discussed their perception that certain problems were increasing or affected numerous people and used this as a rationale for choosing to prioritize research in that category. For instance, one participant articulated “It seems like that is more and more these days too. People don’t just have one thing” (rural, Missouri). Another participant expressed “Well, how often does that happen? That doesn’t happen often” (urban, Missouri). And when discussing research into special needs a third participant stated: “Given the ramifications of some of the mental/emotional/physical problems to the rest of the community and society, I would opt for a higher amount here.” (rural, Michigan).
Gap/no gap in knowledge
Participants frequently mentioned their reasons for both choosing and not choosing categories of funding due to the need for more information, or sometimes the lack of this need. This was heartening, since the research team sometimes struggled to make clear to participants that we were not merely asking them which categories they thought were important for health, but which they thought were important for research. The fact that many participants could distinguish between areas where they saw problems but not a lack of knowledge and those where there was a problem and a lack of knowledge reflected an understanding of the purpose of the discussion.
One example of a participant identifying problems without need for research referred to quality of life as well as caregiver issues. “I just think that when people give up, that no matter how much money you put in front of them for healthcare aides to help them, they’ve already given up. They just want somebody to be there with them” (rural, Missouri). Another participant illustrated the identification of a problem and a lack of knowledge when stating “I see it this way: we are talking of investment and research, and what I think is that we know so little about how to improve the quality of life, we have so little consciousness that a good research will make us more aware about how to be better, without having need of drugs, or doctors, or many other things, for me this is very important.” (urban, Michigan).
It was also clear that participants saw research as the solution to certain types of problems but not the solution to other types of problems that they faced. For example, one participant articulated “I think that for the most part with the not-rare stuff (diseases), we understand how they work. It’s prevention and support and resources that we need to put more money into rather than just researching them further when we already know how they work.” (rural, Missouri).
Weighing & distribution
Our participants often spoke of looking relationally at different categories to determine how much they should spend on them, either by weighing different categories against each other or by emphasized how they wanted to distribute their funds as widely across categories as possible. One participant clearly articulates this when stating “It’s not that I don’t think that that’s important. I just think that there are other things that are more important than that” (rural, Missouri). Similarly, when discussing the patient–doctor area of research, another participant explicated:
“You could possibly go to another doctor if you have a problem with (your doctor). So I would rather we spend more money promoting health so I won’t get sick and then even if I am sick promoting my quality of life, like to the max, before I start worrying necessarily about having a relationship with one person or not.” (urban, Michigan).
Justice
Our research team tried to better understand our participants’ views of health disparities by defining a research category as “Health inequities.” Surprisingly, very few participants chose to fund this category [6]. On the other hand, participants did mention issues of justice in their discussions, but these were more generally in terms of fairness and need, especially in funding categories such as “Access to care” and “patient–doctor.” When discussing access, one participant articulated: “Access because it just can’t be the people who can afford it to go”. (urban, Michigan). Another participant stated:
“I think we hear all the time that we have the greatest healthcare system in the world, and I believe we…I mean in terms of knowledge and all that, but I think, you know, everybody can’t…doesn’t…isn’t able to take full advantage of it, and that’s what I would like to see that the advantages that we have working for us we take advantage of. So that’s…that’s my deal.” (urban, Missouri).
Regarding patient–doctor research, another participant stated:
“They’re making a lot of money taking care of these people. So learn how to treat them right, you know. You can’t mistreat a patient because he’s black or white or rich or poor or …Treat them all.” (urban, Missouri).
Foundational/not foundational
A very frequent criterion voiced by participants for their choices was their belief that certain issues were the ‘root’ of others, and that by funding research into one area, they would be able to solve several downstream issues. For example, one participant stated:
“If we promote health, we know the causes of diseases, then we are going to have a good quality of life, and if we improve the studies, the quality of life will be better.” (rural, Michigan).
This position was also clearly articulated by another participant who stated:
“It doesn’t matter how much research is done. if I don’t have the access. I don’t have the means to go and see a physician. So, this is … I think this is very important because if I don’t have insurance, I don’t have transportation, I’m worried about the cost and things like that … it doesn’t matter. All these other things are null and void.” (urban, Michigan).
Implementation in practice
Finally, many of our participants voiced skepticism toward the ability of research to change their everyday lives as patients. They emphasized how they wanted to fund research not merely based on its importance, but based on the likelihood that the research would have relevance and implementable consequences. For example, one participant argued “I think, it’s important to recognize this is money spent on research, not on services.” (rural, Michigan).
Another participant articulated:
“I know our government is on a tight budget, but I think we can re-prioritize where we’re spending our money and put more into things that are going to help the population people…When you look at actually implementing studying pilot projects, that’s actually putting things into practice and making things happen rather than just simply researching and maybe that information would just sit somewhere and never be applied.” (rural, Missouri).
Comparison of transcripts with ACA & PCORI criteria
In a second step, we attempted to align the participant codes with those articulated by the ACA and PCORI. The results of this attempted alignment are in Tables 4 & 5.
| CHAT codes | ACA prioritization criteria |
|---|---|
| Patient-centered: personal experience Patient-centered: community-centered Patient-centered: generic patient | ‘Patient needs, outcomes, and preferences’ and ‘Relevance to patients (and clinicians) in making informed health decisions’ |
| Cost: weighing & distribution | ‘Effect on national expenditures associated with healthcare treatment, strategy, or health conditions’ |
| Impact | ‘Disease incidence, prevalence, and burden’ |
| Gap in knowledge | ‘Gaps in evidence in terms of clinical outcomes’ and ‘Potential for new evidence to improve patient health, wellbeing and quality of care’ |
| Justice | ‘Practice variations and health disparities in terms of delivery and outcomes of care’ |
| Foundational/not foundational | ‘Effect on national expenditures associated with healthcare treatment, strategy or health conditions’ |
| Implementation in practice | No equivalent |
ACA: Affordable care act; CHAT: Choosing All Together.
| CHAT codes | PCORI prioritization criteria |
|---|---|
| Patient-centered: personal experience | Patient-centeredness |
| Patient-centered: community-centered | Patient-centeredness |
| Patient-centered: generic patient | Patient-centeredness |
| Justice | Inclusiveness of different populations |
| Cost | Efficient use of research resources |
| Impact | Impact on healthcare system performance |
| Gap in knowledge | Probability of improvability via research Address current gaps in knowledge/variation in care |
| Weighing distribution | Efficient use of research resources |
| Foundational/not foundational | Efficient use of research resources |
| Implementation in practice | Impact on health of individuals and populations (adapted with permission from National Priorities p.13) Potential to influence decision-making |
CHAT: Choosing All Together; PCORI: Patient-Centered Outcomes Research Institute.
Discussion
“Scientists do not have a blank check to conduct research; they must answer to the public’s demands and needs. To satisfy the demand for public accountability, publicly funded research needs public oversight.” – D Resnik 2001 [13].
Engaging stakeholders for input on research priority setting is a new, but increasingly valued project. While not using public deliberation per se, other methods such as focus groups, citizen juries and surveys have been utilized by different groups to elicit this type of input [3,11,14,15]. Adding to this literature, this study found that by engaging stakeholders in a deliberative exercise about research funding priorities, numerous insights emerged.
First, the high level of content overlap between our CHAT codes and the PCORI prioritization criteria as well as the ACA prioritization criteria (displayed in Tables 4 & 5) suggests that the prioritization criteria articulated by the ACA and PCORI are likely to be fairly representative of public research prioritization opinion. This indicates that the limited public input process that PCORI utilized to generate these criteria was not entirely off base. This alignment is also encouraging as it indicates that changes suggested by broad public involvement in these questions are likely to be manageable in scope.
Second, upon deciding that certain participant codes paralleled ACA and PCORI prioritization criteria, we also realized that they did not perfectly mirror each other. This finding is not surprising considering that results of previous engagements with different groups of stakeholders have also found that they tend to prioritize research differently [16–18]. Discrepancies provide an opportunity to reflect on how the participants’ discussions could deepen and expand current prioritization criteria.
For example, while the ACA’s criteria of “Patient needs, outcomes, and preferences’ and ‘Relevance to patients (and clinicians) in making informed health decisions.” can be understood as equivalent to PCORI’s criteria of “patient centered,” our participants’ comments illustrated several different interpretations that could be made of this concept. Is the ‘patient’ the particular patient’s particular perspective, the general patient with a generalized perspective, or could it even refer to different patient communities’ perspective? Each interpretation could yield different applications of the criterion “patient centered” as applied to research funding.
Likewise, while our participants’ emphasis on funding research on the “Foundational” issues could align with existing criteria about cost-effective research, it also shows how our participants understood many different health problems to be interrelated, and how many of them took a much broader and more holistic view of health than the compartmentalized way that many of our funding organizations function. Participants frequently articulated a belief that we need not fund research into every important problem, but that we must often dig down to the root cause of the current problems to spend money where it can have the most impact. This is potentially a critique of our current funding mechanisms that often divide up research into various areas, whether disease systems or topic areas, without looking at the potential relationship between these areas to implicate funding choices.
Finally, our findings show that in some areas, the ACA criteria are more aligned with our findings, and in other areas, the PCORI criteria show more commonality. This suggests that the simple translation of the ACA criteria to the PCORI criteria may not be as straightforward as is claimed. While PCORI claims that their prioritization criteria reflect the ACA criteria once illuminated by stakeholder input, our analysis challenges this claim in that certain ACA criteria (such as “disease incidence, prevalence and burden” as well as the more nuanced articulation of how research should relate to patients) is lost in the PCORI criteria.
This study has several limitations which limit the generalizability of our findings. The stakeholders who participated in this study were all geographically located in the Midwestern US region, albeit in two different states. In order to better capture the interests of patients and stakeholders across the USA, it will be important to expand these conversations to other geographic regions.
While we intentionally recruited as diversely as possible in terms of age, race, gender, ethnicity and urban/rural residence, as well as oversampled people from underserved communities, the small number of total participants limits the potential generalizability of our findings.
Our qualitative analysis was based on deidentified transcripts of focus groups. As such, we were able to analyze what was said, but not by whom. We are unable to analyze our data by race, gender or other demographic characteristic that did not characterize the whole focus group (e.g., urban or rural). Given the limitations of the study, further empirical work will need to be done in order to explore how various subgroups within the larger public prioritizes certain criteria over others.
Conclusion
This research suggests that given the right context and framework, nonexpert and nonprofessional members of the public may be fully capable of reflecting upon the reasons for prioritizing some research over others. More than half of our research participants (71.6%) had incomes less than $35,000 and at least 39% were under the federal poverty level. Their education levels were on average low and yet they were able to engage meaningfully with this discussion. Clear in the participant responses in the CHAT exercise is that patients and other stakeholders from underserved communities are capable and willing to be engaged in these types of deliberations. For example, many participants were able to articulate complex understanding of the financial implications of their choices, and several participants called upon their professional experience to justify research prioritization.
This qualitative analysis does demonstrate that engaging with stakeholders directly about research prioritization is both important and feasible. It aligns closely with PCORI’s mission and would enable stakeholders to be able to help define the terms and frame the direction of the research conversation. While PCORI requires stakeholder engagement in its funded research, it is important to realize that by incorporating stakeholder views into funding decisions, the impact of public engagement can be greatly increased. Further, several distinct insights from participants should encourage further exploration of some funding assumptions, such as that research domains should be funded distinctly from their relationship to each other, and that “patient-centered” research can have several distinct meanings with very different implications.
The Choosing All Together exercise is a useful and effective way to incorporate stakeholder input on complex issues such as research funding priorities.
The prioritization criteria of underserved populations may largely reflect those currently utilized by the Patient-Centered Outcomes Research Institute.
Important differences between prioritization criteria used by Patient-Centered Outcomes Research Institute and those employed by underserved populations can illuminate and challenge existing practices.
Financial & competing interests disclosure
Research reported in this article was funded through a Patient-Centered Outcomes Research Institute Award (1IP2PI000521–01). The authors acknowledge the grant-funded research team: TR Campbell, CC Collins, M Danis, HM Kim, C Ledón, S Martinez, CD Myers, Z Roe, K Ryan and L Szymecko. The statements presented in this article are solely the responsibility of the author(s) and do not necessarily represent the views of Patient-Centered Outcomes Research Institute, its Board of Governors or Methodology Committee. Goold and her institutions could benefit from future paid licenses (royalties) for the Choosing All Together tool used in this study. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Ethical conduct of research
Institutional Review Board approval for the overall study was obtained from both the University of Michigan (Goold) and Saint Louis University (SS Cargill and LL Baker). The qualitative data analyzed for this manuscript were in a deidentified state and, therefore, exempt from Institutional Review Board oversight at the time of data analysis.
References
1.
Frank L, Basch E, Selby JV. The PCORI perspective on patient-centered outcomes research. JAMA 312(15), 1513–1514 (2014).
2.
Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA 307(15), 1583–1584 (2012).
3.
Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q. 81(3), 363–388 (2003).
4.
Owens C, Ley A, Aitken P. Do different stakeholder groups share mental health research priorities? A four-arm Delphi study. Health Expect. 11(4), 418–431 (2008).
5.
Gooberman-Hill R, Horwood J, Calnan M. Citizens’ juries in planning research priorities: process, engagement and outcome. Health Expect. 11(3), 272–281 (2008).
6.
Goold SD, Myers CD, Szymecko L et al. Priorities for patient-centered outcomes research: the views of minority and underserved communities. Health Serv. Res. 52(2), 599–615 (2017).
7.
Solomon S, Abelson J. Why and when should we use public deliberation? Hastings Cent. Rep. 42(2), 17–20 (2012).
8.
Goold SD, Biddle AK, Klipp G, Hall CN, Danis M. Choosing healthplans all together: a deliberative exercise for allocating limited health care resources. J. Health Polit. Policy Law 30(4), 563 (2005).
9.
Burkhalter S, Gastil J, Kelshaw T. A conceptual definition and theoretical model of public deliberation in small face-to-face groups. Commun. Theory 12(4), 398–422 (2002).
10.
Compilation of Patient Protection and Affordable Care Act: extracted sections concerning patient-centered outcomes research and the authorization of the Patient-Centered Outcomes Research Institute (PCORI) 2010. www.pcori.org/sites/default/files/PCORI_Authorizing_Legislation.pdf
11.
Fischer MA, Allen-Coleman C, Farrell SF, Schneeweiss S. Stakeholder assessment of comparative effectiveness research needs for Medicaid populations. J. Comp. Eff. Res. 4(5), 465–471 (2015).
12.
Patient-Centered Outcomes Research Institute. National priorities for research and research agenda 2013. www.pcori.org/research-results/research-we-support/national-priorities-and-research-agenda
13.
Resnik DB. Setting biomedical research priorities: justice, science and public participation. Kennedy Inst. Ethics J. 11(2), 181–204 (2001).
14.
MacLean M, Anderson J, Martin BR. Identifying research priorities in public sector funding agencies: mapping science outputs on to user needs. Technol. Anal. Strateg. Manag. 10(2), 139–155 (1998).
15.
Gooberman-Hill R, Horwood J, Calnan M. Citizens’ juries in planning research priorities: process, engagement and outcome. Health Expect. 11(3), 272–281 (2008).
16.
Viergever RF, Olifson S, Ghaffar A, Terry RF. A checklist for health research priority setting: nine common themes of good practice. Health Res. Policy Syst. 8, 36 (2010).
17.
Bowling A, Jacobson B, Southgate L. Explorations in consultation of the public and health professionals on priority setting in an inner London health district. Soc. Sci. Med. 37(7), 851–857 (1993).
18.
Kapiriri L, Tomlinson M, Chopra M, El Arifeen S, Black RE, Rudan I. Setting priorities in global child health research investments: addressing values of stakeholders. Croat. Med. J. 48(5), 618–627 (2007).
Information & Authors
Information
Published In
Copyright
© Future Medicine Ltd.
History
Published online: 7 July 2017
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Show me the money! An analysis of underserved stakeholders’ funding priorities in Patient Centered Outcomes Research domains. (2017) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2017-0008
Export citation
Select the citation format you wish to export for this article or chapter.