How different is research done by the Patient-centered Outcomes Research Institute, and what difference does it make?
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: To determine whether research funded by the Patient-centered Outcomes Research Institute (PCORI) is consistent with the original aims of Congress and unique among other major USA funders. Methods: We compared a sample of funded projects from PCORI, NIH (Phase IV) and agency for healthcare research and quality (AHRQ; American Recovery and Reinvestment Act [ARRA]-based comparative effectiveness research funding) from 2014 to 2018 on number of outcomes/study, patient-centeredness of outcomes (those related to survival, function, symptoms and health-related quality of life) and other features that may characterize patient-centered research (e.g., whether conducted in a real-world setting) using PCORI portfolio data and ClinicalTrials.gov. Results: The mean number of outcomes in PCORI studies (≥9) appeared higher than NIH (≥3)/AHRQ (5.5); a higher percentage of outcomes/study were patient-centered: >85% PCORI versus 50% AHRQ and ≤30% NIH. The majority of PCORI studies (≥74%) were conducted in a real-world setting; this characteristic could not be identified for NIH/AHRQ studies. Conclusion: PCORI-funded studies appear to have unique aspects relative to NIH and AHRQ that are consistent with PCORI’s aims of patient-centeredness.
We are the largest single research funder that has comparative effectiveness research (CER) as its main focus, and we incorporate patients and other stakeholders throughout the process more consistently and intensively than others have before. We call this ‘research done differently’, so states the Patient-centered Outcomes Research Institute (PCORI) [1].
The PCORI’s legislative mandate is to fund patient-centered CER [2]. PCORI operationalizes this concept by funding CER trials and involving patients, patient advocates and other appropriate stakeholders (e.g., payers, clinicians, manufacturers) in topic priority setting, requiring applicants to recruit and incorporate patients (especially) and other stakeholders as members of the research team for proposal development, research execution and ultimately dissemination. Further, patients and other stakeholders are fully empowered members of the merit review (study section) panels that score applications on ‘patient-centeredness’, ‘patient and stakeholder engagement’ and more typical evaluation criteria, such as study importance, technical merit and study team qualifications. The expressed implication, compared with research funded by other organizations, is that research results are more relevant to the needs of patients, their clinicians and other healthcare decision-makers. Due to that increased relevance, it is more likely that resulting evidence is timelier, implemented more quickly and ultimately has a greater positive impact on clinical practice and patients’ health [3].
PCORI’s mission is quite focused: ‘to (help) people make informed healthcare decisions, and (improve) healthcare delivery and outcomes, by producing and promoting high-integrity, evidence-based information that comes from research guided by patients, caregivers and the broader healthcare community’ [1]. All PCORI-funded CER studies are required to compare two or more alternatives that patients, providers and/or other health system decision-makers face in real-world conditions. Although there is increasing evidence of changing research efforts toward the direction that PCORI has taken, to our knowledge, no empirical research has evaluated whether, or the degree to which, PCORI research is different from that of other major USA funders.
An understanding of the unique value of PCORI’s approach to research funding is critical since its funding expires in 2019 if not reauthorized by Congress.
Objective
To evaluate PCORI’s contribution to patient-centeredness in research and to determine whether this contribution is unique to PCORI or employed by other funding organizations.
Approach
Choosing comparator funding organizations
To best address the relative impact of PCORI’s requirements for patient-centeredness in research, other publicly funded agencies were identified based on similar therapeutic area funding as well as coverage of the translation continuum – clinical and health services research rather than basic science research. In addition, data on the characteristics of studies funded by each agency were required for comparative evaluation.
Several publicly funded, health-related agencies were considered to be appropriate comparator candidates: the NIH, the Agency for Healthcare Research and Quality (AHRQ), the Veterans’ Administration and potentially the Department of Defense and the Centers for Disease Control and Prevention.
Following a review of organizational missions and websites, and using our judgment as to public policy relevance, we selected the NIH and AHRQ for comparison to PCORI. AHRQ has a limited history of supporting CER. The agency specifically funded 11 CER – so-called clinical and health outcomes initiative in comparative effectiveness (CHOICE) – studies prior to 2010 as part of the American Recovery and Reinvestment Act of 2009 (Table 1) [4]. However, since PCORI was legislated in 2010, AHRQ no longer funds CER. We considered the AHRQ-sponsored CHOICE studies the most similar to PCORI-sponsored research and selected them for our comparative analysis.
| Clinical topic per study |
|---|
| 1. Back pain |
| 2. Asthma |
| 3. Nephrolithiasis |
| 4. Diabetes |
| 5. Heart failure |
| 6. Gestational diabetes |
| 7. Bipolar disorder |
| 8. Prostate cancer |
| 9. Asthma |
| 10. Cardiovascular disease |
| 11. Cardiovascular disease |
Among other agencies that may fund CER, AHRQ’s mission is more broad than PCORI’s and generally oriented to health services (as opposed to clinical) research: ‘to produce evidence to make healthcare safer, higher quality, more accessible, equitable and affordable, and to work with (other elements of) the Department of Health and Human Services and other partners to make sure that the evidence is understood and used’ [5].
The NIH is a large and established agency that funds more than $39 billion in research annually, substantially more than PCORI and AHRQ [6]. Its mission is: ‘to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life and reduce illness and disability’ [7]. NIH-funded observational and interventional studies largely focus on areas distinct from AHRQ and PCORI, addressing emerging technologies not yet in clinical practice (and thus not CER candidates). However, a portion of NIH funding does target interventions in clinical practice, and it is this set that was considered to be potentially appropriate for comparison to PCORI CER studies.
Other organizational differences relevant to the comparison are the criteria that each applies for selecting research to fund, and the funding mechanism itself. PCORI utilizes a contract mechanism, which provides for substantial control of its applicants and the research it funds, whereas NIH and AHRQ fund research under the grant mechanism approach.
Study review criteria & suggested therapeutic areas
For a comparative analysis to be useful, clinical conditions of interest should be similar across agencies, and each funded research project would need to be coded similarly for the selected criteria to be compared.
We developed an initial set of criteria (Table 2) to compare research portfolios. We recruited a panel of 11 representatives from key stakeholder groups (patient advocacy organizations, academics, payers, manufacturers) and senior officials from two funding agencies (PCORI, AHRQ). All panel members were familiar with the respective missions and research agendas of all three funders.
| Characterize each study by funding agency in terms of the following aims: |
|---|
| a. Study design: (RCT, observational, prospective, retrospective) |
| b. Patient populations (and subgroups, if applicable) |
| c. Outcomes: principal and secondary |
| d. Degree of ‘real-worldness’ (e.g., based on practice site [community vs research setting]), trial type (e.g., degree of ‘pragmatism’), degree of protocol restrictiveness, comparator (e.g., placebo vs active) |
| e. Comparators (if any) |
| f. Budget (e.g., direct cost), as available and or best available surrogates (e.g., sample size) |
| g. Analytical methods (if results reported, publicly available) |
| h. Timeframe (i.e., proposed or actual length of study) |
| i. Partners, groups engaged (if applicable) |
RCT: Randomized controlled trial.
We e-mailed each stakeholder a synopsis of the rationale for the planned comparative evaluation and the list of criteria (Table 2), and solicited feedback using open-ended questions and interviews through e-mail, conference calls or both. We also asked about suggested therapeutic topics that might be appropriate for eliciting the comparative nature of research portfolios. Criteria for portfolio comparisons, as originally recommended by our stakeholder panel, tended to fall into four broad categories: patient-centeredness, study representativeness, trial efficiency and post-study issues (Table 3). In Table 4, we present selected stakeholder comments concerning criteria for comparison.
| Criteria |
|---|
| Patient-centeredness |
| – Study aim(s) |
| – Patient group representativeness |
| – Patient-centered outcomes (types, numbers) |
| Engagement processes |
| • Trial representativeness |
| – Research setting |
| – Geographic spread |
| Trial efficiency |
| – Rates of recruitment, retention, completion |
| – Cost per patient |
| – Study length |
| Post-study issues |
| – Dissemination plans |
| – Time to publication |
| – Implementation plans |
| Abbreviated comments |
|---|
| PCORI studies may include more outcomes per study than other sources of funding due to its emphasis on patient-centered measurement and guidance to investigators that applicants address the full range of outcomes about which patients care. |
| PCORI studies may have a wider geographic reach and spread of professional disciplines involved in the work. |
| Shared decision-making is important to PCORI and may be a differentiator. |
| Due to PCORI’s employment of stakeholders throughout the research process (e.g., topic selection, design, outcomes, process, etc.), its approach to CER tends to be more ‘demand driven’ than is the case for typical clinical research, which tends to be ‘supply driven’ by researchers and/or funders. |
| A comparator differentiator will likely be the research aim since PCORI’s mission and mandate are to inform healthcare decision-making, which is not necessarily the case with other funder agencies. |
| Compare plans for disseminating results and sharing data in ways other than through peer-reviewed publications. |
| Contrast the nature and extent of engagement with partner patients and other groups since this is central to PCORI’s mission. |
| Try to contrast issues of trial efficiency. My experience has been that the intense up-front and continuous patient engagement throughout the entire research investigation leads to faster and more effective recruitment, improved retention rates and I expect quicker study results and lower cost per patient enrolled. |
CER: Comparative effectiveness research; PCORI: Patient-centered Outcomes Research Institute.
Selecting conditions
The therapeutic areas most often cited by the stakeholder panel for comparison were oncology and mental health, and studies with this focus were selected for comparison across agencies. The rationale behind this choice was threefold: at the initiation of the study, these two conditions were most frequently listed on studies funded by PCORI, enabling a sample size large enough for comparison; limited resources required us to select only a sub-sample of studies for comparison; and this restriction by therapeutic area does not apply to AHRQ CHOICE studies, as all 11 were selected for comparison.
Data sources
Each agency’s website was reviewed for applicable content; agency officials and other individuals who were familiar with the respective research portfolio data sources were contacted. All three organizations maintain a database of their funded portfolios.
PCORI’s Portfolio Taxonomy Dataset 2.0 is a highly detailed descriptive taxonomy (Table 5) that includes many, but not all, criteria we were seeking [8]. The NIH and AHRQ maintain public use and accessible databases, Research Portfolio Online Reporting Tools (RePORT) [9] and Grants On-Line Database (GOLD) [10], respectively, which capture their entire funded portfolios. Unfortunately, neither RePORT nor GOLD includes a descriptive taxonomy or standardized abstracts by which reasonable inferences can be made as to our criteria of interest. However, all three organizations require investigators to register clinical trials with ClinicalTrials.gov (CT.gov), which does include an abstract and a limited (relative to PCORI’s) descriptive taxonomy [11].
| Condition |
|---|
| Health services |
| Study population |
| Care condition |
| Approach/design |
| Comparator(s) |
| Stakeholder engagement |
| Products and tools |
| Outcomes |
†
Each category contains multiple subcategories. See PCORI Board of Governors for expanded taxonomy.
PCORI: Patient-centered Outcomes Research Institute.
CT.gov does not systematically code for ‘effectiveness’ trials, although it does provide a ‘Phase 4’ code for drugs that have been approved for marketing. Therefore, we found no systematic means of identifying non-drug interventions (e.g., procedures, medical devices/diagnostics) that had entered clinical practice. NIH categorizes studies as interventional or observational, and only interventional studies have a phase listed; this variable is blank for observational studies. However, we assumed that observational studies included interventions that were generally not considered experimental and thus are likely used in clinical practice. Nearly all NIH interventional studies were categorized as Phase 2–3 or Phase 3. Only a small subset of studies was listed as Phase 4 interventional; when they were combined with the observational studies, the majority of NIH studies available and selected for the comparative evaluation were of observational design. This limitation applies to NIH-sponsored studies but not to AHRQ CHOICE CER trials or to PCORI’s entire clinical portfolio. The NIH project search strategy is presented in Table 6.
| Oncology | Mental health | |
|---|---|---|
| Interventional studies search | ||
| Search date | 22 March 2017 | 23 March 2017 |
| Search terms | Oncology OR cancer OR neoplasm OR tumor OR neoplasia OR malignancy OR oncologic | mental health OR mental OR mental healthcare OR depression OR mental health services OR anxiety OR emotional health OR mental hygiene OR mental health problems OR mood OR anxiety OR psychological |
| Study types | Interventional studies | Interventional studies |
| Additional criteria | Phase 4; Funder type: NIH† | Phase 4; Funder type: NIH† |
| Number of studies with start date 2014–2016 | 146 | 86 |
| Observational studies search | ||
| Search date | 22 March 2017 | 23 March 2017 |
| Search terms | Oncology OR cancer OR neoplasm OR tumor OR neoplasia OR malignancy OR oncologic | mental health OR mental OR mental healthcare OR depression OR mental health services OR anxiety OR emotional health OR mental hygiene OR mental health problems OR mood OR anxiety OR psychological |
†
Search was replicated selecting ‘Funder Type: Other U.S. Federal agency’ in order to potentially identify PCORI and AHRQ funded studies within clinicaltrials.gov.
AHRQ: Agency for Healthcare Research and Quality; PCORI: Patient-centered Outcomes Institute.
Based on the extreme imbalance in available descriptive coding across the respective portfolios and the differences in the nature and makeup of the respective available databases, a comprehensive, direct, unbiased comparison across the three portfolios was not possible. Due to the above opportunities and limitations, the following decisions were made:
•
Selection of datasets:
◦
PCORI: Behavioral/mental health (B/MH) and oncology projects from 2014 to 2018 were selected using the PCORI Portfolio Taxonomy Dataset 2.0.
◦
AHRQ: All 11 CHOICE trials from 2014 to 2018 were selected using CT.gov.
◦
NIH: Phase 4 (interventional) or observational B/MH and oncology studies from 2014 to 2018 were selected using CT.gov.
•
CT.gov datasets (i.e., AHRQ and NIH), data were compiled and presented for the available coded categories of interest (e.g., study design and number of reported outcomes).
•
PCORI data were compiled and reported for all available criteria of interest (e.g., study settings, priority populations, decision tools, shared decision-making).
•
For selected key categories that were not readily coded, a random subset of studies was selected, and criteria of interest (e.g., presence of comparator arm, and for those with comparators, use of ‘usual care’ versus active comparators) were hand-coded and compiled.
•
‘Patient-centeredness’ was identified for reported primary and secondary outcomes across all datasets using all available information. (Note: a detailed description of this process is presented below in the subsection titled ‘Estimating Patient-centeredness of Reported Outcomes’).
Compiling study criteria results
Whenever possible, all data were used to evaluate the degree to which the respective portfolios addressed the available criteria of interest. For instance, PCORI’s dataset allowed us to evaluate study criteria that were not coded in the CT.gov data.
The full set of studies from PCORI was randomly reordered and the first 20 were selected as the portfolio set for selected criteria. The same procedure was used to select 20 NIH studies. All 11 AHRQ CER trials were included.
Estimating patient-centeredness of reported outcomes
The degree to which study outcomes were ‘patient-centered’ was paramount in addressing our study aim (the degree to which PCORI research is different). However, patient-centeredness is not coded in any of the databases, (including PCORI’s), and the term is not precisely defined in the literature. PCORI’s broad definition is ‘outcomes that matter to patients’ [3]. We started our comparison with the more specific definition of ‘…outcomes that people notice and care about such as survival, function, symptoms and health-related quality of life,’ [12] but determined that further operationalization of the concept was required for consistent and reliable coding, given that patients may care about outcomes such as rehospitalization rate, correct diagnosis and other outcomes. Specifically, the following rules for coding patient-centeredness were established:
If the stated outcome is, in your opinion ‘survival, function, symptoms and health-related quality of life’, code ‘Yes’. If it is clearly a biomedical parameter (e.g., blood pressure), a test result, a tumor partial response or anything you judge as clearly inconsistent with what the PCORI site suggests as a patient-centered outcome, code ‘No’. If it is not a term connoting ‘survival, function, symptoms or health-related quality of life’, but as a potential patient, it is an outcome you would consider clearly important (or could be considered ‘patient-centered’), code ‘Leaning Yes’. All others, code ‘Leaning No’.
Through a modified-Delphi approach, three researchers independently coded the outcomes as described and compared results. Inter-rater agreement was >80% in the first round; raters then shared rationale on each item of initial disagreement, and inter-rater agreement was >90% in the second round. Next, all coders reviewed the discrepancies and recoded. Only the first and second outcomes listed in the respective databases were coded by assuming that these corresponded to the primary and secondary outcomes/aims of study findings.
Findings
Table 7 presents several summary findings across study design, including the presence of a comparator and comparator type (active or usual care) and the number of reported outcomes.
| PCORI | NIH | AHRQ CHOICE | |||
|---|---|---|---|---|---|
| Behavioral health n = 87 | Oncology n = 70 | Behavioral health n = 107 | Oncology n = 146 | n = 11 | |
| Study design (n, %) | |||||
| – Experimental | 77 (88) | 50 (71) | 21 (20) | 2 (1) | 9 (82) |
| – Quasi-experimental | 4 (5) | 1 (1) | 0 (0) | 0 (0) | 0 (0) |
| – Observational | 6 (7) | 19 (27) | 86 (80) | 144 (99) | 2 (18) |
| Conducted in real-world setting (n, %) | ∼90% | ∼74% | Not determined | Not determined | Not determined |
| Comparative in nature (n, %) | 87 (100) | 70 (100) | n = 20; 15 (75) | n = 20; 6 (30) | n = 10; 9 (90) |
| Usual care specified (n, %) | 64 (73) | 45 (64) | Not determined | Not determined | n = 10; 5 (50) |
| Mean number of outcomes | 10 | 9 | 5 (n = 20) | 3 (n = 20) | 5.5 |
AHRQ: Agency for Healthcare Research and Quality; CHOICE: Clinical and Health Outcomes Initiative in Comparative Effectiveness; PCORI: Patient-centered Outcomes Research Institute.
Study design
The majority of PCORI studies were experimental or quasi-experimental in design (93% B/MH; 72% oncology), as were AHRQ CHOICE studies (82%), whereas NIH Phase 4 studies were more likely to be observational in design (80% B/MH and 99% oncology).
Comparativeness
All PCORI and 90% of AHRQ CHOICE studies were comparative in design; that is, the intervention of interest was evaluated relative to an active comparator intervention; 75 and 30% of B/MH and oncology NIH studies, respectively, were comparative trials.
Usual care as a comparator
Seventy-three percent (B/MH) and 64% (oncology) of PCORI studies and 50% of AHRQ CHOICE studies listed ‘usual care’ as a comparator. We did not attempt to evaluate the degree to which NIH studies included a usual care arm since the preponderance of the studies were observational, many of which were not comparative.
Number of outcomes per study
PCORI studies evaluated a relatively large number of outcomes on average (mean = 10 B/MH; 9 oncology), approximately twice the number reported in NIH (mean = 5 B/MH; 3 oncology) and AHRQ studies (mean = 5.5).
Patient-centered outcomes
According to our evaluation, PCORI study outcomes were much more likely to be classified as ‘patient-centered’ than those of the NIH and AHRQ (Table 8). Specifically, we considered 17 of the 20 (85%) PCORI B/MH and 20/22 (91%) oncology outcomes to be patient-centered or so leaning; only 54% (12/22) of AHRQ’s outcomes were so classified. For NIH, 3/35 (9%) B/MH and 9/32 (28%) of oncology study outcomes were classified as patient-centered or so leaning.
| Patient-centered outcomes: PCORI–NIH–AHRQ | PCORI B/MH n = 10 studies; 20 outcomes | PCORI oncology n = 11 studies; 22 outcomes | NIH B/MH n = 21 studies; 35 outcomes | NIH oncology n = 19 studies; 32 outcomes | AHRQ CHOICE n = 11 studies; 22 outcomes |
|---|---|---|---|---|---|
| Determination of patient-centered outcomes; n | |||||
| Primary outcome? Yes + Leaning Yes | 8 | 10 | 1 | 5 | 6 |
| Secondary outcome? Yes + Leaning Yes | 9 | 10 | 2 | 4 | 6 |
| Total; n (% of outcomes) | 17/20 (85) | 20/22 (91) | 3/35 (9) | 9/32 (28) | 12/22 (54) |
AHRQ: Agency for Healthcare Research and Quality; B/MH: Behavioral/mental health; CHOICE: Clinical and Health Outcomes Initiative in Comparative Effectiveness; PCORI: Patient-centered Outcomes Research Institute.
Several characteristics of interest, detailed below, were only available for studies funded by PCORI.
Study setting
Table 9 displays the frequency of the different PCORI study settings, most of which we considered to be ‘real-world’ settings. For instance, 67 and 63% (B/MH and oncology, respectively) included ambulatory care settings; 21/20% home settings; 28/23% via phone; and 20/26% via ‘virtual’ settings. Overall, we estimated that 74% of oncology and 90% of B/MH studies were conducted in ‘real-world’ settings. Each PCORI study was conducted across a mean of two settings.
| Behavioral health n = 87 | Oncology n = 70 | |
|---|---|---|
| Study setting: PCORI; N (%) | ||
| – Ambulatory care | 58 (67) | 44 (63) |
| – Community | 16 (18) | 2 (3) |
| – ER | 0 (0) | 1 (1) |
| – Home | 18 (21) | 14 (20) |
| – Hospice | 0 (0) | 18 (26) |
| – Long-term care | 1 (1) | 0 (0) |
| – Mobile | 0 (0) | 1 (1) |
| – Phone | 24 (28) | 16 (23) |
| – Rehab | 1 (1) | 0 (0) |
| – School | 2 (2) | 0 (0) |
| – Urgent care | 0 (0) | 0 (0) |
| – Virtual | 17 (20) | 18 (26) |
| – Non-specific | 13 (15) | 6 (9) |
| Number of settings/study | ||
| – Mean | 2 | 2 |
| – Median | 2 | 1 |
ER: Emergency room; PCORI: Patient-centered Outcomes Research Institute.
Priority populations
Our sample of PCORI studies included a large array of what PCORI terms ‘priority populations’ (Table 10). Particularly notable high-volume categories among B/MH and oncology studies, respectively, were racial and ethnic minorities (78, 66%), low income (59/30%), age 65 years and older (31/41%) and multiple chronic conditions (26/10%). On average, B/MH studies included 3.2 priority populations and oncology studies included 2.5 priority populations.
| Behavioral health n = 87 | Oncology n = 70 | |
|---|---|---|
| Priority patient populations: PCORI; n (%) | ||
| – Racial and ethnic minorities | 68 (78) | 46 (66) |
| – Low income | 51 (59) | 21 (30) |
| – Women (only) | 30 (34) | 28 (40) |
| – Children (0–17) | 17 (20) | 5 (7) |
| – 65 and over | 27 (31) | 29 (41) |
| – Rural | 20 (23) | 8 (11) |
| – Urban | 23 (26) | 12 (17) |
| – Special needs | 6 (7) | 1 (1) |
| – Multiple chronic conditions | 23 (26) | 7 (10) |
| – Rare disease | 7 (8) | 3 (4) |
| – Genetic makeup affecting outcomes | 1 (1) | |
| – Low health literacy and/or numeracy | 13 (15) | 8 (11) |
| – LGBT | 2 (2) | 2 (3) |
| – Veterans | 7 (8) | 2 (3) |
| Number of priority patient populations/study | ||
| – Mean | 3.2 | 2.5 |
| – Median | 3 | 2.5 |
LGBT: Lesbian, gay, bisexual, transsexual; PCORI: Patient-centered Outcomes Research Institute.
Decision tools & shared decision-making
Table 11 provides information as to the inclusion and/or evaluation of decision tools and shared decision-making across PCORI studies. PCORI reported three classes of decision-related studies. We could not distinguish whether or the degree to which the decision-related tools/activities were the intervention of interest or were simply included in the study that evaluated a nondecision intervention. Patient/caregiver decision tools were included in 16% (B/MH) and 34% (oncology) of studies, provider decision tools in 7 and 20%, respectively, and shared decision-making was reported in 17 and 26%, respectively. Thirty percent of B/MH studies and 51% of oncology studies evaluated or included at least one decision-related tool or process.
| Behavioral health n = 87 | Oncology n = 70 | |
|---|---|---|
| Decision tools and shared decision-making; n (%) | ||
| – Clinical decision tools (patients/caregivers) | 14 (16) | 24 (34) |
| – Clinical decision tools (providers) | 6 (7) | 14 (20) |
| – Shared decision-making | 15 (17) | 18 (26) |
| Studies with any shared decision-making/≥1 above categories; n (%) | 26 (30) | 36 (51) |
Discussion
Congress created PCORI because it believed that patients, their physicians, payers and other healthcare decision-makers had inadequate patient-centric evidence to choose among care options. PCORI declared that it would fund a different type of research to achieve the mission Congress set for it and proceeded to pioneer processes that involved patients and other healthcare decision-makers intensively throughout the research endeavor. Now that PCORI needs Congressional reauthorization, it is important to address these questions: is PCORI’s research different? If it is, is it providing value, particularly patient-centric value? And is that value materially different from that of other investments? In this study, we attempted to characterize PCORI’s approach to clinical research as it relates to Congress’ mandate, and to address whether and/or the degree to which PCORI research appears to be different from efforts of the NIH and AHRQ.
There were many limitations with our study.
Comparative criteria development potential bias
Stakeholders’ comments in identifying useful comparative criteria were helpful. Additionally, the lead author (BR Luce) was formerly the Chief Science Officer of PCORI prior to the initiation of this study. Since we intentionally selected stakeholders who we believed were familiar with, and likely supportive of, PCORI’s mission and research agenda, we were fully cognizant of their and our potential bias that PCORI-funded research may be unique among research funded by major USA funding agencies. However, our objective was to identify the appropriate criteria against which to evaluate PCORI’s research uniqueness, and thus we judged that whatever bias may have existed could be useful and valid for our purpose. Unfortunately, many criteria that the panel recommended were not available in the databases we reviewed. For instance, one rater with substantial knowledge of the NIH and PCORI suggested comparing recruitment and retention rates, study duration and cost per patient, with the expectation that PCORI’s requirements would compare favorably. Understanding these questions is part of PCORI’s evaluation plan (www.pcori.org/sites/default/files/PCORI-Evaluation-Framework-3.0.pdf), but publicly available data do not permit the inclusion of these variables in a comparison with other funders.
Lack of comparative data
In proposing the present evaluation, we expected to be able to query existing portfolio databases of the three federally funded research organizations. However, we encountered great difficulty in developing useful, comparative data, mainly because only PCORI (by far the newest of the three agencies) had invested in a detailed taxonomy and coded each funded study according to it. Data available through CT.gov are rather rudimentary compared with PCORI’s taxonomy. To enable a more direct comparison across similarly coded datasets, we could have chosen to use only CT.gov data for PCORI studies as well. However, we chose to use the more detailed data available through the PCORI taxonomy to appropriately characterize PCORI’s research portfolio. Of the many caveats associated with this study, the lack of directly comparative data is important to highlight. This limitation also emphasizes the lack of transparency among funders who do not invest in a robust taxonomy of their funded research.
Different clinical conditions
PCORI and NIH studies were restricted to those from B/MH and oncology therapeutic areas. As only 11 AHRQ CHOICE studies were available for analysis, we were not able to restrict them by therapeutic area (Table 1). While the conclusions from this study pertain specifically to studies in these therapeutic areas, we can only speculate that results are generalizable across other therapeutic areas.
Reported number of outcomes per study
We note in the Findings section that PCORI studies reported many more outcomes than NIH or AHRQ studies. We speculate that patient-centered outcomes research such as PCORI purports to fund is more likely to include a larger array of outcomes that patients (especially), their caregivers and perhaps their clinical providers care about. However, this difference could be, but is not necessarily, an artifact of differential coding between PCORI and CT.gov. Additionally, it is common for clinical trial objectives to limit outcomes to one or two (even commonly aggregating somewhat disparate outcomes into an artificial composite outcome category) with the rather limited aim of demonstrating statistical significance.
Study abstracts
We did not have the resources to deeply evaluate the abstracts of the selected studies we used for the comparative analysis. An evaluation of the abstracts from each study may have led to fewer data gaps reported in the comparative analysis, although this benefit may have come at the expense of introducing more subjectivity to analyses.
Coding patient-centeredness of outcomes
Our coding of outcomes in terms of patient-centeredness was necessarily subjective. We could find no authoritative source for a definition with sufficient detail to be used for coding, and therefore developed our own coding definition. The concordance at the first round, however, suggests that our definition was consistent with that of coders deeply familiar with the concept of patient-centeredness. We believe that the end results provide a reasonably accurate characterization as to reported outcomes being patient-centered. Also, we assumed that the first and second outcomes reported in the respective databases corresponded to the primary and secondary study outcomes; however, the order of outcomes from PCORI and CT.gov may have been arbitrary.
Reporting only PCORI criteria
Data for several criteria were only available from the PCORI dataset (e.g., study settings and whether studies were conducted in real-world settings, priority populations, shared decision-making, decision-making tools and usual care specified as a comparator), although we could and did code the last category for AHRQ CHOICE studies. While we cannot make comparative statements, we do believe that reporting the degree to which PCORI studies addressed these criteria is indicative of the potentially unique nature of its portfolio.
Identifying relevant NIH studies
The NIH portfolio created from CT.gov was likely limited by our inability to systematically identify studies of interventions that were already in clinical practice. The Phase 4 query used to identify potentially comparative NIH studies is a US FDA term to identify drugs that have been approved for marketing. Nevertheless, the NIH file we created for analysis does include many entries that are not strictly drug related, so it is unclear whether or the degree to which this approach was a limiting factor. We were also surprised at finding the relatively high frequency of NIH study designs being classified as observational and wonder if that may be a function of our search strategy, given a large percentage of B/MH- and oncology-related interventional studies were excluded after restricting to Phase 4 studies only.
There are also several structural factors that likely influence the nature of the respective research portfolios.
Application guidelines
An obvious and important determinant of portfolio differences across the three agencies were signals/requirements they transmit to potential applicants (e.g., via its application instructions and the criteria it uses to select applications for funding). Of note are two of PCORI’s six merit review criteria: patient-centeredness and patient/stakeholder engagement. Neither criteria were included by AHRQ or NIH for scoring applications.
Grants versus contracts
Another factor is that all PCORI-funded research is, by law, via contract whereas NIH and AHRQ fund the vast bulk (possibly all) of CER or CER-like research via a grant mechanism. The contract mechanism provides the funding agency with direct control, for instance, to ensure adherence to guidelines and timelines, whereas the grant mechanism allows for greater investigator control. A case in point is that only PCORI explicitly requires deep involvement of patients and other stakeholders in the research itself, and only PCORI evaluates submitted applications relative to its ‘patient-centeredness’ and engagement. Thus, to some extent, PCORI forces its portfolio to be different.
Agency missions
The portfolios of three health research agencies we identified for comparison represent a pipeline of research based on their inherent missions, which, although clearly different from one another, include varying degrees of overlap to evaluate PCORI’s assertion of ‘research done differently’.
Conclusion
PCORI-funded clinical research appears to be different from similar research funded by the NIH and AHRQ. The characteristics of PCORI-funded research appear to address the objectives inherent in patient-centered CER, which are to provide patients, providers, payers and other healthcare decision-makers with scientifically valid evidence of what works best for different classes of patients in real-world settings. Future efforts should attempt to determine whether or the degree to which PCORI-like patient-centered research makes a difference in patient care and health. Our findings suggest that more detailed information about funded projects from other major funders would aid in understanding the contributions and value of different health research funders.
While mounting anecdotal evidence suggests that patient-centeredness is increasing with other funders, PCORI has likely played an important leadership role in this movement. An unanswered question remains: will the attention to patient-centeredness in research by other funders be sustained in the absence of a patient-centered research funder, or does this change require the ongoing existence of such a research funder to ensure USA health research addresses the interests and needs of patients and other stakeholders?
PCORI’s focus on CER, including its unique approach, does not appear to duplicate the research efforts of the NIH or AHRQ.
•
To evaluate the degree to which Patient-centered Outcomes Research Institute (PCORI)-funded research is different, we selected funded research of interventions that were deemed to be in clinical practice (e.g., not experimental) across three federally funded agencies (PCORI, NIH and agency for healthcare research and quality [AHRQ]) and evaluated unbiased samples of the respective comparative effectiveness research portfolios relative to criteria associated with patient-centered outcomes research.
•
We found the following, subject to the many caveats previously noted:
–
NIH, AHRQ and PCORI missions are different: NIH focuses on basic research, AHRQ on health services research and quality of care and only PCORI focuses on CER.
–
NIH, AHRQ and PCORI CER portfolios differ in key respects:
–
PCORI is the only agency focusing on patient-centered CER: to meet Congress’ objective of funding CER to assist patients, doctors, other healthcare providers, payers and other healthcare decision-makers in choosing between alternative clinical options based on scientifically valid evidence, only PCORI stands out.
–
Research design differs: the PCORI portfolio and AHRQ’s limited previous CER are much more likely than the NIH sample we reviewed to be experimental or quasi-experimental in design. NIH studies were mostly observational in design after the exclusion of Phase 3 and earlier interventional studies.
–
Comparativeness differs: all PCORI studies were (by design) comparative, as were 10 of the 11 AHRQ clinical and health outcomes initiative in comparative effectiveness trials; both cases were much more likely to be comparative than the NIH Phase 4 sample of trials we reviewed.
–
Patient outcomes differ:
–
Numbers/study: PCORI-funded studies reported at least twice as many outcomes as studies funded by the NIH and AHRQ.
–
Patient-centeredness: PCORI studies that reported primary and secondary outcomes were judged to be twice more likely than AHRQ and thrice more likely than NIH to be patient-centered.
–
PCORI studies are characterized by:
–
Being conducted in real-world settings
–
Focusing on priority populations
–
Including or evaluating shared decision-making and decision tools
Author contributions
BR Luce was responsible for study conception and drafting of the manuscript. BR Luce and JC Simeone were responsible for study design, acquisition of data, data analysis and revision of the manuscript.
Acknowledgments
The following individuals contributed to this report and the authors most appreciate their efforts: L Frank, H Edwards, K Kistler, S Graham, J Vandigo, as well as various staff at ClinicalTrials.gov [1] and AHRQ and members of a stakeholder advisory panel. The authors also are indebted to PhRMA and Pfizer for funding support.
Financial & competing interests disclosure
Funding for this study was provided by PhRMA and Pfizer. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/
References
1.
Patient-Centered Outcomes Research Institute. PCORI: About Us (2019). www.pcori.org/about-us
2.
National Institute for Health and Care Excellence (NICE). NICE DSU Technical Support Document 9: the identification, review and synthesis of health state utility values from the literature (2010). http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD9-HSUV-values_FINAL.pdf
3.
Frank L, Basch E, Selby JV, Patient-Centered Outcomes Research Institute. The PCORI perspective on patient-centered outcomes research. JAMA 312(15), 1513–1514 (2014).
4.
National Institute for Health and Care Excellence (NICE). NICE DSU Technical Support Document 7: evidence synthesis of treatment efficacy in decision making: a reviewer's checklist (2012). http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD7-reviewer-checklist.final_.08.05.12.pdf
5.
Agency for Healthcare Research and Quality. Mission and budget (2018). www.ahrq.gov/cpi/about/mission/index.html
6.
NIH. Budget (2019). www.nih.gov/about-nih/what-we-do/budget
7.
NIH. Missions and goals (2017). www.nih.gov/about-nih/what-we-do/mission-goals
8.
Patient-Centered Outcomes Research Institute. Board of governors meeting (2015). https://www.pcori.org/events/2015/board-governors-meeting
9.
NIH. Research portfolio online reporting tool (RePORT). https://report.nih.gov/
10.
Agency for Healthcare Research and Quality. Grants on-line database (GOLD) (2005). https://psnet.ahrq.gov/resources/resource/1101/Grants-On-Line-Database-GOLD
11.
NIH. Requirements for registering and reporting NIH-funded clinical trials in clinicaltrials.gov. https://grants.nih.gov/policy/clinical-trials/reporting/index.htm
12.
Patient-Centered Outcomes Research Institute. Patient-centered outcomes research (2013). www.pcori.org/research-results/patient-centered-outcomes-research
13.
Concannon TW, Meissner P, Grunbaum JA et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 27(8), 985–991 (2012).
14.
Mullins CD, Abdulhalim AM, Lavallee DC. Continuous patient engagement in comparative effectiveness research. JAMA 307(15), 1587–1588 (2012).
15.
Vandigo J, Oloyede E, Aly A, Laird AL, Cooke CE, Mullins CD. Continuous patient engagement in cardiovascular disease clinical comparative effectiveness research. Expert Rev. Pharmacoecon. Outcomes Res. 16(2), 193–198 (2016).
16.
Frank L, Forsythe L, Ellis L et al. Conceptual and practical foundations of patient engagement in research at PCORI. Qual Life Res. 24(5), 1033–1041 (2015).
17.
Thompson AG. The meaning of patient involvement and participation in healthcare consultations: a taxonomy. Soc. Sci. Med. 64(6), 1297–1310 (2007).
18.
Carman KL, Dardess P, Maurer M et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff. 32(2), 223–231 (2013).
19.
Shippee ND, Domecq Garces JP, Prutsky Lopez GJ et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 18(5), 1151–1166 (2015).
20.
Deverka PA, Lavallee DC, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 1(2), 181–194 (2012).
22.
PCORI engagement rubric (2015). www.pcori.org/sites/default/files/Engagement-Rubric.pdf
23.
HSRProj (2019). wwwcf.nlm.nih.gov/hsr_project/home_proj.cfm
24.
Consensus definition of ‘patient-centered outcomes’ (2012). www.pcori.org/assets/PCOR-Definition-Revised-Draft-and-Responses-to-Input.pdf
Information & Authors
Information
Published In
Pages: 1239 - 1251
PubMed: 31436471
Copyright
© 2019 Bryan R Luce.
History
Received: 25 April 2019
Accepted: 23 July 2019
Published online: 22 August 2019
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
How different is research done by the Patient-centered Outcomes Research Institute, and what difference does it make?. (2019) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2019-0054
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Linda Truong, Tegan Stettaford, Katrina Watt, Nafiseh Ghafournia, Pavel Anetko, Jiyoung Shin, Kris Pierce, John A. Lawson, Jennifer H. Martin, PAtient-Centric epilepsy clinical trIal model For Improved health outcomes using Cannabidiol (PACIFIC study)—a methodology for developing patient-centred clinical trials in rare epilepsy syndromes, BMC Medical Research Methodology, 10.1186/s12874-025-02757-1, 26, 1, (2026).
- LeRoy Graham, Mary Hart, Michael Stinson, Rhoda Moise, Lynda Mitchell, Tonya A. Winders, Donna D. Gardner, “Black People Like Me”: A virtual conference series to engage underserved patients with asthma in patient centered outcomes research, Research Involvement and Engagement, 10.1186/s40900-023-00428-3, 9, 1, (2023).
- Joni C. Carroll, Cindy Jiang, Hailey Mook, Melissa A. Somma McGivney, Kim C. Coley, Patients’ opinions on participation in patient-centered outcomes research in community pharmacies: A qualitative study, Journal of the American Pharmacists Association, 10.1016/j.japh.2022.09.001, 63, 1, (182-187), (2023).
- Karen J. Sherman, The Trials and Tribulations of Selecting Comparison Groups in Randomized Trials of Nonpharmacological Complementary and Integrative Health Interventions, The Journal of Alternative and Complementary Medicine, 10.1089/acm.2019.0460, 26, 6, (449-455), (2020).