Skip to main content
Free access
Research Article
16 June 2017

Attitudes toward comparative effectiveness research and patient engagement among reproductive health clinicians

Abstract

Aim: To assess reproductive health clinicians’ knowledge of and attitudes toward comparative effectiveness research (CER), patient-centered outcomes research (PCOR) and patient engagement in research. Materials & methods: Web-based survey of reproductive health clinicians. Results: Among 103 responding clinicians, familiarity with CER and PCOR was moderate (35 and 44%, respectively). Once definitions were provided, most respondents agreed with the potential positive impacts of CER and patient engagement (65–87%), the importance of PCOR (95–99%) and that their patients might be interested in engaging in research as more than subjects (93%). Conclusion: We found positive attitudes toward PCOR and CER, and a range of experiences with patient engagement in research. There may be untapped potential for PCOR and CER in the reproductive health field.
First draft submitted: 29 September 2016; Accepted for publication: 21 December 2016; Published online: 16 June 2017
Patient-centered outcomes research (PCOR) aims to enable patients and their caregivers to make informed decisions about their health and the care they receive. PCOR often involves comparative effectiveness research (CER), which aims to give people the information they need to make healthcare decisions by providing scientific evidence on the effectiveness, benefits and harms of different treatment options and comparing these options in a real-world setting [1]. Additionally, PCOR strives to meaningfully engage patients, clinicians and other stakeholders as members of the research team and to conduct research on topics that matter to them [2,3]. Patients and other stakeholders can be involved at all stages of research, from setting research agendas and priorities to the dissemination of findings, and the level of engagement may range from consultation to full partnerships between researchers and stakeholders [4–6].
By engaging patients and other stakeholders, the ensuing research and results may be more relevant and responsive to patients’ and stakeholders’ needs, more applicable to diverse groups, more ‘actionable’ and may provide a greater level of accountability and transparency in research [7]. Yet, many research studies do not align with the research areas identified and prioritized by patients and other stakeholders [8], and evidence suggests that there may be a gap between the information people need to make healthcare decisions and what researchers have studied in CER [9].
Stakeholder engagement in research has long been a component of research internationally [10,11], primarily in the UK [12,13], and is a significant component of community-based participatory research frameworks [2,14]. In the USA, the creation of the Patient Centered Outcomes Research Institute by the Patient Protection and Affordable Care Act of 2010 formalized the concept of patient and stakeholder engagement with the goal of producing research that can inform and improve healthcare decision making [14]. While not widely or uniformly evaluated, there appears to be a positive impact on research that engages patients and stakeholders, enhancing both the quality and relevance of studies [7,15,16].
PCOR strives to produce evidence that allows clinicians and patients to make more informed healthcare decisions; thus, clinicians are among the primary consumers of PCOR and an important stakeholder group to engage in research [14]. As key decision makers in healthcare – both with and on behalf of patients – clinicians may be attuned to the needs of patients and aware of relevant gaps in clinical knowledge; therefore, clinicians represent a critically important group to understand and engage. And, because they are among the most common intermediaries between patients and healthcare, any research study or clinical intervention would need adequate clinician buy-in to succeed. However, clinicians are not often engaged as stakeholders in research [16,17], and few studies have assessed clinicians’ general experiences with and attitudes toward CER and/or patient engagement in research; research on the views of nonphysician clinicians is even more sparse [18–20]. Available evidence suggests that clinicians have favorable yet complex views toward CER and PCOR [18–20].
To our knowledge, no research has specifically assessed familiarity with or attitudes toward PCOR, CER or patient engagement among clinicians providing reproductive healthcare. This area of clinical practice and research may be particularly amenable to PCOR methods because of the preference-sensitive nature of reproductive health decision making. To address this knowledge gap, we conducted a survey of clinicians, clinical administrators and researchers at reproductive health centers to explore knowledge of and attitudes toward CER, PCOR and patient engagement in research as well as to inform future opportunities for PCOR and reproductive health.

Materials & methods

Sample

Planned Parenthood is a national network of reproductive healthcare providers across the USA. There are 56 unique, locally governed affiliates that operate approximately 650 health centers, serving over two million people each year to prevent unintended pregnancies through birth control, reduce the spread of sexually transmitted infections through testing and treatment and screen for cervical and other cancers. About half of these health centers also offer abortion care. This voluntary, web-based survey was deployed to Planned Parenthood clinicians, clinical administrators and researchers to share their attitudes toward and experiences with CER, PCOR and patient engagement in research. The survey was sent in June 2015 to a convenience sample of 500 valid email addresses. To increase survey reach, we employed modified snowball sampling by including a forwardable link for recipients to share with colleagues that may not have received the unique survey link, such as per diem clinicians. Response rates and tests for nonresponse bias could not be calculated because survey invitations could be shared with other potential respondents, and we did not collect any information on those individuals who declined to participate.
The survey invitation requested that recipients take a brief survey about their opinions of and experiences with research; it did not mention PCOR, CER or patient engagement. The survey was anonymous and did not collect any identifying information; consent was implied by survey completion. We sent two reminder emails during the 1-month study period. 183 respondents completed the survey. We excluded respondents who did not complete items beyond the demographics questions (n = 66). Due to a small number of respondents who were not clinicians or clinical administrators, we excluded nonclinical respondents (n = 14). Our final analytic sample included 103 respondents. This research was determined to be exempt by the Chesapeake Institutional Review Board.

Measures

The survey consisted of 23 close- and open-ended questions, several of which were adapted from previous surveys about familiarity with and attitudes toward CER and PCOR and attitudes toward, experience with and interest in patient and clinician engagement in research [5,18,19,21]. Input on the survey instrument was provided by the Consumer Patient Researcher Roundtable, an initiative launched in 2011 and maintained by AcademyHealth, but driven by collaboration between consumers, patients, researchers and delivery system leaders.

Demographics & professional characteristics

Demographic questions included respondent age and gender. Professional characteristics included current professional role (e.g., clinician, clinician/researcher or clinical administrator or director), degree, percentage of the average work week spent in clinical care, patient volume and experience conducting research.

Comparative effectiveness research

We assessed familiarity with the concept of CER with one item using a four-point Likert scale (very familiar, moderately familiar, slightly familiar and not at all familiar) [21]. We then provided a definition of CER (research comparing the effectiveness and safety of preventive, diagnostic and treatment options to produce evidence that is useful for making informed decisions in real-world clinical settings), and assessed respondents’ attitudes toward CER by asking them to indicate the extent to which they agreed with four statements about CER measured with a five-point Likert scale (strongly agree, agree, neither agree nor disagree, disagree and strongly disagree) [18,19].

Patient-centered outcomes research

We assessed familiarity with the concept of PCOR with one item using a four-point Likert scale (very familiar, moderately familiar, slightly familiar and not at all familiar) [19]. We then provided a definition of PCOR (CER is considered to be PCOR if it answers questions that help patients and/or their caregivers make the best decisions), and assessed attitudes toward PCOR with seven items using a four-point Likert scale (very important, moderately important, slightly important and not at all important) [19].

Patient & stakeholder engagement

We assessed attitudes toward patient engagement by measuring agreement with four items using a five-point Likert scale (strongly agree, agree, neither agree nor disagree, disagree and strongly disagree) [19]. We assessed clinician experience engaging patients in research by asking them to select as many of the ways they had engaged patients as applied [5]. We also asked respondents about perceived interest of patients in patient engagement in research with one item (yes, no or maybe). Finally, we assessed perceived barriers to patient engagement in research with one item where respondents could select all that applied from a list of options [19].

Data analysis

In this descriptive analysis, we calculated counts and frequencies for each survey item. Missing responses (n = 2–8 across items) were excluded from the total counts before calculating frequencies. Analysis was performed using IBM SPSS v21 [22].

Results

Survey participants

The majority of survey respondents identified as clinicians (66%); the remainder identified as clinician administrators (27%) or clinician/researchers (7%; Table 1). Respondents were physicians (24%); advanced-practice clinicians (78%), including nurse practitioners, certified nurse midwives and physician assistants; and ‘other’ (9%). 9% also held another advanced degree, such as an MPH, MS or PhD/DrPH. Overwhelmingly, respondents were female (89%), which is typical of Planned Parenthood settings. The vast majority of respondents provide direct patient care, with 59% spending more than 75% of their typical week seeing patients. The majority of respondents (79%) had some experience conducting research, such as helping to recruit patients for a study or delivering the clinical intervention in a study.
Table 1. Demographic and professional characteristics of clinician survey respondents, 2015 (n = 103).
Characteristicn%
Age (years):  
– 20–2943.9
– 30–392625.2
– 40–492322.3
– 50–592524.3
– 60 or above2524.3
Gender:  
– Female9289.3
– Male1110.7
Percentage of average week spent seeing patients:  
– I do not see patients54.9
– Less than 25%1918.4
– More than 25% but less than 50%87.8
– More than 50% but less than 75%109.7
– More than 75%6159.2
Professional role:  
– Clinician6866.0
– Clinician/researcher76.8
– Clinical administrator or director2827.2
Number of patients seen in a typical month:  
– Fewer than 300 patients per month5859.2
– 300–500 patients per month3333.7
– Greater than 500 patients per month77.1
Degree:  
– PhD/DrPH11.0
– MD/DO2524.3
– NP6058.3
– CNM1110.7
– PA98.7
– MPH/MS87.8
– Other98.7
Prior experiences with research:  
– Yes8178.6
– No2221.4
Among respondents who typically see patients.
Respondents could select more than one degree.
CNM: Certified nurse midwife; DO: Doctor of osteopathic medicine; DrPH: Doctor of public health; MD: Doctor of medicine; MPH: Master of public health; MS: Master of science; NP: Nurse practitioner; PA: Physician assistant; PhD: Doctor of philosophy.

Comparative effectiveness research & patient-centered outcomes research

Familiarity with CER and PCOR was moderate; 35 and 44% of respondents were moderately or very familiar with CER and PCOR, respectively (Table 2). Once a definition of CER was provided, 87% agreed that CER should be used to develop practice guidelines, which can improve the quality of patient care, and can improve how patients and providers make healthcare decisions (Table 3). Attitudes toward PCOR were positive, with 95–99% of respondents stating that it is moderately or very important that research informs clinical decision making and patient care (Table 4).
Table 2. Familiarity with comparative effectiveness research and patient-centered outcomes research among clinician survey respondents, 2015 (n = 103).
Response categoryCER (%)PCOR (%)
Very familiar7.913.4
Moderately familiar26.730.9
Slightly familiar21.828.9
Not at all familiar43.626.8
Missing responses (n = 6).
CER: Comparative effectiveness research; PCOR: Patient-centered outcomes research.
Table 3. Attitudes toward comparative effectiveness research among clinician survey respondents, 2015 (n = 103).
ItemStrongly agree (%)Agree (%)Neither agree nor disagree (%)Disagree (%)Strongly disagree (%)
CER should be used to develop practice guidelines38.148.510.33.1
CER can improve how patients and providers make healthcare decisions32.054.611.32.1
CER can improve quality of patient care27.858.811.32.1
CER is not important to direct care delivery1.04.114.451.528.9
Missing responses (n = 6).
CER: Comparative effectiveness research.
Table 4. Perceived importance of patient-centered outcome research characteristics among clinician survey respondents, 2015 (n = 103).
ItemVery important (%)Moderately important (%)Slightly important (%)Not at all important (%)
Research to inform clinical decisions78.418.62.11.0
Research that helps patients make best healthcare decisions78.419.61.01.0
Research that compares different treatment options76.322.71.0
Research that answers questions providers care about73.224.71.01.0
Research that measures outcomes patients care about72.225.81.01.0
Research that facilitates dialog with patients about treatment options66.032.01.01.0
Research that facilitates shared decision making64.929.94.11.0
Missing responses (n = 6).

Patient & stakeholder engagement

Attitudes toward patient engagement in research were positive but showed more variation than attitudes toward CER or PCOR. For all items, the majority of respondents agreed with statements about the ways in which patient contributions can improve research and healthcare (Table 5).
Table 5. Attitudes toward patient engagement among clinician survey respondents, 2015 (n = 103).
ItemStrongly agree (%)Agree (%)Neither agree nor disagree (%)Disagree (%)Strongly disagree (%)
Patients working with researchers can improve the value of research23.252.622.12.1
More research studies should actively engage patients20.047.429.52.11.1
Patients working with researchers can improve healthcare16.867.414.71.1
Patients do not have expertise to contribute meaningfully to research2.19.523.251.613.7
Missing responses (n = 8).
Almost half of respondents (48%) reported experience engaging patients in research as more than study subjects in at least one way. For example, 18% had engaged patients in recruiting or retaining study participants, 16% in data collection and 13% in identifying research topics. Few had ever engaged patients in research activities like developing the study budget (2%) or networking and expanding the research team (4%). Potential obstacles to patient engagement selected by respondents included lack of time (81%), lack of knowledge about research (63%) and lack of payment for patients’ time (62%, Table 6). When asked if they believed that some of their patients would be interested in engaging in research as more than subjects, 45 and 48% of respondents responded ‘yes’ and ‘maybe’, respectively.
Table 6. Experience with and perceived barriers toward patient engagement in research among clinician survey respondents, 2015 (n = 103).
Experience engaging patients in research%
Identifying research topics or agenda12.6
Developing the research question4.9
Proposal development6.8
Developing the study budget1.9
Networking and expanding the research team3.9
Leading training activities8.7
Participating in training activities9.7
Study design4.9
Recruiting or retaining study participants17.5
Data collection15.5
Data analysis5.8
Results review, interpretation or translation10.7
Dissemination/sharing study findings12.6
Other3.9
None of the above52.4
Perceived barriers to patient engagement
Lack of time80.6
Lack of knowledge about what research is63.1
Lack of payment for their time62.1
Concerns about sharing sensitive information61.2
Don’t think they have anything to contribute55.3
Lack of interest in research47.6
Language barriers43.7
Don’t see the value of research39.8
Lack of training in research35.9
Lack of access to researchers35.0
Distrust of researchers32.0
Don’t think research will improve their health31.1
None of the above
Statements are ordered according to the steps in study design and conduct.

Discussion

This study assessed a sample of reproductive health clinicians’ knowledge of and attitudes toward CER, PCOR and patient engagement in research. The findings expand upon the limited literature on providers’ attitudes toward CER and patient engagement more generally, and offer further insight into attitudes among nonphysician clinicians and among reproductive healthcare providers, specifically.
Survey respondents were overwhelmingly female and predominantly composed of nonphysician clinicians, such as nurse practitioners, certified nurse midwives and physician assistants. This demographic composition is representative of clinical staff at Planned Parenthood health centers, but differs from other clinician populations previously studied, which included more male and physician representation [18,19], and may not be representative of clinicians in reproductive healthcare more broadly. For instance, the gender distribution among obstetrician–gynecologists in the USA is nearly equal [23].
Familiarity with CER and PCOR was moderate in our sample and was higher for PCOR than CER. The higher familiarity with PCOR could be due in part to the emphasis on patient-centered care at Planned Parenthood. Nonetheless, familiarity with CER was somewhat higher among this population than in prior research with primary care clinicians. While similar proportions of respondents from the two studies reported not being at all familiar with CER (44% in this study vs 45% in Forsythe et al.), a higher proportion of our respondents reported being very familiar (8 vs 5%) or moderately familiar (27 vs 17%) with CER [19]. This increased familiarity with CER may reflect trends of increasing awareness of CER in clinical practice in general.
Overall, Planned Parenthood clinicians in our sample held positive attitudes toward CER, PCOR and patient engagement in research once definitions were provided. Across all questions, the majority of respondents agreed or strongly agreed with the potential positive impacts of CER and patient engagement on research, as well as the importance of the principles of PCOR. When compared with research conducted among physicians prior to passage of the Affordable Care Act and among primary care clinicians in 2012–2013, a higher proportion of our survey respondents agreed that CER can improve the quality of patient care [18,19]. Similarly, views on patient engagement in research were positive; however, there was more variability in attitudes toward patient engagement within our study population than in attitudes toward CER or PCOR.
Planned Parenthood clinicians may have highly positive views on these measures for several reasons. First, our survey participants were overwhelmingly comprised of nonphysician clinicians; past research suggests that physicians report less favorable views on CER than other types of clinicians, and that nurse practitioners and physician assistants are particularly positive toward the value of engaging patients in health research [19]. Thus, strong agreement in these areas may be reflective of the make-up of the underlying clinician population. Second, Planned Parenthood health centers are often involved in research; clinical staff may, therefore, be more receptive to these research methods and the value of research in clinical practice and decision making. Third, as stated earlier, the preference-based nature of much of reproductive health decision making may make this area particularly receptive to CER and PCOR methods and principles. Fourth, those individuals who completed the survey may have had more positive views toward the content of the survey than individuals who chose not to complete it, with potential for response bias. However, this is mitigated by the fact that the survey invitation and introductory language did not include any mention of PCOR, CER or patient engagement. Finally, some of the more positive views toward CER and patient engagement in our study as compared with prior studies may also be due to the later date of our survey and increased visibility of these concepts.
Past research has assessed barriers for clinicians to engage in research as stakeholders [19] or researchers’ perceptions of barriers for stakeholder engagement in research [5]. Here we sought to assess how clinicians – who may understand the willingness and interest of their patients to engage in research – perceive barriers to patient engagement. Additionally, nearly half of Planned Parenthood clinicians had experienced engaging patients in research as more than study subjects. While clinician perceptions of patient attitudes should not be interpreted as a direct proxy for patient attitudes, it is important to consider, and perhaps compare in future research, the difference between provider-mediated attitudes and attitudes directly from patients.
Despite identifying potential barriers, it is notable that when asked if they thought their patients would be interested in future research engagement, 93% of clinicians responded “yes” or “maybe”. Planned Parenthood clinicians also reported an interest in conducting research themselves. The results from our study may indicate a patient and clinician population who are interested in and willing to engage in research on reproductive healthcare; future engagement should be considered and advanced.
Our study had several limitations. The demographic characteristics of respondents, and the underlying population of Planned Parenthood clinicians, may not be reflective of the reproductive healthcare or general healthcare provider community, limiting our ability to generalize our findings to other populations. However, findings may be more easily extrapolated to health centers with similar clinician groups, such as Title X or other safety net family planning providers. Further, the modified snowball recruitment methodology allowed us to increase our reach but limited our ability to assess the response rate. The number of respondents was relatively small, and it is possible that those who completed the survey may not be representative of all Planned Parenthood clinicians.
Where possible, questions were adapted from existing survey instruments on awareness of and attitudes toward PCOR, CER and patient engagement in research, allowing for comparison between our results and others’ findings. Highly positive responses throughout our survey could be due to phrasing of questions or reporting bias related to social desirability of responses. Past surveys using similar statements have found greater variability in agreement, indicating that past respondents were willing to report more neutral or negative views [18,19]. Nonetheless, we can be confident that attitudes toward CER, PCOR and patient engagement are generally positive among this population and the results are comparable to other studies. The present study was quantitative in nature; further exploration of attitudes using qualitative methodology may offer insights into the complexities or nuances of clinician attitudes.
Studies of providers’ attitudes toward CER, PCOR and patient engagement in research have been limited, especially among nonphysician clinicians and reproductive health specialists. However, available information indicates that providers are a stakeholder group that sees the value in CER and PCOR and could benefit the research community through its engagement in research. We did not assess either potential barriers or facilitators to clinicians’ engagement as stakeholders or patient-identified barriers or facilitators for engagement in research activities. While these have been assessed to some extent elsewhere [19,5], future research could explore these factors in the reproductive health field. Assessments of patient attitudes and willingness to engage in reproductive health-related research should also be explored. Though lack of engagement in research may be due to challenges in resourcing and operationalizing engagement activities, clinician attitudes toward engagement are positive. Researchers should take advantage of this support to meaningfully engage clinicians and patients in research.

Conclusion

Given the preference-based nature of reproductive healthcare (e.g., choosing a contraceptive method), reproductive health research may be a natural fit for PCOR and CER. Understanding providers’ views on these topics is an important step in ensuring reproductive health research translates to evidence that is useful for both clinicians and their patients. Attitudes toward CER, PCOR and patient engagement in research were highly favorable among this nonrandom sample of Planned Parenthood clinicians and may demonstrate a population interested in and willing to engage in future research. The principles and methods of CER, PCOR and patient engagement could be harnessed within the reproductive health field to facilitate patients’ and their caregivers’ ability to make informed decisions about their health and the care they receive.
Summary points
Patient-centered outcomes research (PCOR) aims to enable patients and their caregivers to make informed decisions about their health and the care they receive by engaging patients, clinicians and other stakeholders and conducting research relevant to these stakeholders.
While not widely or uniformly evaluated, there appears to be a positive impact of engagement in research, enhancing both the quality and relevance of studies.
There is limited information on clinicians’ general experiences with and attitudes toward comparative effectiveness research (CER) and patient engagement in research, and no previous studies focused on reproductive health clinicians’ familiarity with or attitudes toward PCOR, CER or patient engagement in research.

Materials & methods

A voluntary, web-based survey was sent electronically to Planned Parenthood clinicians in June 2015 to assess attitudes toward and experiences with CER, PCOR and patient engagement in research.

Results

Overall, the study population held positive views toward CER, PCOR and patient engagement in research.
Familiarity with CER and PCOR was moderate. Once definitions were provided, most respondents agreed with the potential positive impacts of CER and patient engagement and the importance of PCOR.
While respondents noted potential barriers to patient engagement, most thought their patients may be interested in engaging in research as more than subjects.

Conclusion

Given the preference-based nature of reproductive healthcare decisions, the reproductive health field may be a natural fit for CER and PCOR. Understanding providers’ views on these topics is an important step in ensuring that reproductive health research translates to evidence that is useful for both clinicians and their patients.
We found positive attitudes toward PCOR and CER, and a range of experiences with patient engagement in research. There may be untapped potential for PCOR and CER in the reproductive health field.

Acknowledgements

We would like to acknowledge AcademyHealth's Consumer Patient Researcher Roundtable members, who contributed to the development of this survey instrument.

Financial & competing interests disclosure

The first author (J Kohn) has received an engagement award from the Patient-Centered Outcomes Research Institute (PCORI). The other authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
No writing assistance was utilized in the production of this manuscript.

Ethical conduct of research

The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.

References

Papers of special note have been highlighted as: • of interest
1.
Agency for Healthcare Research and Quality. What is comparative effectiveness research. http://effectivehealthcare.ahrq.gov/index.cfm/what-is-comparative-effectiveness-research1/.
2.
Sofolahan-Oladeinde Y, Mullins CD, Baquet CR. Using community-based participatory research in patient-centered outcomes research to address health disparities in under-represented communities. J. Comp. Eff. Res. 4(5), 515–523 (2015).
3.
Frank L, Basch E, Selby JV. The PCORI perspective on patient-centered outcomes research. JAMA 312(15), 1513–1514 (2014).
4.
Shippee ND, Domecq Garces JP, Prutsky Lopez GJ et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 18(5), 1151–1166 (2015).
5.
Forsythe LP, Ellis LE, Edmundson L et al. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J. Gen. Intern. Med. 31(1), 13–21 (2016).
6.
Mallery C, Ganachari D, Fernandez J et al. Innovative methods in stakeholder engagement: an environmental scan. www.effectivehealthcare.ahrq.gov/tasks/sites/ehc/assets/File/CF_Innovation-in-Stakeholder-Engagement_LiteratureReview.pdf.
7.
Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J. Comp. Eff. Res. 4(2), 133–145 (2015).
8.
Crowe S, Fenton M, Hall M, Cowan K, Chalmers I. Patients’, clinicians’ and the research communities’ priorities for treatment research: there is an important mismatch. Res. Involvem. Engagem. 1(2), 1–10 (2015).
9.
Workman T, Maurer M, Carman K. Unresolved tensions in consumer engagement in CER: a US research perspective. J. Comp. Eff. Res. 2(2), 127–134 (2013).
10.
Canadian Institutes of Health Research. Strategy for patient-oriented research: patient engagement framework. www.cihr-irsc.gc.ca/e/documents/spor_framework-en.pdf.
11.
Sarrami Foroushani P, Travaglia J, Eikli M, Braithwaite J. Consumer and community engagement: a review of the literature. www.aci.health.nsw.gov.au/__data/assets/pdf_file/0010/165592/Consumer-and-community-engagement-literature-review.pdf.
12.
National Institute for Health Research. INVOLVE. www.invo.org.uk/find-out-more/.
13.
James Lind Alliance. About the James Lind Alliance. www.jla.nihr.ac.uk/about-the-james-lind-alliance.
14.
Frank L, Forsythe L, Ellis L et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual. Life Res. 24(5), 1033–1041 (2015).
15.
Brett J, Staniszewska S, Mockford C et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 17(5), 637–650 (2014).
16.
Forsythe LP, Szydlowski V, Murad MH et al. A systematic review of approaches for engaging patients for research on rare diseases. J. Gen. Intern. Med. 29(Suppl. 3), S788–S800 (2014).
17.
Concannon TW, Fuster M, Saunders T et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J. Gen. Intern. Med. 29(12), 1692–1701 (2014).
18.
Keyhani S, Woodward M, Federman AD. Physician views on the use of comparative effectiveness research: a national survey. Ann. Intern. Med. 153(8), 551–552 (2010).
• First study exploring physicians’ attitudes toward comparative effectiveness research and its use in clinical practice.
19.
Forsythe LP, Frank L, Walker KO et al. Patient and clinician views on comparative effectiveness research and engagement in research. J. Comp. Eff. Res. 4(1), 11–25 (2015).
• Previous study exploring primary care providers’ and chronic and/or rare disease patients’ attitudes toward comparative effectiveness research and patient engagement in research.
20.
Williams SS, Esposito D, Rich EC. Patients and clinicians as stakeholders in comparative effectiveness research: multiple perspectives and evolving roles. J. Comp. Eff. Res. 3(6), 573–575 (2014).
21.
National Pharmaceutical Council. The state of CER and the environment for decision making. www.nationalpharmaceuticalcouncil.com/publication/state-comparative-effectiveness-research-and-environment-decision-making.
22.
IBM Corp. SPSS Statistical Software: release 21. www-01.ibm.com/software/analytics/spss.
23.
Rayburn WF. Characteristics and Distribution of Obstetrician-Gynecologists in the United States. The Obstetrician-Gynecologist Workforce in the United States: Facts, Figures, and Implications 2011. ACOG, DC, USA, 43–66 (2011).