Nine years of comparative effectiveness research education and training: initiative supported by the PhRMA Foundation

In March 2009, the PhRMA Foundation's Health Outcomes Research Committee recommended establishment of a Center of Excellence for CER Education Grant Program, and commenced the writing of an RFP to fund academic Centers of Excellence that would develop and teach CER curricula. Leading pharmaceutical CER/PCOR scientists were invited by E Cannon, then Executive Director (now President) of the PhRMA Foundation and JP Gagnon to join this workgroup [9]. The 17-member panel for the program consisted of academicians, industry scientists, professional association leaders and a federal agency program officer. Members represented the disciplines of medicine, pharmacy, clinical epidemiology, pharmacoepidemiology, pharmacoeconomics, informatics, information sciences and health services research. Consensus development involved completion of a premeeting survey and attendance at an in-person meeting, and culminated in a written consensus report. The premeeting survey used a nominal group process to identify knowledge areas, skills and abilities required to competently conduct or expertly utilize CER/PCOR [10].

In December 2009, the panel met at the PhRMA Foundation's headquarters in Washington DC to discuss the results of the survey and design the model curriculum. Key knowledge areas identified including an understanding of observational research methods, data resources, and the USA healthcare system. Identified skills and abilities included data management, critical evaluation of the literature, development of analytic plans and the ability to interpret findings and the impact of those findings on insurance products [7,9]. It was here where the concepts of dissemination and implementation science and translational science came into focus [7]. The panel saw the need for both didactic and active learning experiences and emphasized a role for real world data. Although, the experience of the panel was largely in the realm of pharmaceuticals, members clearly saw the need to broaden the reach of CER/PCOR to include diagnostics, devices, behavioral and educational interventions and health-system delivery changes. Although, there was sentiment among panel members of the need for a structured doctoral program in CER/PCOR, they also realized that several academic centers already teach these concepts; for them, simple adjustments to current offerings would be all that would be needed.

In an effort to integrate this multidisciplinary field into a cohesive whole, in 2011, the PhRMA Foundation put forth a call to academic centers to develop degree-granting programs in CER/PCOR. To be funded by a 3-year grant to develop and implement a detailed CER curriculum, requirements of the RFP were directly aligned with the report of the Model Curriculum Project. The goals of the RFP were eightfold: to develop programs that train scholars who conduct rigorous, useful and effective CER; to produce high quality researchers who are capable of interpreting and integrating CER into practice; to provide resources to develop corroborating evidence on the usefulness and value of CER; to convene public forums to educate the broader public about CER; to collectively establish a core CER curriculum for the nation; to sponsor lectures in regional and national venues to promote thoughtful discussions about important CER topics; to determine workload needs for CER; and to use electronic means to make the results of CER training materials widely accessible to the public. Eligible academic institutions were those who offered degree granting programs with an existing NIH- or Agency for Healthcare Research and Quality (AHRQ)-funded CER research or training program and the capacity to develop this degree granting program using existing tenure-track faculty. Award selection criteria were also established. The RFP was available for four annual cycles and a total of six academic Centers of Excellence were funded during that time: University of Washington (UW) and Johns Hopkins University (2012), University of Utah and Harvard TH Chan School of Public Health (2013), University of Maryland (2014), and University of Illinois at Chicago (2015). Each Center of Excellence tailors their program offerings to their institution-specific needs and resources. A short description of each is provided below (Table 1).

The University of Washington Center of Excellence in CER leverages existing expertise from within the multidisciplinary UW Centers for Comparative and Health System Effectiveness Alliance. The Center trains across the disciplines of epidemiology, biostatistics, health economics and outcomes research, informatics and health services research. The Comparative and Health System Effectiveness Alliance links capacity and resources across UW, Fred Hutchinson Cancer Research Center, Veterans Affairs Puget Sound Health System and Kaiser Permanente Washington (formerly Group Health) Research Institute. Led by EB Devine, L Garrison, and A Basu, faculty developed a CER Certificate of advanced study for graduate students already enrolled in the Pharmaceutical Outcomes Research and Policy Program or the Department of Health Services. The 16–18-credit certificate requires numerous prerequisites and students take advanced coursework in decision modeling/value of information, evidence synthesis (network meta-analysis), Bayesian statistics, and causal inference. Incorporation of a course in data science is currently under discussion. To date, ten students have completed their PhD having earned the CER certificate; two additional students are currently enrolled in the certificate program. A series of weekly CER seminars throughout each year provides a venue for students and faculty alike to present CER-related ‘works in progress’.

The Hopkins CER Center of Excellence is housed in the existing Center for Health Services and Outcomes Research, led by Director, A Wu and Associate Directors, J Segal and J Marstellar. J Segal leads the CER Center of Excellence. The centerpiece of the Hopkins' CER program is a 21-credit certificate in CER, which can either be completed as a stand-alone certificate online, or used to fulfill requirements for a CER track within the existing Masters of Public Health degree. The curriculum brings together coursework in health status and patient outcomes, epidemiology, informatics, research ethics, evidence synthesis, econometrics, pharmacoepidemiology, decision sciences and research methods. Two courses, developed with the new funding from the PhRMA Foundation, rounded out the offerings – an “Introduction to Comparative Effectiveness and Outcomes Research” and “Evaluating Quality Improvement and Patient Safety Programs”. The certificate can be completed online. The CER center also sponsors an ongoing series of seminars under the umbrella of the Center for Health Services and Outcomes Research.

The Harvard CER Educational Center of Excellence is led by S Hernandez-Diaz. As with UW and Hopkins, Harvard also has a long-standing presence in conducting research and offering training in epidemiology, biostatistics and health economics and outcomes research. The goal of this CER Educational Center is to provide an integrated training, including specialized curriculum with core and elective coursework and supervised research by multiple faculty at the Program on Pharmacoepidemiology and CER at the Harvard School of Public Health and at the Division of Pharmacoepidemiology and Pharmacoeconomics at Brigham and Women's Hospital. The Dean's 2010 initiative in CER positioned Harvard well to receive the PhRMA Foundation award in 2013. Since then, the Doctoral and Master programs offer courses specifically developed to teach cutting-edge methodology for CER, and some existing courses were expanded to include more PCOR aspects. Over time, the theme of big data science has featured more prominently with the addition of courses on healthcare database analytics. Moreover, students can take advanced courses on related areas such as causal inference, decision science, clinical trials or meta-analysis. Harvard has also offered symposia on CER methods for therapeutic and preventive interventions, drug safety management and patient oriented benefit risk evaluation. Five graduate students have received direct support from the PhRMA Foundation grant; many others have benefitted from the program.

With their PhRMA Foundation support, the University of Utah established the “Technology-Oriented CER (TOCER) Training Program”. Led by D Brixner, to meet center objectives, the Utah faculty established a new course in healthcare data analytics, a second course in decision analysis and cost–effectiveness analysis, and a third in the use of secondary data sources for TOCER. They also enhanced an existing course in CER methods. Utah draws students form the Department of Pharmacotherapy, Division of Public Health in the Department of Family Medicine and from the Master of Science in Clinical Investigation Program, School of Medicine. Faculty at Utah have been active in the CER Collaborative, a joint initiative of the Academy of Managed Care Pharmacy (AMCP), the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), and the National Pharmaceutical Council (NPC) [11]. The Collaborative has developed tools, resources and training materials to promote greater uniformity and transparency in the evaluation and use of evidence for coverage and healthcare decision-making. Three students are currently pursuing the CER certificate while completing their PhD in the Department of Pharmacotherapy.

The University of Maryland launched the Patient-Centered Research for Outcomes, Effectiveness, and Measurement (PROEM) center with Dr Eleanor Perfetto as lead faculty. This Center of Excellence focuses on patient centeredness, and faculty strongly advocate for patient engagement as the first step in any CER activity. The PROEM trains both CER/PCOR scientists and decision-makers who interpret and use CER/PCOR, including patients and advocates. Program offerings include a CER/PCOR concentration within existing PhD and Masters (MS) programs (started in 2016), an annual summer institute, a basic, online CER/PCOR training program offering continuing education credits, and post-doctoral fellowships. Three new graduate courses were developed: Introduction to CER/PCOR, Quality and Quality Measurement, and Health Outcomes Measurement. Novel features of the PROEM Center of Excellence include a PCOR training program for rare-disease patient advocates [12], and collaboration with the currently AHRQ-funded, PATIENTS Program (PI: Mullins) [13] and US FDA-funded M-CERSI (PI: Polli) [14]. It also serves as the coordinating center for the CER collaborative' s online, continuing-education courses for its CER certificate program [11]. PROEM has benefitted from formal feedback provided by both internal and external advisory boards, comprised of colleagues from academia, industry, and the public sector, which have provided suggestions for improvement as PROEM's offerings were developed.

The sixth and final center to be funded was the University of Illinois at Chicago (UIC). Based in the Pharmacy Systems, Outcomes and Policy group, co-principal investigators Drs Glen Schumock and Simon Pickard lead this Center of Excellence. UIC's accomplishments featured the development of a new, online MS degree in CER. The purpose of the program is to provide individuals with the skills and knowledge relevant to conducting CER for application in the pharmaceutical and healthcare industries, and is intended primarily for individuals already working in a pharmaceutical or medical product company, government agency, or in a healthcare provider organization. Because it is 100% online, students can be located anywhere in the USA or abroad, and can finish the program at their own pace. The first class will matriculate in Fall 2017. The curriculum is comprised of 20 core and 12 elective credit hours (32 hours in total), and requires completion of a CER capstone project. Among its other aims, the UIC Center had a lead role in the second invitational conference in 2017. It conducted a weekly seminar series on CER and related research for faculty, graduate students, and other researchers. The center also convened workshops of clinical pharmacists and residents to train them in CER best practices. With direction from and external advisory board, the Center was successful in meeting its core aims.

In 2013, the directors of the first two CER Centers of Excellence that were funded (UW, Hopkins), joined with E Cannon and JP Gagnon to plan a national invitational conference intended to strengthen the proposed elements of a CER/PCOR curriculum and to allow the funded centers to share their experiences in implementing these in real-world and diverse academic settings. Funding for the conference was provided by a small conference grant awarded by AHRQ (PI: Segal), and a similar grant from PCORI (PI: Cannon). The PhRMA Foundation also supported the conference. Invited stakeholders were 120 key individuals from academia, educational accrediting bodies, industry, federal and state governments, professional societies, and the private sector; together, representing over 50 organizations. Attendees convened at the offices of the Pew Charitable Trust in January 2014 to address the five conference objectives: review existing competencies proposed for CER; describe existing curricula used to achieve these competencies; identify curricular enhancements; identify workforce needs; and discuss unique methodologic issues of CER unaddressed by related disciplines.

Over the two days of the conference, the directors of the five then funded, Centers of Excellence shared their curricular and training approaches, and leaders from AHRQ, PCORI, FDA, Centers for Medicare and Medicaid Services, industry and academia shared their perspectives. Keynote speakers addressed incorporating CER into policy-making (G Wilensky) and discussed the future of CER (M McClellan). Attendees broke into small groups to discuss the issues at hand. Conference findings included that CER/PCOR is an increasingly multidisciplinary science for which researchers must possess both a breadth of overall knowledge and sufficient depth in one or more areas of expertise. Identified gaps in methods training included patient engagement, decision sciences, dissemination and implementation science, and the use of big data.

In early 2015, the PhRMA Foundation solicited a strategic evaluation of the CER/PCOR Centers of Excellence Training Programs from Avalere Health [15]. Published in May 2015, findings from the report revealed that the Centers of Excellence have operationalized the objectives outlined in the PhRMA Foundation's original RFP. The report further suggested that faculty at the Centers of Excellence were appreciative of the support provided by this initiative, and vision of the Foundation for development of CER curriculum nationally, as this initiative still appears unique in the academic landscape. The flexibility afforded each Center of Excellence in crafting its program around current offerings, existing strengths and available resources has contributed to the success of the program, as opposed to a ‘one size fits all’ approach. All Centers of Excellence have been able to build on strong foundations and pre-existing programs in CER/PCOR. The report noted specifically that several Centers of Excellence have leveraged resources from the existing graduate programs to train researchers in CER; those that developed online programs focused primarily on training users of CER. As the Centers of Excellence have employed a variety of approaches, the combined offerings of all six centers meets, and often exceeds, the eight objectives outlined in the original RFP. A major finding of the report was that addressing the needs of users of CER remains an important gap. Report authors recommended the conduct of a follow-on conference, one that focused on training users of CER.

Funding for the second invitational conference was again provided by the PhRMA Foundation, this time with in-kind support from AMCP. The conference planning committee was led by the directors of the UIC Center of Excellence (Schumock and Pickard), assisted by Dr Perfetto (University of Maryland, MD, USA), along with  E Cannon, JP Gagnon, a faculty liaison from the first conference (Devine) and a representative from AMCP (Saha). As the conference title makes clear, the conference focused on enhancing the uptake and use of CER/PCOR by patients, clinicians and payers. Details of the conference proceedings are found in the two companion papers in this series [16,17].

In launching the CER Education and Training Initiative, the PhRMA Foundation has added to its long-standing and highly successful funding portfolio another landmark program. In sponsoring three seminal conferences and one conference report, and providing seed funding for six academic Centers of Excellence, the foundation has made significant inroads into moving the field of CER/PCOR forward for both research scientists who conduct CER/PCOR and clinicians and patients who use CER/PCOR.

In turn, the six Centers of Excellence each function within their own sphere of influence in training scientists and advancing knowledge and use of CER/PCOR. Each Center of Excellence offers similar coursework and extends its reach through seminars and symposia. Yet, each tailors its offerings to the unique strengths of the offering academic institution. Four Centers of Excellence enhance existing PhD and MS tracks with a CER/PCOR concentration, but do not offer degree-granting programs. Four programs focus on training researchers to conduct CER, two train clinician users of CER. Three centers offer their coursework online. Overall, the Centers of Excellence have largely met the intent and objectives of the PhRMA Foundation's initiative. Today, each center continues to train both those who conduct and use CER, continually evolving to meet the training needs of both in this dynamic healthcare environment. The identification of barriers, existing shortfalls in the use of CER, and suggestions for how to bridge these, were provided at the second conference. It is now up to the six centers, as well as many other academic centers, professional organizations and academic institutions to continue to meet these challenges.

The PhRMA Foundation initiative complements several other major CER/PCOR initiatives that were undertaken within this same timeframe. One effort was led by the members of the Clinical and Translational Science Award (CTSA) Workgroup for CER Education, Training and Workforce Development, of the National Center for Advancing Translational Sciences (NCATS) of the NIH. These investigators proposed preliminary competencies for CER. The group distinguished applied competencies (skills needed by those conducting CER) from Foundational competencies (those needed by both practitioners and users of CER) [18]. The list developed for the PhRMA Foundation's initiative is comprised of the same overarching categories as those developed by those involved in the NCATS initiative; indeed some of the scientists served on both groups.

A second initiative was led from the AHRQ Effective Healthcare Program and took the form of useful guidance. Two of these publications are titled, “Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide” [19] and “Methods Guide for Effectiveness and Comparative Effectiveness Reviews” [20]. More recently, a third initiative has been led by PCORI and has taken the form of the PCORI Methodology Standards, which has now evolved to include 48 standards covering 12 topical areas [21]; a companion academic curriculum has since been developed [22]. These AHRQ and PCORI guidance documents are widely used by research scientists who conduct CER/PCOR. The fourth initiative has been led by the AMCP. This professional society created a CER addendum to their existing AMCP format for Formulary Submissions [23]. The AMCP format is used to facilitate formulary decision-making on the part of Pharmacy & Therapeutics Committees. Relatedly, the AMCP/ISPOR/NPC CER Collaborative rounds out the list [11]. The AMCP format and the CER collaborative both focus on training users of CER/PCOR. Importantly, further efforts are needed to better integrate patients into CER/PCOR studies – from identifying research questions of meaning to patients to disseminating these findings in ways that are easily interpretable and accessible to patients.

While the field of PCOR is still in its relative infancy, its scope is enormous and it continues to expand. Much work remains to be done, and will involve many. For example, Academy Health now offers an annual conference on dissemination and implementation science [24]. In 2016, AHRQ launched the Patient-Centered Outcomes Research Clinical Decision Support Learning Network, to give patients and providers tools to make more informed decisions [25]. Clearly, one of the key priorities for the future is identify and prioritize areas where CER/PCOR evidence has the greatest potential to guide decision making and investments in healthcare.

Beginning in 2009, the PhRMA Foundation launched an ongoing, robust initiative to develop training and educational programs in the field of CER/PCOR. Now completed, while the results of this unique initiative are evident and have made a significant contribution to the national landscape of CER/PCOR, the work is ongoing and requires the efforts of numerous academic institutions, government programs, professional organizations and foundations.

In the past 8 years, methods for conducting CER have advanced and are now widely accepted for use in healthcare decision-making. PCOR is rapidly evolving, with the realization of the importance of the patient's perspective being increasingly recognized over time. With the era of ‘big data’ upon us, the evidence available to conduct CER will become voluminous, and methods to properly and appropriately analyze these data will continue to advance. The patient's voice will likely become ever more central to healthcare decision making. The field continues to evolve and expand, requiring the commitment of numerous academic centers, governmental programs, professional organizations and foundations to improve patient-centeredness and increase efficiencies in the healthcare system.

We thank E Cannon, President of the PhRMA Foundation for her generous support and outstanding leadership, JP Gagnon for his commitment and vision, and our many colleagues who have helped to realize the vision of this national initiative in CER education, specifically A Basu (UW) and SH Diaz (PI: Harvard). We thank E Law (UIC) for his helpful comments on this manuscript.

EB Devine, LP Garrison, E Perfetto, AS Pickard, GT Schumock, JB Segal and D Brixner are principal investigators for their respective CER Centers of Excellence, originally funded by the PhRMA Foundation. E Perfetto is an employee of the National Health Council, a multistakeholder membership organization that receives dues and sponsorship funding from a wide range of organizations. Members and sponsors are listed on the NHCouncil.org website. E Cannon is President of the PhRMA Foundation. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/