Involving the public in systematic reviews: a narrative review of organizational approaches and eight case examples
Abstract
This paper reviews the recent literature on public involvement in the systematic review process. We examine how relevant organizations involve the public in their review processes and how the public are involved in individual reviews. We identified nine surveys or reports of public involvement in systematic reviews at an organizational level and eight examples of public involvement in individual reviews. The public was found to be involved in the following stages of the review process: topic prioritization; refining the scope of the review; suggesting, locating and appraising the literature; interpreting findings; and writing up the review. Numerous tensions, facilitating strategies and recommendations for good practice were identified. Future research directions are delineated.
Public involvement is now recognized in the health research policies of many developed countries [1–3,101–105]. In the UK, for example, research programs such as Research for Patient Benefit funded through the National Institute for Health Research, require researchers to demonstrate how the public were involved in the design and development of their grant applications, and how the public will be involved in managing the research, undertaking the analysis and disseminating the findings if funding is awarded [103]. In the USA, the recently enacted Patient Protection and Affordable Care Act states that the Patient-Centered Outcomes Research Institute will have public representatives on the Board, as well as on expert advisory panels, which will identify research priorities and establish a research agenda [2]. In Australia, the shared vision of the National Health and Medical Research Council and the Consumers Health Forum of Australia is for ‘consumers and researchers [to work] in partnerships based on understanding, respect and shared commitment to research that will improve the health of humankind’ [104]. Internationally, the Cochrane Collaboration has advocated for the direct involvement of the public since the Collaboration’s foundation in 1993 [106].
Because of its high status within evidence hierarchies [4], one of the most important types of health research in which the public can get actively involved is the systematic review. The Cochrane Collaboration, the international organization set up in the 1990s to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of healthcare interventions [107], has provided much of the impetus for involving the public in systematic reviews. Members of the public have been part of the Collaboration since its creation [106], have two seats on its Steering Committee [5,6], are part of various Cochrane Review Groups (CRGs) [6] and make up the membership of the Cochrane Consumer Network (CCNet) [5].
Despite such important developments internationally, Kreis et al. have pointed out that ‘the literature on public involvement in systematic reviews is still relatively sparse’ [6]. The purpose of this paper therefore is to review how relevant organizations involve the public in their review processes and how the public are involved in individual systematic reviews. Building on and updating a recent review of case examples carried out by the authors [3], it is anticipated that this paper will be of use to researchers who wish to actively involve the public in the systematic review process, to commissioners of systematic reviews and to the public thinking of getting actively involved in reviews of interest to them.
Overview of review method
This section of the paper provides an overview of the review method. A full account of the method used to identify relevant documents for this review is set out in an earlier paper [3].
▪ Definition of terms & scope
We were guided by INVOLVE’s definitions of ‘the public’ and ‘public involvement in research’ [3]. INVOLVE defines the public as, ‘patients and potential patients; people who use health and social services; informal carers; parents/guardians; disabled people; members of the public who are potential recipients of health promotion programmes, public health programmes and social service interventions; organizations that represent people who use services’. Public involvement in research is conceptualized as doing research ‘with’ or ‘by’ the public, rather than ‘to’, ‘about’ or ‘for’ the public.
Public involvement in the systematic review process is defined as the involvement of the public in one or more of the following review activities:
▪ Topic identification, prioritization and commissioning
▪ Refinement of review question
▪ Development of review protocol (including search strategy and inclusion/exclusion criteria)
▪ Literature search
▪ Appraisal of the literature, including assessment of study relevance, data extraction and evidence synthesis
▪ Initial drafting of findings
▪ Interpretation of findings
▪ Final report writing
▪ Literature search & data extraction
A literature search was undertaken in January 2011 of the following databases, with the aim of identifying all peer-reviewed journal articles published in English on public involvement in the systematic review process at the organizational and individual review level: PsycINFO, MEDLINE and MEDLINE In-Process, CINAHL, EMBASE, Cochrane Methodology Register, HMIC and invoNET. Recognizing that other terms are used to describe the public (e.g., the Cochrane Collaboration uses the term ‘consumer’), a variety of search terms were used in the literature search, including ‘consumer’, ‘user’, ‘participant’, involvement’, ‘inclusion’, ‘review’, ‘systematic review’ and ‘research’. Note that we did not employ search terms relating to specific organizations that undertake systematic reviews. A further search of a comprehensive bibliography of references on public involvement in health and social care research [108] was undertaken, to identify relevant reports of systematic reviews that involved the public.
On completion of the literature search, titles of papers and (where available) abstracts were scrutinized for possible inclusion in the review by the first author, and these inclusions were checked for accuracy by the second author. Disagreements were resolved through discussion. The search yielded nine surveys or reports of public involvement in systematic reviews at an organizational level and eight case examples of public involvement in individual systematic reviews.
A data extraction table was developed to extract relevant data from each case example of public involvement in an individual systematic review. Data were extracted relating to: country where the review was undertaken; subject area of the review; level, method and stage of public involvement; description of the public actively involved; the contributions made by the public; tensions identified; facilitating strategies and recommendations.
Public involvement in systematic reviews at an organizational level
Nine surveys or reports of public involvement in systematic reviews at an organizational level were identified [5–11,106,109], most of which relate to public involvement in the Cochrane Collaboration and its various CRGs. In this context therefore, we define an organization to be an institution, body or review group that undertakes systematic reviews on a range of different topics. In 1994, Bastian set out the vision for public involvement in the Collaboration [106], and an initial survey by Kelson in 1999 identified the extent to which the Collaboration involves the public through its CRGs [7]. Since Kelson’s survey was undertaken, reports of how four CRGs involve the public in their review processes have been published [8–11]. The most recent review of public involvement in the Cochrane Collaboration is that provided in Wale et al. [5], which discussed findings from two surveys – a CCNet-led evaluation of 2006 and an external consultant-led evaluation of 2009 [109]. Using collaboration-wide surveys and interviews with managing editors of the review groups, the external evaluation reported that just over half of the 36 CRGs felt they were benefiting as expected by involving the public (e.g., increased readability and/or quality of reviews, usefulness of summaries in a public-friendly language), but that all six members of the CCNet interviewed found it very hard to comment on whether their involvement had had any impact. Wale et al. concluded that, while the public are effectively involved in commenting on prepublished Cochrane reviews, opportunities still exist to strengthen the public’s participation in prioritizing, producing and disseminating Cochrane reviews. In particular, Wale et al. reported that public involvement in the prioritization of review topics was not one of the high priorities for many Cochrane review groups [5].
To the authors’ knowledge, only one multiagency comparative study of public involvement in systematic reviews at an organizational level has so far been published [6]. Through in-depth, semistructured interviews with key informants and a review of organizations’ websites, Kreis et al. explored current approaches to involving the public in US-based and key international organizations, and groups conducting or commissioning systematic reviews [6]. Of the 17 organizations contacted, only seven reported that they actively involve the public in their work. These seven organizations are: the Agency for Healthcare Research and Quality’s (AHRQ) Effective Health Care (EHC) Program, including the Center for Evidence-based Policy, the Oregon Evidence-based Practice Center, the Johns Hopkins Evidence-based Practice Center, the Cochrane Collaboration, the Cochrane Musculoskeletal Group, the Cochrane Pregnancy and Childbirth Group and the Campbell Collaboration [6]. Appendix S3 of the online version of Kreis et al.’s paper gives a full description of these seven organizations [6].
At an organizational level, Kreis et al. found the public to be involved in systematic reviews in different ways [6]; for example, within the Cochrane Musculoskeletal CRG, the public are represented on the Editorial Board, the decision-making body for CRGs. At AHRQ, the public are represented in the Stakeholder Group for the EHC Program, a group of 15–20 individuals who represent a variety of constituents, including healthcare providers, payers and the public. This Stakeholder Group meets regularly and provides input into the programme (e.g., on the topic nomination or selection process). The public was found to be involved in topic identification and prioritization of review topics in different ways at an organizational level; for example, AHRQ provides an opportunity for the public to nominate research topics through the EHC Program’s website, and these topics may result in a systematic review. The Cochrane Musculoskeletal CRG reported that it has carried out two brainstorming workshops with the public to identify priorities in musculoskeletal health [6].
Public involvement in individual systematic reviews
This section of the paper updates Boote et al.’s narrative review of public involvement in seven individual systematic reviews [3] by including a further case example that appeared in print in 2012 [12]. As set out in Table 1, of the eight case examples now identified, two provided details of the public involvement within the main systematic review document itself [13,110]. For the remaining six examples [12,14–16,111,112], a reflection on the public involvement appeared in a separate document to the review itself. As Table 2 sets out, the public was found to be involved in a wide variety of review topic areas, and a wide variety of people provided a public perspective to the systematic review process.
▪ Levels & methods of public involvement in individual systematic reviews
Applying INVOLVE’s classification schema, which differentiates between three different levels of public involvement (consultation, collaboration and publicly led) [17], only one case example is classified as ‘publicly led’ [111]. The review of patients’ perspectives on electroconvulsive therapy (ECT) was led by the Service User Research Enterprise at London’s Institute of Psychiatry. This review was conducted and co-authored by two service user researchers with direct experience of ECT (one of whom was the lead investigator) and two clinical academics at Service User Research Enterprise [111]. Four examples involved the public collaboratively [12,15,110,112]. Two approaches to collaboration were noted: a service user researcher being a full member of the review team [15] and public representation on the review’s advisory group [12,110,112]. Six examples involved the public consultatively [13–16,110,111,]. Consultative methods included: a workshop to discuss the systematic review protocol [14,15] and the review’s emergent findings [14,110,111]; a service user reference group [111]; a virtual forum [13]; and a three-round Delphi process conducted through email [16].
▪ Contributions of the public to individual systematic reviews
Five main contributions that the public can make to individual systematic reviews were identified, as discussed below.
Refining the scope of the review
Most examples reported that the public helped to define the scope of the review prior to the search strategy being developed. For example, a service user reference group worked with one review team to set inclusion criteria and a framework for the analysis [15]. Three examples reported public involvement in the choice of outcomes, population, interventions or study type to be included in the review protocol [16,111,112]. For example, Serrano-Aguilar et al. outlined how a Delphi process, involving 53 patients with degenerative ataxias (DA), identified and prioritized DA-related health problems, which were used as DA-related outcome measures on which information was retrieved from included papers [16]. The review of patients’ perspectives on ECT was guided by a reference group, which recommended that emotional impacts of ECT be included as a review outcome [111]. The publicly led nature of this review was believed to have influenced the type of evidence that it included: a decision was made to review user/voluntary group studies, as identified by the user researchers and the reference group, and also testimonies from those receiving ECT, which were sourced from websites, various print media and video archive [111]. In the review of HIV health promotion for men who have sex with men (MSM), the public made suggestions about the types of interventions that should be prioritized for inclusion in the review, as well as the subgroups of MSM and outcomes on which the review should focus [112].
Suggesting & locating relevant literature
Three examples reported that the public was invited to suggest and locate relevant literature or ongoing studies [12,13,111]. For example, Nilsen et al. reported that a key function of the panel of CCNet members was to alert reviewers to unpublished studies that could be included in the review [13]. In the review of patients’ experiences of ECT, the user group ‘Communicate’ located an important unpublished study of ECT for the benefit of the review [111]. In the review of chemoradiotherapy for women with cervical cancer, the Patient Research Partners helped to trace contact details for investigators of ongoing trials relevant to the review [12].
Appraising the literature
In the review of patients’ experiences of ECT, the user researchers were involved in appraising the identified literature [111]. It was reported that the critical perspective of the service user researchers revealed important methodological inadequacies in the identified studies of ECT. The variation in the complexity of the questions, the timing of the assessments and the professional status of the interviewer were all highlighted [111]. For example, it was reported that all the included studies in the review started with certain assumptions and values concerning the intervention that led to differences both in participant selection and in interpretation of data. A template for analyzing the substantive themes was developed that ensured that no single approach to ascertaining consumers’ views on ECT was privileged. The different strategies used by clinical and consumer research to summarize data were also examined to see how diverse conclusions may be reached on the basis of very similar data [111].
Interpreting review findings
Five examples reported public involvement in interpreting review findings [12,14,110–112]. For example, in the review of chemoradiotherapy for women with cervical cancer, the Patient Research Partners attended the collaborators’ meeting, at which the first results of the review and meta-analysis were presented, along with the clinicians and statisticians who had provided trial data for inclusion in the meta-analysis [12]. The review team reported that the Patient Research Partners had brought, ‘the results of the study to life as it evidenced the experience of real people’ [12].
Writing up the review
Three examples reported public involvement in writing up the review, either as first author [111], co-author [15] or as the author of the review’s foreword [110]. In addition, in the review of chemoradiotherapy for women with cervical cancer, the Patient Research Partners contributed to a ‘key findings’ document aimed at clinicians, researchers and patients [12], and provided input into the lay summary for the Cochrane review [18]. The review of patients’ perspectives on ECT is, to the authors’ knowledge, the only published systematic review in the field of health and social care first-authored by a member of a publicly led organization [111].
Tensions identified when involving the public in the systematic review process
This section considers the tensions concerning public involvement in the systematic review process, which were reported in the literature at the individual review level. No tensions were reported at the organizational level, in the surveys and reports that we identified.
▪ Time pressures
Four examples highlighted the time implications of involving the public in the review process [14,15,110,111]. Smith et al. spoke of a lack of time to develop a network of the public able to contribute to the grant application that funded their review [15], while Braye and Preston-Shoot were frustrated at being unable to hold a third consultation event with the public during the review process, due to a tight timescale imposed by the review’s commissioner [14].
▪ Resourcing problems
Two examples reported problems adequately resourcing public involvement in the review process [15,112]. For example, Smith et al. reported a lack of funding to involve the public at the grant-development stage, when the team were applying for funding to carry out their review [15]. In the review of HIV health promotion for MSM, a lack of funding for advisory group work, and the way in which staff time was not allocated explicitly to the management of the advisory group, potentially had an adverse impact on the public’s involvement in the review process [112].
▪ Researchers’ concerns about involving people who are not well & discussing ‘taboo’ subjects
In the review of chemoradiotherapy for women with cervical cancer, the review team spoke of their initial concerns about involving cancer patients who were unwell, as well as other concerns, such as the proposed level of public involvement being seen as tokenistic, and also about discussing potentially ‘taboo’ or sensitive subjects: “We aren’t medical or nursing professionals so don’t have experience of working with patients. We had no idea how well or poorly the women were going to be or whether any of them were coming into it with preconceptions that would make it difficult to work with them” [12].
▪ Continuity issues
For a myriad of reasons, it is not always possible for the public to attend all meetings that they are asked to do so by reviewers. For example, in the review of chemoradiotherapy for women with cervical cancer, it was reported that one Patient Research Partner, due to the advanced nature of her illness, became too unwell to attend the advisory group meetings during the conduct of the review [12].
▪ Concerns about group dynamics
Two examples reported concerns about group dynamics within the review’s advisory group when members of the public were present [111,112]. For example, during the review of patients’ experiences of ECT, some advisory group members (who were clinical and university-based researchers) voiced concerns about the power and authority of the service user researchers [111]. It was noted that time and commitment was needed to foster trust between the user researchers and clinical researchers and for equitable working patterns to be established [111]. During the review of HIV health promotion for MSM, it was reported that some clinical and university-based members of the advisory group were overly dominant, and the contributions from the public at these meetings varied [112].
▪ Research Ethics Committee involvement
For two examples, a favorable opinion was sought from a Research Ethics Committee to involve the public in the review process [15,16]. For one example, the Research Ethics Committee insisted the review team obtain informed consent from the public actively involved in the review process using an informed consent form. The review team was unhappy with this requirement as it changed the status of the public from that of advisors to that of research participants [15].
▪ Concerns about the selection & representativeness of the public
Three examples highlighted tensions regarding how the public involved were selected [15,112] and the degree to which they were representative [16]. In one example, members of service user organizations were invited to join the review’s service user reference group [15]. The review team reported that they had not sufficiently anticipated the emotional responses that the notion of joining the group on behalf of an organization could provoke [15]. In the review of HIV health promotion for MSM, the researchers expressed concern that the public who took an active part in the review process were drawn exclusively from members of organizations representing the interests of MSM, and not from MSM themselves [112]. The authors of the review of treatments for DA expressed concern that the characteristics of the public who were actively involved in the review process might be different from patients with DA as a whole [16].
▪ The public’s perceptions on their degree of influence
In the review of chemoradiotherapy for women with cervical cancer, one Patient Research Partner expressed reservations about the degree to which the public can influence systematic reviews, compared with primary research [12]. This person expressed concerns at the outset regarding how much input ‘users’ could really make in the context of a systematic review, given that the outcomes are to a degree, “preset by the outcomes that were collected within the individual trials”. She stated that, “for a meta-analysis where the outcome measures have already been collected, I am not sure how much difference we have really made overall.” This cervical cancer patient doubted that she would get involved in another systematic review, as she felt that there is more opportunity to shape primary research [12].
Facilitating strategies & recommendations for good practice
This section outlines facilitating strategies and recommendations for good practice for involving the public in the systematic review process, which have been articulated in the literature at either the organizational or individual review level.
▪ Funding & payment
Although at an organizational level, the public was found to be involved mainly on a voluntary basis [6], at an individual review level, some review teams, in the cases we identified, did pay the public for their time. Braye and Preston-Shoot recommend that systematic reviewers should budget for the cost of public involvement, in order to pay the public for their time and to meet any out-of-pocket expenses [14]. Reflecting this, two examples reported that they had budgeted for public involvement in their review’s funding application [12,15]. Smith et al. further recommend that reviewers should discuss payment early on with service users: this is because payment for involvement can affect the benefits that some service users may receive and it can take time to get payment systems set up within host institutions [15].
▪ Identifying a lead or advisory group for public involvement
Braye and Preston-Shoot recommend a member of the review’s team should take lead responsibility for public involvement, and that the time for this activity should be adequately budgeted [14]. An alternative approach was suggested by Vale et al., who reported that the review team was advised on public involvement by an expert Reference Group [12]. This Reference Group, which included the Consumer Liaison Lead from the UK’s National Cancer Research Network, two experts in public involvement in health research and a cervical cancer patient, advised the review team on how best cervical cancer patients could be recruited to be actively involved in the review, as well as what support and information should be provided, and the activities that the cervical cancer patients might undertake in the review process [12].
▪ Recruiting the public through relevant networks
In the review of chemoradiotherapy for women with cervical cancer, working with relevant charities was considered to be an effective way of recruiting patients to be involved in the review process. This was because the people who use such charities are, ‘very involved in their disease and its treatment and they will also potentially have wider networks of people that they are in contact with to draw experiences from’ [12].
▪ Training, briefing & information provision
In order to maximize the contribution of the public at the organizational or individual review process, the provision of training, briefing notes and background information has been recommended [14]. At an organizational level, Skoetz et al. reported that an introduction to statistical methods for evaluation of clinical trials and the critical appraisal of trials are offered to public members of the hematological malignancies CRG [8]. At the individual review level, in the review of HIV health promotion for MSM, members of the advisory group were offered a glossary of key terms used in the review process, and briefing papers were sent out before each advisory group meeting [112]. In the review of chemoradiotherapy for women with cervical cancer, the Patient Research Partners were provided with a detailed information folder and an accompanying workshop aimed at describing systematic reviews and meta-analyses [12]. The women actively involved in this review valued such support, and one provided this feedback: “The presentations and the material took great care to unravel the complex world of research acronyms and concepts and they explained complex ideas simply but without dumbing down. That made me feel that we were equal partners in a really important piece of work” [12].
▪ Structured methods of involvement
To refine the population, interventions and outcomes of interest with the involvement of the public at the start of the review process, the use of structured group processes was considered helpful [16,112]. For example, the use of voting within review advisory group meetings was considered a useful way of prioritizing interventions and the population subgroups on which the review should focus [112]. Structured methods of involvement help to reduce the influence of overly dominant members of a review’s advisory group, while ensuring that the public’s contribution is considered alongside that of health professionals [112]. Serrano-Aguilar et al. reported that structured Delphi questionnaires, administered through email, enabled the review team to involve the public rapidly and inexpensively [16]. The Delphi method also meant that face-to-face interactions between researchers and the public were unnecessary, thus preventing any negative group dynamics from occurring [16].
Discussion
This review has identified eight case examples of public involvement in individual systematic reviews, as well as reports and surveys of how relevant organizations involve the public in their systematic review processes. At the organizational level, the literature mainly details how the public is involved in the work of the Cochrane Collaboration and its various CRGs [5–11,106,109]. Refereeing of protocols and reviews were found to be the most popular and well-established mechanisms for involving the public in CRGs, and many CRGs now ensure that all review protocols and the reviews themselves are refereed by at least one member of the public [5–11]. Internationally, seven organizations have been identified that routinely involve the public in their review processes, including topic prioritization [6].
At the individual review level, the public was found to be involved at the consultation, collaboration and publicly led levels of involvement, and at all the key stages of the systematic review process: refining the scope; suggesting, locating and appraising relevant literature; interpreting the review’s findings; and writing up the review [12–16,110–112]. A wide variety of members of the public were involved in the individual examples identified, including patients, carers, user researchers and representatives from third sector organizations. Different methods of public involvement in the individual review process were noted, including consultation workshops, membership of review advisory groups, membership of the review team, email discussion lists and the Delphi process. Numerous tensions concerning, and facilitating strategies and recommendations to assist, public involvement in the systematic review process were identified, mainly at the individual review level, rather than the organizational level. It should also be noted that most of the tensions identified at the individual systematic review level were predominantly those voiced by health professionals. The purpose of the following discussion is to reflect on the implications of some of the key findings of this review, to consider the review’s limitations, and to set out future research directions.
Two inter-related concerns that researchers voiced about involving the public in the individual review process were funding and time pressures. For example, one example reported that the research team could not involve the public as much as they had hoped due to the tight delivery deadline imposed on them by the commissioner [14]. Commissioners need to recognize that meaningful public involvement in the review process may mean that reviews take longer to deliver while being more expensive, compared with reviews not involving the public. Commissioners need also to consider how best to fund the involvement of the public in the development of grants to conduct systematic reviews [15].
Two case examples reported that ethical approval was obtained for those members of the public actively involved in the systematic review process [15,16]. In one case, this requirement was queried [15]. In England, recently published guidance states that, ‘ethical approval is not needed for the active involvement element of the research, where people are involved in planning or advising on research’ [113]. Researchers in England are not therefore obliged to obtain written informed consent from the public actively involved in the design and conduct of systematic reviews. Researchers outside England are advised to follow best practice guidance in their respective country on this issue. It is recommended that the Cochrane Collaboration, and other international organizations that undertake systematic reviews such as the Campbell Collaboration, should consider issuing their own guidance on whether or not ethical approval is required for the public to be actively involved in the review process.
▪ Limitations of the review
Due to none of the major health databases currently having a subject heading to describe public involvement specifically in the research process, it is possible that papers on public involvement in the systematic review process may have been overlooked. Only publications written in English were searched for, and it is acknowledged that further evidence may be found in work published in other languages.
▪ Recommendations for reporting public involvement in systematic reviews & for future research
Only eight examples were identified of public involvement in individual systematic reviews; we would therefore encourage further case examples to be published in order to strengthen the evidence base. To ensure that these case examples are identified in any future systematic review on this topic, we recommend that a systematic review’s abstract and main body of text should include details of the contribution(s) of the public (if any) to the review process, together with the stage(s) at which, and the method(s) by which, the public was involved.
No quality standards currently exist to guide best practice for the involvement of the public in the systematic review process. Therefore, we endorse Braye and Preston-Shoot’s recommendation for consensus-development work to be undertaken to develop such standards [14], using the consensus-derived principles and indicators of successful public involvement in NHS research [19,20] as a starting point. Quality standards for the involvement of the public in the systematic review process would be of value to all key stakeholders: to review funders and commissioners, to help assess the proposed public involvement in applications to carry out systematic reviews; to systematic reviewers, to guide best practice; and to the public involved in the review process, as a tool to monitor the quality of their involvement.
At the organizational level, with the notable exception of the external CCNet evaluation [109], very little formal evaluation of the effects of public involvement in the systematic review process has been carried out [6]. As Kreis et al. state: ‘at the current time, good quality evidence is lacking about whether or not public involvement has an impact on outcomes such as relevance and accessibility of a review or acceptance of the results. There has been no formal evaluation of the impact, and one would expect that different approaches (e.g., providing the opportunity for public comment, involving the public as key informants or forming an advisory board including the public) have different effects, depending on which aim is pursued’ [6]. Further research at the organizational level is therefore needed to address these issues.
Conclusion
This paper has examined the published evidence relating to how the public is involved in the systematic review processes of relevant international organizations, and how the public is involved in individual systematic reviews. We would encourage other review teams to publish their own accounts of how they have involved the public in the review process in order to strengthen the evidence base, and we would encourage organizations that involve the public in their systematic review processes to rigorously evaluate the impact that the public are having, in order that we might learn about what works best, for whom and in what circumstances.
Future perspective
If the suggestions made in this paper are taken up by the research community, then it is likely that, in 5–10 years time, consensus-development work will have been undertaken to develop quality standards for the involvement of the public in the systematic review process, and that the reporting of public involvement in a review’s abstract, and the main body of a systematic review, will become mainstream. In addition, it is hoped that a comparative, realist evaluation of the involvement of the public in systematic reviews at a programmatic level within relevant international organizations will have been undertaken.
| Case example | Systematic review publication(s) | Publication describing the public’s contributions to the review process |
|---|---|---|
| Teaching, learning and assessment of law in social work education | [21,114] | [14] |
| User involvement in nursing, midwifery and health visiting research | [22,23] | [15] |
| Degenerative ataxias | [24] | [16] |
| Patients’ perspectives on electroconvulsive therapy | [25] | [111] |
| HIV health promotion for men who have sex with men | [26] | [112] |
| The conceptualization, measurement, impact and outcomes of public involvement in health research | [110] | [110] |
| Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material | [13] | [13] |
| Chemoradiotherapy for women with cervical cancer | [18,27] | [12] |
| Case example, country (year) | Level, method and stage of public involvement | Details of the public actively involved | Contributions made by the public to the review process | Ref. |
|---|---|---|---|---|
| Teaching, learning and assessment of law in social work education, UK (2005) | Consultation: two conferences comprising lay people and health professionals at the early and late stages of the review process | 15 participants identified themselves as working from a service user or carer perspective | Finalized the research question and the review protocol. Evaluated the data collected during the review process, reviewed emerging findings, made recommendations for the report, and considered the wider implications of the findings | [14] |
| User involvement in nursing, midwifery and health visiting research, UK (2009) | Collaboration: a service user researcher was a part of the research team Consultation: a service user reference group was established | Members of 24 national service user organizations | Refined the scope of the review, to set inclusion criteria and developed a framework for the analysis | [15] |
| DA, Spain (2009) | Consultation: three-round Delphi process through email, to inform the development of review protocol and literature search | 53 patients with DA | Identified and prioritized DA-related health problems, which were used as DA-related outcome measures on which information was retrieved from the papers included in the review | [16] |
| Patients’ perspectives on ECT, UK (2007) | Public led: carried out by the SURE at the Institute of Psychiatry in London Consultation: the team was guided by a review reference group that met at the early and late stages of the review process with several points of communication in between | The review was conducted and co-authored by two service user researchers. The review reference group included representatives of user and voluntary groups with a stake in ECT research. The reference group had a majority of people with direct experience of ECT | The review reference group advised on the location of the literature, and reviewed drafts of the report. The group also recommended that the reviewers consider the emotional impact of ECT among the review outcomes. The user group ‘Communicate’ located an important unpublished study of ECT | [111] |
| HIV health promotion for MSM, UK (2007) | Collaboration: the review’s advisory group included people from organizations campaigning on behalf of, and providing services for MSM. The advisory group contributed to the design of the review protocol | Representatives of Terrence Higgins Trust, Gay Men Fighting AIDS, Sigma Research and The National AIDS Trust | The advisory group met three times. Meeting 1 agreed terms of reference and the plan for the review. Meeting 2 prioritized the types of interventions to be reviewed, the subgroups of MSM and outcomes on which to focus. Meeting 3 was held to comment on initial findings from the syntheses, to advise on communicating reports of the review, and the draft of the review | [112] |
| The conceptualization, measurement, impact and outcomes of public involvement in health research, UK (2010) | Collaboration: lay presence on the review’s advisory group and the public were involved as advisors at all the design, conduct and analysis stages of the review process Consultation: towards the end of the review, a workshop with lay people and health professionals was held to discuss emerging findings | Three lay people on the review’s advisory group. Lay people attended a consultation workshop to help interpret the review’s findings | At the final writing-up stage, a lay member of the review advisory group wrote the foreword to the review. The lay people on the advisory group and those who attended the workshop helped to interpret the findings | [110] |
| Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material, Norway (2006) | Consultation: through email discussion list. Consulted on the review protocol, literature search, and the initial draft of the review | A panel consisting of members of the Cochrane Consumer Network | The panel was invited to make the reviewers aware of unpublished studies that could be included in the review at the literature search stage | [13] |
| Chemoradiotherapy for women with cervical cancer, UK (2012) | Collaboration: the Patient Research Partners attended review progress meetings | A group of five Patient Research Partners – women who have had experience of chemoradiotherapy | The Research Partners: helped to trace contact details for investigators of ongoing trials relevant to the review; helped to interpret review findings and the meta-analysis; contributed to a ‘key findings’ document aimed at clinicians, researchers and patients; and provided input into the lay summary for the Cochrane review | [12] |
DA: Degenerative ataxias; ECT: Electroconvulsive therapy; MSM: Men who have sex with men; SURE: Service User Research Enterprise.
Background
▪ Public involvement is a key part of health research policy in many developed countries throughout the world.
▪ One of the most important types of health research in which the public can get actively involved is the systematic review.
Public involvement in systematic reviews at an organizational level
▪ Nine surveys or reports of public involvement in systematic reviews at an organizational level were identified, most relating to public involvement in the Cochrane Collaboration.
▪ One multiagency comparative study of public involvement in systematic reviews at an organizational level revealed seven organizations routinely involve the public in their systematic review processes, including topic prioritization.
Public involvement in individual systematic reviews
▪ Eight case examples of public involvement in individual systematic reviews were identified.
▪ Five main contributions that the public can make to individual systematic reviews are: refining the scope; suggesting and locating relevant literature; appraising the literature; interpreting the review’s findings; and writing up the review.
▪ Key tensions associated with involving the public in the review process include: time pressures; resourcing problems; continuity issues; concerns about group dynamics; and concerns about the selection and representativeness of the public.
▪ Facilitating strategies and recommendations for involving the public in systematic reviews include: funding and payment; identifying a lead or advisory group for public involvement; recruiting the public through relevant networks; training, briefing and information provision; and structured methods of involvement.
Conclusions & recommendations
▪ Commissioners need to recognize that meaningful public involvement in the review process may mean that reviews take longer to deliver whilst being more expensive compared with reviews not involving the public.
▪ Reviews that have involved the public need to make public involvement explicit in both the abstract and the main body of the review. This will allow future reviews of public involvement to identify all the available evidence.
▪ Quality standards should be developed for the involvement of the public in the systematic review process.
▪ More research is needed to evaluate the effectiveness of public involvement in the systematic review process at the organizational level.
Financial & competing interests disclosure
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
No writing assistance was utilized in the production of this manuscript.
References
Papers of special note have been highlighted as: ▪▪ of considerable interest
References
1.
Department of Health. Research Governance Framework for Health and Social Care (2nd Edition). Department of Health, London, UK (2005).
2.
Patient Protection and Affordable Care Act. Public Law 111–148 (2010).
3.
Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy 102,105–116 (2011).
▪▪ Presents the first published narrative review of case examples of public involvement in systematic reviews in health and social care.
4.
Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J. Clin. Nurs. 12,77–84 (2003).
5.
Wale J, Colombo C, Belizan M, Nadel J. International health consumers in The Cochrane Collaboration: fifteen years on. J. Ambul. Care Manage. 33(3),182–189 (2010).
▪▪ Most recent evaluation of the contribution of the public to the work of the Cochrane Collaboration.
6.
Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. doi:10.1111/j.1369–7625.00722.x (2012) (Epub ahead of print).
▪▪ The only multi-agency comparative study of public involvement in systematic reviews at an organizational level that has so far been published.
7.
Kelson M. Consumer collaboration, patient-defined outcomes and the preparation of Cochrane Reviews. Health Expect. 2(2),129–135 (1999).
8.
Skoetz N, Weingart O, Engert A. Resources review. A consumer network for haematological malignancies. Health Expect. 8,86–90 (2005).
9.
Shea B, Santesso N, Qualman A et al. Consumer-driven health care: building partnerships in research. Health Expect. 8(4),352–359 (2005).
10.
Ghersi D. Making it happen: approaches to involving consumers in Cochrane reviews. Eval. Health Prof. 25(3),270–283 (2002).
11.
Sakala C, Gyte G, Henderson S, Neilson JP, Horey D. Consumer-professional partnership to improve research: the experience of the Cochrane Collaboration’s Pregnancy and Childbirth Group. Birth 28(2),133–129 (2001).
12.
Vale C, Tierney J Spera N, Whelan A, Nightingale A, Hanley B. Evaluation of patient involvement in a systematic review and meta-analysis of individual patient data in cervical cancer treatment. Syst. Rev. 1(1),23 (2012).
13.
Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst. Rev. 3,CD004563 (2006).
14.
Braye S, Preston-Shoot M. Emerging out of the shadows? Service user and carer involvement in systematic reviews. Evid. Policy 1(2),173–194 (2005).
15.
Smith E, Donovan S, Beresford P et al. Getting ready for user involvement in a systematic review. Health Expect. 2,197–208 (2009).
16.
Serrano-Aguilar P, Trujillo-Martín MM, Ramos-Goni JM, Mahtani-Chugan V, Perestelo-Pérez L, Posada-de la Paz M. Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Soc. Sci. Med. 69(6),920–925 (2009).
17.
Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy 61(2),213–236 (2002).
18.
Chemoradiotherapy for Cervical Cancer Meta-analysis Collaboration. Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: individual patient data meta-analysis. Cochrane Database Syst. Rev. 1,CD008285 (2010).
19.
Telford R, Boote J, Cooper C. What does it mean to involve consumers successfully in NHS research? A consensus study. Health Expect. 7(3),209–220 (2004).
20.
Boote J, Barber R, Cooper C. Principles and indicators of successful consumer involvement in NHS research: results of a Delphi study and sub-group analysis. Health Policy 75(3),280–297 (2006).
21.
Braye S, Preston-Shoot M, Johns R. Law in social work education: reviewing the evidence on teaching, learning and assessment. Social Work Education 24(5),547–563 (2005).
22.
Smith E, Ross F, Donovan S et al. User Involvement in the Design and Undertaking of Nursing, Midwifery and Health Visiting Research. NCCSDO, London, UK (2005).
23.
Smith E, Ross F, Donovan S et al. Service user involvement in nursing, midwifery and health visiting research: a review of evidence and practice. Int. J. Nurs. Studies 45,298–315 (2008).
24.
Trujillo-Martín MM, Serrano-Aguilar P, Monton-Alvarez F, Carrillo-Fumero R. Effectiveness and safety of treatments for degenerative ataxias: a systematic review. Mov. Disord. 24(8),1111–1124 (2009).
25.
Rose D, Wykes T, Leese M, Bindman J, Fleischmann P. Patients’ perspectives on electroconvulsive therapy: systematic review. BMJ 326,1363–1367 (2003).
26.
Rees R, Kavanagh J, Burchett H et al. HIV Health Promotion and Men Who Have Sex with Men (MSM): a Systematic Review of Research Relevant to the Development and Implementation of Effective and Appropriate Interventions. EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, London, UK (2004).
27.
Chemoradiotherapy for Cervical Cancer Meta-Analysis Collaboration. Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials. J Clin. Oncol. 26,5802–5812 (2008).
▪ Websites
101.
NIH Director’s Council of Public Representatives. http://copr.nih.gov/
102.
Consumers’ Health Forum of Australia. History. www.chf.org.au/history.php
103.
NIHR. Patients and public. www.crncc.nihr.ac.uk/ppi
104.
National Health and Medical Research Council and the Consumers Health Forum of Australia. Statement on Consumer and Community Participation in Health and Medical Research. Commonwealth of Australia 2002. www.nhmrc.gov.au/_files_nhmrc/publications/attachments/r22.pdf
105.
Canadian Institutes of Health Research. Citizen Engagement Framework. www.cihr-irsc.gc.ca/e/41753.html
106.
Bastian H. The power of sharing knowledge: consumer participation in the Cochrane Collaboration 1994. http://consumers.cochrane.org/sites/consumers.cochrane.org/files/BastianPowerofSharingKnowledge_1994.pdf
107.
The Cochrane Collaboration. www.cochrane.org
108.
Boote J. Patient and public involvement in health and social care research: a bibliography. www.rdsyh.nihr.ac.uk/file.ashx?id=3959
109.
CCNet. CCNet external review of consumers in the Cochrane Collaboration, focusing on review groups (supplement to December 2009/January 2010 CCNet Newsletter). www.ihe.ca/documents/CCNet%20external%20review_AppendixJan2010.pdf
110.
Brett J, Staniszewska S, Mockford C, Seers K, Herron-Marx S, Bayliss H. The PIRICOM Study: a systematic review of the conceptualization, measurement, impact and outcomes of patients and public involvement in health and social care research. UK Clinical Research Collaboration (2010). www.ukcrc.org/index.aspx?o=3233
111.
Carr S, Fleischmann P. Systematic review of consumers’ perspective on electro-convulsive therapy. In: Collection of Examples of Service User and Carer Participation in Systematic Reviews. Carr S, Coren E, (Eds). SCIE, London, UK (2007). www.scie.org.uk/publications/researchresources/rr02.pdf
112.
Rees R, Oliver S. An example from sexual health promotion. In: Collection of Examples of Service User and Carer Participation in Systematic Reviews. Carr S, Coren E (Eds). SCIE, London, UK (2007). www.scie.org.uk/publications/researchresources/rr02.pdf
113.
National Research Ethics Service and INVOLVE. Patient and public involvement in research and Research Ethics Committee review 2009. www.invo.org.uk/pdfs/involvenresfinalstatement-310309.pdf
114.
Braye S, Preston-Shoot M, Cull L-A, Johns R, Roche J. Teaching, learning and assessment of law in social work education. Institute for Excellence, London, UK (2005). www.scie.org.uk/publications/knowledgereviews/kr08.asp
Information & Authors
Information
Published In
Copyright
© Future Medicine Ltd.
History
Published online: 11 September 2012
Keywords:
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Downloaded 231 timesCitations
How to Cite
Involving the public in systematic reviews: a narrative review of organizational approaches and eight case examples. (2012) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer.12.46
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Anna Gavine, Albert Farre, Fiona Lynn, Shona Shinwell, Phyllis Buchanan, Joyce Marshall, Sara Cumming, Louise Wallace, Angie Wade, Elayne Ahern, Laura Hay, Marianne Cranwell, Alison McFadden, Lessons for the UK on implementation and evaluation of breastfeeding support: evidence syntheses and stakeholder engagement, Health and Social Care Delivery Research, 10.3310/DGTP5702, (1-206), (2024).
- Piotr Teodorowski, Elisa Jones, Naheed Tahir, Saiqa Ahmed, Sarah E Rodgers, Lucy Frith, Public Involvement and Engagement in Big Data Research: Scoping Review, Journal of Participatory Medicine, 10.2196/56673, 16, (e56673), (2024).
- Nila Sathe, Colleen Ovelman, Naykky Singh Ospina, Omar Dewidar, Elizabeth Terhune, Damian Francis, Vivian Welch, Patricia Heyn, Tiffany Duque, Meera Viswanathan, Centering Racial Health Equity in Systematic Reviews Paper 6: Engaging racially and ethnically diverse stakeholders in evidence syntheses, SSRN Electronic Journal, 10.2139/ssrn.4735939, (2024).
- Benjamin D Scher, Juliet Scott-Barrett, Matthew Hickman, Benjamin W Chrisinger, Participatory Research Emergent Recommendations for Researchers and Academic Institutions: A Rapid Scoping Review, Journal of Participatory Research Methods, 10.35844/001c.74807, 4, 2, (2023).
- Daniel J. Gould, Marion Glanville-Hearst, Samantha Bunzli, Peter F. M. Choong, Michelle M. Dowsey, Research Buddy partnership in a MD–PhD program: lessons learned, Research Involvement and Engagement, 10.1186/s40900-023-00414-9, 9, 1, (2023).
- Peter Tugwell, Vivian Welch, Olivia Magwood, Alex Todhunter-Brown, Elie A. Akl, Thomas W. Concannon, Joanne Khabsa, Richard Morley, Holger Schunemann, Lyubov Lytvyn, Arnav Agarwal, Alba Antequera, Marc T. Avey, Pauline Campbell, Christine Chang, Stephanie Chang, Leonila Dans, Omar Dewidar, Davina Ghersi, Ian D. Graham, Glen Hazlewood, Jennifer Hilgart, Tanya Horsley, Denny John, Janet Jull, Lara J. Maxwell, Chris McCutcheon, Zachary Munn, Francesco Nonino, Jordi Pardo Pardo, Roses Parker, Kevin Pottie, Gabriel Rada, Alison Riddle, Anneliese Synnot, Elizabeth Tanjong Ghogomu, Eve Tomlinson, Karine Toupin-April, Jennifer Petkovic, Protocol for the development of guidance for collaborator and partner engagement in health care evidence syntheses, Systematic Reviews, 10.1186/s13643-023-02279-1, 12, 1, (2023).
- Nadia Corp, Lucy Bray, Carolyn A. Chew‐Graham, Kay Polidano, Tamsin Fisher, Adam D. Farmer, Megan McDermott‐Hughes, Benjamin Saunders, Self‐directed self‐management interventions to prevent or address distress in young people with long‐term physical conditions: A rapid review, Health Expectations, 10.1111/hex.13845, 26, 6, (2164-2190), (2023).
- Archana Shrestha, Dipesh Tamrakar, Bhawana Shrestha, Biraj Man Karmacharya, Abha Shrestha, Prajjwal Pyakurel, Donna Spiegelman, Stakeholder engagement in a hypertension and diabetes prevention research program: Description and lessons learned, PLOS ONE, 10.1371/journal.pone.0276478, 17, 10, (e0276478), (2022).
- Laura J. McGowan, Emma C. Joyes, Emma A. Adams, Aishah Coyte, Richard Gavin, Catherine Richmond, Hosein Shabaninejad, Fiona Beyer, Angela Broadbridge, Kevin Dobson, David Landes, Suzanne Moffatt, Richard G. Watt, Falko F. Sniehotta, Ruth Freeman, Martha Paisi, Clare Bambra, Dawn Craig, Eileen Kaner, Sheena E. Ramsay, Investigating the Effectiveness and Acceptability of Oral Health and Related Health Behaviour Interventions in Adults with Severe and Multiple Disadvantage: Protocol for a Mixed-Methods Systematic Review, International Journal of Environmental Research and Public Health, 10.3390/ijerph182111554, 18, 21, (11554), (2021).
- Ursula Ellis, Vanessa Kitchin, Mathew Vis-Dunbar, Identification and Reporting of Patient and Public Partner Authorship on Knowledge Syntheses: Rapid Review, Journal of Participatory Medicine, 10.2196/27141, 13, 2, (e27141), (2021).