Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities
Abstract
Patient-reported outcomes (PROs) are reports of the status of a patient's health condition that come directly from the patient. While PRO measures are a well-developed technology with robust standards in research, their use for informing healthcare decisions is still poorly understood. We review relevant examples of their application in the provision of healthcare and examine the challenges associated with implementing PROs in clinical settings. We evaluate evidence for their use and examine barriers to their uptake, and present an evidence-based framework for the successful implementation of PROs in clinical practice. We discuss current and future developments for the use of PROs in clinical practice, such as individualized measurement and computer-adaptive testing.
Figure 1. Conceptual model for the use of standardized and individualized patient-reported outcomes in clinical practice.
PROM: Patient-reported outcome measure.
First draft submitted: 17 November 2015; Accepted for publication: 8 June 2016; Published online: 18 July 2016
Patient-reported outcome (PROs) measures (also sometimes referred to as PROMs) are health status assessments elicited directly from patients. PRO measures (PROMs) usually take the form of standardized questionnaires. To date, PROs have been used mainly for research purposes and occasionally also for monitoring the health of a given population, mostly as part of population surveys. In some countries, notably in the National Health Service (NHS) in England, there has been an explicit effort to put them at the center of the development of a more patient-centered, outcomes-oriented performance model [1].
Aggregated measurements of PROs for well-defined populations have a potentially huge role to play as a driver of patient-centered quality improvement. A critical step toward that aim, however, will be whether success is achieved in their use at the level of individual clinical decision-making, which currently constitutes an underdeveloped area of implementation. There is therefore a pressing need to evaluate the challenges of using PROs in clinical practice, scrutinizing implementation and the evidence supporting their use.
In this paper, we will:
Provide an overview of what PROs are, including the different types of measure and applications;
Evaluate the research evidence for the use of PROs in clinical practice;
Review previous experiences of utilizing PROs at both the aggregated and the individual level;
Provide a framework for the successful implementation of PROs in clinical settings;
Explore potential future developments for the use of PROs in clinical practice.
Overview of PROs
Over recent years there has been a great expansion in the use of PROs, especially for research purposes, yet there have been some confusion as to what the term ‘PRO’ actually refers to. For the purpose of this review we will use the definition provided by the US FDA, which defines PRO as an umbrella term that incorporates “any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else” [2]. The appeal of this terminology is rooted in the fact that it avoids much of the theoretical debate surrounding the conceptualization and definition of many key-related constructs, such as perceived health, health status, (health-related) quality of life and well being, while at the same time emphasizing the genuine importance of the individual's own perspective when making an evaluation on their health [3]. In this paper we refer to PROs in relation to overall constructs and PROMs where we refer to instruments operationalizing the measurement of the construct.
A previous model-based classification system for research and clinical practice has established three guiding concepts for PROMs: construct, population and measurement model [3]. First, construct refers to the range of characteristics (traits and states) measured by the instrument along with its measurement object (e.g., symptom status, functional status, health perceptions, health-related quality of life). Second, the population of a PROM is the range of people for who the instrument is suitable (e.g., age, gender, condition and culture). Third, measurement incorporates metric (the method used to assign numeric values to the responses given by the individuals and the construction of the scores), dimensionality (the number of scores produced for each individual), adaptability (the extent to which the instrument can be tailored to the specific circumstances and preferences of each individual – the degree to which measurement is standardized or individualized).
The majority of PROMs tend to be presented in the form of fully standardized questionnaires with fixed items and response options. The first widely used PROMs included the Sickness Impact Profile [4], the Nottingham Health Profile [5] and the Instrumental Activities of Daily Living [6]. Most widely used measures today include the EuroQOL EQ-5D [7] and the Short Form SF-36 [8]. These were originally classified as generic measures, for example, applicable across and within populations, regardless of gender and disease or condition. However, the traditional distinction between generic and specific instruments has been challenged based on the fact that many so-called generic measures do not actually cover all major constructs (symptoms, function, health perceptions, health-related quality of life), nor are applicable across all age groups (children, adults, elderly) [9].
Less commonly, instruments may also allow patients to modify their content or scoring system to adapt to their own circumstances. These include individualized PROMs (iPROMs), such as the Person Generated Index, [10] and the Canadian Occupational Performance Measure [11]. Computer-adaptive PROMs, in which the most relevant items are iterative selected and administered by a learning computer algorithm until sufficient information about the patient is collected, also provide an alternative. Individualized and computer-adaptive PROMs will be considered in greater depth later in the review.
The number of PROMs has greatly expanded in the recent years, and the two most comprehensive databases identify a large number of instruments (PROQOLID [711] [12] and BIBLIOPRO [1356] [13]). The co-existence of competing measures aimed at the same populations has triggered the development of standardized systems for the evaluation of the instruments focusing on psychometric properties (validity, reliability, responsiveness, interpretability and cross-cultural adaptation) and ease of use (including administration burden) with the aim of supporting head-to-head comparisons, such as EMPRO [14–16] and COSMIN [17,18].
PROs have most commonly been utilized for research purposes. They have also been utilized, albeit less frequently, for monitoring the health of a stated population, typically through periodic surveys [19,20]. In clinical practice, PROs allow for objective measurement of the patient's subjective view of their health status to complement other clinical measures. Numerous benefits for using PROs have been cited, such as the screening, diagnosing, monitoring of conditions, along with the promotion of patient-centered care [21]. In theory, however, there are many more potential clinical applications (Box 1).
Each of these potential applications can be considered plausible because PROs are measures of health status. Unlike measures related to processes of care, which are intermediate products of the health system, these are end points relevant to the patient. Unlike measures of patients’ satisfaction with care, they do not focus on an evaluation of the standards of care, but rather on the impact that care itself has on the health of individuals. PROs have now, however, joined measures of processes of healthcare, along with patient experience and satisfaction, in the armamentarium of feedback and measurement tools for improving the quality of healthcare. There have also been barriers to their use in clinical settings, for example, rather than being considered a helpful addition they have often been regarded as disruptive to healthcare, and an impediment to doctor–patient communication [22]. Evidence also suggests that clinicians may lack awareness of when to incorporate PROMs in consultations [23] and may also lack expertise with regards to the interpretation and application of such measures [24].
Evidence for using PROs in clinical practice
A number of systematic reviews have been conducted aiming to evaluate the impact of using PROMs in clinical practice [9,25–29] and a wide reaching Cochrane review on the routine provision of PROMs to healthcare providers and patients is currently underway [30]. Other PROM-related systematic reviews have focused on mental health [31,32]; oncology [33] and palliative care [34] (one with a focus on barriers and facilitators for implementation, and two scoping reviews have concentrated on aspects of routine feedback, including a typology of the diverse ways in which PROMs have been used in mental health services [35], the impact on health outcomes and implementation factors in routine cancer care [36].
Out of those systematic reviews which have taken a broad perspective including all type of conditions, settings and measures [25,27,30] the most cited is Valderas et al. [9] which critically appraised the impact of routinely measuring patient-reported outcomes in clinical practice across 34 randomized clinical trials. These reviews have found evidence for impact upon the processes of care, specifically, an increase in the rate of diagnoses and chart notations for the conditions targeted by the interventions. This observation was confirmed for the specific use of PROMs for screening and case finding of depression in another systematic review [31], although this significant although limited effect (relative risk [RR]: 1.38; 95% CI: 1.04–1.83) was mostly due to the effect of a reduced number of studies in which patients were considered eligible only when scoring above a certain severity threshold. Similarly, there was also a positive effect on the advice and education provided by the healthcare professionals. However, no changes were observed in terms of medication and similarly, no impact was observed on the management of depression. The impact of the use of PROMs on outcomes of care remained inconclusive for the vast majority of the systematic review, except for the specific use of PROMs for feedback on treatment outcome in mental health specialist settings [32]: a significant positive short-term effect on mental health was observed which did not prevail in the long run (effect size d: 0.10; 95% CI: 0.01–0.19).
Valderas [9] identified a total of 36 end points for the randomized controlled trials (RCTs) included in the systematic review, which appears to reiterate the lack of consensus among researchers in relation to how interventions should work and thus what constitutes a relevant indicator when using PROMs in clinical practice. In addition, these systematic reviews have consistently observed methodological limitations with regards to design and analysis. Critically, many interventions did not consider providing training on how to interpret the information collected by the PROMs. More importantly, many studies demonstrated an incomplete understanding of the mechanisms by which the intervention operates [9,21,37]. In this way, a considerable methodological limitation is that they have attempted to establish the impact of the use of PROs on distal primary outcomes, such as improvement on health status, without understanding impact upon proximal outcomes, for instance communication between patients and the healthcare professionals [38,39].
The potential effect of the use of PROMs on health outcomes is crucially mediated by the modification of the behavior of both patients and professionals [9,40]. ‘Patient centeredness’ is a critical component in this model, and while all PROMs are self-reports of the patient's health status, only some of them explicitly focus the evaluation on those areas that are most relevant for each patient. iPROMs are concerned with areas which are directly nominated by patients themselves [3], research conducted in the UK has highlighted the need to incorporate individualized assessment alongside standard measures [37].
Lessons learned from using PROs at the aggregate level
The extent to which PROs have been implemented in clinical settings globally has, to date, been somewhat varied. PROMs are currently at the center of an extensive program for the orientation of the NHS in England, the largest and oldest single-payer healthcare system in the world, toward a performance model based on health outcomes. This marks the most ambitious program implementing PROMs to date globally in terms of scale, marking the first time that all patients undergoing specific interventions have been invited to complete measures before and after specific treatments [41]. Given the scale of the NHS program, a detailed examination of implementation in clinical practice in England, and lessons that can be learned from this, is particularly relevant.
The NHS PROMs program has been operating since 2009 [41]. Under this mandatory program PROMs (the EQ-5D alongside condition-specific instruments), have been collected for all NHS patients undergoing one of four elective surgical procedures (hip replacement, knee replacement, varicose vein and groin hernia surgery), both before and after surgery [42]. The PROMs program has focused on the collection of the aggregation of patient scores for comparison of providers rather than to inform individual patient management and clinical decision-making. Information is collected at an individual level while feedback takes place at a group level, while such feedback has limitations, the resources involved in gathering such information in the first place are extensive, and many lessons can potentially be learned from this to inform clinical decision-making. A substantial body of knowledge on the routine collection and feedback of these measurements is developing as a result of its evaluation, particularly in terms of rationale, feasibility and impact (Box 2) [28,40,43–49].
Policymakers now possess evidence about many aspects of the use of PROs in order to drive quality improvement. First, a weak association is observed between scores from PROMs and scores from patient-reported experience measures (PREMs) in elective surgery [44]. Second, it is feasible to organize data collection and feedback on a national scale, with modest recruitment rates (45–68% depending on the intervention) with a high postoperative response (65–85%) [45,46]. Third, the choice of metric derived from PROMs makes a substantial difference to the results, such as the proportion of providers defined as ‘poor’ [45].
Although the PROMs program has been positive in the sense that many participating patients reported valuing their experience, there was, however, no discernible impact on patient outcomes over its first 3 years [43]. This may be because providers only started receiving feedback in the third year of the program, in addition, the presentation of feedback was, at the time, not of high quality, and there was little room for improvement for the selected procedures. However, the evidence for using feedback on PROMs for quality improvement at a group level is severely limited. A systematic review [28] was only able to identify one study with methodological limitations that, not surprisingly, did not observe a positive impact [47].
The underlying assumption is that measurement of PROs, along with clear feedback and dissemination of this information, will stimulate and incentivize health professionals and ultimately health trusts to provide better care [40]. A key challenge is how to bridge the gap between the information that is provided at the aggregate level with changes in care delivery based on individual patients [1,49]. So far, these two agendas have developed relatively independently of each other, without joined up thinking about how they might come together. For example, it will be essential to maintain high response rates for aggregate use of PROMs to monitor quality; this is much more likely if there is engagement by health professionals and patients. More importantly it is difficult to anticipate how clinicians will be able to identify opportunities for improvement of care as measured against the standard of PROM scores unless these scores are part of their daily experience of managing patients. The expansion of the clinical applications of PROMs will be pivotal to the success of the whole PROMs program. A realist synthesis of the literature is underway [50] to help identify users’ aims and the theories underlying their expectations and will put these observations in an appropriate context.
A framework for implementing PROs in clinical practice
Using available evidence from systematic reviews and from qualitative research with patients and health professionals [51–53], along with specific guidelines for implementing outcome measures in clinical practice [54–56], and other available evidence including how PROMs feedback is intended to improve the care of individual patients [57], we have developed a framework of key requirements for the successful implementation of PROs in clinical practice (Box 3).
Based on this, five key issues have been identified that need to be considered when implementing PROs in clinical practice: the purpose of using the measure, the characteristics of the PRO itself, the setting, the feedback system and additional support to the implementation. A strategy that is key to success is targeting patients and health professionals that are most likely to benefit from it. Specific requirements include training for clinicians on the administration and interpretation of PROMs; the use of existing information systems for feedback to clinicians, that is frequent and timely; and the provision of information that is linked to specific actions, among others [9,21,58–59].
We present a practical example of how these requirements can be operationalized, based on a study evaluating the use of PROs for children with advanced cancer in the USA (Box 4). This randomized controlled trial [60], conducted at three large pediatric cancer centers, comprehensively considers the issues of implementing PROMs in clinical practice with lessons that can be applied elsewhere. In this study, the use of PROs was used to support the routine monitoring of children and teens undergoing treatment, and allowed clinicians to systematically track symptoms and health-related quality of life over time. The design of the study incorporated the key issues we have identified above (and in Box 3), in particular special consideration was given to feedback for both families and providers, with printed reports and emails provided immediately after survey completion. Families and providers were also offered training in how to interpret the results. The study authors acknowledged potential limitations to their feedback intervention, in that it may have lacked leverage required to affect clinical behavior and subsequent referrals. The limitation of feedback linked to treatment decision-making is observed in comparable studies [21] and constitutes an ongoing barrier to the take up of PROs in clinical practice.
Our framework for the implementation of PROs in clinical practice also provides insight into the ultimately unsuccessful implementation PROs in clinical practice on a large scale in a rather unique context. As part of the Quality and Outcomes Framework, an incentive scheme for primary care services covering the vast majority of the population of England, the use of PROs was promoted in the management assessment of depression in primary care (Box 5). Neither training nor specific indications were given on which measure to choose from a choice of different measures, the rationale for choice, when or how to introduce the measure, when to collect the score, or how to introduce it within the flow of a clinical appointment, and scarce information was provided regarding the instruments themselves [61]. The feedback system was also considered poor, with the scores not being available in a timely fashion, lack of time for integrating the information provided by PROs with other clinical information and irregular feedback. The only elements considered were the selection of instruments tailored to the setting and the linking of PRO scores (depression severity) to specific action (treatment options), through explicit reference to the NICE clinical practice guidelines for the management of the condition [62]. As such, the framework, as outlined, provides a means of learning from the lessons of the past, indicating ways PROs can be more successfully implemented in clinical practice in the future.
Future developments
The potential of individualized PROs
A key challenge facing healthcare systems across the world concerns how to promote patient-centered care in the context of the increasing demands of patients managing multiple, long-term conditions [68]. It is estimated that between a quarter and half of primary care patients have more than one health problem [69,70]. Even though clinically complex patients have an increased use of healthcare services than other members of the population, they face worse health outcomes [71,72]. The use of iPROMs, where the patient can explicitly prioritize which conditions or symptoms to address, offers the potential of aligning the priorities of patients and professional [73].
iPROMs are measures for which individual respondents are actively involved in the specification of content and/or scoring of the instrument [74]. Such measures empower patients to evaluate the relevance of different aspects of their health conditions [75], either by selecting the specific domains, weighing their importance or both [3,76]. By tailoring the questionnaire to the person's own perspective, iPROMs have a higher likelihood of detecting issues that may be relevant in clinical practice, as well as being responsive to the individual aspects of health-related quality of life [77]. iPROMs are more likely to be able to detect a change in patient's health status than standardized tools alone (in turn, among standardized tools condition-specific instruments will tend to prove more responsive than more generic ones). They may be particularly useful for patients with multiple conditions in helping them to convey to practitioners which aspects of their health they would like to be prioritized.
Previous efforts to incorporate PROMs into clinical practice have focused almost exclusively on the use of standardized measures, with implicit focus on the measurement of outcomes amenable to healthcare. This is consistent with a clinical management approach based on the disease-oriented biomedical model [78]. Alternatively, a patient-centered care approach would place the focus on whole person care and the elicitation of health and healthcare goals. This approach, more consistent with a biopsychosocial model, would benefit from the use of iPROMs [79]. Although conceptually different, these two approaches are complementary (Figure 1).
Despite their ability to convey personal preferences and idiosyncratic aspects from the patient's perspective, there are some disadvantages of using iPROMs, both from a structural and a methodological perspective. Structural disadvantages include the burden associated with administration, at the researcher, clinical and patient level [76]. Methodological issues include the absence of a common metric, which raises concerns regarding analyses and comparison of results (not only between patients but also for the same patient across time), and the possibility of successively obtaining floor effects if the items generated by the patient are too idiosyncratic [77]. Moreover, distinct processes are used for item and content generation, resulting in wide ranging levels of individualization [77]. This results in measures with different degrees of individualization, ranging from those individualized at the content level, where the respondent can nominate the items (e.g., the Goal Attainment Scaling) to those where the items are provided to the respondent, who is then able to rate the importance and relevance of each item, thus being individualized at the scoring level (e.g., the Audit of Diabetes Dependent Quality of Life) [80]. Given these issues it is vital to ensure that iPROMs are methodologically rigorous and fit for purpose. Currently none of the major systems for the standardized evaluation of PROs considers degrees and quality of individualization [9,18,81]. There is, therefore, a pressing need to establish, through expert consensus, rigorous evaluation standards for assessing the methodological quality of iPROMs.
The administration of different combinations of standardized and individualized PROMs could provide an alternative to the current, compartmentalized remit of health provision, allowing for multiple conditions to be addressed within a single healthcare consultation and help to drive forward patient-centered prioritization. But the question still remains about whether the same PRO can be used for both purposes.
Item banks & computer-adaptive tests
As we move further into the information age, the greater availability of ‘smart’ computing solutions alongside increasing application of advanced psychometric models to healthcare PROMs grants a number of new and exciting opportunities to enhance PRO use in clinical practice.
In the field of psychometrics, ‘modern’ measurement theories are increasingly being utilized to create questionnaires that are both shorter and more accurate than those developed using ‘classical’ methodologies [65,82]. Modern measurement methods, including item response theory [83] and Rasch analysis [84] are being increasingly adopted in health services research and have recently been used to develop PROs which measure wide-ranging phenomena including pain, depression, fatigue and the impact of multimorbidity [85–88]. Strikingly, in other fields where questionnaires are used to capture ‘high stakes’ information, such as educational testing and psychometric assessment of employees, they have been considered the industry standard for decades [89].
One key attribute of item response theory is the ability to calibrate precise item parameters (the degree of the underlying phenomena they measure), rather than simply applying a numeric value to a response. In practice, this means that the validated scales may be administered using any number or combination of items, rather than it always being necessary to administer the entire scale to each respondent. This had led to the increasing development of large ‘banks’ of psychometrically calibrated questionnaire items, known as item banks [90].
One advantage of item banks is their ability to capitalize on ‘intelligent’ computerized questionnaire administration algorithms, especially computer-adaptive tests (CATs); which can improve the quality of measurement while administering fewer items than a standard paper-based questionnaire [91]. Computer adaptive testing improves measurement by selecting the most appropriate items for each patient, based on their previous responses. For example, when administering a functional impairment questionnaire using CATs, the computer would be unlikely to administer an item that represents a very high level of functional impairment (e.g. “I have problems getting dressed”) once it had decided (following previous item responses) that the respondent probably had very little impairment (e.g., if they responded positively to a statement about exhausting exercise). Questionnaires using CAT usually take a fraction of the time to complete and, because of the rigorous item bank development, can be even more reliable, valid and sensitive to change than compared with their paper-based counterparts [92]. Despite the complexity of the underlying CAT algorithms, their administration is straightforward and has been shown to be acceptable and accurate across diverse populations, including older adults [93] and children [94].
Taken at face value, the increased precision and brevity of CATs may appear to be their most appealing feature in busy clinical environments. But there are other significant benefits to using this technique in clinical practice. The ability of CATs to accurately ‘target’ questionnaire items means that patients may be spared from responding to items which are either plainly obvious or potentially distressing; such as an item asking a patient in the very early stages of a progressive degenerative disease if they are experiencing the high levels of impairment that are typical of later disease stages.
The computerized nature of CAT grants a number of other significant benefits for PRO administration. These include the immediate provision of feedback, which can be automatically tailored to the individual responses of the patient. More practical advantages provided by computerized administration include no missing data and automatic scoring, inputting and storing. These issues have all been shown to be important barriers to the uptake of PROs in clinical practice [95,96].
Future perspective
The coming years have exciting potential for the use of PROs in clinical practice, and we predict a number of ways in which the field will evolve over the next decade. As technology plays an ever greater part in society and healthcare systems, we anticipate that computerized devices incorporating CAT, such as tablets, will be used for the increased collection and feedback of PROs, making effective use of both standardized and individualized measures, incorporating personalized feedback for both clinicians and patients. We also anticipate such application will be implemented in an increasingly longitudinal fashion, for PRO scores to become part of clinical reviews over time, linked to changes in patient outcomes, activation and goal setting. As part of this it will be important that clinical staff are trained in the understanding and application of PROMs, to avoid mistakes of the past, which will involve developing expertise in how to interpret measures and how to feed this information back to patients in order to inform treatment. At present there is a lack of link up with PROs and clinical decision-making, as to how interpretation of measures can directly influence clinical decision-making at the ‘bed-side’. This constitutes an area that needs to be addressed in coming years, a more comprehensive framework needs to be developed in order to provide greater guidance to clinicians as to how confident they can be in measured values, and how interpretation of such values can be linked to a range of appropriate treatment options.
Conclusion
The evidence for the role of PROs in clinical practice is currently more compelling for impact upon the processes of care, than on outcomes of care. A key barrier for implementation continues to be lack of alignment with current clinical processes, particularly in terms of insufficient linkage between interpretation of scores and clinical decision-making.
Based on available evidence, we have developed a framework of the key requirements for the successful implementation of PROs in clinical practice, encompassing the instrument itself, purpose of the PRO, setting, feedback system, support to implementation (specifically training for clinicians on the administration and interpretation of PROs) as well as characteristics of patients and health professionals involved. In the future this framework could be further developed into a ‘bed-side’ tool kit providing clinicians with confidence in values, how to interpret them and link them to treatment options (in a way that is responsive to the needs of the patient).
We are now starting to see effective administration and instant feedback of PROs through computerized devices. The next logical step in this is to develop enhanced relevance to each patient. Most previous efforts have focused on the use of fully standardized measures thereby limiting opportunities to tailor measurement to individual preferences. To this end, individualized measurement and efficient administration through the use of CAT represent very promising developments for further uptake of PROs in clinical practice, administering measures in a way that is appropriate and responsive to each patient and different aspects of their health conditions. Crucially CATs offer a streamlined approach which can reduce burden compared with the time required to complete PROMs through traditional means.
Supporting decision-making in the diagnostic process:
Screening
Diagnosis
Informing risk stratification and prognosis (identification of vulnerable patients and patients ‘at risk’)
Supporting prioritization and goal setting
Supporting decision-making in indication for treatment (medical/surgical)
Facilitating monitoring of:
General health status
Response to treatment/management
Facilitating communication:
Between patients and health professionals
Within teams and between professionals: consistent use along the care pathway
Within a single service:
Longitudinal evaluation of the service as a whole and for audit
Comparing outcomes between different user groups
Contributing to the evaluation of individual clinicians’ practice and/or for comparing performance across clinicians in the same service
Across multiple services:
Longitudinal evaluation of the performance of a system as a whole
Comparing performance of individual services
Comparing outcomes between different user groups
Within a single service:
Evaluation of performance of the service as a whole along the time and for audit
Comparing outcomes between different user groups
Contributing to the evaluation of individual clinicians’ practice and/or for comparing performance across clinicians in the same service
Across multiple services:
Evaluation of performance of the system as a whole along the time
Comparing performance of individual services
Comparing outcomes between different user groups
Clinical activity:
Screening
Diagnosis
Risk stratification and prognosis
Prioritization
Goal setting
Indication for treatment (medical/surgical)
Monitoring of:
General health status
Response to treatment/management
Facilitating communication:
Between patients and health professionals
Within teams and between professionals (along care pathways)
PROM:
Simple (low burden of administration)
Valid
Reliable
Responsive
Interpretable (explicit link between PROMs outputs and clinical activity)
Tailored to the particular setting and clinical activity
Setting:
Questionnaire-friendly environment
Feedback system:
Integrated in clinical information systems
Structured, with explicit interpretation of the individual scores
Targeting both all relevant healthcare professionals and patients
Frequent and timely
Support to the implementation:
Training of patients and health professionals on the interpretation of scores and outputs
Additional resources
In addition, the effectiveness of the implementation can be maximized by targeting specific groups most likely to benefit from this approach
Patients:
Male
Those previously unknown to the clinician
Those with perceived borderline status of measurement in relation to the clinical activity
Health professionals with low familiarity/degree of confidence in the clinical activity
PROM: Patient-reported outcome measure.
This is the first RCT designed to test a PRO feedback intervention among children and teens with advanced cancer. Wolfe and colleagues [60] assessed 339 children for eligibility, with 104 allocated to either the intervention arm of the study or the control group. The control group completed questionnaires using tablet computers no more than once a week: PQ survey consisting of age and respondent-adapted versions of the MSAS, PedsQL4.0, and an overall sickness question. Oncologists and families received printed reports summarizing PROs; e-mails were sent to oncologists and subspecialists when predetermined scores were exceeded. The control group completed the questionnaires but no feedback was provided. The feedback intervention had two components: PQ reports and PQ e-mails. PQ reports were printed reports that were given to providers before the visit and to families immediately after survey completion. PQ reports consisted of bar plots of PedsQL4.0 and MSAS symptom scores from current and four prior administrations, a summary highlighting changes since the last report, and a list of available resources for symptom control (for families) or generic pain management recommendations (for providers). Training on how to interpret PQ reports was offered to families at enrollment and annually to providers. Primary outcomes included linear trends of MSAS, PedsQL4.0 total and subscale scores and sickness scores during 20 weeks of follow-up, along with child, parent and provider satisfaction with PQ feedback. Although routine feedback of PROs did not significantly affect the child's symptoms or HRQL, changes were in expected directions and improvements observed in emotional HRQL through exploratory analyses were encouraging. Importantly, children, parents and providers valued PRO feedback. Post hoc subgroup analyses found larger and statistically significant improvements on the HRQL emotional domain scores and the sickness question for patients who survived longer than 20 weeks.
HRQL: Health-related quality of life; MSAS: Memorial Symptom Assessment Scale; PedsQL4.0: Pediatric Quality of Life Inventory 4.0 Generic Core Scale; PQ: Pediatric Quality of Life and Evaluation of Symptoms Technology; PRO: Patient-reported outcome; RCT: Randomized controlled trial.
In addition to PROMs being implemented as part of the NHS PROMs program, PROMs have also been implemented in primary care as part of the QOF, a voluntary annual reward and incentive scheme in primary care. Incentives for the assessment of the severity of depression at diagnosis and at follow-up using PROMs were introduced in 2006 [61], with minor changes subsequently introduced in subsequent years. Other than being directed to three specific scales validated for using in primary care (PHQ, BDI-II and HADS), [62], GPs received no specific guidance regarding the implementation of the data collection and feedback system.
The indicators were quickly adopted by the practices, as evidenced by data provided by the Health and Social Care Information center. In 2006–2007, two out of three practices achieved the maximum amount of points, a proportion that increased to four out of five by 2011–2012. Although these figures may indicate a degree of success, qualitative research with patients and health professionals suggested that the use of PROs was not having the desired impact on either group.
Patients overall favored the use of PROs, which they considered a useful supplement to clinical judgement that could aid the GP in the diagnosis, management and monitoring of depression [52,63]. They also noted their usefulness in providing feedback and raising awareness of symptoms of depression [52,63]. Of particular interest, patients perceived the administration of these instruments as confirmation of their problems and as evidence that their GP was taking their problems seriously [52].
The perceptions of clinicians, however, were substantially different. Multiple concerns were raised in relation to the applicability, administration and interpretation of the PROMs. Lack of specific training [64] and simply not knowing when and how to administer the measures were major problems. Faced with a lack of guidance on the questionnaires might be used in clinical practice and the challenge of fitting their administration into a 10-min consultation GPs often introduced significant changes to established administration procedures, affecting where and how the measure was completed, and the number of questions asked [64] and compromising the validity of the scores [65]. It was also not always clear to GPs how to interpret the scores [53]. The PROMs were often not considered as an integral part of assessment and diagnosis, and their administration was felt to disrupt the normal flow of the consultation, [53,64], while adding an additional burden to the workload and roles of GPs and practice nurses [66]. Perhaps not surprisingly GPs concerns on the use of these PROMs have resulted in their use no longer being specifically incentivized as part of QOF [67].
BDI-II: Beck Depression Inventory Second Edition; GP: General Practitioner; HADS: Hospital and Anxiety Depression Scale; NHS: National Health Service; PHQ: Patient Health Questionnaire; PRO: Patient-reported outcome; PROM: Patient-reported outcome measure; QOF: Quality and Outcomes Framework.
Overview of patient-reported outcomes
Patient-reported outcomes (PROs) is an umbrella term incorporating any report of the status of a patient's health condition that comes directly from the patient, without interpretation of responses by a clinician or anyone else.
The majority of PRO instruments are fully standardized questionnaires with fixed items and response options.
The number of PRO measures has greatly expanded in the recent years. There are many potential clinical applications and evidence is evolving to support this.
Evidence for using PROs in clinical practice
Systematic reviews have found evidence for PRO measures impacting upon the processes of care. Evidence is less compelling for impact on outcomes of care. A new Cochrane review is currently underway.
Many PRO interventions have had methodological limitations in design, such as incomplete understanding of mechanisms and lack of training to clinicians in how to interpret scores.
Five key issues require attention for the successful implementation of PROs: i) the purpose of using the measure, ii) the characteristics of the PRO itself, iii) the setting, iv) the feedback system and v) additional support to the implementation.
The potential for individualized PROs
Individualized PROMs are measures for which individual respondents are actively involved in the specification of content and/or scoring of the instrument.
Such measures have a higher likelihood of detecting issues that may be relevant in clinical practice, as well as being responsive to the individual aspects of health-related quality of life.
For patients with multimorbidity individualized PROMs, combined with standardized measures, may be effective for addressing multiple conditions within a single healthcare consultation, helping to drive patient-centred care plans, goal setting and prioritization.
Item banks and computer-adaptive tests
Item response theory is supporting the increasing development of large ‘banks’ of psychometrically calibrated questionnaire items, known as item banks.
The ability of computer-adaptive tests (CATs) to accurately ‘target’ questionnaire items means that patients may be spared from responding to items which are either redundant or potentially distressing.
The computerized nature of CAT benefits from immediate provision of feedback, which can be automatically tailored to the individual responses of the patient.
Practical advantages of CAT include no missing data and automatic scoring, inputting and storing – issues that have been shown to be barriers to the uptake of PROs in clinical practice.
Conclusion
The evidence for the role of PROs in clinical practice is currently more compelling for impact upon the processes of care, than on outcomes of care.
Based on available evidence we have developed a framework of the key requirements for the successful implementation of PROs in clinical practice. In the future, this framework could be further developed into a ‘bed-side’ tool kit linking scores to clinical decision-making.
Individualized measurement and CAT are very promising developments for further uptake of PROs in clinical practice, administering measures in a way that is appropriate and responsive to each patient and different aspects of their health conditions.
Financial & competing interests disclosure
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
No writing assistance was utilized in the production of this manuscript.
References
Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Black N. Patient reported outcome measures could help transform healthcare. BMJ 346, f167 (2013).
•• A comprehensive review on the potential of patient-reported outcomes (PROs) to transform healthcare provision by an author who has played a leading role in the development and establishment of the National Health Service national patient-reported outcomes measures program in England.
2.
US Food and Drug Administration: guidance for industry patient-reported outcome measures: use in medical product development to support labeling claims. US: Department of Health and Human Services Food and Drug Administration (2009). www.fda.gov/downloads/Drugs.
3.
Valderas JM, Alonso J. Patient reported outcome measures: a model-based classification system for research and clinical practice. Qual. Life Res. 17(9), 1125–1135 (2008).
4.
Bergner M, Bobbitt RA, Carter WB, Gilson BS. The Sickness Impact Profile: development and final revision of a health status measure. Med. Care 19(8), 787–805 (1981).
5.
Hunt SM, McKenna SP, McEwen J, Williams J, Papp E. The Nottingham Health Profile: subjective health status and medical consultations. Soc. Sci. Med. 15(3), 221–229 (1981).
6.
Lawton MP, Brody Em. Assessment of older people: self-maintaining and instrumental activities of daily living. Gerontologist 9(3), 179–186 (1969).
7.
Rabin R, de Charro F. EQ-5D: a measure of health status from the EuroQol Group. Ann. Med. 33(5), 337–343 (2001).
8.
Ware J Jr, Kosinski M, Keller SD. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med. Care 34(3), 220–33 (1996).
9.
Valderas JM, Kotzeva A, Espallargues M et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual. Life Res. 17(2), 179–193 (2008).
•• This systematic review provides comprehensive evidence on the impact of measuring PROs in clinical practice and has been widely cited.
10.
Ruta DA, Garratt AM, Leng M, Russell IT, MacDonald LM. A new approach to the measurement of quality of life. The patient-generated index. Med. Care 32(11), 1109–1126 (1994).
11.
Carswell A, McColl MA, Baptiste S, Law M, Polatajko H, Pollock N. The Canadian Occupational Performance Measure: a research and clinical literature review. Can. J. Occup. Ther. 71(4), 210–222 (2004).
12.
Emery MP, Perrier LL, Acquadro C. Patient-Reported Outcome and Quality of Life Instruments Database (PROQOLID): frequently asked questions. Health Qual. Life Outcomes 3, 12 (2005).
13.
Garcia-Duran Huet M, Ferrer M, Herdman M et al. “BiblioPRO: online library of PRO instruments in Spanish”. Qual. Life Res. 19(1), 62–63 (2010).
14.
Valderas JM, Ferrer M, Mendívil J et al. Scientific Committee on “Patient-Reported Outcomes” of the IRYSS Network. Development of EMPRO: a tool for the standardized assessment of patient-reported outcome measures. Value Health 11(4), 700–708 (2008).
15.
Schmidt S, Ferrer M, González M et al. Evaluation of shoulder-specific patient-reported outcome measures: a systematic and standardized comparison of available evidence. J. Shoulder Elbow Surg. 23(3), 434–444 (2014).
16.
Schmidt S, Garin O, Pardo Y et al. Assessing quality of life in patients with prostate cancer: a systematic and standardized comparison of available instruments. Qual. Life Res. 23(8), 2169–2181 (2014).
17.
Mokkink LB, Terwee CB, Knol DL et al. Protocol of the COSMIN study: COnsensus-based Standards for the selection of health Measurement INstruments. BMC Med. Res. Methodol. 6, 2 (2006).
18.
Mokkink LB, Terwee CB, Patrick DL et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J. Clin. Epidemiol. 63(7), 737–745 (2010).
19.
National Health Interview Survey 2014. Survey description. Centers for Disease Control and Prevention (2015). www.cdc.gov/nchs/nhis/quest_data.
20.
Black N, Burke L, Forrest CB et al. Patient-reported outcomes: pathways to better health, better services, and better societies. Qual. Life Res. 22(9), 1103–1112 (2015).
21.
Greenhalgh J. The applications of PROs in clinical practice: What are they, do they work, and why? Qual. Life Res. 18(1), 115–23 (2009).
•• Adopts a rigorous theory-driven approach to describe six different applications of PROs in clinical practice, summarizing evidence on processes and outcomes of care.
22.
Kyte DG, Calvert M, Van der Wees PJ, Hove R, Toland S, Hille JC. An introduction to patient-reported outcome measures (PROMs) in physiotherapy. Physiotherapy 10(2), 119–125 (2015).
23.
Greenhalgh J, Abhyankar P, McCluskey S, Takeuchi E, Velikova G. How do doctors refer to patient-reported outcome measures (PROMS) in oncology consultations. Qual. Life Res. 22(5), 939–950 (2013).
24.
Williams JW, Mulrow CD, Kroenke K et al. Case-finding for depression in primary care: a randomized trial. Am. J. Med. 106(1), 36–43 (1999).
25.
Espallargues M, Valderas JM, Alonso J. Provision of feedback on perceived health status to health care professionals: a systematic review of its impact. Med. Care 38(2), 175–186 (2000).
26.
Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J. Eval. Clin. Pract. 5(4), 401–416 (1999).
27.
Marshall S, Haywood K, Fitzpatrick R. Impact of patient reported outcome measures on routine practice: a structured review. J. Eval. Clin. Pract. 12(5), 559–568 (2006).
28.
Boyce MB, Browne JP. Does providing feedback on patient-reported outcomes to healthcare professionals result in better outcomes for patients? A systematic review. Qual. Life Res. 22(9), 2265–2278 (2013).
29.
Boyce MB, Browne JP, Greenhalgh J. The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Qual. Saf. 23(6), 508–518 (2014).
30.
Gonçalves Bradley DC, Gibbons C, Ricci-Cabello I et al. Routine provision of information on patient-reported outcome measures to healthcare providers and patients in clinical practice. Cochrane Database Syst. Rev. 3, CD011589 (2015).
31.
Gilbody SM, House AO, Sheldon TA. Routine administered questionnaires for depression and anxiety: systematic review. Br. Med. J. 322, 406–409 (2001).
32.
Knaup C, Koesters M, Schoefer D, Becker T, Puschner B. Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis. Br. J. Psychiatry 195, 15–22 (2009).
33.
Khalid A. Routine administration of standardized questionnaires that assess aspects of patients’ quality of life in medical oncology clinics: a systematic review. J. Egypt Natl Canc. Inst. 25(2), 63–70 (2013).
34.
Etkind SN, Daveson BA, Kwok W et al. Capture, transfer, and feedback of patient-centered outcomes data in palliative care populations: does it make a difference? A systematic review. J. Pain Symptom Manage. 49(3), 611–624 (2014).
35.
Krägeloh CU, Czuba KJ, Billington DR, Kersten P, Siegert RJ. Using feedback from patient-reported outcome measures in mental health services: a scoping study and typology. Psych. Serv. 66(3), 224–241 (2014).
36.
Howell D, Molloy S, Wilkinson K et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann. Oncol. 26(9), 1846–1858 (2015).
37.
Valderas JM, Espallargues M, Kotzeva A, Alonso J. Assessing the impact of routinely measuring patient-reported outcomes in clinical practice: critical appraisal of 34 randomized clinical trials. International Society of Quality of Life Research. Qual Life Res. 19(1), 11 (2010).
38.
Whooley MA, Stone B, Soghikian K et al. Randomised trial of case-finding for depression in elderly primary care patients. J. Gen. Intern. Med. 15(5), 293–300 (2000).
39.
Bergus GR, Hartz AJ, Noyes R Jr et al. The limited effect of screening for depressive symptoms with the PHQ-9 in rural family practices. J. Rural Health 21(4), 303–309 (2005).
40.
Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Soc. Sci. Med. 60(4), 833–843 (2005).
41.
Devlin N, Appleby J. Getting the most out of PROMs putting health outcomes at the heart of NHS decision-making. London, the King's Fund (2010). www.kingsfund.org.uk/sites/files.
42.
Department of Health. Guidance on the routine collection of Patient Reported Outcome Measures (PROMs) (2009). www.mstrust.org.uk/competencies.
43.
Varagunam M, Hutchings A, Neuburger J, Black N. Impact on hospital performance of introducing routine patient reported outcome measures (PROMs) in surgery. J. Health Serv. Res. Pol. 19(2), 77–84 (2014).
44.
Black N, Varagunam M, Hutchings A. Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery. BMJ Qual. Saf. 23, 534–542 (2014).
•• Provides a comprehensive assessment of whether the reporting of better experiences by elective surgical patients was associated with better outcomes (effectiveness and safety).
45.
Hutchings A, Grosse Frie K, Neuburger J, van der Meulen J, Black N. Late response to patient reported outcome questionnaires was associated with worse outcome. J. Clin. Epidemiol. 66(2), 218–225 (2013).
46.
Hutchings A, Neuburger J, van der Meulen J, Black N. Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons. BMC Health Serv. Res. 14, 66 (2014).
47.
Weingarten SR, Kim CS, Stone EG, Kristopaitis RJ, Pelter M, Sandhu M. Can peer-comparison feedback improve patient functional status? Am. J. Manag. Care 6(1), 35–39 (2000).
48.
Peters M, Crocker H, Dummett S, Jenkinson C, Doll H, Fitzpatrick R. Change in health status in long-term conditions over a one year period: a cohort survey using patient-reported outcome measures. Health Qual. Life Outcomes 12, 123 (2014).
49.
Valderas JM, Ferrer M, Mendivil J et al. Development and testing of standards for individualization of patient reported outcome (PRO) measures using EMPRO. Qual. Life Res. 21(Suppl. 1), 42–43 (2012).
50.
Greenhalgh J, Pawson R, Wright J et al. Functionality and feedback: a protocol for a realist synthesis of the collation, interpretation and utilisation of PROMs data to improve patient care. BMJ Open 4(7), e005601 (2014).
51.
Gonçalves DC, Gibbons EJ, Fitzpatrick R, Valderas JM. A systematic review and narrative review of qualitative studies about the use of PROs in clinical practice. Qual. Life Res. 21(S1), (2012).
52.
Dowrick C, Leydon GM, McBride A et al. Patients’ and doctors’ views on depression severity questionnaires incentivised in UK quality and outcomes framework: qualitative study. BMJ 19(338), b663 (2009).
53.
Leydon GM, Dowrick CF, McBride AS et al. Questionnaire severity measures for depression: a threat to the doctor-patient relationship? Br. J. Gen. Pract. 61(583), 117–123 (2011).
54.
Snyder CF, Aaronson NK, Choucair AK et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual. Life Res. 21(8), 1305–14 (2012).
•• Summarizes the key issues in the user's guide for implementing Patient-Reported Outcomes Assessment in Clinical Practice developed by the International Society for Quality of Life Research.
55.
Valderas JM, Alonso J, Guyatt GH. Measuring patient-reported outcomes: moving from clinical trials into clinical practice. Med. J. Aust. 189(2), 93–94 (2008).
56.
Valderas JM, Alonso J, Prieto L, Espallargues M, Castells X. Content-based interpretation aids for health-related quality of life measures in clinical practice. An example for the visual function index (VF-14). Qual. Life Res. 13(1), 35–44 (2004).
57.
Santana MJ, Feeny D. Framework to assess the effects of using patient-reported outcome measures in chronic care management. Qual. Life Res. 23(5), 1505–1513 (2013).
•• The authors set out a framework of how feedback of PROs is intended to improve the care of individual patients.
58.
Valderas JM, Rue M, Guyatt G, Alonso J. Systematic Use of Quality of Life Measures in the Clinical Practice Working Group. The impact of the VF-14 index, a perceived visual function measure, in the routine management of cataract patients. Qual. Life Res. 14(7), 1743–1753 (2005).
59.
Valderas JM, Fitzpatrick R, Roland M. Using health status to measure NHS performance: another step into the dark for the health reform in England. BMJ Qual. Saf. 21(4), 352–353 (2011).
60.
Wolfe J, Orellana L, Cook EF et al. Improving the care of children with advanced cancer by using an electronic patient-reported feedback intervention: results from the PediQUEST randomized controlled trial. J. Clin. Oncol. 32(11), 1119–1126 (2014).
• A randomized controlled trial evaluating whether feeding back PROs electronically to providers and families of children with advanced cancer improves symptom distress and health-related quality of life.
61.
British Medical Association - NHS Employers. Revisions to the GMS Contract 2006/07. Delivering Investment in General Practice. Department of Health (2006). www.nhsemployers.org/SiteCollectionDocuments.
62.
NICE. Depression: the treatment and management of depression in adults. National Institute for Health and Clinical Excellence (2009). www.nice.org.uk/nicemedia/live/12329.
63.
Malpass A, Shaw A, Kessler D, Sharp D. Concordance between PHQ-9 scores and patients’ experiences of depression: a mixed methods study. Br. J. Gen. Pract. 60(575), 231–238 (2010).
64.
Mitchell C, Dwyer R, Hagan T, Mathers N. Impact of the QOF and the NICE guideline in the diagnosis and management of depression: a qualitative study. Br. J. Gen. Pract. 61(586), 279–289 (2011).
65.
Prieto L, Alonso J, Lamarca R. Classical Test Theory versus Rasch analysis for quality of life questionnaire reduction. Health Qual. Life Outcomes 1, 27 (2003).
66.
Toop L. The QOF, NICE, and depression: a clumsy mechanism that undermines clinical judgment. Br. J. Gen. Pract. 61(588), 432–433 (2011).
67.
British Medical Association –NHS Employers. 2013/14 general medical services (GMS) contract quality and outcomes framework (QOF). Guidance for GMS contract 2013/14. www.nhsemployers.org/Aboutus/Publications.
68.
Valderas JM, Starfield B, Sibbald B, Salisbury C, Roland M. Defining comorbidity: implications for understanding health and health services. Ann. Family Med. 7(4), 357–363 (2009).
69.
Barnett K, Mercer SW, Norbury M, Watt G, Wyke S, Guthrie B. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet 380(9836), 37–43 (2012).
70.
Salisbury, Johnson L, Purdy S, Valderas JM, Montgomery AA. Epidemiology and impact of multimorbidity in primary care: a retrospective cohort study. Br. J. Gen. Pract. 61(582), e12–e21 (2011).
71.
Fortin M, Bravo G, Hudon C et al. Relationship between multimorbidity and health-related quality of life of patients in primary care. Qual. Life Res. 15(1), 83–91 (2006).
72.
Smith SM, Soubhi H, Fortin M, Hudon C, O'Dowd T. Interventions for improving outcomes in patients with multimorbidity in primary care and community settings. Cochrane Database Syst. Rev. 4, CD006560 (2012).
73.
Valderas JM, Mercer SW, Fortin M. Research on patients with multiple health conditions: different constructs, different views, one voice. J. Comorbidity 1(1), 1–3 (2011).
74.
Browne J. www.kingsfund.org.uk.
75.
Garratt A, Schmidt L, Mackintosh A, Fitzpatrick R. Quality of life measurement: bibliographic study of patient assessed health outcome measures. Br. Med. J. 324(7351), 1417–1419 (2002).
76.
Dijkers MP. Individualisation in quality of life measurement: instruments and approaches. Arch. Phys. Med. Rehabil. 84(4), S3–S14 (2003).
77.
Jolles BM, Buchbinder R, Beaton DE. A study compared nine patient-specific indices for musculoskeletal disorders. J. Clin. Epidemiol. 58(8), 791–801 (2005).
78.
Engel G. The need for a new medical model: a challenge for biomedicine. Science 196, 129–136 (1977).
79.
Engel GL. The clinical application of the biopsychosocial model. Am. J. Psychiatry 137(5), 535–544 (1980).
80.
Bradley C, Todd C, Gorton T, Symonds E, Martin A, Plowright R. The development of an individualized questionnaire measure of perceived impact of diabetes on quality of life: the ADDQoL. Qual. Life Res. 8(1–2), 79–91 (1999).
81.
Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based outcome measures for use in clinical trials. Health Technol. Assess. 2(14), 1–74 (1998).
82.
Wright B. Raw scores are not linear measures: rasch vs. classical test theory CTT comparison. Rasch Meas. Trans. 6(1), 208 (1992).
83.
Lord FM. Applications of Item Response Theory to Practical Testing Problems. Routledge (1980). https://books.google.com/books.
84.
Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danish Institute for Educational Research (1960).
85.
Lai J-S, Dineen K, Reeve BB et al. An item response theory-based pain item bank can enhance measurement precision. J. Pain Symptom Manage 30(3), 278–288 (2005).
86.
Gibbons CJ, Mills RJ, Thornton EW et al. Development of a patient reported outcome measure for fatigue in motor neurone disease: the Neurological Fatigue Index (NFI–MND). Health Qual. Life Outcomes 9(1), 101 (2011).
87.
Gibbons C, Kenning C, Coventry P, Bee P. Development of a multimorbidity illness perceptions scale (MULTIPleS). PLoS ONE 8(12), e81852 (2013).
88.
Forkmann T, Boecker M, Norra C et al. Development of an item bank for the assessment of depression in persons with mental illnesses and physical diseases using Rasch analysis. Rehabil. Psychol. 54(2), 186–197 (2009).
89.
Hambleton, Ronald K. Principles and selected applications of item response theory. In: Educational measurement (3rd Ed.). Linn RL (Ed.). The American Council on Education/Macmillan series on higher education, NY, USA, 147–200 (1989).
90.
Holland P, Wainer H. Differential Item Functioning. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, USA (2012).
91.
Lai J, Cella D, Chang C-H, Bode RK, Heinemann AW. Item banking to improve, shorten and computerize self-reported fatigue: an illustration of steps to create a core item bank from the FACIT-Fatigue Scale. Qual. Life Res. 12(5), 485–501 (2003).
92.
Haley SM, Raczek AE, Coster WJ, Dumas HM, Fragala-Pinkham MA. Assessing mobility in children using a computer adaptive testing version of the pediatric evaluation of disability inventory. Arch Phys. Med. Rehabil. 86(5), 932–939 (2005).
93.
Jette AM, Haley SM, Ni P, Olarsch S, Moed R. Creating a computer adaptive test version of the late-life function and disability instrument. J. Gerontol A Biol. Sci. Med. Sci. 63(11), 1246–1256 (2008).
94.
Haley SM, Fragala-Pinkham MA, Dumas HM et al. Evaluation of an item bank for a computerized adaptive test of activity in children with cerebral palsy. Phys. Ther. 89(6), 589–600 (2009).
95.
Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ. The routine use of patient reported outcome measures in healthcare settings. BMJ 340, c186 (2010).
•• Provides a guide for individuals keen to use PRO measures at a local level.
96.
Alonso J, Bartlett SJ, Rose M et al. The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative. Health Qual. Life Outcomes 11, 210 (2013).
Information & Authors
Information
Published In
Copyright
© Future Medicine Ltd.
History
Published online: 18 July 2016
Keywords:
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Downloaded 1,225 timesCitations
How to Cite
Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities. (2016) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2015-0014
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Maria A. Fiol-deRoque, Georgina Vidal Mansilla, José A. Maderuelo-Fernández, Olaya Tamayo-Morales, Francisco Martín-Luján, Pilar Astier-Peña, Macarena Chacón-Docampo, Carola Orrego, Montserrat Gens-Barberà, Pilar Andreu-Rodrigo, Ignacio Ricci-Cabello, Evaluation of SINERGIAPS, an intervention to improve patient safety in primary healthcare centers in Spain based on patients’ perceptions and experiences: a protocol for a hybrid type I randomized clinical trial, Frontiers in Public Health, 10.3389/fpubh.2024.1324940, 12, (2024).
- Nura Abdel-Rahman, Orly Manor, Einat Elran, David Siscovick, Ronit Calderon-Margalit, Implications of Patient-Reported Outcome Measures among patients with recently diagnosed type 2 diabetes, Israel Journal of Health Policy Research, 10.1186/s13584-024-00592-1, 13, 1, (2024).
- David T. Eton, Kathleen J. Yost, Jennifer L. Ridgeway, Bayly Bucknell, Mike Wambua, Natalie C. Erbs, Summer V. Allen, Elizabeth A. Rogers, Roger T. Anderson, Mark Linzer, Development and acceptability of PETS-Now, an electronic point-of-care tool to monitor treatment burden in patients with multiple chronic conditions: a multi-method study, BMC Primary Care, 10.1186/s12875-024-02316-5, 25, 1, (2024).
- Jose M Valderas, Ian Porter, Jimmy Martin-Delgado, Mieke Rijken, Judith de Jong, Oliver Groene, Janika Bloemeke-Cammin, Rosa Sunol, Rachel Williams, Marta Ballester, Katherine de Bienassis, Candan Kendir, Frederico Guanais, Dolf de Boer, Michael van den Berg, Development of the Patient-Reported Indicator Surveys (PaRIS) conceptual framework to monitor and improve the performance of primary care for people living with chronic conditions, BMJ Quality & Safety, 10.1136/bmjqs-2024-017301, (bmjqs-2024-017301), (2024).
- Lucas Gallo, Charlene Rae, Sophocles Voineskos, Nina Vestergaard Simonsen, Andrea L. Pusic, Lotte Poulsen, Jens Ahm Sørensen, Anne F. Klassen, Stefan J. Cano, Further psychometric evaluation of the WOUND‐Q: A responsiveness study, Wound Repair and Regeneration, 10.1111/wrr.13179, 32, 4, (451-463), (2024).
- Gayanika M Hettiarachchi Senarath, Pari Delir Haghighi, Lu Bai, Michelle M Shannon, Nadine E Andrew, Velandai Srikanth, David A Snowdon, Denise A O’Connor, Barriers and facilitators to the uptake of electronic collection and use of patient-reported measures in routine care of older adults: a systematic review with qualitative evidence synthesis, JAMIA Open, 10.1093/jamiaopen/ooae068, 7, 3, (2024).
- Anouk Groenewegen, Maud M. van Muilekom, Cenne H.A.M. Sieben, Michiel A.J. Luijten, Sharon L. Janssen, Nynke Venema-Taat, Mira Korte, Caroline B. Terwee, Piet M. ter Wee, Hedy A. van Oers, Lotte Haverman, The Amsterdam PROM Implementation Strategy: Policy and Pathway, NEJM Catalyst, 10.1056/CAT.23.0414, 5, 7, (2024).
- Zoe Clothier, Jo Armes, Agnieszka Kehinde, Victoria Mumford, Kate Upshon, Clare Williamson, Rachel Macarthur, Rachel Stevenson, Sophie Otter, May Teoh, Jenny Harris, Patient reported experience measures to assess psychosocial cancer care: a rapid review of current instruments, Measurement and Evaluations in Cancer Care, 10.1016/j.ymecc.2024.100007, (100007), (2024).
- D. Figueroa, R. Guiloff, F. Figueroa, E. Stocker, S. Rocha Piedade, [Translated article] 4 Domain Sports PROM in Spanish: Cross-cultural adaptation in the Chilean population and reliability analysis, Revista Española de Cirugía Ortopédica y Traumatología, 10.1016/j.recot.2024.07.001, 68, 5, (T497-T501), (2024).
- D. Figueroa, R. Guiloff, F. Figueroa, E. Stocker, S. Rocha Piedade, 4 Domain Sports PROM en español: adaptación transcultural en la población chilena y análisis de confiabilidad, Revista Española de Cirugía Ortopédica y Traumatología, 10.1016/j.recot.2024.02.001, 68, 5, (497-501), (2024).
- See more