The use of real-world effectiveness measurements in clinical settings at Helsinki University Hospital: a systematic literature review
Publication: Journal of Comparative Effectiveness Research
Abstract
Aim: Patient reported outcomes collected alongside clinical trials do not reflect real-world effectiveness (RWE). This review assessed the use of RWE measurements in routine clinical treatment and the instruments applied to collect that data. Materials & methods: The RWE articles published from HUS (Helsinki University Hospital) were extracted from several databases. Results: Out of 170 eligible articles, generic health-related quality of life instruments were used in 87 (51.2%) and disease-specific health-related quality of life instruments in 58 (34.1%) articles as a primary measurement. Most of the articles pertained to surgery, gynecology and pediatric surgery. Conclusion: The number of articles assessing RWE is very limited compared with all the articles published from HUS. Thus, we still have limited information about the effectiveness of the treatment in real life.
In the clinical and economic evaluation of healthcare services, it is important to measure the effectiveness of the treatment. Patient-reported outcomes are nowadays collected routinely alongside randomized controlled trials, but those results do not reflect their real-world effectiveness (RWE), since the patients are carefully selected for the trials. Little is known about the use of routinely collected effectiveness measurement in real-life clinical work.
Specialized medical care is organized by hospital districts in Finland. The country is divided into 20 hospital districts, out of which five are university hospital districts. One hospital district can run several hospitals. Hospital District of Helsinki and Uusimaa (HUS) is the largest of these with 24 hospitals throughout the province of Uusimaa in southern Finland. The provision of the most demanding medical operations is centralized on the national level to the university hospitals. In accordance with a national agreement on the division of tasks, the treatment of rare and expensive illnesses (e.g., organ transplant surgery, pediatric heart surgery) have been centralized to HUS university hospital. The HUS catchment area for highly specialized healthcare services covers the area of 2.1 million inhabitants (∼40% of the Finnish population). More than half a million individual patients receive medical care annually and surgical procedures (ca. 100,000 per year) are performed in 100 operation rooms in 14 hospitals. HUS provides services in all medical and surgical specialties [1].
With the development of healthcare technology and care processes/protocols, there are more and more treatment options for different diseases to be meant to benefit various patients. However, the resources in healthcare are scarce, and choices need to be made to identify the relevant alternatives [2]. It is important to make sure that an assessment of best practices includes both costs and patient outcomes [3].
It is a well-known fact that information is needed to support the decision making of how the scarce resources in healthcare should be used. It is also important to be able to make choices of which methods or procedures are to be used for the best care of the patients. However, there are no common rules or regulations of how the choices should be made [2,4,5].
In health economics, the common concept is the term ‘effectiveness’. Measuring the effectiveness of the treatment provides a tool to assess the health gains for the patient [2]. Assessing the outcomes of healthcare interventions by measuring the health status of patients is now a standard feature of clinical and economic evaluations. An extension of this is to measure patient-reported outcomes routinely within health services [6]. Routinely collected patient-reported outcome measures provide important information about the disease or its treatment on the patient’s wellbeing [7]. It is shown that systematic use of patient-reported outcomes is a key to better communication and by extension better decision-making between doctors and patients, therefore enhancing the patients to report better satisfaction with their care [8].
Hundreds of standardized measures are developed during the years to capture patient reported outcomes, in other words, symptom status, physical function, mental health or wellbeing. Additionally, various generic health-related quality of life (HRQoL) instruments (such as EQ-5D, 15D, SF-36 or HUI-3) were developed to measure the overall HRQoL of the patients.
A randomized controlled trial (RCT) provides the least biased information on the efficacy of medical interventions and creates the basis for systematic reviews on the efficacy of interventions. However, RCTs face problems due to the often-narrow eligibility criteria for the patients and the better than average know-how of the units providing the interventions. Thus, the efficacy shown by trials and systematic reviews is often better than the effectiveness provided by average healthcare units for average patients, and the extrapolation of the results to ordinary settings is often uncertain [9].
The objectives of this study were to assess the use of RWE measurements in routine clinical treatment in the largest university hospital in Finland according to the published studies and the instruments which were applied to collect that data.
Materials & methods
The search strategy is shown in Table 1 and the number of abstracts and articles in each step can be seen in Figure 1. The first screening was done by reading the abstracts. Two investigators (KK, MH) independently read the abstracts. Based on abstracts, the articles that might/could/probably fulfill/fulfilled the inclusion criteria were selected for further inspection. Two investigators (KK, MH) read the articles independently. If there was any discrepancy regarding whether or not the article would fulfill the inclusion criteria, the third investigator (PR) read the article and inclusion or exclusion was debated and determined by the whole study group.
| Database: Ovid MEDLINE(R) Epub ahead of print, in-process & other non-indexed citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) <1946 to Present) | |
|---|---|
| Search strategy: | |
| 1 | exp Comparative Effectiveness Research/(2954) |
| 2 | effectiveness*.mp. (377883) |
| 3 | exp Cost-Benefit Analysis/ (72784) |
| 4 | (cost* benef* or cost utilit* or cost* effect* or cost* of illnes*).mp. (176801) |
| 5 | exp quality-adjusted life years/ (9935) |
| 6 | quality* adjusted* life* year*.mp. (14526) |
| 7 | QALY.mp. (6775) |
| 8 | routine* practi*.mp. (6028) |
| 9 | exp “Quality of Life”/ (159460) |
| 10 | quality* of* life*.mp. (269716) |
| 11 | qol.mp. (30271) |
| 12 | HrQoL.mp. (12495) |
| 13 | (instrument* or survey* or questionnaire*).mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept word, rare disease supplementary concept word, unique identifier, synonyms] (1259857) |
| 14 | exp “Surveys and Questionnaires”/ (878009) |
| 15 | (15D or EQ-5D or EORTC or HAQ or HHS).mp. [mp = title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept word, rare disease supplementary concept word, unique identifier, synonyms] (19843) |
| 16 | 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 (2189063) |
| 17 | helsinki univ* hosp*.af. (3918) |
| 18 | ((HUH or HUS or HUCH) and helsinki).af. (4723) |
| 19 | (Hospital district of Helsinki and Uusimaa).af. (213) |
| 20 | 17 or 18 or 19 (7712) |
| 21 | 16 and 20 (1296) |
| 22 | limit 21 to (yr = “2000 -Current” and (english or finnish or swedish)) (1254) |
Figure 1. Flowchart of systematic literature review process.
HRQoL: Health-related quality of life; HUS: Helsinki University Hospital; RWE: Real-world effectiveness.
The inclusion criteria were: articles was HUS-related; article reported the RWE of the patient treatment; generic or disease-specific quality of life (QoL) measurement or direct valuation method had been applied; the assessment had been done by a patient or a close relative; effectiveness was measured before and/or after the treatment; the patients reported on were consecutive patients; the number of patients was at least 20; and the article was published between 2000 and August 2017.
Results
Our systematic literature search identified 3107 publications, out of which 834 were duplicates. That left us with 2273 publications. However, 1804 were either letters, editorials, congress abstracts, animal studies, clinical trials, cross-sectional studies and as we were looking for original studies, those were not included for further review. Thus, we had 469 abstracts out of which we read full-text articles to assess their eligibility for the review. A total of 299 full-text articles were excluded since they did not meet the inclusion criteria. RWE was used as an end point in 170 studies, and those were included in our qualitative synthesis (Figure 1).
The mean number of yearly RWE articles was 9.4 (170/18 years). The number of RWE articles published by HUS-connected researchers, however, has increased during the years included in our systematic review. In the year 2000, only one RWE article was published, whereas in the year 2016, the number was 19. Although the number of RWE studies has increased during the years included in the review, the number is very small compared with the total number of studies published from HUS.
A generic HRQoL instrument (e.g., 15D, EQ-5D, SF-36) was used in 87 out of 170 articles (51.2%) as the primary measurement. Disease-specific HRQoL instruments were used in 58 out of 170 articles (34.1%) as a primary measurement. We found 91 different questionnaires used in articles.
The rest (14.7%) employed a direct valuation instrument, such as a visual analog scale. RWE was measured both before and after the treatment in 56.0% (n = 95) of the articles. Due to the nature of the disease (e.g., strokes), RWE was measured only after the treatment in 44.0% (n = 75) of the articles included. The included articles, arranged by their main measurement setting are presented in more detail in Table 2.
| Measurement setting | HRQoL instrument n (%) | Disease-specific instrument n (%) | Direct valuation method n (%) | General health status n (%) | Total n (%) |
|---|---|---|---|---|---|
| Before–after the treatment | 55 (63 %) | 30 (52 %) | 8 (57 %) | 2 (18 %) | 95 (56 %) |
| After the treatment | 32 (37 %) | 28 (48 %) | 6 (43 %) | 9 (82 %) | 75 (44 %) |
| Total | 87 (100 %) | 58 (100 %) | 14 (100 %) | 11 (100 %) | 170 (100 %) |
HRQoL: Health-related quality of life.
The most frequently used HRQoL instrument in our study was the 15D, which was used in 53 articles and its variants 16D for children and 17D for adolescents was used in 7 articles. Other HRQoL instruments were the EQ-5D (n = 13) and the SF-36/RAND-36 (n = 21).
A total of 32.9% of the studies were primarily concerned with surgery, 12.4% with gynecology and 11.8% with pediatric surgery. The remainder covered various types of clinical specialties. The articles were grouped by the clinical specialties in Table 3.
| Clinical specialties | n | % |
|---|---|---|
| Surgery | 56 | 32.9 |
| Gynecology | 21 | 12.4 |
| Pediatric surgery | 20 | 11.8 |
| Anesthesiology and intensive care | 13 | 7.6 |
| Psychiatry | 13 | 7.6 |
| Internal medicine | 13 | 7.6 |
| Otorhinolaryngology disease | 12 | 7.1 |
| Respiratory medicine | 4 | 2.4 |
| Pediatrics | 3 | 1.8 |
| Neurology | 3 | 1.8 |
| Over two specialties | 3 | 1.8 |
| Emergency medicine | 2 | 1.2 |
| Dental and oral diseases | 2 | 1.2 |
| Ophthalmology | 2 | 1.2 |
| Cancer disease | 2 | 1.2 |
| Skin disease | 1 | 0.6 |
| Total | 170 | 100.0 |
Discussion
Randomized controlled trials (RCTs) are the cornerstone when assessing medical treatments. Especially in specialized healthcare, double-blind, placebo-controlled studies are highly respected. However, in those studies, the patients are carefully selected. The treatments in RCT studies are also very well standardized and usually given by experienced clinicians. However, the real world often differs considerably from a set-up in RCT.
Our systematic review gathered a total of only 170 articles in which RWE measurements have been used in routine clinical work or practice in HUS during the years 2000–2017. Thus, our study shows that the number of RWE studies is very low when compared with the total number of studies published in HUS. As an example, the number of peer-reviewed HUS-connected publications was 2883 during the year 2016 [10]. The articles reveal that the RWE studies published have often been projects or studies of researcher origin, but meanwhile, RWE measurements have not become a stable routine practice in everyday clinical work since the project’s completion.
In the HRQoL assessment, it is generally considered whether to use both general and disease-specific instruments. In this review, most of the RWE articles (44/87) published not only disease-specific QoL measurement but also generic measurements. This enables the comparison of the burden and treatment results across different clinical specialties.
The ultimate aim of RWE is to produce as much good and as little harm as possible for each patient, with reasonable costs to the society. Good medical know-how and patient view, up-to-date scientific evidence, continuous documentation and benchmarking are utilized to reach this goal [9].
Why were there so few RWE articles published in HUS? In Finland as in any other small linguistic area, there is always some challenge entailed in developing or validating a measurement suitable for clinical use [11]. New inventions in medicine as well in other sciences seem to get more publicity than reporting the effectiveness of the treatment in common diseases, for example, elevated blood pressure. A competition for increasingly fewer grants may lead the researchers to choose their research fields among those that are probably considered ‘ideal’ for a grant committee. The citation index may be more important than the research result itself.
The title of the HUS’s strategy is ‘effective treatment for the best of patient’. Thus, according to HUS’ strategy, it is very important that the effectiveness – and not only the effect in RCTs – of the treatment is measured. However, based on the results of our review, this primary strategy of the HUS seems to be only a hypothetical wish at present. Just as in healthcare there is always a challenge to use limited resources reasonably; the RWE data would be very valuable for the persons involved in making the decisions. Further to this, now and in the future, the patients themselves are and will continue to routinely participate in their treatment decisions more than previously; the RWE data is also very desirable from their point of view.
The HRQoL instrument models 15D, SF36 and EQ5D are validated in Finnish. Electronic medical records, smartphones, etc. can simplify the entailed data collection and storage. The study was conducted in a large university hospital with multiple specialties, which could have been made clear by the use of a disease-specific QoL questionnaire.
Although the use of RWE and patient-reported outcomes is nowadays strongly encouraged, only a surprisingly small minority (10–15 articles out of 2900 peer-reviewed articles from the hospital in year 2016) actually included real-word data. We find this alarming and worth reporting to a wider audience as it is likely that the situation may be similar also in other university hospitals around the world.
As far as we know, this is the first systematic review where the number of RWE studies published from a large referral hospital providing secondary and tertiary healthcare services has been compared with the total number of studies published in the same area. Although the term effectiveness is often cited in the keynote address and goal of the strategy, we still have limited information about the effectiveness of the treatment in real life.
•
Only 170 eligible articles were gathered in which real-world effectiveness (RWE) measurements have been used in routine clinical practice in Helsinki University Hospital (HUS) during the years 2000–2017.
•
The number of published RWE studies is very low compared with the total number of studies published in HUS.
•
Most of the articles used both general and disease-specific health-related quality of life instruments, which enables the comparison of the treatment results across different clinical specialties.
•
The RWE studies published have often been projects or studies of researcher origin, but RWE measurements have not become a stable routine practice in everyday clinical work since the project’s completion.
•
We still have limited information about the effectiveness of the treatment in real life.
Author contributions
Data were collected and analyzed by K Koskinen and M Hytönen with the contribution of P Räsänen. K Koskinen wrote the manuscript with the support of M Hytönen and P Räsänen. All authors edited subsequent versions and approved the final copy.
Acknowledgments
The authors would like to thank the audit committee of the Hospital District of Helsinki and Uusimaa for their valuable comments during the study as well as the information specialist Kaisa Larmo from Terkko for her guidance in constructing the search strategies and the use of the search engines.
Financial & competing interests disclosure
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
No writing assistance was utilized in the production of this manuscript.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/
References
Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Helsinki University Hospital (2019). www.hus.fi
2.
Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes (4th Edition). Oxford University Press, Oxford, UK, 20–73 (2015).
•• Provides essential background for the theory of decision making in healthcare.
3.
Gutacker N, Bojke C, Daidone S, Devlin NJ, Parkin D, Street A. Truly inefficient or providing better quality of care? Analyzing the relationship between risk-adjusted hospital costs and patients’ health outcomes. Health Econ. 22(8), 931–947 (2013).
4.
Sintonen H, Pekurinen M, Linnakko E. Terveystaloustiede (Health Economics) (2nd Edition). WSOY, Porvoo, Finland, 45–49 (1997).
•• Provides background for decision making in healthcare especially in Finnish settings.
5.
Drummond MF, McGuire A. Economic Evaluation in Health Care: Merging Theory with Practice. Oxford University Press, Oxford, UK, 25–32 (2001).
• Provides valuable background information for the article.
6.
Nuttall D, Parkin D, Devlin N. Inter-provider comparison of patient-reported outcomes: developing an adjustment to account for differences in patient case mix. Health Econ. 24(1), 41–54 (2015).
• Gives an idea of how case mix should be taken into account while comparing patient-reported outcomes (PROs).
7.
Santana MJ, Feeny D, Weinkauf J et al. The use of patient-reported outcomes becomes standard practice in the routine clinical care of lung–heart transplant patients. Patient Relat. Outcome Meas. 1, 93–105 (2010).
• Gives a nice example of how PROs are used in routine care.
8.
Nelson EC, Eftimovska E, Lind C, Hager A, Wasson J, Lindblad S. Patient reported outcome measures in practice. BMJ 350, g7818 (2015).
• Describes how PROs should be measured.
9.
Malmivaara A. Real-effectiveness medicine-pursuing the best effectiveness in the ordinary care of patients. Ann. Med. 45(2), 103–106 (2013).
•• Presents a concept for real-life effectiveness and was a driving force for this article.
10.
Helsinki University Hospital. HUS Annual report 2017 (2017). www.hus.fi/en/about-hus/Pages/default.aspx
11.
Hytönen M, Hammarén-Malmi S, Myller J et al. Tautikohtaisen elämänlaatumittarin validointi – esimerkkinä nenä- ja sivuontelotautikohtainen SNOT-22 mittari. Duodecim. 133, 1317–1325 (2017).
•• An example of country-specific validation of disease-specific quality of life instrument.
Information & Authors
Information
Published In
Pages: 321 - 326
PubMed: 32141305
Copyright
© 2020 Kati Koskinen, Maija Hytönen & Pirjo Räsänen. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 29 October 2019
Accepted: 8 January 2020
Published online: 6 March 2020
Keywords:
Topics
Authors
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
The use of real-world effectiveness measurements in clinical settings at Helsinki University Hospital: a systematic literature review. (2020) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2019-0159
Export citation
Select the citation format you wish to export for this article or chapter.