Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers
Publication: Journal of Comparative Effectiveness Research
Abstract
As part of the HTx (Next Generation Health Technology Assessment) project, this study was aimed at identifying the main barriers for application of real-world evidence (RWE) for the purposes of health technology assessment in the Central and Eastern European countries. A mixed methods approach was employed to identify the main barriers: a scoping review of the literature and a series of discussions with stakeholders. Based on the applied approaches, we attempted to summarize the main barriers and challenges related to transferability of RWE in five main groups: technical, regulatory, clinical, scientific and perceptional barriers. Further research should pursue the development of detailed, consensus-based guidelines to improve the harmonization and standardization of RWE.
Real-world data (RWD) and real-world evidence (RWE) have an increasing role in healthcare decision-making. From RWD, analysts can develop RWE, which is already being used for health technology assessment (HTA) purposes to some extent [1,2]. With the increased availability of RWD, there is potential for more widespread use. However, a number of challenges about ensuring the reliability of RWE for HTA [3] can be identified in the literature. This may lead to disparities among stakeholders when discussing RWD and RWE use in decision-making [4,5].
Regulatory bodies and HTA organizations for descriptive analyses (e.g., of treatment patterns) and burden of illness as well as epidemiology data and monitoring the safety of marketed therapies, especially in the more advanced healthcare systems in Western Europe, often use RWD. Decision-makers can benefit from RWD collected to assess the (comparative) effectiveness of health technologies in nontrial settings and populations [6]. Use of RWE, however, is limited in the Central and Eastern European (CEE) countries [7] because in most cases the decision-makers prefer to rely on traditional sources of evidence, such as randomized controlled trials (RCTs) or expert opinion.
Generating RWE is a resource intensive process; CEE countries are usually short of financial and human resources and lacking the necessary infrastructure for RWE generation [8]. Because new technologies are usually introduced in Western European countries, CEE countries have an opportunity to benefit from RWE generated in early-technology-adopter countries. Moreover, the current use and interest in using RWE stimulates the regulatory use of such kind of data. In such cases, HTA bodies and decision-makers in CEE have no access to the original RWD collected in other countries; hence, they need to make judgments on the transferability of published RWE.
As part of the European Commission-funded H2020 project HTx (Next Generation Health Technology Assessment), the current study aimed to identify the main barriers for application of RWE derived in the Western European countries for the purposes of CEE decision-making in healthcare. The main aim of HTx is to create a framework for the Next Generation Health Technology Assessment (HTA) to support patient-centered, societally oriented, real-time decision-making on access to and reimbursement for health technologies throughout Europe.
Materials & methods
For the purposes of the current study, a mixed methods approach to identify the main barriers was employed: first, a scoping review of the literature; second, a series of internal discussions and a webinar with stakeholders from CEE countries. For the purpose of conducting the scoping review, several search strings in PubMed database were applied using different keywords. The other sources searched were Google Scholar, including publications, conference papers and projects dedicated to RWE; the Value in Health database; and reports of HTA agencies (National Institute for Health and Care Excellence [NICE] from England, Haute Autorité de Santé [HAS] from France, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen [IQWIG] from Germany). The keywords used in the literature search were transferability, AND implementation, AND barriers OR challenge, AND real-world evidence OR effectiveness, AND health technologies OR therapies used in different variations.
Specific inclusion and exclusion criteria were defined for the scoping review. Studies were not excluded by publication date. To be considered for this review, studies had to meet the following inclusion criteria: English-language articles; the articles had to identify barriers and challenges for RWE transferability; the articles needed to describe the possibilities for transferability of RWE; and the articles were required to discuss the importance of the RWE transferability process. Studies that presented different aspects of RWE data or covered any recommendations in the area of decision-making in the healthcare sector without considering RWE were excluded.
After the scoping review was completed, the research team compiled a list of barriers to the use of RWE in CEE countries identified from the included studies. Next, a series of discussions with multidisciplinary HTx project members was organized, followed by a webinar with CEE stakeholders. The draft list of barriers was discussed, with the aim of reaching a consensus on the main barriers, including additional obstacles not previously identified.
Results
The initial workflow of the scoping review consisted of screening and identifying studies that met the selection criteria. 234 studies were screened based on titles and abstracts. Of these, 20 candidate full texts were reviewed a third time by an independent reviewer, and a further 214 were excluded.
The result was 20 papers, including full systematic review articles, reports, conference papers and other resources focusing on the study question that met the inclusion criteria. A flow diagram of the literature search is presented in Figure 1 [36].
Figure 1. PRISMA flow diagram of the literature search and process of study selection, adapted from the PRISMA statement.
*Reporting the number of records identified from each database or register searched (rather than the total number across all databases/registers); **If automation tools were used, indicates how many records were excluded by a human and how many were excluded by automation tools.
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RWE: Real-world evidence.
Information on barriers and challenges of RWE transferability identified in the 20 included studies is summarized in Table 1.
| Publications | Barriers and challenges | Ref. |
|---|---|---|
| Akhras et al., 2019 | - Challenges related to lack of knowledge, technical and confidentiality issues, lack of trust between stakeholders | [9] |
| Buckle et al., 2019 | - Concerns about external validity of the RWD - Concerns about the quality of transferable RWD | [10] |
| Cole et al., 2015 | - Lack of a favorable governance framework - Considerable variation in local approaches to data governance - Heterogeneity across jurisdictions in data protection | [11] |
| Dreyer et al., 2018 | - No agreed-on framework for assessing reliability of RWD | [12] |
| EC, April 2018 | - Lack of standardized data RWD collection - Lack of data quality standards and validation processes - Lack of representative databases - Lack of enough studies demonstrating how RWD can be used depending on the purpose in healthcare systems | [13] |
| Hampson et al., 2018 | - Lack of collaboration between manufacturers and regulatory agencies - Incomplete data - Data mining - Access to data - Lack of universally accepted methodological standards - Lack of universally accepted standards or principles for the design, conduct, analysis and/or reporting of RWE - Lack of investigator expertise - Obsolete evidence hierarchies | [14] |
| Justo et al., 2019 | - Problems with the data - Gaps in expertise - Lack of confidence in observational research - Trust issues between users and data holders | [15] |
| Kamphuis et al., 2018 | - Data sources are varied and fragmented across Europe - Data infrastructure – data are only in national languages and may not be usable - Heterogeneity of health data across the EU, different definitions and varying typologies for structure data - Cultural and national influences may lead to different data interpretations or data input - Data access and security - Lack of joint stakeholder collaboration - Various evaluation methodologies - Institutional inertia - The interests of academics or researchers acting as (principle) investigators on the data | [16] |
| Makady et al., 2017 | - Lack of harmonization of policies for RWD use from HTA agencies - No approach for data collection form from marketing authorization holders - Lack of collaboration between EMA and HTA agencies on RWE requirements | [17] |
| Malone et al., 2018 | - Organizational barriers to the use of published RWE - Lack of skill, training and timely study results | [18] |
| Mitton et al., 2007 | - Knowledge transfer and exchange | [19] |
| Olariu et al., 2016 | Several challenges are noted in collecting RWE: - The limited availability of policies on this topic - There is no consistent/standardized approach for data collection | [20] |
| Oortwijn et al., 2019 | - Differences in structure, setup and content of databases - No guidance for when and how to use RWD/RWE - Lack of sufficient expertise in HTA agencies to advise on RWE studies or to critically appraise them | [21] |
| Rincon et al., 2012 | - Need of information technology platform and organization for effective adoption of evidence-based practice in healthcare | [22] |
| Szkultecka-Dębek et al., 2018 | - Need for improvement of the data availability and quality in the Central and Eastern European countries - More work is required in the areas of data generation, interpretation and use | [23] |
| Tordrup et al., 2015 | - Differences in practice and local inputs | [24] |
| Tunis et al., 2018 | - RWD is produced for different purposes by different stakeholders - Differences in structure, setup and content of databases - Challenges in sharing RWD and RWE across countries and regions - o accepted guidance when and how to use RWD and RWE | [25] |
| Vincenten et al., 2019 | - Lack of communication and or collaboration - Lack of funding - Capacity deficiencies | [26] |
| Wilk et al., 2015 | Main challenges of RWE development in Poland: - Low awareness of RWE concept - Unstable legislation - High costs - Low computerization level - Lack of cooperation standards and data integration | [27] |
| Wise et al., 2017 | - RWD are non-standardized - RWD are often noisy or unstructured - There is no formal regulatory requirement for RWD - Cyber security issues | [28] |
HTA: Health technology assessment; EMA: European Medicines Agency; RWD: Real-world data; RWE: Real-world evidence.
The next step of the study was a series of three internal group discussions among experts from independent research institutions and academia during the second half of 2020, as preparation for a webinar with stakeholders from CEE countries. During these discussions, to complement the relevance of the RWE transferability barriers derived from the literature search, the final wording was formed and the barriers grouped into main categories to be presented at the workshop (Table 2).
| Barrier | Examples |
|---|---|
| Technical barriers | 1. Lack of expertise and capacity in the HTA agencies to critically evaluate RWE 2. Lack of available financial resources for using and administrating RWE 3. Deficient methods and practices for reporting of RWD curation |
| Regulatory barriers | 1. Lack of unified, widely accepted and implemented guidance documents for all EU countries on how to publish and share RWE 2. Lack of cooperation standards and data integration for common HTA across Europe 3. Requirements for using only local evidence in HTA 4. Lack of clear and accepted requirements of how and when to use RWE 5. Lack of a favorable local/ national governance framework related to using RWE 6. Frequently changing regulations on RWE |
| Clinical and scientific barriers | 1. Unique demographic, racial, ethnic and genetic characteristics 2. Differences in epidemiological data across countries 3. Variations in disease severity 4. Differences in medical practice limiting RWE transferability for specific patient groups 5. Differences in predefined criteria for evaluation of the effectiveness of medicines 6. Lack of transparency in the design, execution and report of studies using RWD 7. Lack of established methodological standards for RWD curation 8. Uncertainty related to the results from RWE studies |
| Perceptional barriers | 1. Uncertainty in the quality of RWE 2. Differences in HTA agencies perceptions and preferences for RWE 3. Limited trust in RWE due to lack of access to the underlying RWD 4. Variability of impact and importance of RWE in decision-making in different Central and Eastern European countries |
HTA: Health technology assessment; RWD: Real-world data; RWE: Real-world evidence.
The online workshop took place on 4 December 2020, with 57 participants joining in from 12 CEE countries. The Medical University of Sofia with the help of Syreon Research Institute and the European Organization for Rare Diseases organized this webinar. The webinar focused on implementation barriers of practices guiding to the HTx project’s vision of next-generation HTA in Eastern European countries. Presentations were given by researchers from several key institutions across Europe, including the Medical University of Sofia (Bulgaria), Syreon Research Institute (Hungary), Utrecht University and Zorginstituut Nederland (The Netherlands), Dental & Pharmaceutical Benefits Agency (Sweden), as well as NICE and the University of York (United Kingdom). 32 attendees represented academic institutions, five came from public payers and other governmental bodies, three represented patient organizations and the remainder were from various independent research institutes, HTA offices and consultancy companies of the CEE region. Input from participants were collected during the workshop, as well as afterward in written form.
A master file was continuously updated, with both methods influencing the final selection and wording of barriers. On the basis of the applied parallel approach, we attempted to summarize the main identified barriers and challenges related to transferability of RWE into four main groups:
•
Technical barriers
•
Regulatory barriers
•
Clinical and scientific barriers
•
Perceptional barriers
Discussion
The first main group of barriers for transferability of RWE is technical or organizational barriers, which were subdivided into four subgroups. The first is related to human capabilities or human resources. Unfortunately, in some CEE countries, a serious problem is the lack of enough experts in the field who might be able to advise on RWE studies or critically appraise them and assess their appropriateness at a national level. This could significantly interfere with correct interpretation of the various RWE coming from other countries. Moreover, the possibilities for RWE transfer from Western European countries in specific CEE countries could be biased and not precisely assessed. A large number of obstacles hinder the application of RWE data created in Western European countries for the purposes of CEE decision-making according to other sources as well. A US study listed some barriers for the limited use of published RWE, including the lack of skills, training and limited availability of timely study results, and highlighted the need for continuous education on the interpretation of study methods and findings to evaluate and use RWE [24].
Financial limitations are a significant problem especially during the current pandemic situation. CEE countries have always had more limited healthcare resources than Western European countries, leading to a situation where they cannot afford to make bad decisions. However, extremely limited funding is available for detailed and comprehensive analysis and appraisal related to RWE in healthcare. RWE are mostly generated in Western European countries due to their advanced technological status and greater available resources. When considering medical, administrative, regulatory data acceptability and transferability in practice in CEE countries, some additional issues surface. Further studies should research the reliability and regional variation issues in existing healthcare databases as well as the transferability and quality of the RWE. Limited financial resources for providing enough valuable scientific evaluation and organization of the process for RWE transferability in CEE countries remains a serious barrier.
Adequate administration of collected RWE from other countries is also important. It is crucial to manage, protect and use the evidence in the right way. Certain obvious problems with maintenance of collected RWE for different health technologies might be identified in some CEE countries; this includes lack of experts’ involvement and lack of sustainability of the administrative process.
RWE should be collected and analyzed electronically, and thus information technology issues in CEE countries are also of high importance because these countries often lack an established and high-level working information technology infrastructure. Low computerization is still identified for some state institutions, and serious efforts are in motion to improve and cope with this issue.
The second main group of barriers includes regulatory barriers. Establishment of a stable government, which is an indicator for good and effective government policy, is a complex and difficult process in some CEE countries. Lack of government stability and frequent political changes in the healthcare sector lead to frequently amended and unstable legislation. The constant changes in local legislative requirements for HTA as well as considerable variations in the approaches for managing the process for RWE use are also results of the frequently changing environment.
Moreover, there are no unified guidelines, prepared based on discussions among a wide range of experts from both CEE and Western European countries. Therefore, no standardized approach exists on how to transfer RWE among countries, especially from early technology adopter to late adopter countries, is still not available. The preparation process for such a guideline is time-consuming and requires the involvement of different stakeholders across multiple countries; therefore, it cannot be conducted without a strong agreement among key decision-makers.
Established HTA policies can form another barrier. In many countries, there are no clear and accepted requirements on how and when to use RWE to inform decision-making in relation to certain technologies. Moreover, there might also be a requirement for adopting only local evidence for the effectiveness of new technology. The impact and importance of RWE in decision-making among CEE countries varies greatly, as some HTA agencies still have doubts regarding the credibility of RWE.
The next identified barrier group comprises Clinical and scientific barriers. Each country has its own specific population characteristics due to geographic, cultural, biological and political reasons. These unique characteristics can lead to significant differences in individuals’ responses to the available health technologies and their effectiveness. Therefore, the process of RWE transferability needs to consider these differences.
Generalization of RWE for clinical results for all patients with a particular disease without differentiation into subgroups could lead to substantial bias in certain cases. To some extent, existing differences in treatment approaches or medical practices in different countries limit the possibilities for RWE transferability for specific patient groups.
Widely adopted pharmacotherapy guidelines are not used in all CEE countries, and some of health technologies are not available in all CEE countries for financial or other reasons. Moreover, a variety in predefined criteria to evaluate the effectiveness of medicines exist, leading to differences among the countries in the process of collecting and transferring specific RWD and RWE. Some parameters such as utility values could be transferred carefully by using the same predefined methodologies.
A key barrier is the lack of internal capabilities to apply highly sophisticated analytical methods for the purpose of transferability and possible reduction of the bias. Development of adequate methods for statistical analysis during the transferability process is urgently required. Moreover, some other barriers could be highlighted, such as lack of transparency in the design, appropriate and suitable statistical methods, execution and report of studies using RWD, lack of established methodological standards for RWD curation and significant uncertainty and skepticism related to the results from RWE studies not supported by RCTs.
The final group of selected barriers is perceptional barriers, which result from uncertainty in the quality of and differences in HTA agencies’ perceptions and preferences for RWE. The quality of collected RWD and related RWE might be different and difficult to evaluate and used for international comparison and analysis in countries where research has not been conducted. HTA agencies in different countries each have their own rules, requirements, beliefs, perceptions and preferences related to RWE use, which could present a reason for RWE transferability delay or rejection. This might result from institutional inertia in state institutions in some CEE countries. In the report ‘Overview of the Development of the Use of RWD Including a Review of International Consensus Methods Currently Developed,’ which was part of the HTx project, a representative from the Estonian agencies shared their opinion that they do not want to use RWD because they believe companies should conduct RCTs [29].
Despite the fact that the HTA regulation entered into force at the beginning of 2022, there is still a lack of cooperation standards, not least in the area of RWE use, and this is not a top priority on the list of preparations for initial joint assessments. Lack of a common EU legislation for HTA processes, lack of cooperation standards and integration of efforts for common HTA among countries could also interfere, to some degree, with the RWE transferability process. These can be the main reasons for nationally oriented requirements and uncertainty of the relevance of RWE. CEE countries often implement their own approaches for access to the use of RWE (even if it may improve efficiency or effectiveness) and put a different value on existing RWE. The forthcoming implementation of regulations on HTA could support voluntary cooperation on HTA between Member States. It will also reduce uncertainty on effectiveness and improve the evaluation of innovative technologies and the assessment of nonclinical domains in RWE use for HTA purposes [30].
This study identified the main barriers for RWE transferability using the combination of two approaches: a literature search, together with internal discussions and consideration of the opinions of stakeholders from CEE countries. The findings revealed that additional education of healthcare providers is needed because of the lack of sufficient experts in the field who might be able to advise on RWE studies and assess them according to national requirements and standards.
To improve transparency, trust and research replicability of RWE studies, a task force of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) recommended that investigators should register RWE studies in publicly available portals [31]. A joint ISPE-ISPOR Special Task Force has been working on different reports related to RWE Good Practices for Real-World Data Studies. On the other hand, real-world studies are less costly than RCTs, which could be a reason for extending post-launch RCTs with RWE studies [32]. Some authors have compared research results from observational studies to results from RCTs with similar objectives. The RCT DUPLICATE initiative (Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) applied a specifically designed process to perform RWE studies emulating RCTs and compare the results from both types of studies. The authors concluded that ‘concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly,’ which could be identified as a barrier for RWE application and transferability [33].
The main limitation of this study is that only barriers considering transferability or application of RWE from Western European to CEE countries were assessed. The limited number of studies included hinders the possibility to present all relevant barriers and their influence on transferability of RWE to CEE countries. The authors attempted to cope with these issues by performing a parallel discussion with stakeholders for the specific barriers in the area. However, further research focused on the most valuable barriers and recommendations of how to overcome them is required.
In addition, several of the challenges identified also exist for HTA agencies in Western Europe, when deciding whether to use RWE – especially the perceptional barriers and some of the regulatory and clinical/scientific barriers. These challenges exist in all decisions about whether to use RWE. Nevertheless, they are amplified when talking about transferability to the CEE setting, where there is an extra layer of additional barriers, such as technical challenges.
Conclusion
Several key barriers to the use of RWE in CEE countries have been identified, including differences in databases, issues with the regulatory framework, different methodological approaches and a lack of effective collaboration. At present, there are few clear solutions to these challenges. Further research should pursue the development of detailed consensus-based guidelines to improve the harmonization and standardization of RWE.
Future perspective
The importance of RWD and RWE is increasing exponentially not only in Western European countries but also in CEE due to the necessity of improving regulatory processes and optimizing decision-making. Many challenges exist not only when converting RWD into RWE but also regarding the process of transferring RWE from Western to CEE countries. Therefore, initiatives and programs should be implemented to improve and foster and optimize the process of transferability of RWE from Western European to CEE countries. At the time of writing, the use of RWE is one of the most important topics in healthcare decision-making. With novel initiatives such as DARWIN-EU [34] and federated data networks such as EHDEN [35], it seems increasingly likely that the focus will not be on whether RWE should be used, but rather how it should be implemented. Many barriers to using RWE are common to all healthcare settings, but they are augmented in CEE by the additional, critical issue of transferability of RWE from Western Europe. We expect – and hope – to see an increasing number of collaborative initiatives in the coming years to foster the processes and infrastructure necessary to support the transferability of RWE to CEE countries.
•
The role of real-world evidence (RWE), generated from real-world data, in healthcare decision-making is growing.
•
As a result of this scoping review and a series of discussions with multidisciplinary HTx (Next Generation Health Technology Assessment) project members, several barriers and challenges related to the transferability of RWE were identified: technical, regulatory, clinical and scientific and perceptional barriers.
•
These challenges exist in all decisions about RWE use but are amplified in the Central and Eastern European setting because of an extra layer of additional barriers.
Author contributions
All authors meet the authorship criteria of the journal: substantial contributions to the conception or design of the work; or the acquisition, analysis or interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All authors conceptualized the design and methodology. M Kamusheva, B Németh and Z Mitkova performed the scoping review and drafting the manuscript. A Zemplényi, Z Kaló, J Elvidge, M Dimitrova, J Pontén and K Tachkov critically revised the manuscript. All authors approved the final version and agreed to be accountable for all aspects of the work.
Financial & competing interests disclosure
The HTx project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 825162. This dissemination reflects only the authors’ view, and the commission is not responsible for any use that may be made of the information it contains. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.
Open access
This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/
References
Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
HauteAutorité deSante. Real-world studies for the assessment of medicinal products and medical devices (2021). www.has-sante.fr/upload/docs/application/pdf/2021-06/real-world_studies_for_the_assessment_of_medicinal_products_and_medical_devices.pdf
2.
Neyt M, Devos C, Thiry N et al. Benefits and costs of innovative oncology drugs in Belgium (2004–2017). Health Technology Assessment (HTA). Belgian Health Care Knowledge Centre (KCE), Brussels, Reports Belgium (2021). https://kce.fgov.be/sites/default/files/atoms/files/KCE_343_Innovative_oncology_drugs_in_Belgium_Report.pdf
3.
US FDA. Realworld data (RWD) and realworld evidence (RWE) are playing an increasing role in health care decisions (2020). www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
4.
Makady A, de Boer A, Hillege H et al. What is real-world data? A review of definitions based on literature and stakeholder interviews. Value Health 20, 858–865 (2017).
• The authors concluded that there is a variability in stakeholder definitions of real-world data (RWD). This could lead to disparities among different stakeholders when discussing RWD use in decision-making.
5.
USFDA. Use of real-world evidence to support regulatory decision-making for medical devices (2017). www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices
6.
Makady A, Ham RT, de Boer A et al. GetReal Work package. Policies for use of realworld data in health technology assessment (HTA): A comparative study of six HTA agencies. Value Health 20(4), 520–532 (2017).
• The differences among policies for RWD use in relative effectiveness assessments of drugs and among the policies between health technology assessment (HTA) agencies might discourage the use of RWD for HTA.
7.
Sherman RE, Anderson SA, Dal Pan GJ et al. Real-world evidence – what is it and what can it tell us. N. Engl. J. Med. 375(23), 2293–2297 (2016).
8.
Kaló Z, Gheorghe A, Huic M et al. HTA implementation roadmap in Central and Eastern European countries. Health Econ. 25(Suppl. 1), 179–192 (2016).
9.
Akhras KS, Alsheikh-Ali AA, Kabbani S. Use of real-world evidence for healthcare decision-making in the Middle East: practical considerations and future directions. Expert Rev. Pharmacoecon. Outcomes Res. 19(3), 245–250 (2019).
• The increase in real-world evidence (RWE) generation could play a key role in driving efficiency in healthcare delivery.
10.
Buckle J, Hayward T. Use of real-world healthcare data to widen the evidence base. Milliman White Paper (2019). www.milliman.com/en/insight/use-of-real-world-healthcare-data-to-widen-the-evidence-base
11.
Cole A, Garrison L, Mestre-Ferrandiz J, Towse A. Data governance arrangements for real-world evidence. Office of Health Economics (2015). www.ohe.org/system/files/private/publications/420%20-%20Data%20Governance%20for%20RWE.pdf
12.
Dreyer NA. Advancing a framework for regulatory use of real-world evidence: when real is reliable. Ther. Innov. Regul. Sci. 52(3), 362–368 (2018).
13.
European Commission. Factsheet on Real-world Data (2018). https://ec.europa.eu/research/health/pdf/factsheets/real_world_data_factsheet.pdf
14.
Hampson G, Towse A, Dreitlein WB et al. Real-world evidence for coverage decisions: opportunities and challenges. J. Comp. Eff. Res. 7(12), 1133–1143 (2018).
15.
Justo N, Espinoza MA, Ratto B et al. Real-world evidence in healthcare decision making: global trends and case studies from Latin America. Value Health 22(6), 739–749 (2019).
16.
Kamphuis B, Avouac B, Colomer R et al. RWE in Europe Paper V: Policy Challenges around Real World Evidence Adoption in Europe 2018. The London School of Economics and Political Science (2018). www.lse.ac.uk/business/consulting/assets/documents/rwe-in-europe-paper-v.pdf
•• The paper outlines the ‘challenges for the adoption of RWE in Europe, the potential role for the pharmaceutical industry to support the adoption of RWE, and how RWE can provide key data in HTA decision-making where other sources are less feasible.’
17.
Makady AA. Real-world evidence for health technology assessment of pharmaceuticals: opportunities and challenges. Utrecht University, Utrecht, The Netherlands (2018). http://dspace.library.uu.nl/handle/1874/364283
18.
Malone DC, Brown M, Hurwitz JT et al. Real-world evidence: useful in the real world of US payer decision making? How? When? And what studies? Value Health 21(3), 326–333 (2018).
19.
Mitton C, Adair CE, McKenzie E et al. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 85(4), 729–768 (2007).
20.
Olariu E, Papageorgakopoulou C, Bovens SM et al. Real world evidence in Europe: a snapshot of its current status. Value Health 19(7), A498 (2016).
21.
Oortwijn W, Sampietro-Colom L, Trowman R. How to deal with the inevitable: generating real-world data and using real-world evidence for HTA purposes – from theory to action. Int. J. Technol. Assess. Health Care 35(4), 346–350 (2019).
22.
Rincon TA. Integration of evidence-based knowledge management in microsystems: a tele-ICU experience. Crit. Care Nurs. Q. 35(4), 335–340 (2012).
23.
Szkultecka-Dębek M, Kawalec P, Jahnz-Różyk K. Real world evidence (RWE) and orphan drug policies in selected Central and Eastern European (CEE) countries. JHPOR 2, (2018). www.jhpor.com/article/2213-real-world-evidence-rwe-and-orphan-drug-policies-in-selected-central-and-eastern-european-cee-countries
24.
Tordrup D, Attwill A, Crosby L, Bertollini R. Research agenda for health economic evaluation. RAHEE Expert Review Meeting (2015). www.euro.who.int/__data/assets/pdf_file/0012/302052/RAHEE-expert-panel-review-mtg-feb15.pdf?ua=1
25.
Tunis S, Sampietro-Colom L. Real-world evidence in the context of health technology assessment processes – from theory to action. HTAi Policy Forum Series Newsletter (2019). https://htai.org/wp-content/uploads/2019/03/HTAi_Global-Policy-Forum_newsletter_20190222.pdf
26.
Vincenten J, MacKay JM, Schröder-Bäck P et al. Factors influencing implementation of evidence-based interventions in public health systems – a model. Cent. Eur. J. Public Health 27(3), 198–203 (2019).
27.
Wilk N, Skrzekowska-Baran I, Wierzbicka N et al. Adoption of real world evidence in decision-making processes on public funding of drugs in Poland. J. Health Pol. Outcomes Res. (2), 23–30 (2015).
•• RWE is of growing importance in Poland; however, the level of the growth depends on building stakeholders' capabilities in the Polish healthcare system.
28.
Wise J, Möller A, Christie D et al. The positive impacts of real-world data on the challenges facing the evolution of biopharma. Drug Discov. Today 23(4), 788–801 (2018).
29.
Overview of the development of the use of RWD including a review of international consensus methods currently developed. Project Title: Next Generation Health Technology Assessment to Support Patient-centred, Societally Oriented, Real-time Decision-making on Access and Reimbursement for Health Technologies throughout Europe (2020). www.htx-h2020.eu/wp-content/uploads/2020/12/HTx-D4.4-Overview-of-the-development-of-the-use-of-RWD-final-for-publication.pdf
•• Most of HTA agencies across Europe ‘are yet not actively working on methods for the use of RWD or access to data.’
30.
Regulation of the European Parliament and of the Council on Health Technology Assessment and Amending Directive 2011/24/EU. Brussels, Belgoum (2018).
31.
Orsini LS, Berger M, Crown W et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing – why, what, and how: recommendations and a road map from the Real-World Evidence Transparency Initiative. Value Health 23(9), 1128–1136 (2020).
32.
Klonoff DC. The expanding role of real-world evidence trials in health care decision making. J. Diabetes Sci. Technol. 14(1), 174–179 (2020).
33.
Franklin JM, Patorno E, Desai RJ et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the RCT DUPLICATE Initiative. Circulation 143(10), 1002–1013 (2021).
34.
Data Analysis and Real World Interrogation Network (DARWIN EU). www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-eu
35.
European Health Data and Evidence Network. www.ehden.eu/
36.
Page MJ, McKenzie JE, Bossuyt PM et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021).
Information & Authors
Information
Published In
Pages: 905 - 913
PubMed: 35726611
Copyright
© 2022 The Authors. This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License
History
Received: 31 March 2022
Accepted: 5 June 2022
Published online: 21 June 2022
Keywords:
Topics
Authors
Funding Information
European Union’s Horizon 2020 research and innovation programme: grant agreement no. 825162
Metrics & Citations
Metrics
Article Usage
Article usage data only available from February 2023. Historical article usage data, showing the number of article downloads, is available upon request.
Citations
How to Cite
Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers. (2022) Journal of Comparative Effectiveness Research. DOI: 10.2217/cer-2022-0065
Export citation
Select the citation format you wish to export for this article or chapter.
Citing Literature
- Carl Björvang, Johan Pontén, Anders Viberg, Andrea Manca, Georgia Salanti, Pekka Siirtola, Gema García-Sáez, Saskia Knies, Diana Delnoij, Dalia Dawoud, Jamie Elvidge, Lifang Liu, Bertalan Németh, Zoltán Kaló, Applicability of academic real-world data research in the case studies of the HTx project to practical health technology assessment work, Frontiers in Pharmacology, 10.3389/fphar.2026.1650552, 17, (2026).
- Zoltán Kaló, Guenka Petrova, Bertalan Németh, Petra Došenović Bonča, Diana Delnoij, Alastair Bennett, Konstantin Tachkov, Yingying Zhang, Noemi Kreif, Saskia Knies, Iga Lipska, Tomas Tesar, Zornitsa Mitkova, Johan Ponten, Anders Viberg, Marian‐Sorin Paveliu, Oresta Piniazkho, Adina Turcu‐Stiolica, Tomáš Doležal, Michal Staňák, Alima Almadiyeva, Rok Hren, Dinko Vitezić, Zsuzsanna Petykó, Wim Goettsch, Andrea Manca, Transferability of Real World Evidence to Support HTA Recommendations in Lower Income European Countries, Health Science Reports, 10.1002/hsr2.71534, 9, 2, (2026).
- G. Price, N. Peek, I. Eleftheriou, K. Spencer, L. Paley, J. Hogenboom, J. van Soest, A. Dekker, M. van Herk, C. Faivre-Finn, An Overview of Real-World Data Infrastructure for Cancer Research, Clinical Oncology, 10.1016/j.clon.2024.03.011, 38, (103545), (2025).
- Konstantin Tachkov, Francisco Somolinos-Simón, Jose Tapia-Galisteo, Maria Elena Hernando, Gema García-Sáez, Maria Dimitrova, Maria Kamusheva, Zornitsa Mitkova, Zsuzsanna Petyko, Bertalan Nemeth, Zoltan Kalo, Tomas Tesar, Marian-Sorin Paveliu, Oresta Piniazhko, Iga Lipska, Adina Turcu-Stiolica, Alexandra Savova, Manoela Manova, Rok Hren, Petra Došenović Bonča, Saskia Knies, Michal Stanak, Tomáš Doležal, Dinko Vitezic, Guenka Petrova, Transferability of new methods for health technology assessment in the field of diabetes between early and late adopters’ countries, Biotechnology & Biotechnological Equipment, 10.1080/13102818.2024.2371354, 38, 1, (2024).
- Konstantinos Zisis, Elpida Pavi, Mary Geitona, Kostas Athanasakis, Real-world data: a comprehensive literature review on the barriers, challenges, and opportunities associated with their inclusion in the health technology assessment process, Journal of Pharmacy & Pharmaceutical Sciences, 10.3389/jpps.2024.12302, 27, (2024).
- Antal Zemplényi, Konstantin Tachkov, Laszlo Balkanyi, Bertalan Németh, Zsuzsanna Ida Petykó, Guenka Petrova, Marcin Czech, Dalia Dawoud, Wim Goettsch, Inaki Gutierrez Ibarluzea, Rok Hren, Saskia Knies, László Lorenzovici, Zorana Maravic, Oresta Piniazhko, Alexandra Savova, Manoela Manova, Tomas Tesar, Spela Zerovnik, Zoltán Kaló, Recommendations to overcome barriers to the use of artificial intelligence-driven evidence in health technology assessment, Frontiers in Public Health, 10.3389/fpubh.2023.1088121, 11, (2023).
- Bertalan Németh, Maria Kamusheva, Zornitsa Mitkova, Zsuzsanna Ida Petykó, Antal Zemplényi, Maria Dimitrova, Konstantin Tachkov, László Balkányi, Marcin Czech, Dalia Dawoud, Wim Goettsch, Rok Hren, Saskia Knies, László Lorenzovici, Zorana Maravic, Oresta Piniazhko, Spela Zerovnik, Zoltán Kaló, Guidance on using real-world evidence from Western Europe in Central and Eastern European health policy decision making, Journal of Comparative Effectiveness Research, 10.57264/cer-2022-0157, 12, 4, (2023).
- Vicky Valla, Konstantina Tzelepi, Paraskevi Charitou, Amy Lewis, Benjamin Polatidis, Angeliki Koukoura, Anna Karapatsia, Kyriaki Antonopoulou, Kanella Prodromidou, Eirini Papadaki, Efstathios Vassiliadis, Use of Real-World Evidence for International Regulatory Decision Making in Medical Devices, International Journal of Digital Health, 10.29337/ijdh.50, 3, 1, (1), (2023).
- Ashley Jaksa, Patrick J. Arena, Kelvin K. W. Chan, Rami H. Ben-Joseph, Páll Jónsson, Ulka B. Campbell, Transferability of real-world data across borders for regulatory and health technology assessment decision-making, Frontiers in Medicine, 10.3389/fmed.2022.1073678, 9, (2022).
- Ildikó Ádám, Marcelien Callenbach, Bertalan Németh, Rick A. Vreman, Cecilia Tollin, Johan Pontén, Dalia Dawoud, Jamie Elvidge, Nick Crabb, Sahar Barjesteh van Waalwijk van Doorn-Khosrovani, Anke Pisters-van Roy, Áron Vincziczki, Emad Almomani, Maja Vajagic, Z. Gulsen Oner, Mirna Matni, Jurij Fürst, Rabia Kahveci, Wim G. Goettsch, Zoltán Kaló, Outcome-based reimbursement in Central-Eastern Europe and Middle-East, Frontiers in Medicine, 10.3389/fmed.2022.940886, 9, (2022).