R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 8

Real-world data (RWD) is being increasingly used in initial health technology assessments (HTA) of medicines where randomized evidence on comparative treatment effects is not available, particularly as more medicines are granted regulatory approval based on uncontrolled single-arm trials. However, concerns persist with the use of RWD for this purpose, including fears over data quality and bias. In response, many HTA bodies have introduced guidelines aiming to describe best practices for real-world studies to improve the quality and transparency of evidence. However, as noted previously [1], numerous studies have identified that guidelines published by HTA bodies fall short of providing detailed recommendations on best practice [2,3], leading to calls for collaborative efforts to improve guidelines.

A new draft real-world evidence (RWE) framework published by the National Institute for Health and Care Excellence (NICE) may represent a step-change [4]. The framework outlines current and potential uses of RWE in NICE submissions, including documenting the natural history of disease and its economic burden, and providing inputs for economic models including baseline rates of events, transition probabilities between health states and resource use and costs. Challenges common to many RWE studies are presented, before the framework provides in-depth guidance surrounding the planning, conducting and reporting of any real-world study. Detailed information is provided on assessing data suitability, with NICE developing a bespoke Data Suitability Assessment Tool and reporting it in the appendix of the framework to aid reporting. Criteria are set out relating to data provenance (collection, coverage and governance), data quality (completeness and accuracy of key study variables) and data relevance (availability of key confounders and outcomes, generalizability to the target population and sample size and follow-up). Specific guidance is provided for comparative effectiveness studies, with detailed recommendations for study design, including the use of the target trial approach [5,6]. Analysis guidelines provide recommendations on how to identify potential confounders, articulate causal assumptions, address confounding, consider and address the impact of bias from informative censoring, missing data and measurement error, and use of sensitivity and quantitative bias analysis. Reporting requirements are also outlined, including justifying the need for non randomized evidence, publication of study protocols and analysis plans prior to analysis, and the use of tools for reporting information on key analytical methods and methods to minimize the risk of bias.

While the draft NICE RWE Framework represents a significant step forward, a recent press release from Germany’s Institute for Quality and Efficiency in HealthCare (IQWiG) provides a timely warning that the existence of comprehensive guidelines does not guarantee that best practices will be followed. In 2020, IQWiG published a rapid report outlining criteria for the collection and usability of RWD in their benefit assessments of pharmaceuticals [7]. The press release outlines how these guidelines are not being adhered to, using recent dossiers for four therapies as case studies [8]. A submission relating to amivantamab monotherapy for the treatment of non-small-cell lung cancer was criticized for RWD sources lacking data on key end points and confounders, and not exploring the potential impact of these limitations on study results. Similar criticisms were levelled at a submission relating to nivolumab + ipilimumab for colorectal cancer, where the absence of laboratory data and data on side effects from previous therapies meant that key inclusion criteria from the trial could not be replicated in the RWD source. The use of US data were also criticized in this submission, citing the lack of generalizability to the German population. IQWiG’s conclusion from these submissions was that manufacturers are not interpreting their requirements correctly.

However, a lack of adherence to guidelines does not necessarily imply ignorance. While electronic health records in the USA, for example, contain rich information, for some markets high-quality RWD for a condition may not exist, leading to a trade-off between the use of local data and other important characteristics such as quality, recency and clinical detail. Therefore, in many cases, deviation from guidelines will instead reflect the difficulty of conducting real-world studies in line with best practice, rather than a misunderstanding of best practice.

A lack of adherence to best practice could also be due to limitations in guidelines. Where key elements of best practice are not covered, or are not described in sufficient detail, those carrying out real-world studies must determine best practices using fragmented information from other sources. In some cases, approaches taken will diverge from those expected by HTA committees making reimbursement decisions, especially in cases where there is limited consensus among academics about what constitutes best practice. For example, although submissions were criticized for not exploring consequences of data limitations on results, IQWiG’s rapid report only states that sensitivity analyses should be conducted and provides little detail on what specific sensitivity analyses are recommended. This includes methods relating to quantitative bias analysis, which are not discussed in the rapid report, but would be particularly informative in this context [9]. The rapid report also concentrates specifically on methods for confounder adjustment, with methods for minimizing information bias due to missing data, informative censoring and measurement error not outlined.

Also, despite the draft NICE RWE Framework representing arguably the most comprehensive guideline published by a HTA agency to date, gaps still exist. For example, although advantages and limitations of broad classes of methods for adjusting for observed confounders are discussed, the framework stops short of providing decision rules to aid in the selection of specific methods. Similarly, although it is stated that a ‘transparent, systematic and reproducible process’ should be used to identify relevant confounders using published literature and expert opinion, questions remain on whether it is sufficient to elicit expert opinion from simple advisory boards or whether more complex consensus methods such as Delphi methods should be used. NICE does recognize this however. In a document accompanying the framework, NICE states that the framework is not positioned as a formal methods guide and will be extended in scope to include additional guidelines on priority topics and exemplar case studies [10]. For methods used to analyze randomized controlled trials (RCTs), NICE’s Decision Support Unit have published technical support documents which provide detailed information on specific methods, along with case studies and example code [11]. Similar technical support documents for methods relating to RWE would perhaps represent the optimal way of demonstrating best practice.

Initial findings from the RCT DUPLICATE study suggest that following best practice can in some cases lead to the production of high-quality RWE that aligns with randomized evidence [12]. If RWE methods are to be applied as consistently as those used for RCTs, the level of RWE guidance needs to mirror that of trial methods. Other HTA agencies may respond to the NICE RWE framework by publishing guidelines of similar detail. Let’s hope they do, as robust RWE can definitely play a key role in ensuring access to innovative treatments for patients.