R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 9

Randomized controlled trials (RCTs) remain the gold standard tool for assessing effectiveness of treatments for use in health technology assessment (HTA). However, the use of real-world data (RWD) for this purpose plays a crucial role, offering the opportunity to estimate comparative effectiveness when well-powered RCTs are not possible and supplementing RCT evidence by providing estimates over the longer-term and in population subgroups typically excluded from RCTs.

However, not exploiting randomization has limitations, most importantly the increased risk of unmeasured confounding and other biases. Quantitative bias analysis methods can help to explore how unmeasured confounding is likely to impact estimates of treatment and cost–effectiveness [1–3], but perhaps more important are approaches that help to limit rather than address residual confounding.

A recently published International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Good Practice Report outlines the opportunities machine learning (ML) methods may provide in health economics and outcomes research and highlights multiple approaches that may specifically help with minimizing confounding in RWD studies, including cohort selection, confounder identification and adjustment and estimating treatment effect heterogeneity [4].

Cohort selection is highlighted in the report as an area where ML methods offer particular promise. RWD sources, particularly electronic health records, often derive variables from unstructured data such as free-text clinician notes within patient records. Manual abstraction has traditionally been used to extract information from such sources, which is costly and time consuming. The report highlights how ML methods such as natural language processing can increase the efficiency of cohort selection. This involves practitioners first labeling the data, training ML models on this data and then applying them to new data to determine probabilities of cohort eligibility. Detailed human abstraction is then only conducted on patients with probabilities over a certain threshold, enabling more targeted and detailed extraction. A concern raised in the report is that this approach maximizes specificity over sensitivity (generating few false-positives, but many false-negatives), leading to selection bias if false negatives are not randomly sampled. However, work by Flatiron Health suggests bias can be avoided, with an algorithm derived to identify patients with metastatic breast cancer demonstrating considerable sensitivity with no differences in average outcomes and baseline/clinical characteristics relative to those in test cohorts generated using only manual abstraction [5].

A second promising use case highlighted in the report is in the identification of confounders. HTA RWD guidelines recommend that confounders are identified using systematic searches of prognostic studies and/or use of expert opinion [6]. However, literature and knowledge gaps are possible, particularly in indications that are less researched, such as rare diseases or disease subgroups. Small sample sizes may also limit the ability to adjust for all potential confounders even if data on them is available. In processes called feature selection and extraction, the report explains how ML methods can be used alongside study sample data to identify new relevant confounders and/or reduce the covariate set to those that are the most prognostic. This may be particularly useful in small samples where traditional statistical techniques are prone to overfitting.

The report also details how ML methods can enhance approaches used to adjust for observed confounding. Assumptions regarding the nature of the relationship between confounders and outcomes should always be guided by causal reasoning and subject matter knowledge. However, just like information on whether a given variable is a confounder, functional forms are often unknown, with many practitioners assuming linear and additive effects and ignoring potential non linearity and interaction effects. The report highlights how common ML methods for prediction and classification can flexibly estimate functional forms semi- or nonparametrically.

Causal ML methods can also flexibly estimate treatment effect heterogeneity [7,8] and the report highlights the usefulness of this in supporting a move toward individualized treatment decisions. However, these approaches may have wider applications. The absence of unmeasured effect modification is a key assumption for methods used to generalize or ‘transport’ treatment effects from a study to different populations [9] and ML methods are being increasingly recommended for identifying effect modifiers in this setting [10]. The issue of transportability is likely to become increasingly important as the use of RWD becomes more prevalent in HTA, particularly as RWD data sources from one country are commonly used to generate evidence for HTA submissions in another [11].

Despite the benefits and potential uses of ML methods, the report highlights some limitations. ML algorithms are only as good as the data from which they learn, meaning if training data are flawed, so too will the algorithms. This can lead to unintended biases, such as reinforcing racial inequalities where input data are impacted by racial bias such as unequal access to treatment [12]. Transportability may also be a concern for certain ML algorithms. For example, a deep-learning system for identifying diabetic retinopathy developed by Google Health demonstrated high accuracy both in the lab and in clinical practice across multiple countries [13,14], but faced implementation difficulties in low- and middle-income countries, where lower quality imaging was more prevalent [15,16].

ML methods are also inherently complex and the report highlights concerns over the transparency of these methods and their ‘explainability’ to patients and decision makers. There is a feeling that RWD in and of itself suffers from a lack of acceptability by HTA agencies [17]. If it is felt that complexity and difficulty in understanding is a key driver of non acceptance, manufacturers may be hesitant to use ML methods in HTA submissions. The report encourages ML developers to be transparent about the development and execution of ML methods and develops a ‘PALISADE’ checklist to aid in this. This may be a helpful first step in encouraging acceptance, but demonstration projects that highlight the usefulness of these methods in a HTA setting may also be needed. More studies examining and addressing potential limitations of ML methods and also exploring in what cases ML methods are superior to existing approaches, may be required to generate the confidence needed to enable wider adoption. Many HTA agencies are yet to provide concrete guidance on when they are willing to accept RWD for guiding their decisions. Reducing or addressing key biases is crucial for improving the acceptance of comparative effectiveness studies using RWD, but will require the application of increasingly complex analytics, including the use of methods such as ML. HTA agencies will therefore need to maintain awareness of methods development in the area and update guidelines accordingly.