R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 10

Regulatory agencies and health technology assessment (HTA) agencies commonly express a preference for the use of randomized controlled trials (RCTs) for estimating the comparative effectiveness of health technologies, driven by the fact that randomization minimizes the risk of confounding from unmeasured patient characteristics [1].

However, evidence-based decision making also requires that evidence is externally valid, in other words, that treatment effects from a study represent unbiased estimates of effectiveness in the population for which the decision is being made (the ‘target population’). RCTs are often conducted in selected samples of patients, with important population subgroups that may receive a treatment in clinical practice excluded. If characteristics that differ between RCT and target populations are correlated with treatment effectiveness (i.e., are ‘effect modifiers’), then evidence will not be externally valid [2].

Studies using real-world data (RWD) are increasingly common sources of comparative effectiveness evidence [3–5]. Despite a lack of randomization increasing the threat of confounding, RWD studies are conducted in samples of patients being treated in clinical practice, and so have the potential to be more externally valid.

However, external validity does not just refer to generalizability but also to transportability [6]. Whereas generalizability relates to whether inferences from a study can be extended to a target population from which the study dataset was sampled (often the concern for RCT evidence), transportability relates to whether inferences can be extended to a separate (external) population from which the study sample was not derived. Variation in the availability and quality of RWD across countries means that submissions to some HTA agencies often include evidence on comparative effectiveness from patients residing in other countries. A recent survey of EUnetHTA member HTA organizations documents a widespread concern over a lack of transportability. Organizations were asked to rate the likelihood of accepting RWD in a range of complex situations, and the use of RWD from countries outside of their region was rated as the least acceptable, even in cases where this was the only data available [7].

Such fears have led to a preference for local RWD by HTA agencies [8,9]. As we noted in a previous update [10], Germany’s Institute for Quality and Efficiency in HealthCare (IQWiG) have historically taken a strong position on this, repeatedly criticizing submissions for using international data [11]. However, statements made within two recent benefit assessments for treatments of non-small-cell lung cancer may suggest a softening of this stance. The benefit assessment for sotorasib criticized the use of the German CRISP registry for its incompleteness, and suggested that data from Flatiron Health could have been used (citing its use in a submission to the National Institute for Health and Care Excellence [NICE]), despite this data source only including data on patients from the US [12]. Similarly, the benefit assessment for amivantanab criticized the use of CRISP and another German registry for lacking essential patient characteristics and outcomes, and again questioned why the submission did not use data from Flatiron Health (citing its use in supporting the regulatory approval of amivantamab in Europe) [13]. This assessment also highlighted the potential use of the US-based Fred Hutch Cancer Surveillance System and the Spanish Thoracic Tumors Register. Such decisions suggest IQWiG’s view may now more closely align with NICE, whose real-world evidence framework recognize that the choice of data source involves a trade-off between locality and other aspects of data quality [9].

In addition, a recent review of transportability methods suggests that differences in key effect modifiers across countries does not necessarily need to result in the absence of external validity [14]. The review highlights that if companies have access to data on the target population, and data on effect modifiers are available in both the target sample and study sample, methods similar to those used to adjust for confounding, such as regression-based, weighting-based and doubly-robust methods, can be applied to ensure balance on effect modifiers. Simpler methods, often used to conduct population-adjusted indirect treatment comparisons [15], can also be used where only aggregate data are available on the target population. The review also documents quantitative bias analyses that can be used to assess the sensitivity of results to unobserved effect modification in ways that can facilitate decision making in cases where a lack of transportability is a concern.

However, to date we are not aware of any application of these methods to demonstrate transportability of RWD in a HTA setting, and their potential usefulness is not described in any HTA RWD guidelines. Demonstration studies highlighting the applicability of these methods, would be useful, particularly in indications such as oncology where the use of international RWD is common. A recognition of the usefulness of these methods in HTA RWD guidelines could stimulate their uptake, or better still, by referencing demonstration projects, HTA agencies could explicitly call out which countries effectiveness estimates could be transported from.

Concerns regarding external validity due to the use of international RWD are valid, but HTA agencies must recognize that there are often trade-offs between locality and other aspects of data quality. There is uncertainty over whether recent requests for the use of US data by IQWiG is indeed truly reflective of a more widespread shift in attitudes toward the use of international RWD by HTA agencies, or if these were special cases due to the lack of appropriate local data. If it is the former, then this may represent an important step toward increasing access to therapies for patients living in countries where there is a paucity of readily accessible high-quality RWD.