Transportability of nonlocal real-world evidence and its relevance to health technology assessment: a primer

When evaluating RWE from different geographic or healthcare contexts, it can be challenging to determine how differences in patient characteristics, clinical practices, access to care, healthcare systems and medication use affect its relevance to the local context. Importantly, RWE from countries outside the HTA jurisdiction is increasingly used in HTA, prompting the development of guidance on its use (e.g., see section on ‘External validity bias’ in NICE’s real-world evidence framework [5]). Examples on the use of nonlocal RWE for HTA include the use of RWD from the US to inform survival extrapolations (e.g., atezolizumab for treatment of metastatic lung cancer; NICE technology appraisal TA520), and to derive external comparator arms (e.g., sotorasib for KRAS G12C mutation-positive lung cancer; NICE technology appraisal TA781).

Nonlocal RWE, by which we refer to RWE originating from outside the specific geographic, health system or population context being evaluated for HTA, is essential when local data are unavailable. For instance, countries with stronger clinical research infrastructure and funding to support extensive RWD collection, such as the US, may offer a broader evidentiary base than local RWD at the time of HTA submission for the following reasons:

In such cases, nonlocal RWE may be the only available or timely source of real-world insights. The goal of transportability analysis is to adapt nonlocal RWE to the local context by using statistical adjustments to ensure its relevance to the specific population and healthcare system of interest.

In practice, drug manufacturers may collect RWD in one country – such as Japan or China – and submit it to HTA bodies in Europe or Canada. Limited local sample sizes or short follow-up lengths may necessitate use of nonlocal RWE to improve precision, particularly in rare disease settings or for drugs that are granted accelerated regulatory approval, which are typically approved due to high unmet disease burden or breakthrough therapeutic activity. If both local and nonlocal data are available, combining them can be challenging due to differences in data collection, processing, governance and measurement methods (e.g., claims vs electronic records) in addition to differences in patient and healthcare characteristics.

Clinical input routinely informs HTAs, expert committee deliberations and guideline development. While no systematic review has specifically examined clinician perceptions of nonlocal RWE in HTA ([6] reviewed HTA recommendations on the topic), clinicians often advise on the appropriateness of end points and the generalizability of data to patients seen in practice. We expect that clinicians often will discriminate between local and nonlocal evidence, considering differences in patient populations, clinical practice, treatment regimens, healthcare infrastructure and lifestyle factors. For instance, they may perceive disease incidence or cost estimates from other countries as less relevant to their own. In contrast, they may more readily accept results from randomized clinical trials conducted in other countries. There is considerable variation in clinician perceptions toward the use of RWE in general; however, most clinicians judge it favorably for addressing uncertainties in the use of novel treatments in clinical practice [7].

As an example of criticism of nonlocal evidence by clinicians, consider differences in treatment patterns across jurisdictions described by Sharp et al. [8]. Based on a systematic review of trials in kidney cell carcinoma, the authors reported low rates of second-line immunotherapy in clinical trial control arms due to high costs of immunotherapy drugs in some countries where patients were enrolled [8]. This can lead to an overestimation of benefits from the new therapies evaluated in the trials when applying their results to settings where such therapies are more common (this corresponds to violation of the consistency assumption described in Table 1). Indeed, intention-to-treat effect estimates may have limited generalizability or transportability when subsequent therapies and adherence patterns differ. Emerging approaches, such as the local average treatment effect [9] or per-protocol effects adjusted for nonadherence [10] can improve comparability across jurisdictions by aligning estimates with realistic adherence patterns, despite some challenges in interpretability and estimation relative to intention-to-treat effects.

Clearly, clinical and HTA decision-making should rely ideally on local data. Findings from studies conducted in different geographies, time periods or comparator landscapes may be difficult to interpret for current clinical decision making. Evaluating the transportability of such studies requires substantive cross-country knowledge of differences in clinical practice, patient demographics and treatment standards.

This primer supplements recent reviews on transportability [6,11] and growing research in the area (Figure 2), building on our previous work [12,13] and experience with CDA-AMC and NICE. We provide an overview of the challenges in using nonlocal data for HTA and demonstrate, using a targeted review of HTA appraisals, that common criticisms by CDA-AMC and NICE span all three assumptions required for transportability, which we describe next.

Once transportability assumptions have been evaluated and assumed to hold for the given setting, population, treatments and outcomes, statistical methods can be used to adjust for population-level factors. These methods include matching, weighting and standardization, and are described in more detail in Table 2. A conceptual overview of how transportability analysis using these methods works is shown in Figure 3.

Empirical validation studies using RWD have demonstrated that, in some cases, these methods can successfully adjust nonlocal RWE to the local context [12,15]. Importantly, good data quality is a prerequisite to the use of statistical transportability methods.

As practical examples, we identified CDA-AMC and NICE appraisals that included criticisms relating to the use of nonlocal RWE and related these criticisms to the foundational assumptions for transportability as framed previously. These are shown in Table 3. To generate these examples, we queried publicly available NICE and CDA-AMC appraisals between 2015 and 2024 using keywords ‘generalizability’ and ‘real-world’. Information from select appraisals that included RWD as primary or supportive evidence was extracted to showcase a variety of criticisms, with disagreements about underlying assumptions resolved through discussion between coauthors. This is not meant to be a comprehensive or systematic review of HTA appraisals.

As shown in Table 3, criticisms about the transportability or generalizability of RWE in HTA appraisals are specific to the RWD, disease and intervention. Thematically, these criticisms often map to concerns about one or more – and frequently all three – transportability assumptions. Occasionally, clinical experts providing input to NICE have also concluded that US RWE may be generalizable to the UK for certain therapeutic areas (e.g., initial decision on mobocertinib in TA855). This further highlights the importance of assessing cross-country transportability on a case-by-case basis.

It can be difficult to quantify the impact of concerns about generalizability on the final decision because reimbursement decisions depend on various aspects of the submission. Nonetheless, if these concerns could have been addressed by study sponsors, it is possible that the submission package would have been viewed more favorably. In cases where the primary evidence was from an external control arm analysis (e.g., Table 3 – pralsetinib for RET+ non-small cell lung cancer), consideration of transportability issues at the study design phase may have considerably improved the understanding of their relevance to the local contexts.

When using nonlocal RWD to potentially inform HTA or clinical decision-making, it is ideal to consider transportability issues early in the study design phase. Outlining key assumptions (see Table 1) can help guide variable selection and RWD assessments of fitness for purpose. As described previously, structured frameworks can help guide the identification of relevant differences in healthcare systems, treatment pathways, disease prevalence and patient characteristics [6,11,22] but not all differences will matter in every analysis. While prior work has focused primarily on clinical outcomes such as survival or disease progression, the same framework for evaluating transportability assumptions applies to other end points, including quality-of-life (QoL) measures. For instance, EQ-5D health states are often assumed to be comparable across countries, but differences in patient interpretation, cultural norms or healthcare expectations may affect their cross-country transportability [23,24]. Extending transportability analyses to QoL outcomes would involve assessing whether such factors modify reporting or valuation of health states across populations. This is especially relevant when QoL is a key driver of cost–effectiveness in HTA.

In practice, some key variables for transportability assessment or adjustment will often be unmeasured or captured with non-negligible levels of missingness in existing RWD. This can pose a challenge for transportability analyses. Quantitative bias analysis (QBA) methods can help address this by exploring how unmeasured or mismeasured variables could impact study findings as a part of sensitivity analyses [12,25]. For example, suppose clinicians note that adherence to oral antivirals for COVID-19 is higher in the nonlocal jurisdiction, but that adherence data are not available in the nonlocal RWD to be able to implement transportability analysis directly. QBA allows analysts to

QBA can be conducted using deterministic (single scenario) or probabilistic (range of scenarios) approaches to test the robustness of findings. This helps decision-makers understand how reliable evidence is despite the lack of measurement of some important variables. For details, see a recent primer on QBA [28] and its application to external control arm analysis [13].

HTA increasingly considers the real-world relevance of study findings and the use of RWE [29]. When high-quality, locally sourced data are lacking, using RWD from other countries can be attractive but requires a nuanced understanding of geographic and clinical contexts. Understanding when and how evidence from elsewhere can be used is essential for making informed, transparent decisions. While methods for evaluating transportability are advancing, clear guidance and practical examples can help HTA agencies, regulators and industry stakeholders use nonlocal data more confidently. Ethical considerations and stakeholder involvement remain critical to ensure that the application of transportability methods aligns with patient and system priorities. Indeed, the thoughtful use of RWE, whether local or nonlocal, can ultimately accelerate patient access to novel therapies.

This work was funded by AstraZeneca Canada.

The authors have no competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No funded writing assistance was utilized in the production of this manuscript.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/