Why is target trial emulation not being used in health technology assessment real-world data submissions?

Target trial emulation (TTE) is a framework for the application of randomized controlled trial (RCT) design principles to the analysis of observational data, thereby explicitly tying the analysis to the trial it is emulating [1]. The TTE approach has been proven to enable inference of causal treatment effects while avoiding the critical biases that often plague observational studies [2,3]. An illustration of this comes from Dickerman et al. [4] who emulated a trial in real-world data (RWD) to estimate the effect of statins on the risk of small cell lung cancer among adults with low-density lipoprotein (LDL) cholesterol below 5 mmol/l. A prior observational study [5] suggested that statins may reduce the risk of small cell lung cancer by 77%, however Dickerman found that the TTE per protocol and intention-to-treat analyses reported no effect of statins on lung cancer. The TTE results were in line with those observed in meta-analyses of RCTs (Table 1). The conflicting results generated by observational studies as compared with RCTs has been one of the key reasons why regulators and health technology assessment (HTA) bodies have been reluctant to accept RWD as a measure of treatment effectiveness. By applying the TTE framework, Dickerman et al. demonstrated that the previous observational study had two critical flaws: using post-baseline information to assign baseline treatment, and including individuals who were using statins before baseline. When Dickerman and colleagues included these methodological flaws in their analysis, the effect estimates replicated the non-TTE observational study results (Table 1). Given the ability of the TTE framework to minimize bias in observational studies, several HTA agencies have recently advocated for TTE because application of the framework increases transparency around study design. However, despite guidance from HTA agencies being in place since 2022, to the best of our (AC, SR) knowledge no HTA RWD submission to date has explicitly employed TTE.

At an issue panel at ISPOR 2024, we debated the reasons for this. S Duffield confirmed that unlike the academic literature, where articles referencing TTE have increased exponentially since 2020 (Figure 1), a rudimentary search of National Institute of Health and Care Excellence (NICE)'s appraisals did not identify any submission explicitly using the TTE approach even though it is central to NICE's real-world evidence (RWE) framework [6]. One caveat may be that some published studies identified by systematic literature review and submitted as part of the appraisal could have used this approach.

Polling of the audience attending the panel session was surprising in that it revealed that the main barrier to conducting TTE studies was lack of expertise and experience with the approach, despite its widespread use in the scientific literature. Furthermore, many attendees expressed distrust that HTA agencies would accept TTE RWE despite explicit support of TTE by NICE and several other HTA agencies [7]. This may reflect past experience where submissions using RWD as a measure of treatment effectiveness have largely been rejected by HTA agencies [8].

R Reynolds provided a manufacturer's perspective. While an advocate for the TTE approach, he cautioned that TTE is not a failsafe, and there may be barriers such as the fact that TTE cannot address questions that cannot be framed in terms of a pragmatic trial (i.e., self-controlled or test negative designs in vaccines). R Reynolds also highlighted the fact that TTE is not the only framework to provide best practice around RWD study design and execution. Notably, the estimand [9] and the SPIFD2 frameworks [10] could also be used to minimize bias. SPIFD2 calls for the selection of fit for purpose RWD, articulation of the hypothetical target trial and transparent documentation of the rationale for decisions made which can then be used to populate the STaRT-RWE or HARPER templates [11,12]. The estimand framework on the other hand provides a systematic approach to defining the treatment effect under investigation in a clinical trial. This covers characterizing PICOs (population, intervention, comparator, outcome) but also includes describing how handling intercurrent events will be handled and how outcomes will be summarized and compared between treatments. R Reynolds pointed out, however, that the estimand framework was developed with a focus on RCTs and therefore is lacking guidance on the description of baseline confounders and the need to specify time zero. S Duffield noted that both the estimand and SPIFD2 frameworks have also not been explicitly used in HTA RWD submissions.

Additionally, R Reynolds noted that there are currently no standard reporting guidelines for TTE (although these are in development [13]) and therefore while TTE may be performed by manufacturers, it may not be documented as such in submissions to HTA bodies. In support of the fact that manufacturers are performing TTE, an example of RWE for post-approval effectiveness that explicitly used the TTE framework was shared [14]. In this example, challenges such as defining the time of treatment implementation and required deep clinical knowledge to enable accurate and appropriate definitions. A recent demonstration project also highlighted this as a challenge when conducting HTA and offered some pragmatic suggestions when defining the hypothetical trial to be emulated [15].

To illustrate the importance of transparency in RWD analyses, S Duffield discussed a case study regarding a recent submission to NICE. The example was an external control arm for a single arm trial (SAT) conducted in a rare subgroup of non-small cell lung cancer [16]. The ECA included US cohort data and routinely collected cancer registry data from the UK (albeit for a very small number of individuals). The resulting adjusted indirect comparison was based on a blended comparator and the committee was concerned that the mix of comparator treatments and subsequent therapies might not be relevant to the UK. Further, results were based on complete case analysis and confounders were taken from a prespecified list which led the committed to question whether all relevant confounders were captured and the magnitude of missing data. The committee also commented that it was not possible to assess whether best-practice study design principles had been followed and whether confounding and residual bias had been robustly addressed. S Duffield noted that the committee's concerns might have been addressed with the completion of the NICE DataSAT tool and by describing how the RWD emulated the SAT, by for example describing it using a TTE protocol.

In conclusion, while the TTE framework has been shown to minimize bias in observational studies and is advocated for by several HTA agencies, its use in HTA submissions has been limited to date. This may be because RWE guidance in the HTA setting has only been around for just under two years, perhaps meaning that manufacturers have not had enough time to employ the approach for submissions. The lack of reporting guidance on TTE may also be a reason why TTE is not explicitly mentioned in HTA submissions. However, the main barrier from an ISPOR audience appears to be a lack of expertise and experience with the approach since its use has only become prominent in the scientific literature recently. Ultimately, the key to successful RWE submissions lies in transparently documenting the rationale for decisions made in data source selection and analysis, which TTE supports. Broadly, there appears to be a need in the health economics and outcomes research community to develop expertise in this area, something which stakeholders should collaboratively seek to address.