Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle? A review of methodological approaches and critical considerations

A series of comprehensive (pragmatic) reviews was conducted to determine what guidance is available for the appropriate identification and selection of EMs when conducting ITCs and how this guidance has been applied in practice; the review included guidance documents from HTA bodies and non-payer organizations, published ITCs and prior HTA submissions in key representative indications. The review focused on how EMs are identified and was not intended to summarize the methodologies used to adjust for EMs. The reviews were restricted to articles in English; however, no geographic restrictions were otherwise applied. The searches of published ITCs and HTA submission documents were also restricted to those published since 2015 and 2017, respectively. This is because one of the key guidance documents in this area – the National Institute for Health and Care Excellence (NICE) Decision Support Unit (DSU) Document 18 – was first published in December 2016, highlighting the relatively recent focus on more advanced methods to adjust for EMs in the context of ITCs [2].

Searches were first conducted in November 2021 and updated in February 2023 across HTA bodies, non-payer organizations and literature databases (Embase and MEDLINE via Ovid SP; Supplementary Table 1) to describe the requirements for the selection of EMs for ITCs and how this was applied in ITCs subsequently published in peer-reviewed journals. Articles on NMA/ITCs identified through the database searches that did not explicitly refer to these concepts in title or abstract were not selected further. Conference abstracts, letters, editorials, or narrative reviews were excluded. Furthermore, pairwise meta-analyses or publications on statistical methods for the adjustment of EMs were not considered. Overview of reviews (also called umbrella reviews) were not considered as they typically do not provide the breadth of information to allow elaboration of issues around EM selection and we considered that any relevant primary ITCs would have already been identified by the literature searches.

The websites of NICE and the Canadian Agency for Drugs and Technologies in Health (CADTH) were reviewed for submissions in non-small cell lung cancer (NSCLC) and psoriasis – an oncology indication and a chronic disease with rapidly changing treatment landscapes – as example case studies to identify how EMs and PVs (given that these terms are frequently used interchangeably) were identified and selected in the context of ITCs for the reimbursement of pharmaceuticals. The search was restricted to NICE and CADTH as other HTA bodies usually only publish submission summaries, which did not provide sufficient information for the purpose of the current review (Supplementary Table 2). Only the latest version of each submission or re-submission was included. Terminated appraisals were not considered.

The reviews were supplemented by Google searches and a snowballing approach using the identified articles. Information on the search strategies is provided in the online appendix.

All screening was conducted by one researcher, with a second, more senior researcher conducting quality assurance checks. The final list of records informing this publication was agreed through discussions among the researchers. Information from the relevant documents was extracted into pre-defined data extraction sheets specifically designed for each review type. All extractions were validated by a second researcher.

Themes across these publications were summarized qualitatively with the aim to identify relevant topics in individual documents and provide a critical assessment of proposed selection approaches for EMs in the context of ITCs for healthcare decision-making.

Comparative treatment effects for novel health technologies commonly rely on the use of indirect comparisons in the absence of head-to-head trials, based on the assumption that the distribution of EMs is similar across the study populations. While imbalance in this EM distribution can considerably bias the validity of analysis results, issues can also arise if PVs are considered interchangeable with EMs, also introducing uncertainty in the decision-making [2,7]. Yet, no clear guidance exists from HTA bodies on the “appropriate” method for identification and selection of EMs.

The current series of pragmatic reviews showed that published guidance, across different organisations and settings, on this topic was generally scarce. This is surprising given the importance of EMs in ITC methods and analysis.

The most explicit guidance was from NICE which recommended a review of the literature or structured discussions with clinical experts to inform this selection [16]. However, the guidance falls short of providing clarity on how literature reviews should be designed; for example, which type of evidence should be considered, which criteria should be applied to define the transferability of findings on EMs and PVs from other sources to the current trial and trial population, and whether a hierarchical approach to the final selection of EMs and PVs to be considered and adjusted for should be applied. Documents from HAS, PBAC and the G-BA referred to qualitative and quantitative methods to test for EMs without providing further details on the exact methods to be employed (the 2009 HAS guidance was the most recent English version that could be identified) [17–19]. Non-payer organisations mainly provided checklists for reporting ITC methods and results. Although many included specific reporting items for the EM selection process, it was not the remit of these checklists to provide clear guidance on how the selection should be conducted in practice.

In line with the lack of clear guidance by payers and non-payer organisations, this trend also continued in practice (published literature). Authors rarely provided justification for the selection of EMs in the published NMAs/ITCs. Some publications listed a-priori selected factors but did not provide additional details on how these lists had been compiled. Six published ITCs noted that the selection of EMs was based on results from a literature review whereas five articles mentioned that experts had been consulted. However, none of these reported whether a systematic approach had been applied for the identification of these factors. It is also unclear to what extent word limits by scientific journals may have contributed to the omission of such information. Despite the short timeframe of the review of published NMAs and ITCs, an early trend suggested increasing awareness of the need to provide more information on the EM selection process, with an increasing number of relevant articles identified year-on-year.

The findings from these reviews were supported by the results of reviewing previous NICE and CADTH submissions for NSCLC and psoriasis—two indications with a rapidly changing treatment landscape. In many instances, EMs appeared to have been defined a-priori (based on prior submissions on the same disease) or the selection was informed by subgroup analyses of clinical trials considered for the analysis. Clinical experts were often involved in validating these assumptions in many submissions, in line with general requirements from HTA bodies, although this was more often explicitly stated in NICE than in CADTH submissions. Clear descriptions of a systematic identification of EMs, an important aspect in providing confidence in the ITC results, were lacking across the identified submissions.

Against this background, further research is needed to address key aspects for building an evidence base with an eye toward supporting regulatory as well as reimbursement submissions (Figure 5). SLRs may need to be tailored accordingly so that a systematic approach to the identification and selection of EMs can be applied. For example, a broad scope SLR of clinical data (both from clinical trials and real-world evidence) can be conducted to identify (1) factors previously described in the literature and prior HTA submissions, and (2) subgroup analyses from both the index trial, previously conducted trials, and published NMAs/ITCs.

A key component in HTA submissions is the integration of expert opinion to describe uncertainties associated with the clinical and cost-effectiveness of competing interventions; structured elicitation exercises have a critical role in validating some of the assumptions in the use of PVs during ITC feasibility studies but a clear documentation of the processes and its results is needed to ensure unbiased conclusions [42–44]. Another component of paramount importance to the study design is the inclusion of clinical experts to validate assumptions made during the feasibility assessment or to identify potentially relevant factors through a formal elicitation exercise. The distinction between EMs and PVs can be challenging when based solely on clinical practice experience, underscoring the need for providing experts with detailed methodological a-priori training to separate the meaning between the two definitions.

The key strength of this research is the combination and comprehensiveness of review activities (guidance frameworks, HTA submissions based on these frameworks, and published ITCs in a real-world setting) to answer the research question. Despite the relatively short timeframe for the reviews of HTA submissions and published ITCs, it is unlikely that more comprehensive information on the EM selection process is available from older publications given the clear recommendation by the NICE DSU published only in 2016 [2].

This research was limited by several factors. First, the literature searches for published ITCs were restricted by terms for EMs and PVs for practical reasons. However, this was deemed unlikely to have biased the trend around the underreporting. Given the importance of addressing interfering variables in ITCs, abstracts were considered to commonly include whether an extensive exercise to identify these factors had been conducted. Second, a pragmatic decision was made to restrict the review of HTA submissions to two representative indications both of which are associated with a rapidly evolving treatment landscape: one chronic disease and one oncology indication. EMs are relatively well established for NSCLC; however, this is not true for psoriasis. This may partially explain the lack of information on interfering factors in many of the HTA submissions for psoriasis. It should be noted that in cases where data are not sufficiently available on interfering variables, additional (systematic) efforts to identify EMs and PVs are unlikely to result in a more comprehensive list of factors to be considered. Third, commercially sensitive information in HTA submissions is often redacted from published documents. In many CADTH submissions, the ITC relevant sections were heavily redacted, providing limited information on the analysis and any EMs and PVs considered relevant for. Some CADTH submissions had to be excluded from the review because even the analysis type was considered commercially sensitive. Finally, the lack of clear information in NICE appraisals on how interfering variables were identified may partially be due to the structure of the NICE submission template. Most NMA/ITC details are included in the appendices which are not immediately publicly available.

Given the evidence-based commitment by HTA bodies for using high-quality, fit-for-purpose evidence, there is a gap in guidance on how to appropriately identify EMs for inclusion in ITCs supporting HTA submissions. Adjusting for the “right” factors should be considered equally as important as choosing the appropriate ITC method. This guidance gap continues as a trend of underreporting of the identification and selection of EMs and PVs in published ITCs and company submissions for reimbursement. More explicit guidance is therefore needed on the wide range of evidence and sources needed to comprehensively identify EMs, including how to conduct SLRs on EMs and validation through formal expert elicitation. Similarly, a transparent reporting of the EM and PV selection process is required when conducting NMAs and ITCs to allow for an independent review and critique of the set of variables included for adjustment.

A Freitag, L Gurskyte and G Sarri were responsible for study conception and design; all authors contributed equally to the literature review, and drafting and revision of the manuscript.

The authors thank Maria Rizzo for valuable input in the design of the review, and Colleen Dumont for reviewing the manuscript and providing comprehensive editorial assistance.

All authors are employed by Cytel. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/