Free access

Review

28 March 2018

Analysis of indirect treatment comparisons in national health technology assessments and requirements for industry submissions

Authors: Ischa J Es-Skali [email protected] and John SpoorsAuthor Info & Affiliations

Publication: J. Comp. Eff. Res.

Volume 7, Number 4

https://doi.org/10.2217/cer-2017-0092

PDF

Abstract

Aim: To determine the preferred methodologies of health technology assessment (HTA) agencies across Europe, Canada and Australia to ascertain acceptance of indirect treatment comparisons (ITC) as a source of comparative evidence. Method: A review of official submission guidelines and analysis of comments in HTA submissions that have used different ITC methodologies. Conclusion: ITC is generally accepted as a technique that allows demonstration of noninferiority to a comparator provided the chosen methodology and underlying assumptions are clear and justified. However, HTA agencies are more likely to closely scrutinize submitted data and evaluate statistical significance of results when superiority is claimed. In addition, the HTA agencies in scope tended to be cautious and only accept ITC data as support for similarity of treatments.

In order to issue clinically relevant guidance, health technology assessment (HTA) agencies have to consider care pathways which often evolve rapidly. Moreover, if the agencies are to comment on relative effectiveness and determine therapeutic positioning, they may require comparative evidence against market-relevant alternatives. However, the development and launch of new therapies can take place quickly or even in parallel, leading to rapidly evolving therapeutic paradigms in clinical practice. As a result, for manufacturers it is challenging, potentially high risk and costly to perform head-to-head trials against the comparator(s) considered most relevant by HTA agencies. One way of addressing this situation is to perform indirect treatment comparisons (ITCs) (Table 1). However, although these are possible technically, a key question relates to their acceptability by HTA agencies. Our paper seeks to qualitatively analyze the preferred methodologies of HTA agencies across Europe, Canada and Australia and their willingness to accept ITC as a source of comparative evidence. We were especially interested to know whether having individual patient data (IPD) sets for one or more trials, allowing for more sophisticated population-adjusted indirect comparisons, has any distinct advantages.

Table 1. Definitions of indirect treatment comparison methodologies.

Indirect treatment comparison/mixed treatment comparison	The estimation of the relative effectiveness of two or more treatments in the absence of any head-to-head trials
Mixed treatment comparison/network meta-analysis	The simultaneous estimation of the relative effectiveness of three or more treatments using a combination of direct and indirect evidence and a common comparator
Bucher's adjusted indirect comparison	Adjusted indirect method of treatment comparison that can estimate relative treatment effects for star pattern networks
Population adjusted ITC	Indirect treatment comparison in which IPD in one or more trials are used to adjust for between-trial differences in the distribution of variables that influence outcome
Matching-adjusted indirect comparison	A form of propensity score weighting, applicable where IPD are available in one population and aggregate data in another
Simulated treatment comparison	A form of outcome regression, applicable where IPD are available in one population and aggregate data in another
Naive indirect comparison	Comparison of competing clinical interventions from data of individual arms of different studies, based on the assumption that the treatment groups are clinically homogeneous in composition

IPD: Individual patient data; ISPOR: International Society for Pharmaco-economics and Outcomes Research; ITC: Indirect treatment comparison.

Source: European Network for Health Technology Assessment. Comparators & Comparisons: Direct and indirect comparisons (2013) and Conducting Indirect-Treatment-Comparison and Network-Meta-Analysis Studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices – Part 2.

Method

A review of published materials was undertaken to analyze the requirements and recommendations of international best practice guidelines (International Society for Pharmaco-economics and Outcomes Research [ISPOR], European Network for Health Technology Assessment [EUNetHTA]) and HTA agencies in the following countries: UK, Sweden, Netherlands, Germany, France, Canada and Australia (Table 2). These countries were chosen because they have well-established HTA systems and to get an idea of policies across Europe and outside of Europe as well. For each of the HTA agencies and organizations in scope, we reviewed the key requirements for acceptance of inferences from indirect comparison and preferences for ITC methodology by looking at published HTA submission guidelines (specific links can be found in the results section). Focus was on the sections for clinical evaluation. A basic literature search was also conducted to look for similar reviews and provide context, but a full systematic literature review was not conducted to inform this study.

Table 2. Countries and product case studies in scope.

Country	HTA Body	Products for case studies + ITC method used in HTA submissions
UK	NICE/SMC
Sweden	TLV
Netherlands	ZIN	• Teriflunomide (Aubagio): Bayesian mixed treatment comparison and Bucher method
Germany	G-BA	• Simeprevir (Olysio): population-adjusted ITC
France	HAS	• Eltrombopag (Revolade): Bucher and naive indirect comparisons
Canada	CADTH/INESSS
Australia	PBAC

CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; HTA: Health technology assessment; INESSS: Institut National D'excellence en Santé et en Services Sociaux; ITC: Indirect treatment comparison; NICE: National Institute for Health and Care Excellence; PBAC: Pharmaceutical Benefits Advisory Committee; SMC: Scottish Medicines Consortium; TLV: Tandvårds-ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

In addition, three products were selected as case studies based on use of indirect comparison in HTA submissions (Table 2). More details on the selection criteria are provided in the case study section. The evaluations available were assessed to gain insight into how the HTA agencies assess information from indirect comparisons, and the role of indirect comparisons in the overall product assessment.

Results

European & international guidelines

ISPOR

The ISPOR has developed a specialized ‘ITC Good Research Practice Task Force’ with the goal to develop guidelines on ITCs that address key issues from the outcomes researcher's perspective as well as key issues from the healthcare decision-maker's perspective. To support this aim, ISPOR has published two key articles: ‘Interpreting Indirect-Treatment-Comparison Studies for Decision Making’ [1] and ‘Conducting Indirect-Treatment-Comparison and Network-Meta-Analysis Studies’ [2]. In addition, there is a supportive guideline called ‘Indirect Treatment Comparison/Network Meta-Analysis Study Questionnaire to Assess Relevance and Credibility to Inform Health Care Decision Making’ [3].

The first part of the report from the Task Force outlines the key concepts of ITC and mixed treatment comparison (MTC) and provides guidance for reviewing and interpreting these studies to inform decision making. It discusses some of the basic concepts of network meta-analysis and its main assumptions and limitations. Furthermore, it details how results of indirect comparisons should ideally be presented so they can be properly interpreted by decision makers. It also discusses the trade-off between relying on lower quality evidence and waiting for direct evidence to become available.

The second ISPOR Task Force report sets out general best practice guidelines for performing network meta-analyses. The underpinning statistical methods are presented and supported by examples that suggest methods for presenting results to nonspecialized audiences. Unfortunately, this article seems to focus mainly on larger evidence networks and does not discuss population-adjusted indirect comparison.

EUNetHTA

The EUNetHTA has also published a guideline which focuses on the methods available for ITCs [4]. It discusses the strengths and limitations, and provides recommendations in order to support indirect evidence development.

This guideline discusses a number of methods of MTC including Bucher's method of adjusted indirect comparison, Lumley's method of network meta-analysis, and Bayesian mixed treatment comparison. The guideline states that there are many issues that must be taken into account when conducting ITCs, which are explained in more detail. EUNetHTA concludes that although Bucher's method of adjusted indirect comparison is the most computationally straightforward of the MTC methods, Bayesian MTC can be used to analyze very complex networks and can incorporate meta-regression to include study-level covariates. The choice of methodology is ultimately context specific and should be appropriate to the data available.

Country-specific guidelines

When looking at HTA submission guidelines and examples of evaluations in which ITC was used, it can be concluded that most agencies prefer adjusted methods for ITC, and some specifically mention that unadjusted indirect comparisons are not acceptable. However, there is variability in the preferred method of ITC between agencies – some agencies (e.g., Gemeinsamer Bundesausschuss [G-BA] and Zorginstituut Nederland [ZIN]) prefer the comparison to be as simple and transparent as possible while others (e.g., NICE and the Pharmaceutical Benefits Advisory Committee [PBAC]) value the reduced uncertainty by adjusting for multiple factors in more complex analyses. An overview of preferred methodology of different HTA agencies is provided in Table 3 and more details are provided in the following country-specific sections.

Table 3. Health technology assessment agency preferred indirect treatment comparison methodology.

Method	NICE	SMC	HAS	G-BA	TLV	ZIN	PBAC
Unadjusted ITC				X	X	✓
Bucher method	✓	✓	✓	✓	✓	✓	✓
Bayesian MTC	✓					✓
MAIC & STC	✓						✓

✓ = recommended; X = not recommended.

CADTH and INESSS did not mention a preference for a particular ITC methodology and are not included in the table; the table is based on both guideline analysis and case studies.

CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; INESSS: Institut National D'excellence en Santé et en Services Sociaux; ITC: Indirect treatment comparison; MAIC: Matching-adjusted indirect comparison; MTC: Mixed treatment comparison; NICE: National Institute for Health and Care Excellence; PBAC: Pharmaceutical Benefits Advisory Committee; SMC: Scottish Medicines Consortium; STC: Simulated treatment comparison; TLV: Tandvårds-ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

HTA agencies also have different recommendations regarding presentation of results and the level of detail required around description of methodology and sensitivity analyses, as summarized in Table 4.

Table 4. Requested data and presentation of results according to health technology assessment guidelines.

Element	NICE	SMC	HAS	G-BA	TLV	ZIN	CADTH	PBAC
Description of methodology	✓			✓	✓	✓	✓
Explanation for underlying assumptions				✓	✓
Justification for choice of anchor comparator				✓
Rationale for study selection	✓	✓				✓		✓
Baseline characteristics and results for included trials	✓		✓			✓
IPD datasets (if applicable)	(✓)		(✓)					✓
Hazard ratios/relative risk + confidence intervals for results		✓				✓		✓
Tables/graphs/diagrams with results	✓	✓	✓	✓		✓
Statistical assessment of heterogeneity	✓	✓		✓
Sensitivity analysis (e.g. exclusion of trials)	✓			✓				✓
Extrapolation to national population		✓
Programming code/software specification				✓				✓
Guideline last updated	2017	2017	2017	2016	2017	2016	2014	2016

✓ = Specifically recommended for inclusion; (✓) = potentially requested.

INESSS is not included in this table as their guidance concerning ITC is very brief and does not include any of the topics mentioned; blank spaces means the criteria is not referred to in guidance.

CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; INESSS: Institut National D'excellence en Santé et en Services Sociaux; IPD: Individual patient data; ITC: Indirect treatment comparison; PBAC: Pharmaceutical Benefits Advisory Committee; SMC: Scottish Medicines Consortium; TLV: Tandvårds- ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

UK

For the UK, HTA submission guidelines for both the NICE and the Scottish Medicines Consortium (SMC) were evaluated.

NICE provides extensive guidelines for manufacturers submitting evidence for their single technology appraisals [5]. The institute discusses trial selection and how to document and present data, methodology and results. Regarding the choice of ITC methodology, NICE is one of the only HTA agencies that states they will accept unadjusted naive comparisons or a simple narrative overview if the submitting company considers that an ITC or mixed treatment comparison is inappropriate. However, NICE does note that if this type of comparison is presented, the data will be treated as observational in nature and associated with increased uncertainty.

The NICE guidelines do not specify in detail a preference for ITC methodology; the Institute is not opposed to statistically advanced methods and prefers a method which is able to adjust for many factors. This is also reflected in the complementary guidelines published by the NICE Decision Support Unit (DSU), which provide detailed technical advice on how to perform and present pairwise and network meta-analysis or matching-adjusted indirect comparison (MAIC) and simulated treatment comparison (STC) [6]. When submitting a population-adjusted indirect comparison, NICE can request to receive full IPD sets so they could perform their own analysis.

The SMC does not provide an opinion on its preferred method for ITC; in the guidelines, the only details provided are on how to describe the search strategy, methodology, results and measures of heterogeneity or sensitivity analyses [7]. The drug committee also provides a detailed checklist which is used to evaluate the quality of indirect evidence. The SMC prefers the most simple and transparent method, with the option to present a more advanced method as a sensitivity analysis.

Indirect comparisons are a well-accepted source of evidence for the SMC; it recognizes that head-to-head trials are not always available, particularly if the goal is to demonstrate noninferiority. Only when considering claims of therapeutic superiority, or requests for a much higher price than the comparator, will the SMC be more likely to criticize the methodological approach and be more restrictive on the level of uncertainty allowed.

The SMC considers ITC evidence for both efficacy and safety end points, so special consideration should be given not only to end points where the biggest difference between treatments is expected, but also to which outcomes have the most impact on physician prescribing and patient preference.

France

In France, the HTA submission guidelines are not extensive regarding ITC [8]; the only guidance offered is that meta-analyses of good methodological quality can be presented as relevant clinical data.

Interestingly, the Transparency Committee (TC, Commission de la Transparence) is flexible regarding recently marketed competitors: products that have been marketed within a 3-year period do not have to be compared with each other. This means both products can have a ‘shared’ Amélioration du Service Médical Rendu (ASMR) rating (rating of improved medical benefit) and are both compared with the same previous standard of care.

Generally, the TC considers placebo-controlled studies as sufficient evidence to demonstrate clinical benefit (Service Médical Rendu [SMR] rating). It is also possible to request an additional clinical benefit over a comparator (ASMR rating) based on indirect evidence. If the product has similar clinical benefit to the comparator, it will be sufficient to demonstrate noninferiority using a simple pairwise comparison. However, if the manufacturer wishes to demonstrate improved clinical benefit through an indirect comparison, it is beneficial to strengthen the evidence by using a more sophisticated method such as a population-adjusted ITC. The TC has the statistical expertise to understand more complex methodology, and would consider the use of IPD as generating more robust results. However, if using IPD, the full dataset should also be submitted so that the TC is able to validate the results of the ITC.

Germany

The German Federal Joint Committee (G-BA) has a very clear position on the appropriate comparative therapy and details about this can be found in the appropriate submission guidelines [9]. It is important to note that this is sometimes a different comparator than the one chosen by other HTA agencies in Europe, a situation which has amplified calls for a more holistic approach to HTA at a European level. Failure to submit sufficient evidence against the appropriate comparator usually leads to a level 5 benefit rating (no proof of benefit), which means that the product will be reference priced against the comparator without price negotiations.

The G-BA guidelines are not clear on how to value indirect evidence against the appropriate comparator when head-to-head trials are not available. However, the guidelines do state a preference for direct evidence and that the use of ITC should be restricted to situations in which it is not possible to perform head-to-head trials. This statement is supported by guidelines from the Institute for Quality and Efficiency in Healthcare (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWIG) [10], which also indicate a belief that real conclusions can only be drawn from direct evidence. Although the G-BA will consider supporting indirect evidence, it will rarely make positive recommendations based on ITC data alone [11]. ITC outcomes would have to be statistically significant for patient relevant outcomes and the manufacturer would have to provide a strong argument for not performing head-to-head trials, other than purely the evolution of care pathways.

In addition, the G-BA provides advice on how to describe ITC methodology and results, but is less explicit about what type of ITC data are preferred. Like most HTA agencies, the G-BA agrees that unadjusted indirect comparison is not a valid method. Only in special cases of high unmet has the G-BA sometimes considered naive ITC data. The guidelines from the G-BA mention that ‘the use of simple indirect comparisons is possible’ and pairwise Bucher comparisons are preferred over mixed treatment comparison, as it is felt that including more studies only increases uncertainty. The G-BA specifically points out that the choice of an appropriate anchor comparator is very important when considering ITC. Performing an indirect comparison will only be considered appropriate if the bridge comparator is proven to be sufficiently similar between studies.

Interestingly, the G-BA is one of the few agencies that will consider ITC data for safety as well as efficacy. As safety data are not often included in studies as specific end points, it is important that manufacturers design the indirect comparison to reflect safety differences as accurately as possible.

It is difficult to predict how the G-BA would respond to a population-adjusted ITC, as this type of evidence has not yet been evaluated by the G-BA to date. It is recommended to only present more complex ‘sensitivity analysis’ if this would add any additional value. The G-BA would not reject a product purely for statistical reasons, but will criticize any lack of clarity in the methodology or unnecessary complexity or uncertainty. In addition, the G-BA is very conservative toward using unpublished data and selecting specific patients for analysis (e.g., using IPD), which might be another reason not to submit a population-adjusted ITC as the main comparison.

Sweden

The guidance provided by the Swedish Dental and Pharmaceutical Benefits Agency (Tandvårds-ochre läkemedelsförmånsverket, TLV) on indirect comparison is limited [12]. The TLV states that direct evidence should be used primarily, but when direct studies have limitations, or are not available, indirect evidence can be accepted. If possible, this should be an adjusted indirect comparison; for example, according to the Bucher method. The only other comment from the TLV is that ITC should be based on a systematic literature review and assumptions and methodology should be clearly described.

Although the TLV only very briefly mentions ITC in their guidelines, the agency is open to this form of evidence. In addition, the TLV has the statistical expertise to understand more complicated population-adjusted methodologies. It is recommended to present both a Bucher and an additional comparison if both are available, as this would strengthen the evidence. It would not be required to submit the complete IPD dataset as the TLV rarely conducts its own analyses. Statistical significance of the results is not considered important for assessing noninferiority, however, large confidence intervals might become an issue in calculations to establish the cost–effectiveness.

The Netherlands

The Dutch Care Institute (ZIN) is also very brief regarding advice on the use of indirect evidence in submissions [13]. ZIN provide some example tables of how indirect evidence should be presented but do not state how to value this type of evidence.

An additional guideline, called the ‘Evaluation of Standard of Science and Practice’ (Bordering Stand van de Wetenschap en Praktijk) [14], is taken into account by ZIN to establish what is considered current standard of care according to science and clinical practice and what the recommended scientific evaluation methods are. This provides some more information on indirect comparisons. It states that ITC can be considered as long as the risk of bias and reliability of results have been properly assessed. Three main methods are discussed: naive or narrative indirect comparison, simple pairwise comparisons or network meta-analysis. Interestingly, ZIN will accept a simple naive or narrative indirect comparison and does not feel a need for adjusted ITC as long as studies are considered sufficiently comparable. ZIN will look at both efficacy and safety data from a general comparative perspective, not requiring statistical outcomes.

More complicated population-adjusted statistical methods are not mentioned in either guideline. ZIN has the capability to understand more complex statistical methods; however, a simple and transparent approach is preferred, and it is recommended to submit more complex evidence as supplemental information only. If using individual patient data leads to significantly different outcomes in favor of the evaluated treatment, it is possible to present this data as the main ITC, however, this should be clearly described and justified. Individual patient datasets would not have to be submitted as ZIN does not perform their own analyses. However, to make sure an ITC is valid, ZIN would strongly prefer the data to be published and peer-reviewed.

Notably, the Evaluation of Standard of Science and Practice guidelines specifically refer to EUNetHTA guidelines as providing recommendations for performing indirect comparison. ZIN is therefore the only HTA agency that specifically recommends following EU level guidance.

Canada

The Canadian Agency for Drugs and Technologies in Health (CADTH) believes that in the absence of head-to-head clinical trials comparing the intervention and relevant comparators, indirect comparison can be used as long as it is based on appropriate techniques [15]. The CADTH does not specify the most appropriate methodology, but stresses that the methods used to synthesize indirect comparisons should be clearly explained and appropriately justified. Any limitations of the methods, potential biases in the parameter estimates, and caveats about the interpretation of results, should be reported in a transparent manner. Sensitivity analyses may also be used to assess the impact of assumptions about comparators.

As CADTH recommendations do not apply to Quebec, we also looked at the Quebec National Institute of Excellence in Health and Social Services (Institut national d'excellence en santé et en services sociaux, INESSS). However, INESS guidelines do not mention anything about indirect comparison [16]. The INESSS guideline purely states that at least one study must be submitted which is a randomized, controlled clinical trial, published or accepted for publication and a maximum of five studies can be included in the evaluation. All other types of evidence are considered supportive only.

Australia

The Australian Pharmaceutical Benefits Advisory Committee (PBAC) does provide extensive submission guidelines [17], including several paragraphs about indirect comparison. Details are provided regarding search strategy, selection of trials, methodology, assessing heterogeneity and presenting results.

The PBAC states a strong preference for direct evidence; however, it also shows an understanding toward situations where this is not available. When presenting ITC data, the PBAC prefers single pairwise comparisons (e.g., the Bucher method) and states that more complex methods, such as network meta-analyses, may be presented as supplementary analyses. Unadjusted indirect comparisons (such as a naive comparison between single arms) or indirect comparisons where differences in trial characteristics may affect the transitivity of the trials are considered difficult to interpret and reduce the confidence of the PBAC in decision making.

The PBAC specifically mentions population-adjusted indirect comparison as a possible method. Where patient-level data are available for at least one study in the comparison, MAIC or STC should be used to correct for trial differences. However, the PBAC does encourage caution when considering these complex types of approach. It is recommended to carefully consider the balance between the additional information requests and challenges these approaches may present with any reduction in uncertainty they may deliver. For methods that require individual patient data, the PBAC asks to attach the IPD in a spreadsheet or to justify when this is not possible.

Case studies

To see how these guidelines are implemented in practice, we looked at the HTA reports for three products that used indirect evidence in their submission and had limited direct comparative evidence available. The products selected were for smaller or orphan indications with limited comparable treatment options and have been recently evaluated by the majority of the HTA agencies in scope. In addition, we wanted to show different application of the main ITC methods, which lead to selection of the following products:

Teriflunomide (Aubagio): indicated for relapsing remitting multiple sclerosis, for which a combination of Bayesian mixed treatment comparison and Bucher method ITC was used to demonstrate comparative effectiveness.

Simeprevir (Olysio): indicated for chronic hepatitis C, focusing only on genotype 4 in which there was no direct evidence and population-adjusted ITC was used as a source of indirect evidence.

Eltrombopag (Revolade): indicated for immune thrombocytopenic purpura, for which either Bucher or naive indirect comparisons were used in submissions.

For each product, we looked at the published HTA evaluations and summarized the indirect evidence that was submitted, the main comments of the HTA body regarding the indirect evidence and the final outcome or reimbursement recommendation, as shown in Tables 5, 6 & 7.

Table 5. Health technology assessment recommendations for teriflunomide (Aubagio).

Agency	Recommendation	Indirect comparisons	Comparators	Claim	Agency comments	Ref.
NICE	(✓) Jan 14 (not recommended for highly active/severe MS)	– Bayesian MTC – Separate ITCs for highly active/rapidly evolving subgroups	β-IFNs, glatiramer (Copaxone), natalizumab (Tysabri) and fingolimod (Gilenya) (base case: studies from 2000 onward)	Lower annualized relapse rates	Pre-2000 trials were excluded, results for disability progression inconsistent with TENERE study, insufficient patient numbers for highly active subgroups, heterogeneity MTC did not provide relevant information and the ITCs were not reliable	[18]
SMC	(✓) Mar 14 (not recommended for highly active MS)	MTC	β-IFNs and glatiramer (Copaxone) (base case: studies from 2000 onward)	Noninferiority for most outcomes (superior only for discontinuation rate)	Heterogeneity (patient populations, baseline characteristics), outcome measures not well defined MTC was accepted to support assumption of comparable efficacy	[19]
HAS	✓ Mar 14 ASMR V	None			Absence of additional comparative data	[20]
G-BA	X Mar 14 No add. Benefit	Bucher comparison	IFN β-1a (Rebif) 44 μg (based on TEMSO, TOWER and PRISMS studies (IFN-β vs placebo)	Not specified	IFN-β-1a (Rebif) 30 μg not included, heterogeneity (study populations) The ITC is not considered acceptable as supportive evidence	[21]
TLV	X May 14	– MTC – Indirect comparison	– Disease-modifying drugs (not specified) – IFN β-1b (Betaseron)	Superiority	Very large confidence intervals but ITC was accepted as evidence for similar effect, justifying cost-minimization analysis	[22]
	(✓) Jun 14 (IFN-β failure)	No new evidence submitted - re-assessment based on re-evaluation MS drugs by TLV (positioning based on side-effect profile and oral formulation)				[23]
	✓ Jun 16	– Updated MTC – Bucher (TLV requested)	– Dimethyl fumarate (Tecfidera)	Noninferiority for most outcomes, superior safety	Teriflunomide (Aubagio) efficacy was considered noninferior, safety benefits versus dimethyl fumarate (Tecfidera) not demonstrated	[24]
ZIN	✓ Mar 14	MTC	Interferons, natalizumab (Tysabri) and glatiramer (Copaxone)	Noninferiority	Heterogeneity (study population, definition disease progression), differences effect when excluding pre-2000 studies, claim accepted	[25]
CADTH	X Jun 14	Bayesian MTC + pairwise meta-analysis	Disease-modifying drugs	Noninferiority for most outcomes and comparisons and superiority for a few selected outcomes	Pre-2000 trials were excluded, heterogeneity (study population), inclusion criteria systematic review unclear, results should be interpreted with caution	[26]
INESSS	✓ Jun 14	– Manufacturer MTC – MTC by CADTH	– No details provided – Alemtuzumab (Lemtrada), fingolimod (Gilenya), dimethyl fumarate (Tecfidera), glatiramer (Copaxone), interferon ß (Avonex), natalizumab (Tysabri)	Not specified	Prefer independent sources, methodology of manufacturer MTC unclear, results were not adjusted for missing data and subsequent treatment, accepted that MTC shows similar efficacy	[27]
PBAC	X Nov 12	Indirect comparison	IFN β-1a/1b (Rebif, Betaseron)	Noninferior effectiveness but better tolerability	Results were not significant and insufficient to demonstrate noninferiority	[28]
	✓ Jul 13 (sub-population)	Indirect comparison	IFN β-1a/1b (Rebif, Betaseron)	Noninferiority	Heterogeneity (study populations) and little statistical power but noninferiority could be assumed	[29]

✓ = recommended (G-BA added benefit 1–4); (✓) = restricted; X = not recommended (G-BA no added benefit 5–6).

CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; INESSS: Institut National D'excellence en Santé et en Services Sociaux; ITC: Indirect treatment comparison; MTC: Mixed treatment comparison; NICE: National Institute for Health and Care Excellence; PBAC: Pharmaceutical Benefits Advisory Committee; SMC: Scottish Medicines Consortium; STC: Simulated treatment comparison; TLV: Tandvårds-ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

Table 6. Health technology assessment recommendations for simeprevir (Olysio) – Hepatitis C genotype 4.

Agency	Recommendation	Indirect evidence			Agency comments	Ref.
NICE	✓ Feb 15	MAIC	IPD from RESTORE trial versus aggregate data from Rumi et al. (PR vs placebo)	Better virological response rates	ITC method and quality of included studies not well described, no statistical uncertainty analysis provided, choice for Rumi et al. not justified, very small effective sample size, results not robust	[30]
SMC	✓ Oct 14	MAIC	Simeprevir (Olysio) + PR versus PR alone using IPD from RESTORE trial	Not specified	Very small patient number, potential bias associated with the single study selected for the comparator PR regimen, only treatment naive patients included, limitations have been partly addressed by sensitivity analyses	[31]
HAS	✓ Dec 14 ASMR IV	None			Data are limited but the efficacy profile in genotype 4 seems comparable to genotype 1	[32]
G-BA	✓ Nov 14 small add. benefit	None			In the absence of an ITC, only a small additional benefit could be established based on the avoidance of side effects	[33]
TLV	(✓) – Conditional Oct 14 (fibrosis stage F3/F4)	None			The absence of comparative evidence leads to great uncertainty about the efficacy of the drug	[34]
	(✓) Jun 15	Genotype 4 not discussed	[35]
ZIN	✓ Sep 14	Supplemental analysis (no details provided)			ZIN would have liked to see an ITC versus sofosbuvir but noted this would be difficult due to methodological shortcomings of the sofosbuvir studies	[36]
CADTH	(✓) Aug 13	No marketing authorization for genotype 4	[37]
INESSS	✓ Jun 14	No marketing authorization for genotype 4	[38]
PBAC	✓ Jul 14	No reimbursement requested for genotype 4	[39]

✓ = recommended (G-BA added benefit 1–4); (✓) = restricted; X = not recommended (G-BA no added benefit 5–6).

CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; INESSS: Institut national d'excellence en santé et en services sociaux; ITC: Indirect treatment comparison; MAIC: Matching-adjusted indirect comparison; NICE: National Institute for Health and Care Excellence; PBAC: Pharmaceutical Benefits Advisory Committee; PR: Peginterferon + ribavirin; SMC: Scottish Medicines Consortium; STC: Simulated treatment comparison; TLV: Tandvårds-ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

Table 7. Health technology assessment recommendations for eltrombopag (Revolade).

Agency	Recommendation	Indirect evidence			Agency comments	Ref.
NICE	✓ Jul 13 (Oct 10 replaced	– Bucher method – ITC (undefined)	– Romiplostim (Nplate) – Standard of care	Similar efficacy	Bayesian approach might be better to deal with heterogeneity of romiplostim trials, appropriate to do ITC versus romiplostim (Nplate) despite differences in patient population between trials; ITC versus standard of care lacked methodological rigor; ITCs were not sufficiently robust	[40]
SMC	(✓) Aug 10 (severe ITP, high bleeding risk)	Bucher method	Romiplostim (Nplate)	Superiority in terms of overall response	Potentially relevant trials were excluded from ITC due to choice of Bucher method, MTC could have controlled for splenectomy status and study design, no conclusion on acceptance of ITC	[41]
HAS	✓ Jun 10 ASMR II	None (not compared with recent comparator romiplostim, ‘shared’ ASMR versus standard of care)	[42]
TLV	(✓) May 11 (conditional evidence development)	ITC (no details provided)			Both preparations are likely to have a similar effect	[43]
ZIN	✓ Aug 10	Naive ITC	Romiplostim (Nplate)	Higher response rates	No conclusions can be drawn from the ITC due to different definitions of outcome measures	[44]
CADTH	X Oct 11	Naive ITC (no details provided)			Based on data from noncontrolled nonrandomized studies, surrogate outcome of platelet count rather than complications of ITP, no conclusion on acceptability of ITC	[45]
INESSS	X Oct 11	ITC (no details provided)			Heterogeneity in the populations, low methodological quality of included studies, efficacy considered proven versus placebo (difference in recommendation is due to recalculation of costs)	[46]
	✓ Oct 12	ITC (no details provided)	[47]
PBAC	X Nov 10	ITC	Romiplostim (Nplate)	Similar efficacy	Surrogate outcome of platelet count instead of bleeding risk, appropriateness of ITC uncertain due to large differences between trials, width of confidence intervals indicated that ITC was not adequately powered for statistical testing of noninferiority	[48]
	(✓) Jul 11 (max. 24 weeks, same as romiplostim)	ITC	Romiplostim (Nplate)	Similar efficacy	Wide confidence intervals, considered noninferior to romiplostim (Nplate) only in post-splenectomy population Resubmission included new cost data and a lower price	[49]

✓ = recommended (G-BA added benefit 1–4); (✓) = restricted; X = not recommended (G-BA no added benefit 5–6).

ASMR: Amélioration du Service Médical Rendu; CADTH: Canadian Agency for Drugs and Technologies in Health; G-BA: Gemeinsamer Bundesausschuss; HAS: Haute Autorité de Santé; INESSS: Institut national d'excellence en santé et en services sociaux; ITC: Indirect treatment comparison; MTC: Mixed treatment comparison; NICE: National Institute for Health and Care Excellence; PBAC: Pharmaceutical Benefits Advisory Committee; SMC: Scottish Medicines Consortium; TLV: Tandvårds- ochre läkemedelsförmånsverket; ZIN: Zorginstituut Nederland.

Summarizing these case studies, manufacturers are already aware of the differences between HTA agencies in their level of understanding and acceptance of indirect evidence and tailored their approach to individual agency requirements. For example, more complex population-adjusted ITCs were only submitted to NICE, and naive ITCs only to CADTH and ZIN. Manufacturers also acknowledged the G-BA preference for the Bucher method. As agencies rarely commented on the choice of methodology, this seems to be an effective strategy.

Regarding the implementation and results of ITCs, HTA agencies often comment on the quality of the ITCs that were submitted and subsequently question the reliability of results. In many cases, HTA agencies are willing to accept methodological challenges, as it is acknowledged that the evidence base placed before them has severe limitations. The issue for HTA bodies is therefore not how ITCs deal with evidence limitations, but simply the lack of clarity and description around the chosen ITC method, underlying argumentation and supporting sensitivity analyses.

It remains challenging to demonstrate the superiority of one treatment over another using an ITC, even if it is only for one or a few selected outcome measures. In addition, HTA agencies are more likely to closely scrutinize submitted data and evaluate statistical significance of results when superiority is claimed. Evidence limitations or uncertainty around results often leads HTA bodies to be cautious about the conclusions that can be drawn from ITC evidence. Even when superiority is claimed, ITC evidence is rarely accepted to support anything more than similarity of treatments.

Conclusion

The first step when contemplating submission of an ITC to an HTA agency is always to consider what is the most important comparator in each market. An ITC can be perfect technically, but if it does not include the comparator the agency considers most relevant, it will be of little use for decision making. Furthermore, the ITC methodology should be tailored to the specific requirements and preferences of the agency. Most importantly, transparency about the limitations of the evidence base with sufficient description and justification of the chosen methodology is essential, with sensitivity analyses added where necessary.

Many agencies prefer the simple and transparent Bucher indirect comparison over more advanced methodology such as a population-adjusted ITC. It is not recommended to present a complex ITC unless it would result in significantly more favorable outcomes. In the Netherlands, it is even considered sufficient to only provide a naive ITC, although other countries prefer some level of adjustment and statistical significance for results.

Germany is the only country in our analysis which has a very strong preference for direct evidence and is unlikely to provide a positive recommendation for a treatment based purely on indirect evidence. For the G-BA, the indirect evidence case and justifications for not having head-to-head evidence need to be very strong and it should be explicitly considered whether it is appropriate to submit a dossier for a product primarily based on ITC.

In most cases, it is challenging to demonstrate an additional clinical benefit using an indirect comparison due to small efficacy or safety differences between trials and wider confidence intervals and uncertainty due to trial differences and statistical methodology. HTA agencies are likely to adopt a more challenging stance regarding outcomes, methodology and assumptions, if a higher price is requested and/or clinical superiority is claimed based upon an ITC. However, it is very commonly accepted to use ITC to demonstrate noninferiority and HTA agencies routinely accept this claim.

There are also differences between HTA agencies in willingness to consider the use of indirect comparisons for safety outcomes. Agencies such as HAS and TLV are generally less focused on safety and will focus on efficacy in clinical benefit assessments. However, for agencies such as NICE, SMC and G-BA, it is recommended to base the choice of end points to use in the ITC on those end points for which the biggest difference is expected, or which have most impact on clinician and patient preference for a treatment, which might be safety end points as well.

ITC is a useful method to generate comparative evidence in the absence of head-to-head trials. The many different ITC methodologies available each offer different possibilities but also have their limitations, which should be taken into account when deciding to perform an ITC. However, ITC is not just a ‘solution’ when direct evidence is unavailable but an option that should already be considered during earlier stages of drug development when it is decided whether to perform a direct comparative trial or to optimize trial design for indirect comparison.

Future perspective

In the fast-evolving pharmaceutical landscape, it will become increasingly difficult to always perform direct comparative studies against the most appropriate comparator considered by HTA agencies before launch, especially considering individual country differences in availability of competitor products and methodology for selecting comparative treatments. Although within Europe there is a move toward more unified processes, and potential joint HTA assessments, this does not eliminate the need for ITC if no direct evidence is available.

Increased use of ITC in HTA submissions should lead to more experience in interpreting this type of data within HTA agencies. Hopefully, this will also result in more extensive guidance regarding preferred ITC methodology and the type of data that should be included in submissions. An example of this can already be found in the increasing use of evidence evaluation criteria such as GRADE, which also includes specific guidelines for evaluating indirect treatment comparisons [50].

In addition, developments in statistics may result in more advanced ITC methods which could overcome various data limitations or lead to more precise outcomes. However, as many HTA agencies currently prefer the most simple and transparent methodologies, it might take some time before newer methods become accepted as part of routine evaluations.

Executive summary

As part of reimbursement planning, it is important to consider whether the head-to-head trials available address the needs of each health technology assessment (HTA) agency in terms of evidence against the most relevant comparator for the desired therapeutic positioning, or whether an indirect treatment comparison (ITC) needs to be conducted.

Most of the HTA agencies analyzed within the scope of this article are very open to the use of indirect comparison, only the G-BA in Germany feels very strongly about the need for direct evidence and would require compelling arguments to justify not providing head-to-head evidence.

The choice of ITC methodology should be tailored to individual HTA agency requirements and preferences; most agencies prefer to see the most simple and transparent evidence possible if the differences between trials are small and do not require additional adjustment and not all agencies routinely integrate complex methods in health technology assessments.

The limitations and possibilities of the available studies must be taken into account, and the availability of IPD opens up the possibility for more sophisticated matching methodologies. However, the additional value and possible reduction in uncertainty must be weighed up against the increased complexity.

It is acceptable to demonstrate noninferiority using ITC, but HTA agencies are more likely to closely scrutinize submitted data and evaluate statistical significance and clinical relevance of results when superiority on efficacy or safety end points is claimed.

Whatever ITC method is chosen, the most important thing for HTA submissions is that the methodology, underlying assumptions and any potential limitations are described in a clear and transparent manner.

Acknowledgements

The authors thank C Brown and N Johnson, colleagues at RJW & Partners for their helpful suggestions.

Financial & competing interests disclosure

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No writing assistance was utilized in the production of this manuscript.

References

Papers of special note have been highlighted as: • of interest

Jansen JP, Fleurence R, Devine B et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1. Value Health 14(4), 417–428 (2011).