Challenges for decision-makers when assessing within-class comparative effectiveness: the case of anticoagulation therapy for atrial fibrillation

In considering how the NOACs compare to warfarin and to each other, we focus on the seven main clinical outcomes of interest which have been subjected to a NMA as part of NICE’s review and which are subsequently used in the CE model.

Looking specifically at Supplementary Tables 18–25 of the draft guidance, we find the odds ratios of each treatment (apixaban 5 mg bd, dabigatran 150 mg bd, edoxaban 60 mg od and rivaroxaban 20 mg od) for each outcome [1]. The point estimates of these odds ratios suggest that apixaban does not outperform the other NOACs across all clinical outcomes. The same conclusion can be made from interpretation of Figure 1A-G, which displays forest plots of the hazard ratio and associated 95% CI for seven main events based on our reproduction of the results of the NMA (Supplementary Table 52, page 172) (Box 1) [8].

Heterogeneity across treatments for the seven events of interest in the competing risk NMA is reported at the bottom of Figures 1A–G (Q-statistic p-values and I² values). The I² values indicate moderate heterogeneity across treatments for myocardial infarction (I² = 52%) and intracranial hemorrhage (I² = 54%). There was substantial or considerable heterogeneity across treatments for systemic embolism (I² = 65%) and other clinically relevant bleeding (I² = 84%).

There are distinct differences between the four pivotal Phase III randomized controlled trials of NOACs for stroke prevention in patients with nonvalvular atrial fibrillation that formed the basis of the NMA. These differences have the potential to make direct comparison of the results misleading even within the context of a carefully conducted NMA [12]. Methodological variations, differences in outcome definitions and other variations between studies contribute to the heterogeneity between studies.

The heterogeneity present between these studies further contributes to the uncertainty in the between treatment comparisons of the results of any NMA. We understand that the use of a NMA is necessary but it is important to highlight that any uncertainty intervals calculated as a result of a NMA assume that the data from the trials included are exchangeable. As described by Camm and colleagues [12], this is unlikely to be the case. Heterogeneity across comparator arm populations in each trial should be considered when evaluating and comparing trial outcomes for within-class treatments. In the case of NOACs, patients taking warfarin across trials may differ in characteristics including age, comorbidities or geographic location – all factors that may contribute to increasing or decreasing costs as well as factors that affect the rates of clinical outcomes to be used in comparative and CEA.

Furthermore, in subgroup analyses of the four pivotal trials some evidence indicative of effect modification was found. For example, effect modification due to age was found in RE-LY [13] and of renal function was found in ARISTOTLE [14]. This means that the transitivity assumption is unlikely to have been met. An additional potential issue is that of ecological bias according to which, the differences seen in subgroup analyses at the individual trial level are not evident at the summary level. Without individual patient data from the individual trials, it will be difficult to detect this effect and adjust for it in NMAs using meta-regression techniques [15]. In conventional (pairwise) meta-analysis, the Cochrane methods guide recommends a minimum of ten studies is necessary to perform a meta-regression [16]. López-López et al. conducted a meta-regression to evaluate potential effect modifiers [7]. While López-López et al. analyzed 23 studies [7], the number of studies providing head-to-head contrasts within the NMA is likely to be too low to form the basis of a robust meta-regression: only 12 trials explored the efficacy of a NOAC against warfarin. The remaining trials explored warfarin compared with aspirin (n = 10) and apixaban compared with aspirin (n = 1). Across NOACs, no treatment had greater than a total of four studies reporting efficacy outcomes compared with warfarin (edoxaban [n = 4], apixaban [n = 3], dabigatran [n = 3] and rivaroxaban [n = 2]). This may be particularly misleading because, for example, of the three studies comparing apixaban to warfarin, only one study compares the low dose of apixaban to warfarin. Similarly, of the two rivaroxaban studies, one study includes only outcomes for the high dose and the other study includes only outcomes for the low dose. It is likely; therefore, that the meta-regression conducted was not powered to adjust adequately for the differences between the treatment comparisons in terms of potential effect modifiers.

The validity of any technique is conditional on the assumptions. In the absence of effect modification, the fixed effect model may still obtain unbiased point estimates, but given the heterogeneity in the trial design the uncertainties are likely to be underestimated and create a false sense of certainty. In the case of effect modification and given differences in trial design, the random effect analyses as suggested by Ren et al. to obtain informative priors of the between trial variance using expert elicitation would be more appropriate [17].

Another limitation resulting from the small number of studies for each comparison was that a NMA using a fixed effect logistic regression approach was conducted [7]. López-López et al. indicated that a study limitation of the NMA was that random effects models could not be fitted due to the limited number of comparisons that had been replicated in two or more trials [7].

In the face of these limitations, we consider that the level of uncertainty reported for the NMA that is the basis of the NICE report should be considered the most optimistic. In reality, the limitations described above would suggest greater overall uncertainty than is represented by the presented CIs.

One of the purposes of a CE model is to synthesize disparate clinical outcomes into a common outcome measure. The standard outcome in CEA is the QALY. The seven key clinical outcomes reported in Figure 1A–G are combined with estimates of life-expectancy, event impacts on quality of life as well as background quality of life to generate an estimated QALY for each treatment in the NICE CEA. Based on the expected QALY estimates and 95% CIs reported in Supplementary Table 65 (page 192) [8], mean QALY differences compared with warfarin were calculated across NOACs. From the mean QALY differences across treatments compared with warfarin in the forest plot presented in Figure 2, one can see that the mean differences are nearly the same for all four NOACs. Only apixaban shows a significant QALY benefit to warfarin; however, the differences between the NOACs are slight and the magnitude of QALY benefits are similar. The conventional tests for heterogeneity do not support differentiation between NOACs on the QALY outcome.

Supplementary Figure 24 from the evidence review (page 193) combines the simulated results of the QALY differences of the NOACs with the estimated additional costs on the incremental CE plane [8]. Each set of simulated points on the CE plane represents the joint density of incremental costs and effects of each of the four NOACs and no treatment relative to warfarin. That it is difficult to distinguish the ‘clouds’ of points from each other, ignoring the values for ‘no treatment’, emphasizes that the NOACs have similar CE.

This result is further emphasized in Figure 3 which shows the net-monetary-benefit forest plot, created using values from the results of the base case analyses in Supplementary Table 65 of the evidence review (page 192) [8]. Net-monetary-benefit is a measure that combines the net cost and net-QALY outcomes together using the decision threshold as an exchange rate.

Where ΔQ_k and ΔQ_k are the additional QALYs and costs for NOAC k compared with warfarin and λ is the cost per QALY used for decision making. For the results presented in Figure 3, a cost per QALY of £30,000 was employed. Note that in constructing this plot, we identified some differences between the values of the CIs in Figure 3 and Supplementary Table 65 of the review likely due to our lack of information on the covariance between costs and effects in the data. Nevertheless, we believe the differences are not large enough to affect our general conclusions on how the evidence should be interpreted.

Supplementary Figure 25 of the evidence review (page 194) displays a cost-effectiveness acceptability curve (CEAC) plotting the probability that each NOAC is cost-effective against the willingness to pay for all four NOAC’s, warfarin and no treatment [8]. While it is common to use a CEAC for health economic assessment, CEACs may give a misleading view of the strength of evidence for each treatment because the CEAC uses ordinal measuring, and because 100% of the simulations are split between the total number of treatment options which means that viable options may nevertheless have an apparently ‘low’ number of times that they are considered optimal. Consider for example that four completely identical treatments in all respects were compared and were all considered cost-effective compared with no treatment. In this case, we would expect the CEACs to show all options as having 25% probability of being cost-effective. Even small apparent differences between treatments would have a profound impact on the rank-order probabilities, which we contend is the issue with the presentation in Supplementary Figure 25 of the evidence review [8].

In the recent NICE consultation process on updating their methods (and in the supporting Task and Finish Report that accompanied the consultation document), the potential confusions arising from relying on CEACs in the presence of many competing options has been highlighted as an area of concern for concealing uncertainty in the ranking process [18,19]. The proposed solution to the problem is to focus on the net-monetary-benefit results of the treatments.