Developing a cross-validation tool for evaluating economic evidence in rapid literature reviews

This qualitative approach of evaluating economic evidence was inspired by the Bayesian framework. The key feature of the Bayesian framework is to allow new findings to be set in the proper context [13]. Briefly, Bayesian posterior distributions were derived by updating prior probabilities with new observed data. Bayesian methods naturally permit the synthesis of all available evidence (i.e., not the analysis of the new data alone); therefore, it can improve the precision of treatment effect estimates and potentially reduce the bias associated with such estimates [14].

Following Bayesian thinking, the purported QALYs gained in the rapid economic review study are put into the context of the existing health economic evidence. The best quality economic study was selected as the reference, which was assumed to reflect the true QALYs gained, to act as a benchmark for the results of other analyses (this selection process is further explained later in the article). These reference studies act as a safeguard, reducing potential biases of interpreting data from an individual economic evaluation. When the results of individual studies are inconsistent with the existing evidence, the findings of new study may be seen as lacking credibility.

Using 83 published articles, Wright and Weinstein categorized the gains in life expectancy from various medical interventions as large or small. The interventions included both treatments and preventive interventions, stratified by the target population and disease [15]. For example, implantable cardioverter-defibrillators lead to a life-expectancy gain of 36 to 46 months for the survivors of cardiac arrest, whereas amiodarone therapy leads to a life-expectancy gain of 14 to 16 months. Also, there is an association between clinical outcomes and generic health related quality of life (HRQOL). Methods of mapping clinical outcomes to health utility have been developed and used in economic evaluations [16]. Thus, it is possible to evaluate, quantitatively or qualitatively, the magnitude of utility improvement by an intervention, on the basis of its impact on severity and frequencies of clinical events. We further extended these concepts to the association between the clinical benefit (set A) and the QALYs gained (set B). Researchers developing this tool must review both clinical and economic literature to populate sets A and B.

Let A be the set of clinical benefits (a_i) for treatments for a given indication. Each element (a_i) represents the magnitude of the difference in clinical effectiveness between two treatments (or between an intervention and placebo).

Let us select numerous interventions whose clinical benefits can be ranked by clinical judgment or clinical outcome measures. Let a₁ > a₂ > a₃ >… > a_n >0. Importantly, specific numeric values do not need to be assigned for the elements in set A because this tool is qualitatively structured. To help illustrate this concept, the following is an example in heart disease. The average clinical benefits (e.g., gain in life expectancy) of heart transplantation are greater than myocardial revascularization with coronary artery bypass grafting (CABG), and the average clinical benefits of CABG are greater than radiofrequency catheter ablation (a surgery that uses radiofrequency energy to destroy a small area of heart tissue, causing rapid and irregular heartbeats) [15,17,18].

Assume that clinical benefits can be directly represented by the differences in QALYs between interventions (i.e., QALYs gained). For simplicity, further assume that clinical benefit is the main contributor to differences in QALYs between select interventions, whereas other factors (e.g., adverse events) have an insignificant impact (note: later a scenario where QALY differences between treatments is driven by the safety profile is discussed). Therefore, clinical benefits will be positively correlated with QALYs gained (i.e., a larger magnitude of clinical benefit is associated with greater QALYs gained). Let B be the set of QALYs gained (b_i).

Each element in set B has a corresponding numeric value of QALYs gained.

Each element in set A (a_i) has a corresponding element in set B (b_i), and the order of the comparisons in set A and B should be the same or similar. Let us assume that the order of all elements in set A are known, and the values in a subset of set B are known. On the basis of the known order of elements in set A and the known values in the subset of B, we can deduce a range of median values for unknown elements in set B.

For most diseases, there are a small number of treatment options available. Therefore, it is unlikely that there will be many elements for set A and B. We can simplify this by using only two elements, one with the largest clinical benefits (e.g., a highly effective treatment that is infrequently used in clinical practice, such as heart transplants) and the other with the smallest clinical benefits (e.g., pharmaceutical therapy). Once we have two elements that represent both the largest and smallest clinical benefit, we can theoretically obtain an interval, and we expect that a new intervention's QALYs gained will likely fall within this interval. When we extend this concept to the simplest condition (only one reference treatment), then, based on the clinical judgment of the new intervention in relation to the reference, we can estimate whether the QALYs gained of the new intervention is larger or smaller than the ‘true’ value from the reference.

We have discussed the relationship between an intervention's gains in QALYs being driven by the difference in clinical benefits between the interventions. For a given condition, the primary clinical effects may be similar across various interventions; however, the secondary outcomes (e.g., the safety profile) may differ. For example, different oral anticoagulants (dabigatran, rivaroxaban and apixaban) have a similar effect on the risk of ischemic stroke in patients with atrial fibrillation, but based on observational research, apixaban is likely to be associated with a lower risk of major bleeding [19,20]. In this case, the QALYs gained are driven by the safety profile. Following the concepts discussed earlier, the elements of set A can be ranked by safety, and elements in set B are the corresponding QALYs gained in this condition.

This tool may be extended to interventions that improve HRQOL but have no effect or marginal effect on survival, clinical events (e.g., stroke) or other objective measures, such as treatments for anxiety and depression [21,22]. When ranking these treatments, researchers need to consider both the HRQOL instruments used (i.e., different HRQOL instruments used for one disease) and the magnitude of HRQOL benefits from an instrument. Furthermore, although used in various diseases in practice, generic HRQOL instruments such as the EQ-5D, may be less responsive to capturing small changes in HRQOL for some conditions, such as in patients with prostate cancer [23]. For these conditions, using this tool may lead to greater uncertainty in estimating the QALYs gained for a given intervention.

When both primary and secondary outcomes have considerable contributions to the QALYs gained, the ranking of health benefits may be complex and undetermined. Therefore, this tool may be unsuitable for this scenario.

The key factor in applying this cross-validation tool is to select references properly and define the true QALYs gained from an intervention. Although efficacy measures are often considered constant, the results of economic evaluations often differ because of varying contexts (e.g., varying clinical practice or different costs), model assumptions or controls of bias parameters (e.g., measurement errors, selection bias, confounding or conflict of interest). Theoretically, however, if effectiveness is estimated based on the same body of key clinical evidence, the findings from different economic evaluations for a specific intervention should be relatively consistent. Therefore, although subject to limitations, we can assume that the best quality economic evaluation for an intervention approximates the ‘true’ but unknown health benefit in QALYs.

Figure 1 shows the illustrated process of implementing the cross-validation tool. Once the tool (i.e., database of clinical benefits and corresponding QALYs gained) is established and validated, we can quickly estimate a new intervention's range of QALYs gained compared with an older treatment once the clinical effect of this intervention is established.

In a hypothetical example in Figure 1, according to clinical judgement (or clinical measures) the benefits of the new intervention would be less than the fourth-ranked treatment in the dataset (associated with 0.6 QALYs gain compared with usual care) and greater than the seventh-ranked treatment (associated with 0.3 QALYs gained compared with usual care). Therefore, the QALYs gained associated with the new intervention likely range from 0.3 to 0.6. If the results of a newly published economic evaluation contradict this value, the validity of the new economic evaluation's findings may be seen as questionable.

This idea can be further explained using a case example. Both continuous glucose monitoring (CGM) and the flash glucose-monitoring system (Flash) measure interstitial glucose and provide data on glucose trends for diabetes insulin therapy. Flash has similar functionality to CGM, and the price is significantly less. However, CGM has the added function of alarms that alert the user to hypoglycemia or hyperglycemia [24]. Thus, CGM should have either better or similar effectiveness than Flash for type 1 diabetes because CGM has more features [24]. A diabetic modeling study prepared by NICE evaluated the cost–effectiveness of CGM versus self-monitoring of blood glucose (SMBG) in patients with type 1 diabetes [25]. The results showed that compared with self-monitoring, CGM had an increase of 0.10 QALYs over a lifetime horizon (continuous subcutaneous insulin infusion [CSII] + CGM: 12.82 QALYs versus CSII + SMBG: 12.72 QALYs). If we assume the NICE study represents the true effectiveness of CGM, we can expect Flash would have a smaller or equal benefit compared with SMBG (QALYs gained of around 0.10). However, a recent CUA in patients with type 1 diabetes found that compared with SMBG, Flash had a large gain of 2.12 QALYs over a lifetime horizon [26]. By comparing the findings with that of the NICE study, we may be curious about the validity of the economic analysis' results. When the study was appraised further, we found that it likely overestimated the QALYs gained associated with Flash because the disutility of nonsevere hypoglycemia may have been overestimated by assuming a linear relationship between the frequency of hypoglycemic events and the total reduction in utility.

To create set A, it is critical to develop a dataset with reference treatments by disease category. Further considerations in developing set A are as follows.

Following are considerations for selecting the best quality economic evaluations of key treatments for a disease (i.e., selecting elements for set B).

Researchers from various settings such as academia, HTA agencies or industry that frequently conduct rapid HTAs or rapid reviews may benefit from the development of this cross-validation tool. Including health economists, researchers with medical or clinical background may contribute to this developing process to ensure the suitability of the clinical judgments for the set A. If the HTA agencies are able to conduct full-HTAs, they may use recent primary economic elevations in HTAs to develop set B, which would be relevant to the setting and context of interest. After establishing the initial version of this tool, researchers can adapt this tool to evaluate economic evidence in their rapid literature reviews and provide the feedback to continue improving this validation tool. It is expected that set A does not need to be modified often, but the estimate of QALYs gained in set B may be modified with the new estimates from recently published economic evaluations.

When applying this tool to a published economic study included in the rapid review, the researcher may provide either positive or negative comments. For example, when the QALY gained of the study is not consistent with the existing evidence, the researcher can state that ‘We did not use a structured validation assessment tool to evaluate the quality of this economic study in this rapid review. However, we used a cross-validation tool to evaluate the cost–effectiveness results and found that the QALYs gained from the intervention in this study was likely beyond the possible range, based on the existing high-quality economic evidence. Therefore, the findings of this study should be interpreted with caution.’

For the individual researchers who occasionally conduct the rapid reviews, it may be impractical to develop the cross-validation tool for a single rapid review. However, they can source established tools that are available from other HTA agencies (given that a tool is widely accepted in the future). Lastly, researchers may consider using the simplified tool, as illustrated earlier for the concepts and the example.