Improving the quality of economic evaluation in health in low- and middle-income countries: where are we now?

An extensive review of published economic evaluations globally found that as of 2016, >230 economic evaluations were produced annually in low and middle income countries (LMIC), a substantial increase from just a decade earlier [1]. Although the review found room for methodological improvement [2], the demand for economic evaluation to inform health policy will ensure that this continues to be a rapidly evolving field. While increases in the volume of economic evaluations in LMIC are encouraging, the quality of these evaluations will be critical. How then, must we define ‘quality’? Is it accuracy, comprehensiveness, adherence with quality checklists? These measures of quality are clearly important in their own right, but they are proxies for the reason we conduct economic evaluation in the first place: to make the ‘right’ allocation and/or investment decisions.

Such decisions are commonly about what should or should not (or, can or cannot) be funded, reimbursed or procured in our health systems by governments, insurers and provider networks, and donors and development partners.

If the quality of an economic evaluation is a function of the decision it intends to inform, we need to acknowledge the dynamics of health policy decisions where economic evaluations are being used: whether these decisions are prospective or retrospective, are routine institutional or one-off decisions, and whether the interventions assessed are individual technologies such as medicines or are complex multi-interventional programmes (or somewhere in-between).

A further contextual element reflecting how quality can be observed and analytical methods chosen is whether the decision is under a general framework of universal health coverage (UHC), within individual disease programs, or at the individual or household spending level; in other words, it is about the budget to be allocated.

For example, when it comes to a decision within the prospective, individual technology, routine decision-making, UHC paradigm, such as that by the Centre for Health Technology Evaluation program at the UK’s National Institute for Health and Care Excellence (NICE), key elements impacting on the quality of economic evaluation include timeliness, consistency and accurate incorporation of UK population health effects. The measure of whether an evaluation is of high quality in this context is the extent to which it meets the informational needs of NICE and of the budget holder, NHS England (NICE decisions predominantly impact on NHS England budget; however, other budgets within the English and Welsh NHS are also impacted by recommendations made by NICE). The quality of an economic evaluation to inform a decision whether to scale up an infectious disease prevention community health worker program in a low-income country context, however, may be weighted more heavily toward comprehensive representation of disease-specific health and nonhealth impacts, impact on various constraints and funding scenarios and analytical transferability to other contexts. In this instance there may be multiple budget holders, including a myriad of international donors and initiatives and local National Treasury or Ministries of Health. This picture is also likely to be substantially different between countries and disease programs, depending on factors such as amount and direction of development assistance for health.

Acknowledging this complex space and the need for high-quality economic evaluation to inform decision making under a UHC framework, in 2016 the International Decision Support Initiative (iDSI) reference case was developed by a wide ranging group of high income country (HIC) and LMIC policy makers and researchers, using a principle-based approach [3]. Principles in the reference case represent common elements that can be applied across intervention types and decision contexts, while matching methodological specifications and reporting standards are supposed to reflect local decision space. For example, the outcomes principle required that the outcomes reported should be appropriate to the decision problem, should capture both positive and negative effects on length of life and quality of life, and should be generalizable across disease states. The methodological specifications of the outcomes principle stated the quality-adjusted life year and the disability-adjusted life year could be considered as options, and the reporting standards required full description of methods for synthesis of the measure chosen. In this way, the iDSI reference case aimed to achieve broad applicability but local relevance to decision maker’s needs. A recent review [4] of the formal take-up of the iDSI reference case in journal publications reporting cost per disability-adjusted life year found limited explicit citation of the iDSI reference case; however, this may reflect the iDSI focus on the direct policy process and supporting analytics, much of which may not achieve publication. It also demonstrates that quality improvement initiatives require dedicated action to encourage uptake from decision makers, who, in the case of global health are often-times development partners and are not always the same as the beneficiaries of improved decisions, which such methodological standards as the reference case are supposed to inform. This is in contrast to HIC and increasingly MIC settings where country-specific reference cases have multiplied and taken hold recently (e.g., Gear4Health.com [5]).

Building on the principle-based approach of the iDSI reference case, two further initiatives aimed at driving quality improvement in economic evaluation have emerged. One of these is the Reference Case Guidelines for Benefit-Cost Analysis in Global Health and Development, which expanded the iDSI framework in a number of areas including to incorporate a welfarist perspective for monetarized valuation of mortality and morbidity risk reduction with a view to facilitate cross sectoral comparisons and decisions [6]. The second one is the Global Health Cost Consortium reference case, focused on the complex area of principles and standards for costing health interventions. These and other initiatives will provide further guidance and specifications for improvement in LMIC economic evaluation quality.

What does the future hold for assessing the quality of economic evaluation especially in the context of global health and what will economic evaluation look like in 5 or even 10 years? While the exact nature of economic evaluation in 2024 is unknown, it is highly likely that the progress made in digital health, digital information systems and artificial intelligence (or more precisely predictive analytics), will have significant impact on the questions asked of economic evaluation, the methods employed [7], and the value that economic evaluation will have to decision making. This will be driven by the positive disruption that digital health may have on pathways of care and the ability of predictive analytics to reduce the transaction costs of evidence synthesis and inference and decision re-evaluation. For example, routine linking of prescribing data, laboratory services and patient-reported outcome measures [8] within a digital health information system would enable real-time and on-going economic evaluation of the cost and health impact of a medicine within the health service, moving away from the one-off ‘approve or decline’ decisions economic evaluation is frequently required to inform, and moving toward a refined, targeted analytical method. As these approaches are adopted, it will have significant implications for how we assess the quality of an evaluation. The World Bank is currently coordinating a multistakeholder effort to think through the specific methodological challenges associated with economic evaluation of digital health and artificial intelligence, and how quality in this area can be maintained and enhanced. Targeted toward in-country decision makers and global development partners, the initiative aims to establish an evaluation framework in digital health and apply the framework in a series of test cases in LMICs.

Although this complex and at times disparate picture of the nature of quality in economic evaluation we have sketched out here reflects the lack of strong empirical evidence of what good quality means in the context of decision making, we assert that the global journey toward UHC generates (or ought to) unprecedented demand for quality economic evaluations. The range of socially accountable decisions that a UHC system necessitates, means that economic evaluation will need to keep up. Decision science involves a range of factors beyond economic evaluation including the political economy and ethical considerations but as long as opportunity cost remains a factor in decision making, quality economic evaluation will be imperative. Development partners will have a critical role to play, but the core drive for quality will inevitably and increasingly come from countries themselves. As countries such as India, China, Ghana and South Africa develop and fund their UHC systems, their own decision needs will define quality and it will be beholden to the international community and research organizations, domestic and global, to respond or become less and less relevant.

The findings, interpretations and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of the World Bank or other partner institutions or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of the World Bank, all of which are specifically reserved.

T Wilkinson is employed to provide consultancy services to the International Decision Support Initiative and the World Bank Group. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.