R WE ready for reimbursement? A round up of developments in real-world evidence relating to HTA: part 5

From the seminal work of Bradford-Hill, it has long been accepted that the results of any single study provide limited evidence regarding a research question and that it is only with the accumulation of evidence from multiple studies that one can gain a degree of certainty [1]. The case for taking such a perspective is perhaps greater for real-world studies on comparative effectiveness, given the greater implications of biases affecting study results. As such, those looking for real-world evidence (RWE) on comparative effectiveness to be accepted by key stakeholders in the regulatory and reimbursement settings must carefully consider the challenges encountered in generating a body of RWE across multiple studies; two recent publications by the Friends of Cancer Research Real-World Data Collaboration have provided interesting insights into the challenges encountered in doing so [2,3].

The two recent publications report the findings of studies in which investigators worked with several data partners across the USA to harmonize definitions of study variables and conduct a set of analyses based on common protocols with the aim of assessing the level of standardization that could be achieved across datasets [2,3]. In one of the studies, real-world comparisons between immunotherapy and chemotherapy for overall survival in non-small-cell lung cancer were made. A number of challenges were encountered in standardizing definitions across data sources while missing data also limited the extent to which analyses could be aligned across the sources. The consistency of results obtained was also a concern, with the direction of relative effects differing across some sources, and relatively wide confidence intervals encountered. Notably, the one study which had an effect estimate in a different direction to other studies was substantially smaller and contained a significant amount of missing data on key prognostic variables including performance status and smoking status.

The potential benefits of collaborative multidatabase studies based on common data models in HTA have been highlighted recently [4]. While we would agree with these benefits, these recent studies serve to highlight some of the challenges that can nonetheless be encountered with such approaches. That is, even when pursuing careful standardization of definitions across sources and using common analytic approaches, significant heterogeneity in outcomes across studies using different data sources may still be encountered. Such heterogeneity poses additional problems in the HTA setting where there is typically an interest in determining not only the direction of effect, but also the magnitude of that effect. As a result, it is important that those conducting these studies carefully consider whether data sources that lack accurate data on key variables should be included; if they are, their limitations should be clearly presented and assessed using appropriate sensitivity and bias analyses. It is also vital that HTA bodies encountering evidence from these types of initiatives as part of HTA (re)assessments approach them in the same manner they would in synthesizing any other body of real-world comparative effectiveness evidence, that is, with thorough consideration of the quality of individual data sources and potential sources of heterogeneity.

After the approval of a number of treatment options for a disease, a question faced by many stakeholders is in what sequence should they be given. In a HTA setting, concerns have been raised regarding the extent to which HTA frameworks are designed to consider treatment sequence [5] and, where they have considered treatment sequences, regarding the scarcity of clinical trial data to inform modeling exercises [6].

The question of treatment sequence has received considerable attention in relapsing-remitting multiple sclerosis (RRMS), where a number of recent observational studies have suggested that earlier treatment with highly efficacious disease modifying therapies may result in better patient outcomes [7–9]. Concerns regarding bias existed with such studies as the extent to which channeling of patients to specific combinations and sequence of treatments can be accounted for using patients from a single healthcare setting and under similar policies is debatable. That is, even where ‘similar’ patients are identified and compared in such studies on the basis of measured characteristics, there is often likely to be unmeasured reasons for the choice of different treatment schedules in these patients. In this regard, a recent paper utilizing RWD from Sweden and Denmark provides interesting insights into this topic [10].

The rationale for the study by Spelman et al. was that while the management of RRMS patients in both countries was quite similar, there were key differences in the treatment strategy recommended and used in RRMS in each country; with Sweden having greater focus on starting patients on highly effective disease-modifying therapies. The investigators controlled for a variety of patient characteristics in an effort to isolate the independent impact of the treatment strategy on outcomes, reporting consistently lower hazards of relapse in Swedish patients compared with Danish. Notably this natural policy experiment overcomes some of the aforementioned limitations of previous studies as the differing policies across countries should increase the possibility that two patients who are truly similar in prognosis receive differing treatment regimens.

In considering the potential application of similar approaches in other indications, we note that the approach used by Spelman et al. is somewhat akin to using geography as an instrumental variable, an approach which has received some criticism when used in the oncology setting [11]. Notably, the criticisms in the oncology space center on the extent to which the geographic regions used satisfied the assumptions of an instrumental variable. Fulfilment of these assumptions is an important consideration, and a potential barrier to the wider use of methods like those applied by Spelman et al. Despite these challenges we believe that, under the correct circumstances, the approach used by Spelman et al. provides a useful and complementary approach to more traditional comparative effectiveness study designs and may support the greater consideration of treatment sequences in future HTA submissions. Going back to the criteria of Bradford-Hill [1], the approach also provides an opportunity to assess consistency of effects across heterogenous designs. Notably, the development of collaborative international multidatabase initiatives may also better facilitate the conduct of such cross-country comparisons.