R WE ready for reimbursement? A round up of developments in real-world evidence relating to health technology assessment: part 15

Oncology is an area of significant innovation, with a total of 115 novel active substances having been launched globally in the past 5 years and 237 over the last 20 years [1]. This explosion in novel therapeutics leads to substantial use and need for real-world data (RWD), including RWD to understand patient characteristics, disease burden, treatment and outcomes, all of which can be useful for regulatory and health technology assessment (HTA) submissions [2–5]. When generating this real-world evidence (RWE), manufacturers not only are considering regulatory and HTA stakeholders, but also healthcare professionals (who may also be involved in RWD collection), so it is important for them to bear in mind the needs of all potential users of the RWE generated. To this end, two of the leading oncology professional societies – the American Society of Clinical Oncology (ASCO) and the European Society for Medical Oncology (ESMO) have recently published guidance on the use of RWD.

In a recent editorial in the Journal of Clinical Oncology (JCO) [6], it was highlighted that the ASCO family of journals has observed a substantial increase in manuscript submissions utilizing RWD in recent years. Between 2018 and 2022, the ASCO journals received 16,133 total submissions, of which 1156 incorporated RWD. However, the majority of RWD studies submitted to the JCO were deemed to be of insufficient quality for review or failed to receive reviewer and editor endorsement for publication. Despite the low acceptance rate for these studies (less than 5%), the JCO editors recognize the potential value of properly conducted real-world research to inform cancer care practice and policy. The editorial cites FDA guidance and the STaRT-RWE template [7] for details on the design, analysis and reporting of RWD studies. However, the editorial also describes three priority areas for consideration on RWD quality:

The three key points raised in the JCO editorial are also highlighted in the RWE guidance from ESMO [8]. The multidisciplinary experts of the ESMO Real-World Data and Digital Health Working Group have developed the first set of recommendations specific to publishing oncology RWE studies in peer-reviewed journals: the ESMO Guidance for Reporting Oncology Real-World Evidence (ESMO-GROW). This new 35-point guidance addresses the unique aspects of oncology research, including disease-specific variables, biomarkers, therapies and outcomes that are not adequately covered in current reporting guidelines. For example, the guidance covers specific oncology variables and details required; e.g., for immunohistochemical markers, the name and clone of antibody, platform and scoring system should be reported. An interesting ask is for the description of the study team, reflecting the fact that real-world research frequently necessitates a multidisciplinary approach, incorporating diverse and complementary scientific perspectives to inform study design, conduct and reporting. Depending on the specific research aims, oncology RWE teams may be comprised of clinical oncologists, other healthcare professionals, public health experts, epidemiologists, statisticians, methodologists, data scientists, patient representatives and more. Such cross-disciplinary collaboration enables rigorous delineation of the research question, methodology, analysis and interpretation of findings. Perhaps by not having specific skills present listed in an RWE team it can allow for scrutiny over study areas where (for example) clinical and statistical input would have been most welcome. Ultimately, ESMO-GROW advocates for transparency and rigor in RWE, which is crucial for patients, the scientific community and healthcare authorities. As with other RWE guidance [9–12], RWD is highlighted in being able to generate evidence when used as an external control arm, and the role of target trial emulation designs in mitigating bias and quantitative bias analyses to address untestable assumptions that may depend on unavailable information, such as informative censoring, missing baseline covariates and unmeasured confounders is also noted.

As compared with just three years ago, we have an explosion in guidance for RWE coming from regulators, HTA agencies and now medical professional societies. Although the availability of many different guidelines can cause confusion for manufacturers to address the needs of all potential stakeholders, by and large, these guidance documents are largely aligned with each other. While some specific details differ, in general, if manufacturers follow a transparent approach – for example, documenting why a (fit-for-purpose) RWD source was used and using an analytical approach that mitigates bias and then quantifies any bias remaining – this will be in line with what most stakeholders are looking for. The plethora of RWE guidelines should be viewed positively, as more stakeholders are being educated on how good RWE can be generated, to hopefully further establish the value of RWE for a range of decision makers as well as the wider scientific community.

RWE guidance by regulators and HTA agencies highlights to manufacturers which methods they consider to be best practice, but they do not offer practical guidance on the design and conduct of these studies to support causal inference. This is perhaps one of the reasons why no HTA submission to date has included RWE generated from a target trial emulation approach [9]. PRINCIPLED (process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs) aims to address this gap by providing granular practical advice on the design and analysis of causal RWD studies using routine healthcare data [13]. The guide was developed by considering insights from the FDA Sentinel Innovation Centre, which conducts post-marketing surveillance for drug safety using large volumes of real-world healthcare data; however, the authors include a range of experts from academia, the FDA and other organizations who have substantial experience in methodology for non-interventional studies. The key steps outlined by PRINCIPLED are:

Overall, PRINCIPLED provides an iterative general approach to resolve issues as they arise during the conduct of RWE studies. The inferential analysis is the final step to ensure a clear demarcation between planning and inference to avoid design or analysis changes prompted by study results. PRINCIPLED could better support manufacturers developing RWE to demonstrate causal inference with its practical steps. However, as causal questions can be diverse, it remains to be seen what the uptake of the guidance is, especially in comparison to already existing descriptions of the target trial emulation approach [14]. While PRINCIPLED had FDA co-authors, it will also be interesting to observe whether it will be endorsed by HTA agencies and incorporated in future updates of their guidance.

Thinking about the future, HTA agency RWE guidance currently does not provide any recommendations around how artificial intelligence (AI) can be applied in RWE generation. Given the explosion in use of generative AI tools such as ChatGPT, it is likely going to be important for HTA agencies to have a position on how these methods can be used. To this end, ESMO-GROW [8] may provide a starting point. ESMO-GROW guidance on AI begins by asking researchers to detail the specific AI methods used rather than relying on broad terms like ‘artificial intelligence’ that can be misused or overused. Comparisons to classical statistical techniques are recommended to demonstrate meaningful improvements are achieved by adopting more complex AI models. Clinical AI applications often utilize natural language processing to extract data from electronic health records, which then feeds analysis models. The acquisition and processing methodology of such data should be fully documented. For AI model development, the architecture of train/validation/test datasets, feature selection and overfitting/underfitting prevention measures should be detailed to enable assessment of generalizability and robustness. Model performance assessment is mandatory and AI models should be interpretable by human experts to instill confidence and identify potential biases, possibly through explainable AI visualizations of feature relevance. All of these elements also feature in ISPOR's PALISADE checklist [15], so perhaps this checklist could be an easy update to incorporate in HTA RWE guidance. However, there are perhaps not yet enough studies showing the clear additional value of the application of AI in best practice frameworks like target trial emulation – without a doubt there is more to come in this space.