Mapping methods gaps between EU joint clinical assessments and local health technology assessment decision-making: an environmental scan of guidance in select EU markets and harmonization challenges

A comparison of methodological guidance on scoping specifications between EU JCA and key European HTA agencies is provided in Table 1.

Overall, the EU HTAR assessment scope sets up the PICO framework as the cornerstone of the JCA process which, as strongly emphasized, should be inclusive and meet the needs of all 27 EU countries [1]. It is meant to translate national policy questions into research questions and outline the evidence required from the HTD to meet the MS needs [30]. Assessors/co-assessors will draft a PICO survey to be distributed to the MS to define their local needs. A MS can choose not to participate in the survey. Differences are expected between MS regarding populations (national decisions may deviate from the label population submitted to the EMA) and comparators (choice is determined by the local clinical care and can include off-label treatments). The MS will be asked to limit their requests to what is necessary for their national decision-making and specify components of clinical care. The assessors/co-assessors will consolidate the PICO(s), while also considering other relevant input (e.g., from patients and clinical experts). The final evidence requirements (consolidated PICOs) will then be communicated to the HTD.

Since the assessment scope is mostly policy driven, it might contain PICO(s) for which no evidence is available. In this case, the HTD must justify why no data were submitted. In other cases, a specific PICO may not be fully covered by the available studies (e.g., the PICO could comprise only a subgroup of the study population or the requested comparator would not be represented in the study). To meet the evidence requirements, available studies will need to be re-analyzed or assessed for suitability for indirect comparisons. Any deviations from the original study protocols must be clearly mentioned in the JCA dossier, and re-analyses resulting from PICO requests must be provided by the HTD (original study analyses should be included in the dossier regardless) [29].

A comparison of methodological guidance on evidence selection between EU JCA and key European HTA agencies is provided in Table 2.

A systematically identified, comprehensive evidence base constructed from a predefined protocol and clear selection criteria is fundamental to the HTA process. The EU HTAR documents emphasize that PICO requests should guide this process and HTDs should perform a timely, relevant systematic literature review (within three months of dossier submission date) outlining prescriptive requirements for information retrieval (e.g., database searches, patient registries and other sources) and evidence selection scenarios based on the number of comparators, PICOs and network connectedness. Evidence of various types (study designs) and from multiple sources may inform the JCA so systematic searches should consider this in the design phase; to ensure the evidence selection is appropriate and robust for comparative effectiveness analysis, all decisions made in this step must be thoroughly justified and transparently presented. Some differences were noted with local HTA guidelines; for example, a systematic review is not a submission expectation in France whereas in The Netherlands, data sources are only accepted if published in peer-reviewed journals or in the European public assessment report.

Evidence from randomized controlled trials (RCT) retains the ‘gold standard’ position in the evidence hierarchy across the EU methods guidelines considered. In EU HTAR guidance, nonrandomized studies are recommended only to fill evidentiary gaps in situations of unanchored indirect treatment comparisons (ITC) or when this is the only evidence source. However, its use in the JCA context is considered limited. The acceptability of data from nonrandomized studies varied across the other HTA agencies, with stricter requirements set forth by IQWiG compared with the criteria outlined by HAS, ZIN and AEMPs allowing consideration of such evidence in HTAs with high unmet need and challenges in RCT evidence generation (such as rare/very rare diseases, first-in-indication products).

One key difference between the EU HTAR and local HTA guidance was the quality assessment step of included studies. Evidence for local submissions must be quality-assessed using the tools recommended by each jurisdiction (Figure 2); for JCA, HTDs must provide a thorough justification and documentation for each quality element and additional contextual information for every study included in the dossier to allow assessors to conduct their own quality assessments.

Other topics covered included the selection criteria for studies to be considered (only if they are directly connected to the evidence networks or contribute as ‘first-order’ loops) for the PICOs set up in the assessment scope. One key difference with the local HTA guidance documents was the EU HTAR request to construct evidence networks at both ‘population’ and ‘comparator’ levels if different, which will have a significant workload impact.

A comparison of methodological guidance on evidence synthesis between EU JCA and key European HTA agencies is provided in Table 3 and a flowchart constructed based on methodological considerations published as part of the EU HTAR is presented in Figure 3.

Assessing the clinical effectiveness and safety of the technology compared with routine clinical care (as defined by each PICO) requires quantification of the relative effects of the technology under evaluation. This must be done alongside an assessment of the degree of uncertainty in the relative treatment effects, while factoring in the strengths and limitations of the supporting evidence.

Overall, the EU HTAR methods strongly advocate for the need of direct treatment comparisons (using a frequentist or Bayesian approach) based on high-quality RCTs conducted on the population of interest. Guidance for when to use each approach is provided based on the outcome types, number of available studies and choice of prior distributions (for Bayesian methods). The discussion of analytic methods, however, is preceded by a detailed description of how to assess the exchangeability and heterogeneity assumptions behind each evidence network. Additional considerations for assessing data robustness (missing values, biases), including how this should be quantitatively assessed and presented for the JCA purposes, are also discussed. The EU HTAR methods emphasize the need to identify effect modifiers a prior through multiple, parallel activities (literature review, subgroup analyses, and expert opinion) which should be transparently presented. This may require additional preparatory efforts for HTDs (e.g., conducting targeted literature reviews) which is not considered a routine preparatory activity for local HTA submissions. There is also a reference to the use of the Instrument for assessing the Credibility of Effect Modification Analyses (ICEMAN) [55] which is less applied in national HTAs but provides a structured way to identify credible effect modifications (subgroup effect, statistical interaction, moderation, or heterogeneity of treatment effects) in the dataset. The IQWiG methods guideline does not mention ICEMAN but it discusses the quality and credibility of subgroup analyses, referring to a publication that formulates usability criteria [54].

The EU HTAR guideline (similar to other local HTA standards) requests the full recording of considerations and decisions made in the feasibility assessment of treatment comparisons that will guide the pre-specification of the appropriate analytical approach to be documented in a statistical analysis plan. Regarding analytical methods preferences, naive comparisons of absolute effects (with no confounding adjustment) across different studies should not be used in JCAs since these methods fail to preserve randomization between patients and studies. This was also clearly noted by IQWiG but not in French and Spanish HTA guidelines. ZIN highlighted that such comparisons may be used in local submissions under exceptional circumstances. However, they should be downgraded in the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) assessment due to a preference for direct comparisons analyzed using random-effects models in the case of a connected network. However, the EU HTAR guidelines become less clear when evidence synthesis is needed for indirect comparisons, either due to disconnected networks or in the presence of high heterogeneity. Although the ITC methods (matching-adjusted indirect comparison [MAIC], simulated treatment comparison) and some novel analytical methods (multilevel network meta-regression [ML-NMR; EU HTAR only]) are mentioned, the EU methods clearly spell out situations in which population adjustment may be less valid (e.g., difficulty controlling for all relevant effect modifiers due to small sample size) and inappropriate to use in the JCA context. Even though the JCA guidelines point out that approaches other than ITCs may be explored to answer the PICO questions in these situations (e.g., splitting evidence networks into subgroups, use of meta-regression when appropriate, sensitivity analyses and exclusion of studies), challenges in the implementation and interpretation of comparative effect estimates from these approaches remain for the local HTA purposes. The same concern applies for the use of nonrandomized studies (e.g., single-arm trials and nonrandomized comparative studies) for evidence synthesis; the EU HTAR guidelines specify that this type of data can only be used if there is full access to individual patient data that would allow appropriate use of adjustment methods for all confounders, prognostic variables and effect modifiers to be applied. The EU HTAR methods for these topics are clear and direct; nonrandomized evidence available only at the aggregate level should not be used in the estimation of treatment effectiveness since there are no adequate methods to account for the necessary adjustments. The use of nonrandomized evidence in the German and Dutch HTA context is only allowed in exceptional cases; however, the absolute requirement of access to full individual patient data is less clear than in the EU HTAR guideline. On the other hand, local real-world evidence is considered important for French and Spanish local submissions.

Based on the EU HTAR methods guideline, uncertainty in the relative effects of a technology may be the result of three, mutually independent domains in data analytics: internal validity (albeit systematic error in results due to study design or patient heterogeneity), statistical precision (as a function of random sample variation in results) which should be evaluated under the JCA scope and external validity (generalizability of findings) to be assessed at the local HTA setting. The dichotomization of these uncertainty elements may impair the HTD’s ability to fully explore the impact of some of these considerations on treatment effect estimates and address specific PICO requests that may impose further challenges (e.g., in patient subgroups lacking consideration in the HTD’s primary clinical trial). The EU HTAR methods highlight that sponsors should thoroughly and transparently present uncertainty sources in evidence or analysis considered in the JCA without claiming the medical context of a specific PICO (e.g., rarity of disease) as an explanation for selecting lower-quality evidence or data outside RCTs. This is a clear distinction between the EU and local authorities which allows flexibility in decision-making through their individual access programs for technologies addressing high unmet patient needs or first-in-indication products for rare and very rare diseases, including targeting pediatric populations. Finally, as highlighted in several EU HTAR documents, the scope of JCA is not to provide a value assessment of the technology under review, but rather to provide clinical assessment reports for a range of outcomes that can be used for local, EU decision-making. For that reason, evidence appraisal systems (such as minimal clinically important difference, GRADE and statistical thresholds) and hierarchy in outcomes assessment are not to be used in JCAs. These steps exclusively remain a MS task and some discrepancies in final assessments may be expected due to procedural differences.

Other topics covered included the use of surrogate (or intermediate) outcomes to assess relative effectiveness estimates. In the JCA context, this topic is controversial since the full validation of surrogate outcomes has not been rigorously demonstrated when the submission is completed and may limit the external validity of the submitted evidence. The EU HTAR guidelines specifically point to the fact that a surrogacy validation exercise (testing association between the surrogate and final outcomes) needs to be assessed at the local, MS level but advises local assessors to include all three levels of evidence; the first level is considered a requirement (i.e., correlation of treatment effects between surrogate and final outcomes in the disease stage and interventions as derived from a meta-analysis of several RCTs preferably identified via a systematic literature review). Multiple statistical testing, considerations for sensitivity and subgroup analyses and post-hoc analyses are also discussed in EU HTAR guidance.