Considerations for understanding systematic reviews and network meta-analyses in neurological applications: a review and critique in chronic migraine

Alignment between the specified PICOS criteria and the search strategy, including search terminology, as well as between the PICOS criteria and the included trials, is a hallmark of a robust systematic review. Both Zhao et al. and Yang et al., however, fall short of this standard in multiple respects.

The PICOS criteria reported in the PROSPERO registration associated with Zhao et al. appear to be misaligned with the NMA evidence base as described in the final publication. For example, with respect to the Population criterion, the registered PICOS indicated only adult patients (≥18 years of age) were considered, however an adolescent patient (12 to <18 years of age) clinical trial appears to have been included based on Table 1 in Zhao et al. [16]. In addition, for the Outcomes criterion, while the registered PICOS indicated a narrowly focused set of outcomes of interest (i.e., migraine days per month, remission rate and adverse events), the NMA assessed outcomes beyond this (e.g., change in migraine disability assessment score).

In Yang et al., the review’s PICOS criteria are not presented in a centralized table resulting in a lack of search strategy transparency. The manuscript states that “[t]he inclusion criteria applied in the current NMA included published randomized clinical trials with either placebo-controlled or active-controlled designs, human study, investigated CGRP interventions applied in patients with CM and patients diagnosed with CM based on the ICHD”. It is also stated that “[t]o provide additional clinical information, we also included trials investigating the efficacy of topiramate or [onabotA] in patients with CM to serve as active controls”. Indeed, multiple placebo-controlled and head-to-head trials of onabotA and topiramate (without CGRP agent arms) are included in the NMA. Yet, the search strategy as outlined in the database search terms presented in eTable 9 explicitly includes only search terms for CGRP agents in CM. It is thus unclear how the onabotA and topiramate trials included in the NMA were identified.

Minimization of errors in the trial selection and data extraction process is a crucial component of ensuring systematic reviews and evidence synthesis produce valid, unbiased results. However, notable instances of data errors and/or insufficient transparency in trial selection can be identified in Zhao et al. and Yang et al.

While Zhao et al. overall concludes that “[onabotA] has the best efficacy and safety profile”, a notable exception is found in the efficacy outcome “≥50% reduction in monthly migraine days” (hereby referred to as migraine response). In this outcome, the NMA finds topiramate to be the most effective treatment in migraine response, with a risk ratio (RR) of 50.06 (95% CI: 3.18, 787.30) compared with placebo. Additionally, the NMA finds topiramate to be statistically significantly more efficacious relative to all other assessed treatments with similarly large RRs. Such inflated RRs with statistical significance across all comparisons in a competitive treatment landscape should immediately raise red flags for a keen reader.

This finding initially appears driven, at least in part, by the inclusion of Diener et al., a relatively small study (N = 59) with corresponding greater uncertainty in effect estimates, as the sole source of topiramate data for the migraine response outcome in the NMA [17]. However, a closer review of Table 1 in Zhao et al. in comparison with the original Diener et al. publication suggests that the findings of the NMA may be driven not merely by small sample size but potentially by a more substantive issue. Table 1, row 21, column “≥ Migraine responders (%)” of Zhao et al. appears to indicate Diener et al. reports migraine response rates of 90.91% for topiramate and 3.57% for placebo. These rates are in stark contrast to the primary Diener et al. publication, in which Figure 4 presents the trial’s findings for achieving migraine response at week 16 as 29% among patients receiving topiramate and 0% among patients receiving placebo.

This disconnect between the values reported in Zhao et al. and Diener et al. is a potential primary driver of the large RR for topiramate in the migraine response outcome produced in the NMA. While presenting the underlying values used in the NMA is indeed aligned with PRISMA NMA extension guidance and allows for transparent review of the synthesized evidence, it is not immediately clear how Zhao et al. came to utilize these disconnected values, though perhaps by interpreting Diener et al. Figure 4 as “n” rather than “percent” and applying a continuity correction to address the zero cell of the trial’s placebo arm. However, Zhao et al. make no mention of applying a continuity correction. Thus, there is seemingly a disconnect between the systematic review’s extracted data and identified sources. In other words, the values synthesized in Zhao et al. cannot be found in the cited clinical trial. Accordingly, findings reported in Zhao et al. for the migraine response outcome may not be reliable and therefore should be interpreted with substantial caution.

Besides the lack of transparency in handling Diener et al., the selective use of only the nonaligned values for the migraine response outcome obfuscates interpretability of NMA results. Table 1 in Zhao et al. also suggests a discrepancy in study inclusion: although Rothrock et al., the primary report of the FORWARD trial comparing onabotA with topiramate, was included for other outcomes, it was omitted from the migraine response analysis without a clear rationale [18].

The exclusion of Rothrock et al. in this outcome is thus a key omission, especially as the findings of the NMA in migraine response appear to contradict the head-to-head FORWARD clinical trial data. Besides Rothrock et al., several key onabotA studies also appear to be omitted entirely from the NMA without clear rationale [19,20]. Furthermore, potential trial duplicates appear in Zhao et al. Table 1 of Zhao et al. is presented as the studies/trials contained in the NMA, yet both Silberstein et al. and Lipton et al. – publications of the same PROMISE-2 clinical trial – are contained in the table as separate rows. It is thus unclear if duplicate data may be included in the NMA [21,22].

Yang et al. entirely omit Rothrock et al. from the NMA. Notably, FORWARD does not appear as an excluded publication in eTable 3 of Yang et al., suggesting that it may not have been identified at all based on the information provided in the manuscript. This omission may reflect the previously discussed disconnect between the PICOS criteria and search strategy. However, given that other onabotA and topiramate trials were identified, it is unclear how Rothrock et al. was missed. This omission is of particular concern because the NMA findings contradict those of the FORWARD trial.

In addition to the opaque handling of onabotA and topiramate trials as noted above, data handling in the NMA is further obfuscated by the lack of reporting of NMA inputs. This is a key requirement in robust and transparent evidence synthesis. For example, the prior-discussed disconnect between Diener et al. and Zhao et al. may not have been identified if the NMA inputs were not published. Indeed, there is no reporting within Yang et al. of what values were purported to be utilized from which of the cited trials in the NMA. While “direct evidence” relative efficacy values are reported in eTable 8 as part of the GRADE assessment presented in Yang et al., there is no indication as to what values or trials underlie the reported direct evidence. Given the demonstrable lack of reported NMA inputs, there is no robust way to verify that the values synthesized in the Yang et al. NMA align with the trials identified in the associated literature review. This is particularly problematic as it contradicts the PRISMA NMA extension checklist (“Results of individual studies: For all outcomes considered [benefits or harms], present, for each study: simple summary data for each intervention group and effect estimates and confidence intervals.”). The tables which Yang et al. point to as fulfilling this requirement in the checklist do not contain the applicable data. This disconnect with the PRISMA NMA extension further obscures interpretation of the NMA and raises additional concerns about the overall validity of the analysis.

To ensure results produced by NMA are valid and unbiased, the trials included in the analysis must adhere to the foundational assumption of exchangeability. However, multiple instances of heterogeneity which potentially violate this exchangeability assumption are present across the trials synthesized in Zhao et al. and Yang et al. Although both Zhao et al. and Yang et al. identify several sources of heterogeneity among the included trials, they do not discuss the potential impact of these differences as recommended by the PRISMA NMA extension guidelines.

For example, Zhao et al. note the wide timespan of included trials and heterogeneity of outcome assessment timepoints. Indeed, the NMA synthesized clinical trials published between 2005 and 2022, with CGRP trials generally conducted more recently than onabotA and topiramate trials. These more recent trials were conducted in a different clinical environment than earlier trials with potential differences in, among other factors, patient expectations, site experience, recruitment patterns and injection devices. As placebo response in migraine prevention is substantial and can vary by route of administration (e.g., placebo response from a more invasive route of administration, such as subcutaneous injection, may be greater than from a less invasive route of administration, such as an orally administered pill), this inherently threatens the exchangeability of trials in relation to network connectivity and thus weakens indirect comparisons that rely on placebo as the common comparator (namely, comparisons between onabotA and CGRPs). Furthermore, more recently-conducted trials include patients with prior preventive treatment failures while older trials were conducted before more recent CGRP agents became available. These trials may thus represent differing populations with different anticipated responses to treatment, given prior treatment failure is often considered a treatment effect modifier in multiple disease areas (with patients failing prior treatment generally at greater risk of failing subsequent treatment).

Heterogeneity in assessment timepoints is also particularly problematic. In addition to the general limitations of timepoint heterogeneity (namely, that treatment effects may differ when outcomes are measured earlier vs later), timing is especially important in CM because treatment administration differs across interventions. OnabotA is administered cyclically, with repeated injections every 12 weeks and an assessment timepoint of 24 weeks [19,20,23]. By contrast, CGRP trials generally assessed outcomes at 1–12 weeks [24,25]. As a result, a single 12-week assessment of onabotA may not fully capture the cumulative benefit achieved after repeated treatment cycles, potentially biasing the NMA against onabotA. The study authors could have explored sensitivity analyses of differing timepoints (as available) in order to elucidate potential relationships between assessment timepoints and NMA findings.

Similarly, Yang et al. identify notable outcome definition differences across trials (e.g., mean monthly migraine days vs mean monthly headache days). Evidence synthesis that pools outcomes across definitions may obscure clinically important distinctions and compromise comparability. For example, onabotA trials general report headache-day outcomes, while CGRP trials generally report migraine-day outcomes. While these outcomes are related, they are inherently distinct as a migraine is a specifically defined form of headache. As such, change in headache days may not be reflective of change in migraine days. Such discrepancies can introduce bias and threaten the exchangeability of trials, undermining the assumption of homogeneity. The study authors could have explored sensitivity analyses here as well by considering separate evidence synthesis of headache-day and migraine-day outcomes as to mitigate outcome definition heterogeneity.