Network meta-analyses for EGFR mutation-positive non-small-cell lung cancer: systematic review and overview of methods and shortcomings

Our search identified 127 articles with 49 duplicates. We reviewed 78 abstracts/titles, excluded 66 in the initial eligibility assessment and included 12 publications for the full-text eligibility review. Two publications were excluded in this process, leaving ten publications for the full review (Figure 1) [15–24].

Our structured review revealed that authors apply different strategies when performing NMAs in first-line EGFR mutation-positive NSCLC. These differences are important to consider given their influence on results and the potential to impact healthcare decision-making. Although the broad evidence base of RCTs in this therapy area makes evaluation of the relative effectiveness of EGFR TKIs using NMAs possible, our review has demonstrated that the clinical and statistical complexity of such analyses makes it challenging to achieve methodological consistency. The post hoc analysis of heterogeneity and inconsistency in the reconstructed archetype networks revealed that substantial significant heterogeneity and inconsistency were present in most networks and were dependent on network construction. Our review highlights two key methodological issues that we believe explain the differences in the findings of these analyses and should be given consideration in future research, as well as several additional considerations.

First, the inclusion of trials proved to be the most influential factor explaining fluctuations in the observed magnitude of treatment effects and rankings. In our review, we found that this was inconsistently applied among the publications. Even though valid causes can explain this difference – for instance, the increasing number of trials performed over time and the removal of trials not contributing to the network – these reasons alone do not fully explain the observed variance. Examples of noticeable differences between the studies include: the combination of first- and later-line patient populations given the accepted differences in outcome in these populations; the inclusion of retrospective studies; and the inclusion of trials with molecularly selected patients. Some studies restricted trials performed on an EGFR mutation-positive population, others to type of EGFR mutations, whereas one trial allowed a broader patient population to overcome challenges associated with lumping (see later). If molecular biomarkers are heterogeneously dispersed across the trials, an imbalance in effect modifiers across the network may be present. Failing to appropriately adjust for biomarker status across the trials may bias the network. Although some NMAs did use sensitivity analyses to evaluate the impact of trial inclusion, we believe that this is an area that could be explored further. In situations where the inclusion of one or more trials is not a clear-cut decision, we encourage the use of sensitivity analyses (with clearly stated rationale) by the exclusion of such trials to evaluate the impact of methodological decisions on NMA outcome.

Second, several researchers lumped treatment arms together to increase the statistical power of the network or to overcome network disconnection. Four out of the five identified network archetypes applied different levels of lumping, ranging from: no lumping; lumping all chemotherapies together; lumping pemetrexed- and nonpemetrexed-based regimens together; lumping first- and second-line chemotherapies together; and lumping chemotherapies and first-generation TKIs (gefitinib and erlotinib) together. We identified this to be a significant source of discordance in the findings. It should be noted that only networks that did not apply any lumping reported low heterogeneity; as such, lumping could explain up to 50% of unexplainable network variance. Although lumping treatments can provide a sound strategy when facing sparsity, as this may reduce the number of connections in a network, it requires strict assumptions of homogeneity that may be difficult to control using statistical methods [5]. Furthermore, specialized statistical methods that account for treatment class effects should be used; none of the authors identified in this review reported having used such models. It should also be noted that studies should generally not be combined, as is the case when lumping treatments, when the therapeutic agents: have a different mechanism of action; include different patient subtypes; or if the studies are conducted in a different setting [30]. The lumping of chemotherapies may be particularly problematic, especially where there are known differences in efficacy between agents [31], as this has the potential for networks that lump chemotherapies being at a high risk of bias. As such, the degree of lumping of therapies should be a key consideration when interpreting the results of analyses and should be avoided where possible.

Although a primary assumption for the conduct of NMAs is that clinical studies are sufficiently homogeneous, clinically and methodologically, to be quantitatively combined, the statistical challenges of addressing heterogeneity and inconsistency in these analyses make it especially important to consider these factors when constructing networks. A detailed discussion of differences in the design of studies in this area has been conducted previously [32] and is beyond the scope of our review. However, key factors that should be considered and are worth highlighting include use of investigator versus independent assessment of PFS, mutation type and important clinical characteristics that are known to impact on the outcome. Our review highlights that these factors are occasionally, but by no means universally, considered. We consider the use of independent-assessed PFS in NMAs to be an appropriate bias-minimizing approach, especially when including open-label studies in networks. This recommendation is based on evidence that open-label studies are at a higher risk of suffering from evaluation bias caused by investigator assessment than double-blinded trials [33,34]. Based on our review of the NMAs, only two authors reported that independent- and investigator-reported PFS were analyzed separately and assessed using sensitivity analyses [17,21]. Mutation type is also an essential consideration as the inclusion of patients with common EGFR mutations only (Del19/L858R) results in a more homogeneous patient population than studies that included patients with both common and uncommon mutations, as outcomes are known to be impacted by mutation type. Only one NMA considered the influence of uncommon mutations on outcome [21]. Finally, the presence of brain metastases is an important clinical factor that is known to impact treatment outcome [35] and, as such, consideration should be given to whether trials included in NMAs enrolled these patients. We identified only one paper that discussed heterogeneity in terms of patients selected for brain metastasis or metastasis in the CNS [17].

Several approaches have been suggested for evaluating incoherence in NMAs [36,37] and have been described elsewhere [38]. It should, however, be noted that tests for incoherence have low power and, therefore, may fail to detect incoherence as statistically significant even when it is present [39,40]. Conclusions should be drawn not just from consideration of statistical significance but by interpreting the range of values included in confidence intervals of the incoherence factors [38].

Although our research was focused on the treatment of EGFR mutation-positive NSCLC, personalized medicine is rapidly becoming a reality for many patients as cancer treatment advances and more targeted treatments become available. As such, comparative effectiveness research needs to carefully consider the implications of these factors in the design of comparative analyses in order to maintain the validity of findings. Areas of future research include the development of robust methods to summarize and synthesize evidence across treatment settings to help determine the optimal sequence of treatments rather than an isolated intervention. Although the discovery of the EGFR mutation has advanced the expected outcomes for patients, the treatment of NSCLC remains a complex area, with many factors potentially influencing the relative effectiveness of treatments, including the presence of brain metastases and tolerability profile; we did not individually evaluate these factors in this review, but they should be considered when interpreting findings. As more treatment options become available for patients with EGFR mutation-positive NSCLC, the ability to evaluate evidence from different trials becomes increasingly important. Combination approaches are increasingly being used in this population. As such, future analyses will also need to include methods for combining and controlling for the inclusion of monotherapy and combination trials.

In conclusion, we identified several issues with the published NMAs and advise caution when using NMAs to inform decision-making for patients with EGFR mutation-positive NSCLC. Although future analyses may reveal additional considerations for the conduct of these types of cross-trial comparisons, our review highlights several important considerations for future comparative effectiveness research.

All authors were responsible for study conception and design. C Samuelsen conducted data analysis and all authors contributed to drafting and revision of the manuscript.

The authors would like to thank N Hawkins, University of Glasgow, for his review and input of the manuscript.

C Samuelsen is a former employee of Boehringer Ingelheim International GmbH, Germany, and the owner of Outcomes Analytica AS. He has received funding from Boehringer Ingelheim. I Griebsch is an employee of Boehringer Ingelheim International GmbH, Germany, and is the author of one of the NMAs reviewed in this article. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

The authors would like to thank S Patel, Origin Health Ltd, for editorial assistance, funded by Boehringer Ingelheim International GmbH.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/