Matching-adjusted indirect comparison via a polynomial-based non-linear optimization method

A range of statistical techniques have been developed that attempt to address these challenges, such as extensions of network meta-analysis [2], simulated treatment comparison [3] and MAIC [4,5]. MAIC methodology typically aims to reweight IPD from the experimental treatment in order to match the distribution at the aggregate level (typically the mean and standard deviation [SD]) of baseline factors in the comparator arm of a target study. Once a set of IPD weights has been obtained, weighted analyses of trial outcomes on the experimental arm are then conducted and the results compared with the aggregate-level study outcomes of the target study.

Various MAIC approaches have been developed [6–10], but it is the approach first published by Signorovitch et al. [6] that has been adopted by reimbursement authorities [5]. This approach is now virtually synonymous with the concept of MAIC for most researchers. It is notable that the approach of Signorovitch has seen little improvement since its original publication. This is arguably due to its considerable success in matching target measures of location and spread across a wide range of practical and theoretical settings [5,11–14]. Entropy balancing [8] is another method-of-moments-based technique that has also shown its usefulness [15]. However, recent work [16] would suggest that entropy balancing is methodologically very similar to the Signorovitch approach. An alternative method-of-moments-based approach has recently been developed by Jackson et al. [7] and this appears to improve upon the Signorovitch approach. Simpler approaches to MAIC that reweight IPD using post-stratification techniques have also been developed [10,17]. These approaches have strong parallels with well-established survey sampling techniques such as raking [18], where sample survey results are reweighted in accordance with overall population demographics. These post-stratification approaches to MAIC, while easily applied and readily understood, are limited to the matching of categorical baseline patient characteristics – a limitation which has likely restricted their application within comparative effectiveness research.

The Signorovitch, entropy-balancing and Jackson et al. MAIC approaches are based on method-of-moments, with the aim of matching IPD to aggregate-level data (ALD) on the central moments of the baseline characteristic distributions – typically the first (mean) and second (variance) moments. As such, these methods are somewhat restricted in their ability to match ALD on non-central moments such as the median and other percentile-based summary measures, for example, interquartile range (IQR). Researchers will typically substitute the median for the mean in these instances. However, while this interchange is entirely justified when dealing with symmetric distributions, it could potentially lead to poor matching if the underlying distributions are skewed.

An important limitation of current MAIC methods is that none formally account for the trade-off between the accuracy of the match and the precision that the subsequent weighted analyses will achieve. Matching accuracy can be easily assessed by comparing the weighted baseline characteristics of the IPD with the baseline characteristics of the ALD. Precision is commonly assessed using the (approximate) effective sample size (ESS) [6,19], calculated as $ESS=\frac{{\left({\sum }_{i=1}^n{w}_i\right)}^2}{{\sum }_{i=1}^n{w}_i^2}$, where the i^th patient has an estimated weight w_i. Note that the sum of the weights is constrained to be n (the number of patients) and all weights are non-negative. The effective sample size (ESS) is at a maximum when all w_i are equal to 1 (ESS = n), that is, each patient weights equally in the analysis and there is no loss of information due to reweighting.

The set of estimated weights that achieve a match between a weighted IPD summary statistic against an ALD summary statistic is certainly not unique. For example, take a very simple case where four patients have baseline responses of 2, 3, 4 and 7. Suppose a researcher wished to reweight this IPD in order to match a baseline ALD target of a mean of 4.3 and an SD of 1.8. How are 4 weights (w₁, w₂, w₃, w₄) to be chosen? One approximate solution exists with the following set of weights: w₁ = 0.19, w₂ = 1.23, w₃ = 1.66 and w₄ = 0.92, leading to an ESS of 3.1. However, another solution exists with an alternative set of weights: w₁ = 0.57, w₂ = 0.17, w₃ = 2.38 and w₄ = 0.88, leading to an ESS of 2.4. The latter solution is to be avoided, as it leads to considerable loss of information and weights the ‘third’ patient very highly.

Information loss within a MAIC is to be expected. Indeed, a high ESS in relation to the observed sample size in the IPD would make the application of MAIC somewhat redundant – an unweighted analysis would provide virtually the same result. Unfortunately, it is not uncommon for MAICs to estimate ESSs at relatively small proportions of the original sample size [11]. MAIC approaches should ideally aim to match baseline characteristics within certain tolerances while retaining as much information as possible. The use of matching tolerances, and the trade-off with information retention, has been previously described [20], but much current MAIC methodology does not specifically reflect this tension.

In tandem with the problem of low ESSs are patients with large estimated weights (also called a lack of weight ‘stability’ [20]). These patients pose additional challenges, especially if they are outliers and exert high influence/leverage in the outcome analyses of the weighted IPD. Propensity-score-based methods allow for the reduction in the influence of patients with large weights via techniques such as stratification or trimming. Solutions to this problem have yet to be fully integrated within an MAIC framework, though estimated weights can be trimmed post hoc[8].

The use of polynomial functions within MAIC research was introduced by Regnier et al. [21] as part of an examination of a range of ITC approaches within an ophthalmological setting. The underlying basis for their MAIC approach was to shrink, stretch and warp distributions of baseline characteristics such that the summary statistics of the altered distribution matched the summary statistics of the target distribution. Their weight optimization MAIC method was subsequently modified, extended and formalised as part of this current investigation (Fourth-order polynomial-modified matching-adjusted indirect comparison [polyMAIC]).

The polyMAIC approach proceeds via the following steps:

It is worth noting the following:

The matching tolerances (i.e., the ε_jk in step 5) should be chosen based on a number of potential considerations. For example, they could reflect detection thresholds, minimally important clinical differences or some fraction of standardised differences. It might be prudent to match some baseline characteristics more closely than others, depending on their relative predictive and/or treatment-modifying importance. Furthermore, matching on measures of central location (e.g., means, medians) might be more critical than matching measures of spread (e.g., variances, ranges).

IPD were simulated to compare the relative performance of the polyMAIC approach with that of Signorovitch. Table 1 summarises the distributions of the five baseline characteristics. A variety of distribution types were used to ensure a suitable mix of continuous, categorical and count data. Note that the pain scale data was rounded to the nearest whole number to reflect a typical Likert-type scale. Four different pseudo-IPD scenarios were developed, with an increasing number of baseline characteristics included in each (Table 2). Correlations between baseline characteristics are arguably present in most clinical studies, for example, patient age is often positively correlated with time since diagnosis, which in turn can be correlated with disease severity. Therefore, for each scenario, we incorporated correlations between each of the characteristics (Table 2) via simulations of multivariate normal distributions. The data was then transformed to obtain the desired baseline distribution types. The data that supports the findings of this study is available from the corresponding author upon request.

Note that the four IPD-generating scenarios were constructed to make the MAIC process increasingly challenging, with a greater number of baseline characteristics, smaller sample sizes, higher correlations between characteristics and ALD targets further away from the unweighted IPD summary statistics. Due to the theoretical limitations of the Signorovitch method in explicitly incorporating non-central moments (e.g., medians) into the matching algorithm, ALD targets were specified as means, SDs and proportions. ‘Narrow’ and ‘wide’ matching tolerance limits and pre-specified maximum IPD weights were used when applying the polyMAIC approach (see Table 3). We set the narrow polyMAIC tolerances to be well below the typical rounding errors associated with published ALD summary statistics. The ability of each MAIC method to effectively reweight the IPD was assessed via the proximity of the weighted IPD summary statistics to the ALD target summary statistics. Maximum weights and ESSs were also calculated and compared. The relationship between the IPD weights obtained from the two MAIC approaches was assessed graphically and via Spearman rank-order correlations.

Multivariate normal data was simulated using the PROC SIMNORMAL procedure in SAS/STAT [22]. The Signorovitch MAIC approach was applied using modified R code as per NICE Technical Support Document [4,23]. The polyMAIC approach was applied using the PROC OPTMODEL procedure in SAS/OR [22]. Examples of the polyMAIC SAS code are included in Supplementary Material 1.

Matching performance was first compared between the Signorovitch and polyMAIC methods, where the latter MAIC method utilised narrow matching tolerances. Both methods achieved good agreement between the summary statistics of their weighted IPD when compared with the ALD targets. The polyMAIC approach showed marginal relative improvements in the ESS over the Signorovitch method (Table 3), ranging from around 1% (325.9 vs 323.7, scenario A) to 9% (82.1 vs 75.5, scenario C). Maximum weights were quite similar between the two MAIC approaches, with differences (polyMAIC vs Signorovitch) ranging from 0.5 higher (3.0 vs 2.5, scenario A) to 2.9 lower (8.1 vs 11.0, scenario C). Note that maximum allowable weights were not pre-specified in the polyMAIC approach when narrow tolerances were used.

The use of wider matching tolerances in the polyMAIC approach led to marked improvements in ESS and maximal weights, albeit at the cost of a reduction in matching performance. Increases in ESS over the Signorovitch approach ranged from 15% (371.1 vs 323.7, scenario A) to 192% (41.4 vs 14.2, scenario D). Mirroring the trend observed using narrow tolerances, improvements in ESS from the polyMAIC approach appeared to be larger with increased matching complexity. Maximum weights varied considerably across the two MAIC approaches, where differences (polyMAIC vs Signorovitch) ranged from 0.1 lower (2.4 vs 2.5, scenario A) to 10.0 lower (9.4 vs 19.4, scenario D). A maximum pre-specified weight of 5 was applied in the polyMAIC approach for scenarios A, B and C – the maximum IPD weights ‘obtained’ were 2.4, 2.9 and 5.0 (respectively). In scenario D the polyMAIC approach could not identify a set of IPD weights with a pre-specified maximum weight of 5, so this was subsequently relaxed to 10 (a maximum weight of 9.4 was obtained).

It is worth noting that the ESSs obtained from either of the two MAIC methods ranged between 14% and 74% of the original IPD sample sizes, highlighting a considerable amount of information loss.

Figures 1 & 2 present the Spearman correlation coefficients of the sets of IPD weights between the two MAIC methods for each of the four scenarios. Unsurprisingly, given that these methods are trying to achieve similar goals, the two methods gave rise to moderately-to-highly positively correlated weights (correlations ranged from 0.50 to 0.99). Correlations tended to weaken as the complexity of the matching task increased but remained similar between the two methods regardless of the polyMAIC matching tolerances. The fourth-order polynomial coefficients estimated as part of the polyMAIC methodology are presented in Table 4.

Many limitations of the alternative MAIC approaches also apply here. PolyMAIC cannot match against a baseline characteristic that is not reported at the aggregate level – this is especially a problem with unanchored ITCs [12,13]. Limited overlap in the distributions of important baseline characteristics between IPD and ALD will invalidate the use of MAIC. Indeed, specifying small tolerances in cases where there is poor overlap between the IPD and ALD distributions could result in non-convergence of any MAIC reweighting algorithm.

The polyMAIC approach does not formally combine the relative importance of the matching closeness against the ESS maximisation. Lower levels of matching (i.e., use of higher tolerances) will generally lead to higher ESSs. However, it is not clear what the optimal trade-off between these two objectives should be. For example, should researchers specify tolerances such that they achieve an ESS of at least, say, 20% of the original IPD sample size? What characteristics should they be willing to match less closely against, or potentially even drop from consideration, to achieve this? This sort of trade-off might potentially vary by case and therefore preclude any standard solution.

Potential improvement with the polyMAIC approach lies with its use of fourth-order polynomials to alter the baseline distributions. Even more flexible functions such as fifth- (or higher-)order polynomials, splines, or fractional polynomials could potentially be employed [20]. However, while locally ‘extreme’ complex functions might have statistical advantages in terms of matching closeness and maximising ESS, they would arguably have little clinical justification. Two patients with similar baseline characteristics should arguably provide a similar weight in an MAIC-based analysis.

We conducted comparisons of MAIC methodology upon a limited set of data simulations. Despite our efforts to replicate typical conditions faced by researchers when conducting MAICs, given the number of scenarios tested we cannot generalise these results to conclude that the polyMAIC method will be universally superior to that of Signorovitch. We did not conduct a full factorial design; rather each scenario altered multiple factors (sample size, number of characteristics, correlations between characteristics, ALD target ‘distances’ and matching tolerances), meaning that we are not able to definitively identify what drives differences between the two methods. Other MAIC approaches could have been included in our comparisons, in particular entropy balancing and the approach developed by Jackson et al. [7]. However, given that the Signorovitch approach is virtually an industry standard, we felt justified in our exclusion of other MAIC methodologies.

 Regulatory and reimbursement authorities typically seek data from head-to-head randomised controlled trials when comparing treatments. However, this level of comparative effectiveness data is often unavailable at the time of their assessment, especially when new treatments have only recently become available. MAIC has proven over the past decade to be an extremely useful statistical tool in circumventing this lack of head-to-head data. Improvements in MAIC methodology should lead to more reliable indirect treatment comparisons and, ultimately, better decision making.