A criterion-based approach to systematic and transparent comparative effectiveness: a case study in psoriatic arthritis

To assess potential challenges in satisfying transitivity within a network of apremilast and other PsA therapies, several literature searches were performed, as summarized in Figure 1. Three distinct, targeted literature searches were conducted to source transitivity discussion, definitions, examples and stated criteria in peer-reviewed literature; any discussion of transitivity (including effect modification) in PsA studies and the broader rheumatology literature; and any assessment of transitivity in current NMAs comparing PsA clinical trials. From these targeted searches, any existing methods for evaluating transitivity, including clinical and patient factors contributing to insufficient transitivity, were organized into five general guidelines of transitivity. Next, a systematic literature search, following the Cochrane guidelines for systematic review, was performed to identify PsA studies to be included in the proposed NMA. Clinical and patient factors extracted from the selected studies were applied to the sourced transitivity descriptions to create more specific, centralized transitivity criteria. From these formalized transitivity criteria, the appropriateness of including the selected PsA studies in the NMA was reported.

The systematic literature search identified relevant studies in the databases Medline via PubMed, EMBASE, the Cochrane Central Register of Controlled Trials and clinicaltrials.gov. Eligible studies were RCTs of current PsA treatments published up to 1 March 2015. Treatments included in the search were MTX; the biologics adalimumab, certolizumab pegol, etanercept, golimumab, infliximab, secukinumab and ustekinumab; and the PDE4 inhibitor apremilast.

The Cochrane guidelines were followed during the selection of full-text articles [22]. At least two reviewers selected the articles from a title and abstract review, and a third reviewer was enlisted for any disagreements. English-language studies needed to report at least one of the key efficacy or safety end points and include separate reporting of patients who were at least 18 years of age. Exclusion criteria eliminated nonrandomized clinical studies, single-arm studies, studies without a control arm (placebo) and studies without full-text publications. A PRISMA flow diagram for study selection was created (Supplementary Figure 2).

To evaluate the RCTs against the newly centralized transitivity criteria, the following study characteristics were extracted: inclusion criteria, exclusion criteria, drug comparator, participant age, sex, duration of PsA disease in years, mean tender joint count at study baseline, prior use of DMARDs or biologics, time of efficacy assessment/crossover design, concomitant DMARD or MTX use and placebo response rates based on the American College of Rheumatology response criteria (ACR20 and ACR50). Also, it was recorded if the study authors evaluated each of these variables for potential treatment effect modification or prognostic significance and what their methods were for addressing effect modification and prognostic factors.

From the targeted literature searches, it was found that the term transitivity was associated with similarity assumption or used interchangeably with comparability [4,23–27]. The similarity assumption was used to describe both finding a true (i.e., similar) treatment effect between two interventions across multiple studies as well as used to describe a network having similar characteristics between studies. However, for the similarity assumption and discussions of comparability, no formal definition or criteria to assess appropriateness of an ITC were found. Furthermore, applying concepts of similarity or comparability alone can lead to a narrower understanding that all study and patient characteristics must be mostly uniform to assume transitivity. In fact, valid ITCs can be performed as long as there is similar distribution of the mutual effect modifiers between studies [1,5,12,27]. From this literature, effect modification is defined as the presence of one or more variables that changes the effect estimate for treatment on a given scale of measure [28]. Specifically, the alteration in treatment effect is different within different subgroups of the same patient characteristic, which may be statistically demonstrated through a test for interaction. Statistical interaction is considered if the combined effect of two factors, such as PsA treatment and PsA disease subtype, is greater than would be expected by the sum or multiplication of their independent effects [29]. Across studies, similar distribution of effect modifiers can be constructed through population-adjustment methods, including propensity score weighting and simulated treatment comparison, if at least one individual, patient-level dataset is available for adjustment [28]. However, addressing effect modification is only one of the five different guiding principles found from different sources during the targeted literature searches that describe transitivity [1,5,30]. Ideas related to transitivity were found from different literary sources and organized below into the five general guidelines.

Search results are organized into five guiding themes: between studies, there exists an anchor treatment, such as a placebo, that is similar in all studies being compared; the selection of studies for the network should not be solely motivated by an expected outcome, or results can be subject to bias; there are true comparative treatment effects between two interventions that can be measured across multiple studies, so that a network of multiple studies is justified; similar distribution of effect modifiers is necessary between studies, such as proportion of older versus younger patients, or poorer versus fitter baseline performance status; and any of the treatments could be appropriately compared in a randomized study based on similar disease indication.

After title and abstract review, 82 publications were selected for full-text review. Some studies were associated with more than one publication, which resulted in the identification of 25 unique clinical trials (Supplementary Figure 2). Applying the inclusion and exclusion criteria and designating placebo as the comparator for a potential NMA, we were able to link 18 unique RCTs (Table 1) [15,16,31–46] for apremilast and seven active comparators (adalimumab, certolizumab pegol, etanercept, golimumab, infliximab, secukinumab or ustekinumab). Of note, two studies of MTX monotherapy were excluded because they lacked the required clinical response variables [47,48], and one apremilast abstract from 2014 (PALACE 4) was excluded for a similar lack of needed extraction variables [49]. During analysis, additional results from McInnes et al. for secukinumab and two follow-up studies for apremilast [15,16,31] were discovered and added. A graphic of the final network is provided in Supplementary Figure 1.

Extraction of the patient and clinical trial characteristics of the selected PsA RCTs helped to identify specific variables that should be evaluated for transitivity across studies. This identification supported the development of the following transitivity criteria, with the aim of providing newly formalized, specific guidance. These centralized transitivity criteria were enlisted to determine if an ITC of the linked PsA RCTs could be appropriately performed. The seven transitivity criteria follow:

Among the selected studies, sufficient pre-identified study variables were available to perform cross-study comparisons (Table 2) [15,16,31–46]. The apremilast trials (PALACE 1 [33], PALACE 2 [15] and PALACE 3 [16]) were considered the reference studies used to evaluate transitivity in comparison with the remaining studies in the network. Patients from these studies were permitted past exposure to biologics and DMARDS and could have concomitant use of MTX, sulfasalazine, leflunomide, low dose oral corticosteroid or nonsteroidal anti-inflammatory drugs during the studies. Further study data that supported the identification of transitivity violations are summarized in Table 2.

Data extraction revealed potential sources of between-study clinical and methodological heterogeneity, and most study authors did not report statistical evaluation of treatment effect modification within their studies. Across the network, transitivity violations related to all seven tenets were found, and major themes included clinical trial design (crossover, timing of efficacy assessment and follow-up) and population differences (baseline disease characteristics, prior and concomitant medications and placebo response rates). Table 3 summarizes the most evident transitivity violations (or none) for each included study [15,16,31–46].

Study differences in data reported that contributed to transitivity violations were especially notable when comparing apremilast with the biologic studies. For instance, the apremilast studies included treatment crossover (early escape), enabling nonresponders in the placebo groups to be re-randomized [15,16,33,34]. However, efficacy end points occurred prior to re-randomization, which limited a potential source of measurement bias. Of the biologic studies, six did not report a crossover design (Transitivity Criteria #2); however, two studies permitted early escape to investigational treatment after initial treatment response measurements. Of the eight biologic studies that did have patient crossover, different methods were used to account for missing variables, making them difficult to compare. Of note, it is possible that crossover or re-randomization was performed but not explicitly reported in the article. As shown in Table 2, duration of PsA disease at baseline ranged from 3 years to nearly 12 years (Transitivity Criteria #3), while the literature demonstrates that outcomes are less favorable with a longer PsA symptom duration or time to diagnosis [59–61]. Across studies, mean tender joint counts at baseline ranged from 12 to 29 (Transitivity Criteria #4). Almost all participants in three of the four apremilast studies (PALACE 1, 2 and 3 [15,16,33] but not PSA-001 [34]) had prior conventional DMARD use, while 43–71% in all four apremilast studies had concomitant DMARD use [15,16,33,34]. Six of the nonapremilast studies also distinguished between proportions of patients with prior (59–100%) and concomitant (20–90%) use, while it was unclear if the remaining studies distinguished between prior and concomitant use (Transitivity Criteria #5). Concomitant treatments were similarly heterogenous (e.g., MTX use 20–90%) across all studies (Transitivity Criteria #6), as was placebo response (9–44%), measured as ACR20 (Transitivity Criteria #7). Furthermore, many studies did not report prior or concomitant medication use. In addition, there was variability in the years the studies were published and the countries from which patients were recruited. While a specific measured variable representing access issues was not available, year of publication or country of study can convey different accessibilities based on the treatment's availability in a healthcare system at that time. These factors may have resulted in differences such as, but not limited to, disease severity, prior therapies received and concomitant medications, making it difficult to compare results across studies. In summary, different inclusion and exclusion criteria may have been used across studies.

Within most of the extracted RCTs, authors enlisted various methods to minimize baseline imbalances in patient, disease and treatment characteristics (Table 4) [15,16,31–46]. Although none of the study authors used the terms ‘prognostic factor’ or ‘effect modifier’ when reporting stratification, 15 of the 18 RCTs specified stratification during randomization. Participants were stratified according to baseline use of DMARDs [15,16,32–35,37–42], prior exposure to biologics [31,36,43] or no stratification factor was reported [44–46]. Of the studies not reporting stratification, all but one reported performing subgroup analyses to determine any difference in treatment efficacy by patient characteristic [44,46].

During analyses, nine studies [15,16,33–35,37–40], as identified in Table 4, specified testing separate treatment effects by baseline MTX or DMARD use with Mantel–Haenszel stratification or analysis of variance regression, including the apremilast studies. However, posttrial, only two studies specifically addressed (and refuted) the significance of a statistical interaction term [34,44]. Of the remaining 16 studies that were included, there were varying levels of reporting rigor for subgroup analyses addressing more general treatment-modifying effects [15,16,31–33,35–43,45,46]. For at least one baseline variable, four studies provided statistical refutation of effect modification [16,37–39], one study [34] reported similar numeric proportions of responders between subgroups, five studies provided similar evaluation plus graphical evidence [32–36] and five studies [15,40,41,43,45] stated in the text that differences in treatment effects by subgroups were not evident. One study [46] did not report analyses for potential baseline imbalances. Some study authors, as noted in Table 4, cited smaller patient populations as challenging their ability to statistically test subgroups [34,42,44].

While most studies reported stratification for one patient characteristic, effect modification was not refuted for remaining variables. As previously noted, effect modification is detected on a specific scale (e.g., additive treatment effect); however, none of the studies specified the models used to estimate efficacy. Some studies [31,33,36,44], as described in Table 4, conveyed potentially higher treatment efficacy in patients unexposed to certain prior therapies. For example, Kavanaugh et al. [33] stratified participants by baseline use of DMARDs and graphically depicted potential effect modification for past exposure to biologic agents. The authors noted relatively lower clinical responses in biologic-experienced patients when compared with biologic-naive patients receiving apremilast or placebo [33]. Similarly, Mease et al. [36] stratified participants by prior TNF inhibitor exposure and reported somewhat dissimilar proportions of clinical response by concomitant DMARDs, which potentially demonstrated effect modification (Supplementary Material). Most study authors did not statistically evaluate treatment-effect modification within their studies, making it difficult to determine within-study, treatment-modifying biases. On the other hand, if an effect modifier was identified at the trial level, the distribution of this variable could be assessed across the network.

Separate from the systematic literature review, our targeted literature search for developing transitivity criteria found studies and published guidelines addressing effect modification within PsA and rheumatoid arthritis (RA) disease studies. Actual detection of effect modification was demonstrated in three RA studies, and effect modifiers included age, swollen joint count, prior DMARD use, concomitant DMARD use, concomitant MTX use and baseline risk (placebo effect) [21,26,29]. Additionally, Christensen et al. [21] potentially detected a modification of the treatment effect based on the disease duration but were unable to demonstrate its independent relative effects from prior DMARD use. Additional PsA, RA and NMA quality guidelines discuss and encourage testing for potential sources of effect modification [5] such as patient comorbidities [51,52] and gender (Consensus Working Party 2013); however, none of the extracted RCTs tested against these potential biases. Furthermore, two full publication NMAs that included apremilast cited variations in study populations and a lack of covariate adjustment in the meta-regressions as limitations; however, potential biases related to transitivity were not measured [18,20]. Within our network, the PALACE 4 abstract was excluded for lack of the reporting variables traditionally available for full publications, while both full publication NMAs included the PALACE 4 abstract in their analysis. Another NMA, available as an abstract, aimed to test treatment efficacy of several treatments, including apremilast, according to biologic-exposed and biologic-naive subgroups [19]. However, due to limited data, trends within the biologic-exposed subgroup could not be developed. Another abstract reporting similar NMA methodology also performed separate analyses by biologic status, although no other patient characteristics were identified [62].

Transitivity considerations are limited by the ability to identify sources of heterogeneity based on individual trial reporting. Often there is potential for unobserved sources of bias or unreported study variables that limit valid comparisons between studies [1]. Particularly in the case of effect modifiers, unknown imbalances can introduce biases to both within- and between-study effect estimates. For example, in the extracted PsA RCTs, year of publication or country of study can convey different accessibilities based on the treatment's availability in a healthcare system at that time. However, a specific measured variable representing access issues was not available. Additionally, comorbidity distributions were not part of the study extraction list because they are often unreported in PsA studies or only severe comorbidities are reported as exclusion criteria. Nevertheless, clinical or more conceptual rather than quantitative judgment can support decision-making when there is no covariate measure available (e.g., regarding differences in study eligibility or disease indication). The greater awareness and transparent reporting of these study characteristics and methodologies will help support robust ITC to inform clinicians, patients and healthcare decision-makers when deciding courses of treatment among the large array of therapies for a given indication.