Methodological considerations for indirect treatment comparisons of HIV incidence for long-acting injectable pre-exposure prophylaxis

TEMs are variables that influence relative treatment effects. Unadjusted ITCs between two trials that share a common comparator (e.g., placebo or standard-of-care arm) are valid only if the trials are similar with respect to TEMs. When TEMs are similarly distributed across studies, a Bucher ITC can be conducted (Figure 1A) [32]. Network meta-analysis (NMA) methodology extends comparisons made in pairwise Bucher ITCs to a wider network of connected interventions (Figure 1B). Importantly, the reliability of Bucher ITCs and NMAs depends on the included trials being sufficiently similar with respect to TEMs. If there is a lack of similarity, relative differences in estimated treatment effects may be due to differences in characteristics considered to be TEMs across studies, rather than one intervention performing better or worse than another. Additionally, between-study imbalance in TEMs may be associated with differences in common comparator efficacy. If the control-arm outcome differs between studies (because TEMs differ), then the observed treatment–control contrasts may not be estimating the same underlying relative effect. Indirectly comparing these within-study contrasts may introduce bias as part of the difference reflects variation in control-arm performance driven by TEMs, not true differences between active treatments. Therefore, it is crucial that TEMs are identified and well-balanced across studies [33,34]. However, given the complex nature of TEMs for PrEP, these variables have not been well-defined. If left unbalanced, unadjusted comparisons may under- or overestimate the relative benefit of an intervention.

Figure 2 provides a hypothetical illustration of differences between study participant characteristics that are identified as TEMs, prognostic factors, or neither. TEMs are different from prognostic variables, which influence the absolute response in the same way across interventions. TEMs are specific to a disease area and are identified through consultation of relevant scientific literature, acquisition of clinical opinion, statistical investigation or any combination of these [33,35]. Figure 2 presents three hypothetical examples of the relationship between HIV incidence and varying participant characteristics in users of injectable or oral PrEP. In each of the three plots, the characteristic on the x-axis is either a TEM, a prognostic factor, or neither. For example, HIV incidence in users of injectable PrEP, relative to users of oral PrEP, varies with oral adherence – that is, oral adherence is effect-modifying. HIV incidence may vary with age for both injectable and oral PrEP users, but the effect of injectable PrEP relative to oral PrEP is always the same, suggesting that age is prognostic but not effect-modifying (in this hypothetical example). Where a characteristic is neither a TEM nor prognostic variable, HIV incidence will not vary for injectable PrEP nor oral PrEP, such as the percent of glycated hemoglobin (HbA1c) in Figure 2.

Follow-up time is an important consideration for rare, time-dependent outcomes in PrEP trials (e.g., HIV incidence), as event counts are meaningful relative to the time participants were at risk. In clinical trials, total person-time for a given treatment arm is calculated by summing each participant’s time from randomization (or first dose) to the earliest of infection, withdrawal/loss to follow-up, death, or study end. For example, if five participants contribute 0.9, 1.1, 1.0, 1.2 and 0.8 years, the total is 5.0 person-years (PY).

Comparable follow-up duration across trials is an important consideration for robust ITCs in an epidemiological context. Given the rarity of HIV acquisition in PrEP trials, accounting for differences in follow-up across studies is critical for producing reliable relative effect estimates. In particular, where exposure times vary, measures that explicitly incorporate event counts per unit of person-time, such as the incidence rate ratio (IRR) and incidence rate difference (IRD), provide relative treatment effect estimates that are less sensitive to unequal follow-up or differences in person-time between studies. Another common effect measure in PrEP trials is the hazard ratio (HR). Although HRs describe the relative instantaneous hazard rate, in the absence of proportionality of hazards when comparing across studies, they do not account for unequal person-time at risk or differences in follow-up duration.

As the IRR is a ratio-based measure, it describes how much higher (IRR > 1) or lower (IRR < 1) the incidence rate is in the intervention group relative to the reference (control) group. The IRD, on the other hand, describes the absolute difference in incidence rates between the intervention and reference groups, where negative values (IRD < 0) indicate lower incidence rates in the intervention group and positive (IRD > 0) higher rates [26,50]. These differences between the effect measures are particularly important within an ITC setting where the event count in the comparator arm may differ. A comparison of ratios based on differing absolute risk in the common comparator arm may result in the apparent treatment effect shifting, requiring caution in the interpretation of results. For example, if incidence rates in the common comparator arm across two studies were 25 events per 100 PY and 10 events per 100 PY, with the active intervention resulting in an absolute reduction of 5 events within both studies, the IRRs would be 0.8 and 0.5, respectively. Therefore, due to differences in incidence across common comparator arms, in contrast to the IRDs (which in this example would both produce a point estimate of -0.05) an ITC of IRRs would suggest differences in effectiveness across studies in terms of the point estimate. While both measures are equally valid, given the inherent differences in the way that the IRR and IRD express treatment effectiveness, it is the duty of the researcher to determine whether it is of clinical relevance to select relative or absolute treatment effects as the primary effect measure [26]. Within the context of HIV, the rare nature of the outcome means particular consideration should be given to the clinical importance of absolute differences in event counts. The ratio of the incidence rates (the IRR ITC estimate) will be more sensitive to changes of just one or two additional events (which may give the impression of inflated differences) than absolute differences (the IRD ITC estimate). In general, the selected effect measure should be presented alongside the incidence rates observed within the intervention and reference arms.

Regardless of the measure being presented, length of follow-up time must be similar between studies for such indirect comparisons to be considered feasible. For example, the Partners PrEP study (Baeten et al. 2012) was conducted in a population of heterosexual men and women with an ART-naive HIV-infected partner in Kenya and Uganda. The median (interquartile range [IQR]) follow-up of the primary data-cut for the study was 23 (16–28) months. In the TDF and F/TDF arms of the study, the incidence rates of HIV infection were 0.65 and 0.50 cases per 100 PY, respectively [43]. In 2014, a follow-up to the Partners PrEP study was published (Baeten et al. 2014), where the median (IQR) follow-up was 35.9 (30–36) months. In the TDF arm, the incidence rate increased to 0.71, while in the F/TDF arm, the incidence rate remained similar at 0.48 cases per 100 PY [42]. While more stable in the F/TDF arm of the study, the increased incidence rate in the TDF arm demonstrates the potential for differences in incidence with variable follow-up duration. Differences in efficacy may further be modulated by differences in adherence to oral PrEP or background HIV incidence, which may compound the effect introduced by differences in follow-up.

In the context of ITCs of oral and injectable PrEP, reliable analyses will need to closely consider follow-up, given the observed differences across trials and the differences in cumulative incidence between oral and injectable interventions. For time-dependent rare events, such as HIV incidence, longer follow-up inherently accrues more person-time at risk, allowing additional events that shorter trials may not capture. In an ITC of two such trials, the between-study estimate can be driven by differences in observed event rates that reflect unequal time at risk rather than true differences in relative effectiveness. For example, in HPTN 084, the cumulative incidence of the cabotegravir arm at one year was approximately 0.002, while that of the F/TDF arm was approximately 0.016. At two years, the cumulative incidence of cabotegravir was approximately 0.003 and F/TDF approximately 0.057. And at almost three years, approximately 0.003 and 0.072. As demonstrated in Figure 5, the cumulative incidence curve for oral PrEP is steeper than for injectable PrEP [14]. Similar trends were seen in HPTN 083, with one-year, two-year and three-year cumulative incidence rates for cabotegravir and F/TDF, respectively, approximately: 0.004 and 0.012, 0.007 and 0.025, and 0.009 and 0.056 [13] (Figure 5).

In an ITC or NMA, when the outcome of interest is very rare, it may be difficult or impossible to robustly capture the effect and uncertainty of a treatment or comparison [27,51]. For example, due to the high efficacy of injectable PrEP, low event counts were observed, with 0, 2, 4 and 13 HIV infections across PURPOSE 1, PURPOSE 2, HPTN 084 and HPTN 083, respectively [10–13]. When studies report very few or no events, called zero-event cells, statistical measures of effect (such as IRRs or odds ratios) and their associated uncertainty calculations may become undefined, rendering it difficult or impossible to determine whether a treatment effect exists [51]. Current methodological guidance for zero-event cells primarily relates to the zero-cell correction in the context of (N)MA, with a common practice involving adding a continuity correction of 0.5 to each aggregate count value [27]. More flexible approaches for zero-cell continuity corrections also exist; however, these may introduce subjectivity, given that the choice of the correction impacts the results of the analysis [51].

In the case of zero events, the traditional Bucher approach to calculation of CIs fails. When zero-event cells occur, the standard errors and CIs of the Bucher ITC cannot be calculated without additional assumptions. Bayesian approaches may also be beneficial in overcoming such challenges [27].

The presence of zero or close to zero events in injectable arms of PrEP studies additionally introduces complications for population adjustment. Population-adjusted ITCs are recommended in the National Institute for Health and Care Excellence (NICE) Decision Support Unit (DSU) Technical Support Document 18 and the European Union Joint Clinical Assessment (EU JCA) evidence synthesis guidelines where there are between-trial differences in TEMs [33,34]. However, regression-based population adjustment methods (such as multilevel-network meta-regression or simulated treatment comparison) are challenged when event count is low or become impossible in the presence of zero events. And while production of propensity score weights may be feasible (e.g., via matching-adjusted indirect comparison), application of such weights to low event counts may have limited benefit.

Additionally, where few events occur, large uncertainty will lead to challenges in establishing statistically significant differences between interventions [52]. It is therefore especially important within this context to consider multiple dimensions beyond efficacy and safety outcomes when evaluating competing interventions, including, for example, mode and frequency of administration, persistence, cost-benefit, quality of life, PrEP user preference, and potential for increased PrEP uptake among underserved communities. Where possible, these dimensions should be considered alongside conventional indirect comparisons as part of a holistic assessment of the additional value that novel PrEP modalities provide.