The mixed randomized trial: combining randomized, pragmatic and observational clinical trial designs

Patients would first be randomized to one of the three trial arms. Those randomized to the RCT or PT arms would then be randomized again, this time to study treatment (e.g., experimental, placebo, SOC or active comparators including best available therapy). This ‘twin’ randomization would be conducted similarly to that performed in a typical standalone RCT. Indeed, the randomization of patients to trial arms is broadly similar to the use of a stratification or blocking factor.

Those randomized to the OT arm would receive treatment as per local clinical practice and at the discretion of the physician/patient (Figure 2). Randomization would ideally be performed under blinded conditions; however, full blinding at the trial level would be problematic due to the unique design elements of each arm (e.g., fixed visit schedule in the RCT arm). Study investigators would also need to know if a patient was randomized to the OT arm, so that they could allocate the patient to a suitable active treatment (e.g., experimental, active comparator[s], SOC). Efforts should be made to maintain blinding at the trial arm level, such as with the use of blinded assessors. This is a relatively common approach in RCTs where unblinding of patients or treating physicians is a potential risk. The use of objective end points (e.g., blood/biomarker concentrations, ambulatory tests, pulmonary function tests, relapses, death) could also help to minimize potential bias. The comparison of interest within an MRT lies between treatments rather than trial arms, and thus, it is not wholly necessary that investigators and patients remain blinded to trial arm allocation. Nevertheless, reducing treatment preference, and resulting bias, demands keeping both trial arms and treatment groups blinded where possible. Blinding to treatment group within each trial arm would in any case be performed as standard practice.

The randomization ratio of patients to each trial arm could be chosen as a simple balanced assignment (i.e., 1:1:1), but there are no obvious barriers to using other ratios with the MRT design. It could, for example, be beneficial to allocate patients to a 1:3:4 (RCT:PT:OT) ratio. The optimal ratio should consider factors such as the balance between the amount of currently available controlled and uncontrolled data. With the multi-tiered MRT structure, that is with three simultaneous and independently running trial arms, the randomization ratio could also be adapted (e.g., from 1:1:1 to 1:3:4) at planned time points. This would reflect the increasing knowledge of the drug’s efficacy and safety profile by proportionally adding more patients to the PT and OT arms. Furthermore, there may be a need to maximize collection of comparative effectiveness data from important comparators (e.g., in cases where a reimbursement authority has required data on a particular SOC or best available therapy).

Alternatively, the patient randomization ratio can allow for appropriately minimizing the number of patients in the PT and OT arms (in the extreme example, with 1:0:0) where a safety risk is unacceptably high with an otherwise unapproved drug. Such a ratio can, for example, be used at the beginning of an MRT so as to stagger the commencement of the RCT, PT and OT arms, over time (Figure 3). The timing of the initiation of the PT and OT arms would thereby reflect the current state of the available controlled and uncontrolled data, and would commence following some form of MA. For example, it might be reasonable to start with a 5:1:0 ratio, and then subsequently to progress toward 1:3:4, over the course of months or years.

Each of the trial arms would be conducted as per standard practice. Patients in the RCT arm would undergo a (typically) wide range of clinical assessments at scheduled visits. Despite being similar to the RCT, the PT arm would allow patients to undergo a less rigorous schedule, choosing to focus on collection of key safety and efficacy data. Patients randomized to the OT arm would also undergo a similar, or less frequent, follow-up regimen as per the patients randomized to PT, and would be allocated treatment according to local treatment guidelines or physician/patient preference. Under certain conditions, this could include the active (experimental) therapy, although it is unlikely that this treatment would be available at MRT onset. It would undoubtedly be preferable to have the experimental treatment group available as soon as possible in the OT. Therefore, in the cases where the RCT, PT and OT arms were staggered over time, there should be a plan for an initial, conditional MA (Figure 3) [23].

Careful and thoughtful planning of the multi-tiered MRT structure is paramount. The MRT’s primary efficacy end point should be chosen to best match the needs of both regulators and payers – optimally following joint scientific advice. In any case, the MRT should be statistically powered on the treatment comparison of most clinical interest. Sample size calculations could be more challenging than for a standard RCT, especially if inter-trial arm variability is considered with respect to the expected treatment effect size. Nevertheless, a typical MRT would enroll substantially more patients than a standard RCT and would thereby address this issue.

Efficacy and safety data from each of the three MRT trial arms would be combined in a formal, pre-planned, statistically appropriate manner in order to support regulatory approval. Statistical adjustment for the effect of the trial arm should be made via the inclusion of a trial arm term (e.g., as a main effect) in the primary-end point analysis. A single, prospective statistical analysis plan, with such issues carefully described, should be agreed before any unblinding. Interim analyses may also be desirable before initiating the PT or OT trials arms, or for adaptions of the randomized treatment allocation ratios within the RCT and PT arms. For example, a placebo arm might be dropped if there was sufficient interim data to indicate superior efficacy of the experimental arm. Indeed, most common Phase II or III trial adaptions (e.g., ‘play-the-winner’ design) can be implemented within the MRT framework [24]. Health economics data (e.g., quality of life) would also typically be collected in some, if not all, trial arms of the MRT in order to support reimbursement submissions.

The MRT would be most valuable when conducted after initial Phase III development, once the safety, efficacy and tolerability of the investigational drug has been tested within a controlled setting. The MRT could then be introduced in Phase III (e.g., replacing a second pivotal Phase III trial) with the aim of achieving full MA. To this end, a staggered approach could be made (Figure 3). For example, interim data from the RCT arm could be used to obtain a limited (conditional) MA. This could then be formally combined with data from the PT using standard analytical techniques in order to obtain wider regulatory approval and conditional reimbursement via an adaptive licensing procedure. Data from the completed PT and OT arms could then provide the necessary real-world comparative effectiveness data to allow for full reimbursement. The optimal timing of the MRT within the overall clinical development and reimbursement program would presumably depend on the therapeutic area, currently available RCT/comparative data and market conditions.

As discussed, comparing the results of trials conducted using different patient populations between pre- and post-MA studies can be a considerable challenge [25–28]. The relatively simple act of randomizing patients to each trial arm (RCT, PT or OT) within a multi-tiered MRT structure would ensure that baseline patient characteristics are balanced across trial arms. Moreover, it puts into place a prospectively defined framework under which randomized controlled data can be directly compared with uncontrolled, observational data. This has the benefit of allowing for the identification and quantification of biases associated with data from controlled-trial versus real-world conditions. Comparing data across these conditions from within the MRT structure should thereby be more acceptable to reimbursement authorities, and payers, faced with the challenge of interpreting real-world data, can quantify and assess the uncertainty by comparing RCT results with those from the PT and OT arms.

The MRT provides authorities with an upfront commitment to obtain real-world safety, efficacy and comparative effectiveness data. Currently, comparative effectiveness data are often provided to payers using indirect comparisons between treatments – that is, not from head-to-head, active-comparator trials. These sorts of comparisons are problematic and prone to bias, and although requested, are open to scrutiny from the reimbursement authorities. The MRT provides a greater provision for direct treatment comparisons, and thereby to clinical data from a broader and comparable set of patients in controlled and real-world clinical settings.

The restrictive nature of patient populations enrolled in Phase III trials has been noted by regulatory authorities and the pharmaceutical industry [29]. Authorities have implemented guidance to push for a broader range of patients in clinical development programs (e.g., pediatric investigational planning) [30]. Here, a wider range of patient demographic and disease profiles is envisaged in the MRT as compared with traditional RCTs. This would increase the generalizability of the data to the wider disease population, and therefore, increase the likelihood of acceptance by both regulatory and reimbursement authorities. In particular, due to the larger size of the MRT, it would allow for a more robust assessment of the effectiveness of the drug within subgroups, as well as in the detection of safety signals.

Historically, the choice of an appropriate active comparator at a Phase III stage was determined primarily by regulatory requirements. Placebo has traditionally been used as the standard non-active comparator whenever ethically possible. More recently, however, the choice of comparator is being determined by payers and there is considerable variation in the active comparators of choice across multiple regions/countries. This is because payers in different regions/countries have different standards of care available to them on the market. The MRT structure allows for a wider range of comparators to be included in an integrated study, which is a significant improvement over standard Phase III RCTs. The MRT has, for example, the ability to capture data from new active comparators within its OT arm. Randomization in the RCT and PT arms can also be reasonably adapted to incorporate active comparators as required.

One of the intentions of the MRT is to remove the necessity of conducting multiple Phase III/IV trials. Although larger (in terms of patient numbers) than any clinical trial of a single type, the MRT could result in an overall reduced setup and running cost. Given the reduced levels of follow-up and monitoring in the PT and OT arms (compared with the RCT arm), it might well be expected that the MRT would result in lower average per-patient costs. Once a drug is determined to be sufficiently safe, a patient randomization ratio favoring the less expensive PT and OT arms could be utilized. Key efficacy and safety data (used specifically for an initial MA) could be collected at a lower financial burden, as compared with a standard RCT. Finally, the MRT’s multi-tiered approach, as compared with the standard sequential Phase III/IV clinical development process, could lead to a considerable reduction in the time required to obtain MA and reimbursement. This could result in faster patient access to new drugs.

Conducting an MRT would require more careful planning and preparation of resources, as compared with a standard Phase III RCT. The individual complexities of each of the separate trial arms would not be reduced by the simple act of randomizing patients under a single trial protocol. Statistical powering of the MRT, when (and if) to perform interim analyses and how to combine data across the trial arms, would all be non-trivial challenges. Determining the optimal randomization ratio of patients to each trial arm would also be far from trivial, and is a central exercise to the multi-tiered and staggered approach of the MRT structure. An additional complexity arises with the operational running of the trial arms in parallel, as compared with in series (as per current standard practice). Overall, the combined data may also provide additional interpretation and understanding challenges to the regulatory/payer reviewers, especially if the efficacy or safety signals differ by trial arm.

The MRT would certainly require more adept planning and preparation of resources on behalf of the sponsor, compared with a single, standalone Phase III RCT. The costs of implementing the MRT could also well exceed the cost of any single trial implementation happening in series. Furthermore, a sponsor could undergo a significant commercial loss with any failed MRT, compared with an ‘equivalent’, negative Phase III RCT. The complexity of a new trial design will always have challenges in the beginning, and may need to be adapted with the learnings in conjunction with regulators and payers.

Exposing patients in an experimental treatment group in an uncontrolled setting (i.e., in the PT and OT arms) adds risk over and above a standalone RCT. Some investigators might be hesitant to expose their patients to an untested drug, and resistance from ethics and regulatory bodies could be expected [31]. In order to address these concerns, patients randomized to the PT and OT trial arms could be blocked from being allocated experimental treatment at study onset, essentially delaying allocation to the experimental arm until sufficient interim RCT data become available. Such a delay would, for example, allow a data review board to evaluate all safety data collected from within the MRT to date. Recommendations from the review might well reassure regulators that expanding the (unapproved) drug to a larger, observational population (i.e., the PT and OT arms) would be ethically sound. Any delay in the onset of recruitment to the experimental treatment group would, however, weaken the value of the data. This would be particularly true if either the available patient population or treatment practices (e.g., SOC) change rapidly over time.

The inclusion and exclusion criteria of the MRT would have to be less strict than its RCT counterpart. In the ideal circumstance, inclusion criteria would be as wide as in any standalone observational trial. While shortcomings of narrow entry criteria are certainly valid with respect to external validity, convincing some ethics committees to expose a wider range of patients to a potentially unsafe or ineffective drug could be difficult [18]. Sponsors usually wish to optimize a study’s success, which is often achieved with a relatively homogeneous disease population. Nevertheless, in the wake of safety concerns arising with currently marketed products, regulator and payer demands dictate testing drugs in a wider range of patients [32]. Given that the MRT would be conducted toward the end of Phase III development, exposure of a novel treatment to a wider patient population would seem to be a justifiable risk in contemporary clinical research. Planned interim analyses could be used to allow for widening of patient inclusion criteria over the time course of an MRT.

There may be concerns regarding the generalizability of the results from combined data from different trial designs. Such concerns could be partially alleviated by measuring end points in a consistent manner between each trial arm. The problem of synthesizing results from multiple trial sources is a problem that authorities face on a regular basis [33]. At the very least, the multi-tiered MRT structure brings an integrated and prospectively planned approach to this dilemma.

While the MRT allows for an integrated assessment of many of the biases outside a strict RCT setting, it cannot completely remedy a situation in which considerable bias is present or when a particular bias affects all trial arms. Attrition bias (e.g., patient drop-out/withdrawal) would still pose a serious challenge to the interpretability of the MRT data regardless of the multi-tiered structure. This would particularly be the case if the drop-out rates varied between trial arms. Selection bias typically present in observational trials will not disappear from the OT arm because it has been incorporated into an MRT structure.

The issue of greater bias in the PT and OT arms is part of a wider between-trial arm heterogeneity issue. Results that differ between the trials arms might be challenging to resolve. In these instances, the natural inclination of regulatory and reimbursement authorities may be to place greater emphasis on the RCT results over those from the other two trial arms. Nevertheless, there is far greater scope to resolve trial arm differences observed within the MRT structure as compared with trying to resolve differences across completely separate trials.

Within-trial arm variability might also be expected – more so in the PT and OT arms due to lower control of the patients’ adherence to treatment and follow-up. The increased risk of bias from both the between-trial arm and within-trial arm variability should generally lead to higher overall sample size requirements as compared with pure RCT approaches. As such, reviewing authorities may be tempted to down-weight the PT/OT results (either in a formal statistical manner or in their interpretation) if a concerning level of heterogeneity was observed. How a regulatory or reimbursement authority might judge disparate results between the trial arms could depend on whether the divergence occurred within safety, efficacy and/or quality of life outcomes.