Evaluation of comparative effectiveness research: a practical tool

The real-world effectiveness of a healthcare intervention can depend on many characteristics such as age, sex, race/ethnicity, disease severity, prior treatments and insurance status [14]. In oncology, these and other characteristics (e.g., tumor stage, tumor characteristics [e.g., genomics and biomarker status] and line of therapy) are likely to have an impact on outcomes. However, due to restrictive inclusion and exclusion criteria in most clinical trials, existing data on innovative therapies at the time of market approval and beyond may be limited to a small subset of the population of patients for which the treatment might be considered. This can limit generalizability (e.g., if only advanced disease is included, results may not be generalizable to patients with earlier stages of disease). CER can address gaps in evidence and inform assessments of real-world drug effectiveness, tolerability and safety. Details on inclusion and exclusion criteria and study time periods should be clearly specified and must be chosen to address the hypothesis being tested to ensure generalizability to the appropriate patient group. Additionally, to support overall conclusions and study generalizability, results can be reported for patient subgroups of interest (e.g., elderly persons, distinct ethnic backgrounds, those with poor performance status or tumor characteristics [e.g., tumor size, biomarker status and histology]).

The evolving landscape of cancer treatment has resulted in the availability of many potential treatment choices, including multidrug treatment regimens. Selection of an active comparator reflective of standard of care is an appropriate choice when evaluating real-world effectiveness of a newly approved therapy. However, standard of care may not always be obvious if there are substantial variations across clinical practice (e.g., regional or health system differences). Selection of one comparator drug or drug regimen is an option in such cases. Choosing appropriate active comparators may reduce the potential for bias, particularly confounding by indication or severity [15]. This is the tendency for the indications that can impact risk of study outcomes to also influence the prescribing tendencies of physicians (e.g., presence of high-risk disease features or overall health status). This can result in biased estimates of intervention-associated outcomes simply because of a differentially higher or lower baseline risk of the outcome among patients who receive a particular therapy. For instance, oncologists may selectively prescribe a single-agent treatment regimen to those who are in overall poor health instead of a combination (e.g., >1 drug) regimen. In this example, the effectiveness of the combination regimen may be artificially inflated relative to the single-agent regimen. In other cases, bias may occur where an active treatment is compared with palliative care, or in instances where tolerability differences, adverse event profiles or other factors differ among treatment choices. Under some circumstances, however, it may be challenging to select an active comparator, or the evidence gap may involve active therapy versus observation. For example, in the case of slow-growing tumors such as prostate cancer, observation is one plausible treatment strategy that involves waiting and closely monitoring the progression of the disease. In such cases, comparing this strategy with active treatments could be appropriate, with careful attention to control for confounding. However, if the data source does not contain enough information on important confounding factors, the waiting/monitoring strategy may appear more effective simply because physicians are reserving this strategy for those with a better prognosis.

Study outcomes must be clearly and consistently defined, clinically meaningful and identically measured. Primary outcomes should be prespecified and defined in detail. Objective measures of clinical outcomes are preferred over outcomes subject to clinical judgment. The accuracy and comprehensiveness of diagnostic and procedure codes used to capture clinically meaningful outcomes are important measures of the quality of a CER study. These should be applied in the same manner in both the treatment and comparison cohorts to avoid bias. In prostate cancer, for example, identification of treatment-related urinary incontinence may involve both a set of diagnostic and procedure codes, such that the exclusion of any relevant code may impact estimates of treatment-associated harms. Validation studies or assessments examining the ability of the coding definition to accurately detect an outcome should be considered whenever feasible, or published validation work should be referenced.

In addition to the accuracy of outcome ascertainment, the clinical meaningfulness of CER outcomes (i.e., how well the outcome provides the evidence needed for patients and their doctors to make treatment-related decisions) is paramount. In oncology, survival (progression-free, cancer-specific or overall) is commonly studied. Other outcomes such as adverse events, side effects and patient-reported outcomes could be of importance to patients. Economic measures including healthcare resource utilization and costs are also often of interest to payers and health system administrators.

The timing of the study treatments in relation to factors such as stage of disease and tumor size should be specified and relevant to the research questions. Certain therapies may have delayed effects that could alter assessments of long-term benefit versus risk. The follow-up period should be consistent with the investigators’ understanding of proposed biological mechanisms that will impact the time to outcome events (e.g., if the gap is related to lack of long-term outcomes, then the period should be long enough to capture those).

CER often relies on analysis of data that were not specifically collected for research, such as data obtained as part of routine medical care or administrative claims. These data can increase the likelihood of bias, particularly where certain variables (representing important confounders or effect modifiers) are not available, and where treatments or outcomes are measured with error [16]. For example, some data sources may not routinely capture important features such as line of therapy, cancer stage, biomarkers (e.g., genetic tests) and this may impact the quality and clinical interpretability of CER. Consequently, it is important to understand the strengths and limitations of data sources, and to consider strategies to address them, such as replication of a study using more than one data source or supplementation of available data with information from external data sources. For example, linkages between claims databases, electronic health records and external cancer registries can provide researchers with information that can be used to validate assumptions regarding tumor stage, line of therapy and other important variables. In some cases, it may be possible to augment the information in a claims database through the collection of information from patients’ medical charts. In head and neck cancer, for instance, infection with human papilloma virus is an important prognostic indicator that is usually not included in claims data but is routinely documented in patients’ medical charts.

A larger sample size increases precision in estimates, provides more statistical power to detect smaller magnitude differences that can be important at the population level and increases the confidence in interpretability and applicability of study results to clinical decision making [14]. A study should provide a clear description of sample size/power estimation, and reviewers should determine whether the underlying assumptions are reasonable (e.g., estimated rates of the outcomes are derived from peer-reviewed literature and can be reasonably assumed to apply to the study population; study power is at least 80%; and the magnitude of the effect of the intervention is of clinical relevance). In cases from the literature where no detail is provided on sample size or study power estimation, the width of the confidence intervals generated around the effect estimates will provide a sense of the range of values in which the true effect is likely to exist.

Among the most important threats to study validity in observational CER are three types of bias that can result from the selection of prevalent users of treatment. These types of bias include: an underestimation of event rates if events occurring when therapy is first initiated cannot be ascertained (e.g., ‘depletion of susceptibles’ bias, where a cohort is depleted of all susceptible individuals experiencing an early event); confounding by risk factors that are modified by the study drugs (which occurs when a risk factor that is affected by the treatment is included as a ‘baseline’ risk factor despite being measured after treatment initiation); and the so-called healthy adherer bias which involves the selection of people who tolerated the treatment and, most likely, for whom treatment appeared to be effective [17]. Restriction of CER to new initiators of therapy (or starters of a new course of therapy) is the most important design feature to limit these sources of bias. The new user design emulates the interventional part of a randomized clinical trial where the treatment arms are restricted to patients who are newly prescribed one of the study drugs. Essentially, patients are followed from initiation of a therapy until the occurrence of specified study outcomes or end of the study period. In addition, there is usually a minimum period of nonuse (washout) prior to the initiation of therapy [18]. In cases where the new user approach may not be suitable (such as in questions pertaining to long-term users), careful consideration of potential biases introduced by prevalent users, as outlined above, are needed.

Use of a historic control may lead to confounding by factors that vary over time [19]. Also known as chronology bias, this situation can arise when long-term trends within the healthcare system affect the severity of the target condition, induce changes in outcome rates or alter how clinicians diagnose disease or identify outcomes. Consequently, selection of a concurrent comparison group is preferred. However historic controls may sometimes be the only option (e.g., when there is rapid high uptake of a new treatment [19]) such as may occur with the approval of a breakthrough therapy. If historic controls are used, they should be as similar to the treatment cohort as possible.

In nonrandomized studies, baseline differences between the treatment and comparator cohorts could result in confounding [20]. Using observational data for CER therefore requires design features (described above) and analytic strategies to control for confounding, such as traditional multivariable regression techniques, propensity score techniques [21] or instrumental variable analysis [22,23]. When the number of outcome events per covariate included in the regression model is sufficient (e.g., ten or more), traditional multiple regression is generally reasonable [24]. In cases of many covariates, it may be tempting to use an automated variable selection method such as stepwise regression. However, such automated procedures apply subjective rules and generally do not align well with the considerations of confounding. Other analytical strategies such as propensity score methods can adjust for large numbers of covariates in CER, and often perform better than logistic regression when the outcome is relatively rare (e.g., seven or fewer events per confounder) [25]. Additional benefits of propensity score methods include their focus on understanding indications for drug selection, the ability to test for interactions between propensity of treatment and drug effects on outcomes and correction for some unmeasured confounding with propensity score calibration. Alternatives such as disease risk scores may be a good choice for analysis of a common outcome and rare or multiple exposures, and instrumental variable analysis is one of several possible approaches to address unmeasured confounders [23]. The forms of the study outcomes, treatments and covariates determine the specific statistical methods to be used. For example, CER studies often investigate time-to-event data with variable follow-up and censoring of outcomes; these are commonly analyzed using Cox proportional hazards regression.

Many observational studies use a cohort approach to emulate randomized controlled trial designs. However, if the exposure period is not correctly classified, there is a risk of immortal time bias [26]. Consider a study in which cohort entry is based on a clinical event such as a diagnosis of lung cancer, with patients followed from the date of diagnosis until death. If the study aims to compare chemotherapy use versus no chemotherapy, and defines exposure by administration of a chemotherapy agent within 90 days of the initial diagnosis, then the time between cohort entry (date of lung cancer diagnosis) and administration of the first chemotherapy agent is immortal since patients must survive this period to receive their first prescription for chemotherapy. Since this immortal period has been misclassified, the chemotherapy-use group will have an artificial survival advantage over unexposed patients [26]. A way to avoid immortal time bias in this example is to align the start of study follow-up with the initiation of chemotherapy, rather than the initial lung cancer diagnosis or more generally to start follow-up only when all study inclusion criteria have been met. An alternative solution is to use time-varying analyses, where all patients are considered unexposed from the time of initial lung cancer diagnosis until the start of chemotherapy. Due to the risk of bias in observational CER studies and the limitations inherent in available methods to mitigate it (e.g., inability to fully control for unmeasured confounding), results should always be interpreted with caution.

All study findings are the result of underlying assumptions made in the design, conduct and analysis of the study. Accordingly, the validity of the inferences that can be drawn will depend on the extent to which these assumptions are valid [27]. Performance of sensitivity analyses that consider alternative assumptions regarding: the data source; criteria to define study cohorts; unmeasured confounders; exposure definitions; outcome definitions; analytical assumptions; and statistical methods can be used to investigate the robustness of study findings to various assumptions and provide a picture of the consistency of an observed result in terms of direction and magnitude. Thus, sensitivity analyses may involve replication of the study in an alternative data source, use of one or more alternative algorithms for cohort selection, treatment and outcome definitions and consideration of alternative choices for analysis. In fact, sensitivity analysis has proven to be an important and consistent predictor of study quality [28].

Additionally, if the biologic response or exposure is expected to differ in certain subgroups (e.g., based on age, race/ethnicity, genetics, baseline severity and prior treatment), consideration should be given to whether these subgroups should be analyzed separately. The research should identify segments of the study population for whom there are concerns about generalizability from prior research. Although many CER studies will lack power to examine all potential subpopulations of interest, if sufficient data are available to define such subpopulations, it can be useful to perform exploratory analyses to examine whether the best choice of therapy may differ for certain subgroups.