A framework to guide the optimal development and use of real-world evidence for drug coverage and formulary decisions

Before beginning an RWE analysis, the prospective research protocol should identify key contextual considerations to determine the evidence level and the corresponding methodological and process standards that should be followed. In some ways, this can be associated with understanding the level of skepticism that will meet the results of RWE and selecting more rigorous standards when skepticism is likely to be high. But the contextual considerations that need to be addressed are much broader than skepticism arising from statistical concerns about sample size or confounding variables. They begin with the type of evidentiary assertion that the RWE is intended to make. As shown in Figure 1, evidentiary assertions to be made by RWE for coverage and formulary policy will generally be either one of superiority for a particular drug in comparison to others, or an assertion of the equivalent effectiveness of a lower-cost agent among two or more drugs. An intended assertion of superiority is one indicator that payers will require a relatively high evidentiary bar – a high level of rigor in evidence and process standards – in order to overcome concerns about the validity of RWE findings.

Other contextual considerations that suggest a high evidentiary bar include RWE use to inform coverage or formulary decisions that would expand use of a drug and increase cost. Assertions that a more expensive drug would lead to lower overall costs across the health system may not be quite as difficult to justify with RWE analyses as assertions of superior health outcomes, but will still require evidence standards that meet a high bar if they are to convince many payers. In addition, RWE that suggests the need for substantial changes to clinical practice, that conflicts with findings from randomized controlled trials (RCTs) or that lacks a clear underlying rationale (a biomedical explanation for the results) will also require stringent efforts to assure its validity. When seeking to use RWE to assert superiority, it may ultimately be wise to consider whether any RWE strategy based on observational studies alone can achieve the degree of trustworthiness required. When the bar for evidence and process standards is agreed to be very high, manufacturers and payers may consider pragmatic clinical trials (PCTs) to be a potentially better option than observational RWE, notwithstanding the current challenges to conducting PCTs.

In contrast, a relatively low evidentiary bar may often be satisfactory when the intended assertion will be that a lower-cost drug offers equal effectiveness in real-world settings when compared with more expensive options. Similarly, a lower bar may often be adequate when the assertion supported by RWE will not expand use of current treatments or require a large shift in clinical practice, when there is a strong underlying rationale behind the findings, such as equivalent outcomes for drugs with similar mechanisms of action, or when the findings complement existing RCT data to fill important gaps in knowledge (e.g., use in a population group not included in the RCTs). RWE findings from analyses with lower evidentiary standards may also be more readily accepted if they signal a new safety concern, since payers are risk-averse and wish to take early steps to minimize potential harms to patients.

This should not be misinterpreted as advocating for lowering the bar for evidence in healthcare decision-making; rather, it is important to recognize that a diversity of decision types exist, and the evidence requirements for each decision type can be different. RWE can be suitable for evidence to support healthcare decisions, either as a partner to RCTs or, in certain circumstances and where data and methodology permit, as the main basis for the decision. The key is to recognize the context of the decision, the desired assertion to be made and the suitability of RWE to support that decision.

The essential role these contextual considerations play in determining the evidentiary bar for acceptance of RWE, and thus the level of standards within each of the steps of RWE development, cannot be overstated. Both payers and manufacturers planning RWE efforts need to consider them carefully before beginning any data analyses. Since payers will be assessing the credibility of manufacturer's RWE when making coverage or formulary decisions, it is particularly important for manufacturers to understand how, and from what perspective, payers will scrutinize the data and methods. This is why active engagement between manufacturers and payers before a manufacturer-led RWE effort commences can be so valuable. Whenever feasible, manufacturers should reach out to payers to develop a shared view of the data, evidence and process standards that can best meet the evidentiary bar for the specific analysis they plan to undertake.

It is also helpful for manufacturers to gain an understanding of the broader perspective payers have on evidence for coverage and formulary decision-making. Payers know that head-to-head trials of drugs are uncommon, and that RWE can be useful in examining drug classes where there are multiple competitors. But the general experience of payers is that when outcomes from drug treatment are evaluated through their own analysis, drugs often do not perform as well in clinical practice as they did in clinical trials. This is why using RWE to claim the contrary is likely to be viewed with skepticism.

It is important to understand that payers greatly value the perceived validity of RCT results and have put great reliance in the FDA's requirements for RCT evidence to justify new treatment approvals. Payers understand RCTs can be flawed, and often fail to address comparative effectiveness, longer term durability of effects or patient and clinician acceptability. Nonetheless, payers are hesitant to adopt an approach to the use of RWE that could threaten to undermine the incentives for manufacturers to conduct RCTs. RWE is viewed favorably by payers as a complement to RCTs, a useful tool to fill in gaps in the evidence base, but deep commitment to RCTs remains. Manufacturers should be mindful of these views when approaching payers to discuss the contextual considerations relevant to their proposed generation of RWE.

Once the level of evidence standards and process have been clarified, decisions can be tailored to meet these standards within each step of RWE development and use. As shown in Figure 1, the first step in this chain is to frame the question and hypotheses to match the intended evidentiary assertion. This may seem like a simple step, but it requires attention to the entire downstream RWE process. Only if the research question can be linked with an adequately curated dataset, with clear methods and with stipulated procedures for verifying the analyses, will the RWE have a chance of meeting the evidentiary bar and thus be fit for informing the policy decisions to be made. As mentioned earlier, it is at this stage that the RWE developers, whether they are manufacturers or payers, need to consider whether the evidentiary bar for the decision can be met through observational studies alone, or whether an RCT or PCT will be necessary. Manufacturers may find it useful to have discussions about the needed evidence and process standards and discuss proposed RWE research questions with multiple payers to make sure that the RWE effort will be accepted by a broad set of organizations. To make these discussions as tangible as possible, it is helpful to delineate the specific PICOTS for the analysis (Patients, Intervention(s), Comparator(s), Outcomes, Time horizon, Setting) [12]. Describing the PICOTS for an analysis serves as a useful technique for ensuring that everyone fully understands the scope and intent of the overall analysis at a granular level that can help avoid later misunderstandings.

When efforts to generate RWE are planned prospectively, data sources and definitions for eligibility and key outcomes can be addressed in a collaborative fashion to maximize the transparency and reliability of the data. However, most RWE analyses are planned and performed using data previously gathered through routine mechanisms. Understanding and curating the data source is therefore a critical part of enhancing the trustworthiness of a retrospective RWE study. A key point here, particularly important when there is a high evidentiary bar, is to ensure that individuals who know the nuances of the data source are involved with the analyses from the earliest stages. Each dataset has quirks, biases and potential variations that could lead to unknown errors in defining eligible patients, the interventions or comparators of interest, and the outcome measures being used. For example, drug utilization patterns and patient behaviors are heavily influenced by benefit design and drug coverage policies enforced by the payer; any RWE that evaluates cost needs to account for these factors as the results may be different in another dataset with different preferred agents and benefit designs. This is one of the reasons why payers are often more likely to believe their own data, rather than analyses using other datasets, even data from well-respected data vendors. If outside databases are used, payers and others believe that transparency in curation is an important goal. End-users should be able to track how original raw data are transformed into a fit-for-purpose RWD dataset, so that the provenance of key data elements is clear. The higher the evidentiary bar is for the overall RWE analysis; the more attention should be given to curating the data source in a way that meets the agreed standards.

As noted in the main body of the ICER Policy Summit background paper [10], there have been numerous multidisciplinary efforts to outline the key approaches through which the analytic methods of RWE can address the perceived limitations of observational data analyses. These specific approaches include sophisticated propensity score matching and other ways to reduce the likelihood of unknown confounders distorting the results. The main point here is that, as with the other elements of RWE development, there are basic methodological requirements that should be core to any RWE analysis, but additional focus and effort will be required when there is a higher evidentiary bar. Steps that can be taken to heighten the transparency or rigor of the methodology include: the engagement of outside academic experts; posting of the analytic protocol at clinicaltrials.gov in advance of any work; and very careful attention to all eligibility and outcome definitions, which, as mentioned earlier, should be woven into the curation of the data source and must involve individuals with a deep understanding of the source data.

One example of the importance of definitions, and of the risks presented by a lack of standardization, is shown by measures of adherence or persistence in use, which can be calculated using multiple methods [13–15]. Both manufacturers and payers should seek to develop and use standard definitions for these and other metrics. Any deviation from these definitions should then be accompanied by transparent justification. Although standardization of key real-world outcome measures such as adherence is not currently the responsibility of any single organization, the Pharmacy Quality Alliance [16], International Society for Pharmacoeconomics and Outcomes Research [17] and the Academy of Managed Care Pharmacy [18] are potential leaders in this area. As ICER welcomes RWE as part of its review process, it may also come to play a role in developing and communicating clear definitions for certain key outcomes.

Among the different steps in RWE development, mechanisms to verify analyses offer perhaps the broadest set of choices from which to select a tailored approach to match the evidentiary bar. For many payers, verification is the most critical step in assessing the credibility of RWE for informing coverage and formulary decisions. Verification options include efforts to replicate results using different methods within the same dataset, or trying to replicate the results using the same methods applied to a different data source(s). Each has a different role to play in helping to verify that the results are robust and trustworthy. It is generally easier to re-run the analyses using different approaches within the same original data source, but when the evidentiary bar is high, replication in alternative data sources, including the payer's own data, may be considered necessary. Sharing the analytic code used for the primary RWE analysis with external parties is another strong measure that can be taken. Some manufacturers have raised the concern that sharing of analytic code could constitute a ‘transfer of value’ to payers raising regulatory scrutiny, and efforts are needed to clarify this point in order for true collaboration in sharing code between manufacturers and payers to proceed [19].

Validation by an independent third party or peer-reviewed journal publication has also been proposed as a way to help verify analyses and add to their trustworthiness. Discussion at the ICER Policy Summit suggested the broad spectrum of RWE capabilities among payers leads to varying opinions about the relative usefulness of these approaches. For some payers without great depth in RWE analytic capacity, academic consultants may be seen as helpful in evaluating the relative credibility of externally derived RWE. For many larger payers, however, third parties, even academic faculty, paid by manufacturers to vet RWE will always be viewed skeptically, especially when payers are evaluating a manufacturer-sponsored RWE project. One reason is that third parties are unlikely to have been involved in data curation and therefore may be blind to important nuances of the data sources used.

As for journal publication, payers often do use publication in highly respected journals as a proxy for RWE quality, and there is a positive social value to peer-review and publication, including the inherent check provided by external experts, broader sharing of methods and results, and the fostering of ongoing dialogue around RWE among all stakeholders. A core commitment to submitting all RWE analyses for peer-review should therefore remain a bedrock of verification. However, some caveats should be noted. As with all third parties, it is unlikely that journal reviewers will understand the nuances of the data source, and the timeline for peer-review and publication usually fails to meet the needs of manufacturers or payers, both of whom are eager to use RWE expeditiously. Consequently, given the limits of peer-review, some uses of RWE to inform coverage and formulary decisions will need to rely on a verification process using more timely approaches that will complement eventual peer-review.

If the contextual considerations have been discussed and a clear understanding reached of the type of assertion intended and the related need for a high or low evidentiary bar, the RWE produced should be fit for informing coverage and formulary decisions. Nonetheless, it is important to note that best practice in using RWE includes the need to integrate it transparently with other evidence sources, and to disseminate the RWE evidence as part of the justification for the decision. Clinicians, patients and other stakeholders should be informed of the role that RWE has played, including the steps that the payer and manufacturer have taken to ensure the trustworthiness of the results.

Underpinning each step of the development process for RWE are several critical principles that must govern the entire process. As noted above, developing RWE for coverage and formulary decisions should involve interaction between a manufacturer and payer(s), and it is key that attention is paid to transparency, communication and collaboration at all steps. Methods to achieve transparency include those previously mentioned of posting RWE protocols in advance, sharing results and analytic code and submitting work for peer-review and publication. But it also includes the more subtle transparency required for effective discussion of the contextual considerations that frame the entire RWE process. Ultimately, if all parties are to accept the value of RWE for a particular coverage or formulary decision, these principles should inform all parts of the process of evidence generation and use.