Building trust in real-world evidence and comparative effectiveness research: the need for transparency

The observational nature of RWE means that, unlike RCTs, patients cannot be randomized to different treatment options in a way that matches these groups for other characteristics (potential ‘confounders’) that may lead to differences in outcome. RWE, and CER studies in particular, are therefore at risk of significant bias if outcomes with different treatment options are compared without adjustment for confounders such as patient demographics, disease severity, comorbidities and concomitant medications. Selection bias is almost inevitable in RWE, because treatment decisions in everyday clinical practice are (hopefully) not random. For example, a highly effective but expensive new medication may initially be reserved by physicians (or its use restricted by payers) for more severely ill patients; hence, comparison of clinical or cost outcomes with a standard of care being administered to less ill patients will clearly be biased by differences in the allocation of the two medications. A question often asked of RWE studies in general, and CER in particular, is “how can you be sure you are comparing apples with apples?”.

It is impossible to eliminate with certainty all sources of bias in RWE, but they can be minimized by following guidelines on the design and validation of RWE studies (such as the guidelines from Good Research for Comparative Effectiveness, Agency for Healthcare Research and Quality, International Society for Pharmacoepidemiology and International Society for Pharmacoeconomics and Outcomes Research) [4–7]. Established methods exist to adjust for potential confounders (e.g., propensity score matching); these should be chosen appropriately, the rationale for selecting a particular adjustment should be communicated and the method of adjustment should be described clearly for the nontechnical expert. Transparency of methodology shows that an RWE study has been well designed and builds trust that the results will be robust despite the lack of randomization.

A challenge for RWE studies in general – and retrospective database analyses in particular – is whether the data source is representative of a wider patient population. Are results from people with multiple sclerosis in a US claims database representative of the wider US multiple sclerosis population? The answer should be yes, if any worthwhile conclusions are to be drawn from the study. Are these same results representative of people with multiple sclerosis in Europe and other parts of the world? Here, the answer may be no, depending on what differences exist between countries in factors such as natural disease course, patient demographics, and the diagnostic, treatment and funding pathway. Indeed, RWE studies are increasingly used to deliver a local, country-level context that multinational RCTs are not designed to provide, reflecting the marked differences among countries and regions in their demands for RWE.

Ultimately, what is being asked is whether the most appropriate data sources (rather than simply the most readily available, or the cheapest or the ones that gave the best results…) have been chosen to address the research question for the study. Demonstrating that the data sources are appropriate and representative means demonstrating due diligence in the evaluation of the available data sources, and being transparent about the methodology used to choose among them. Ideally, a systematic assessment of all of the available real-world data sources in a particular disease area should be performed and published in advance of the RWE studies being conducted; such analyses have been conducted in several therapy areas and have informed subsequent RWE program design [8,9]. Transparency in data selection builds trust in the choice of data source and thus in the results of RWE studies using those sources.

Registration for RCT protocols at sites such as those provided by EUDRACT and ClinicalTrials.gov [10] was originally mandated because of a lack of transparency by some sponsors in disclosing the existence of RCTs, and consequent concerns that RCTs yielding neutral or negative results were not being published. RWE studies now face similar criticism; indeed, suspicion of publication bias is arguably greater for RWE studies than for RCTs, because RWE studies can generally be performed much more quickly and cheaply. Although an average RCT will take at least a year and cost US$30–40 million [11], a retrospective database study could be completed in a few months for approximately 1/100th of that cost. How can investigators presenting an RWE study dispel the suspicion that they ran multiple similar studies and analyses, but published only the one that gave a positive result?

One answer is to adopt institutional and corporate policies on RWE studies that provide a similar level of rigor to policies on the conduct of RCTs. Such a policy may set the definition of RWE studies, mandate posting an outline protocol for each study on an appropriate forum (several sponsors have already used ClinicalTrials.gov [10]) and commit to publication of the study results regardless of the outcome. While most companies have internal publication policies on RCTs that commit to publishing regardless of the study outcome, few currently commit to this for RWE studies [12]. An internal RWE policy may also set out terms for data source selection, data access and analysis/reanalysis, and authorship of publications. Transparency in the overall strategy builds trust that RWE studies that are planned will be published, whether the results are positive or not, and allays concerns over selective reporting of RWE.

Conflicting or contradictory study results are a particular issue for CER and economic- or cost-modeling studies. If one study shows that drug A provides superior outcomes to drug B, but a second study (conducted using a different data source, and/or a different analysis or modeling methodology) shows that drug B provides superior outcomes to drug A – for example, in multiple sclerosis [13,14] – what should a decision maker do? Given the plethora of data sources and analytical approaches, differences in RWE study results are inevitable, and meta-analysis of CER and modeling studies is almost impossible. Unfortunately, conflicting or contradictory study results erode confidence in RWE. Hence, with insufficient technical expertise (or time or inclination) to conduct a critical comparison of the methodological aspects of each study, the average decision maker is likely to ignore both, and revert to the RCTs that he or she understands and trusts.

This issue can be addressed by bringing together and communicating what has been done to address the other challenges regarding the robustness of RWE studies. An RWE study is more likely to be trusted if there has been transparency about the methodology (including steps taken to minimize bias), if a systematic approach to selecting the data source has been demonstrated and published, and if a study protocol has been posted with a commitment to publication of the results. The final report should also follow guidance for the reporting of RWE studies, such as the Strengthening the Reporting of Observational Studies in Epidemiology statement [15], and show concordance of the study with accepted guidelines for the design and validation of RWE studies. Transparency in reporting builds trust in RWE studies that have been planned, designed, conducted and interpreted in a robust fashion, and exposes less rigorous studies.

The demand for RWE will continue to increase, but work needs to be done to break down the current barriers to acceptance of RWE in influencing healthcare decision-making. With access to potentially life-enhancing new treatments at stake, we owe it to patients to act together to build trust in RWE through greater transparency in methodology, data sources, strategy and reporting.

Some of the issues covered in this article have been the subject of presentations given by the author at The International Publication Planning Association Annual Meeting (9–10 February 2015, San Diego, CA, USA) and the International Society for Medical Publication Professionals European Meeting (19–20 January 2016, London, UK) and International Meeting (11–13 April 2016, National Harbor, MD, USA).

R White is an employee and shareholder of Oxford PharmaGenesis. The author has no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.