Evidence generation with external control arms in rare diseases: a multistakeholder perspective

Evidence generated for rare diseases from studies involving external control arms (ECAs) is increasingly being used to support decision-making by regulatory and health technology assessment (HTA) agencies. Yet the standards and guidance for their use varies, making the generation of harmonized fit-for-purpose evidence challenging.

At the recent ISPOR Europe 2023 meeting, a multistakeholder panel convened to discuss ‘Use of External Control Arms in Rare Disease: Are We Moving Towards an International Gold Standard and How Can We Facilitate Progress?’ Here, the panelists, Carla Vossen (Senior Director Epidemiology, RWE, Syneos Health), Pierre Net (Associate Project Manager, HEOR, Syneos Health), Amr Makady (Compound Market Access Lead [Early Products] – Europe, Middle-East & Africa [EMEA], The Janssen Pharmaceutical Companies of Johnson & Johnson), Jasdeep Hayre (Associate Director National Institute for Health and Care Excellence [NICE]) and Patrice Verpillat (Head of the Real World Evidence [RWE] Workstream, European Medicines Agency [EMA]) share some of the discussion points raised during the session.

Jump to:

1. Can you provide an overview of ECAs in the context of rare disease research, and why they are important?
2. How do patients and patient advocacy groups view the use of ECAs in clinical trials for rare diseases, and how can their input be incorporated into the study design process?
3. Could you share examples of successful rare disease studies that have used ECAs, and what were the key takeaways from these studies?
4. Can companies learn from each other to advance the design of SATs using ECAs?
5. What are the biggest challenges that need to be overcome in order to make ECAs a standard part of rare disease research?
6. Could more convergence and guidance from regulatory and HTA bodies help advance the field?

Can you provide an overview of ECAs in the context of rare disease research, and why they are important?

Carla Vossen: It is well known that randomized controlled trials (RCTs) are the ‘gold standard’ to determine efficacy and safety information leading to marketing authorization or reimbursement approval. However, for rare diseases, it might not be ethical or feasible to conduct RCTs and manufacturers then often perform single-arm trials (SATs). ECAs are used to contextualize the results from these SATs.

Patrice Verpillat: The EMA has recently published a reflection paper outlining our current thinking on SATs. From a regulatory perspective, RCTs are and will continue to be the gold standard. However, in some specific situations, which must be discussed with the regulatory agency and payers, SATs may be acceptable. SATs themselves should bring the evidence needed to determine efficacy and be interpretable on its own. ECAs, such as literature reviews, or additional clinical trials or RWD, can help to contextualize the results and provide comparability. The importance of SATs is in their design. If a SAT is badly designed, an ECA will not save it.

Amr Makady: I agree with Patrice and Carla here. It’s always the aim to generate the best evidence within the specific context. If you know that within that context, a RCT comparing the interventional drug with a relevant comparator is feasible, then it is logical to conduct this type of study, not just to meet regulatory or HTA requirements but also because, scientifically, this delivers the most sound evidence. It’s the same with SATs – these should be structured in the best possible way to deliver robust evidence and facilitate comparison to an ECA that will support evidence on comparative efficacy. This can make things complicated in the sense that you have to start thinking about several study designs in parallel early on in development. First, your SAT and how to best design that to deliver as robust evidence as possible. In parallel, you have to think about the optimal ECA design, which presents added complexity; for example, ideally, you would aim to design an ECA that includes a population that is near-identical to your SAT population. However, due to low patient numbers and scarcity of patient-level data, it may become challenging to identify and match patient cohorts to your SAT. Owing to such complexities, you often find yourself obliged to combine different data sources to try and demonstrate comparative efficacy.

Jasdeep Hayre: I agree completely with this. What we're really interested in is the comparative evidence. In a rare disease setting, that comparator can differ from country to country quite radically because the established practice will be quite different. This is why ECAs are needed – to provide a realistic comparison between treatments that consider the different treatment modifiers across various countries, depending on the healthcare system and the treatments available. The external control can supplement information and give us the true treatment effect estimate, which is what NICE is interested in.

“Ultimately, what we are interested in is an unbiased treatment effect estimate. RCTs can produce this but external controls, when conducted properly and used in the right context, such as in rare diseases, can also achieve this goal.”

How do patients and patient advocacy groups view the use of ECAs in clinical trials for rare diseases, and how can their input be incorporated into the study design process?

Amr Makady: Even before engaging in this process, you need to investigate whether these patient advocacy groups exist and whether they're in a position to contribute to the study. Then when you start designing the trial, it's very important to seek dialogue with these groups, especially in the context of rare diseases. They can really give some very good insights into what really matters to them on aspects such as disease progression, impact of clinical symptoms of a disease on patients’ daily lives and what kind of endpoints need to be taken into consideration. With the latter, there’s even extra complexity if new endpoints need to be developed. But in that sense, the importance of involving patients, especially in rare diseases, is higher owing to the sheer rarity of the condition and the lack of knowledge from other sources.

They also gain an increasingly important role in the regulatory and HTA decision processes. They can help convey the story about the added value better because they know how this impacts their patients.

Patrice Verpillat: From a regulatory perspective, we are putting a lot of emphasis on hearing the patient voice and interacting with patient organizations, for example, when advising companies on trial design. We are developing training materials to assist with this, and I definitely think it is an area that will grow.

Jasdeep Hayre: What’s really important is that we have useful and usable guidance for everyone. For that, measuring health outcomes and health-related quality of life in rare diseases is needed. It's important to engage with the patients to understand what's important to them. At NICE, we start early and engage them throughout the process.

We also have an increased role for patients when it comes to collecting RWE at the end the process. If there is uncertainty around the evidence and the product is available through managed access, we will have patient organizations on our steering groups to help to discuss how the treatment and how the data should be analyzed. This ensures there’s patient involvement from the beginning to the end. As Amr said, the patient is central in a rare disease HTA.

The other aspect is there's high interest by patients or carers for rare disease, especially with ultra-rare diseases. This means that in some cases over 50% of affected families will be involved in registries and then involved in a NICE appraisal, and we're really pleased with how patients are involved in our processes.

Could you share examples of successful rare disease studies that have used ECAs, and what were the key takeaways from these studies?

Patrice Verpillat: At the EMA, we are seeing more and more use of ECAs. We have undertaken some preliminary work looking at the use of RWE in initial marketing authorization applications from 2020 to 2023. In a random sample of 52 applications, 42% of applications (22 applications) included RWE. We generally think RWE is used in post-approval but in our study we found that around half of applications in the pre-authorization phase included RWE. Of these 11 applications, most (10) used RWE for efficacy evaluations. These 10 applications, 50% of which were for orphan drugs, included in many cases several studies based on RWD. When we looked into the types of RWE used, we found that 6 (60%) used RWE for disease epidemiology including natural history studies. External controls were used in 80%, which is quite a large proportion. Unfortunately, these 80% were without or with limited confounding adjustment. Only one, which is one of the potential issues, was designed with adjustment for confounding. Amongst these studies, only two were considered as supporting evidence in regulatory decisions. The others brought limited evidence and were taken with caution and in one case the evidence was disregarded due to poor study design.

In addition, we are seeing stakeholders asking us more and more questions about the use of RWD/RWE. At the EMA, we carry out business pipeline meetings that allow us to interact with pharma on their products' regulatory pipeline. When I looked at these interactions over the past 3 years, out of 11 topics related to RWD/RWE, more than half of these questions were related to ECAs.

Jasdeep Hayre: Since NICE issued its RWE Framework last year, there have been over 20 mentions of RWE, with five or more in terms of ECAs in our guidance. A key takeaway is that there isn't often a perfect example or case study. There will be a very strong rationale for why an ECA is used but often the evidence base from the ECA isn't strong enough or they haven't adjusted for confounders or missing data. As long as the manufacturer can make a strong case for using an ECA and have done the best they could, we're willing to consider it. The use of external controls will always be associated with an increase in uncertainty. HTA organizations have to be very careful about assessing its value and how the risk is shared between industry and payers. From our perspective, this is just as important as the design elements. The value proposition will be affected depending on the dossier.

The other element is for companies to take advice early, and the earlier the better. It takes time to design good quality SATs and if it's to supplement an RCT the lead in time can be several years.

Amr Makady: For me, there are two points I would like to mention. As Patrice mentioned, submissions with ECA whereby investigators do not adjust for confounding should not be the norm, especially if submitters potentially had the means to carry out adjustment. If SATs are needed, knowing where the biases are and how these can be mitigated is vital to deliver sound evidence on efficacy. Guidance exists and, as a community, we know the methods available to deal with different forms of potential biases. It’s part of the homework to really make sure these are addressed. This is something that we, as an industry, should always be doing.

Another thing is how do we move together towards identifying case studies whereby robust SATs and ECAs have been conducted to demonstrate comparative efficacy? We need to have a systematic inventory of research questions, with their corresponding data source, rationale, analysis, and metrics, that outlines how they reached a favorable decision from regulatory and HTA perspectives.

“To Jasdeep’s words, we will never agree on a ‘perfect’ case study because it’s so decision- and context-specific. However, there needs to be an overview of ‘good enough’ case studies so we can work from there. This is the only way to really push the discussions in the right direction.”

Can companies learn from each other to advance the design of SATs using ECAs?

Patrice Verpillat: Through the EU PAS Register, the EMA always requests that this information is disclosed in the study protocol. This is publicly available and can help other companies to see how the SAT and the ECA have been used and designed. When companies come to us to discuss a SAT, we also advise they build the ECA at the same time and this should be disclosed within the protocol. This helps us to know what they are planning and then assess the evidence properly.

Carla Vossen: I agree with Patrice here. In my opinion, the design is very much dependent on the indication and on the assessments within a trial. You cannot simply use one design for another indication. You have to look at the right data sources and ensure the design fits the research goal.

Jasdeep Hayre: In rare diseases, especially ultra-rare diseases where the number of affected individuals can be in single digits in some jurisdictions, this is particularly important and the design will differ depending on how easy it is to collect data. But sometimes a SAT is the only approach that can be taken so it’s incumbent on the regulator and HTA agency like NICE to take these specific circumstances into consideration. We mustn’t lose sight of the aim of enabling access to new cost-effective treatments and technologies, which is of course our main aim.

What are the biggest challenges that need to be overcome in order to make ECAs a standard part of rare disease research?

Carla Vossen: I think that the trust in the ECA methods used for contextualizing SATs has increased but one has to reduce the bias as much as possible for the ECA to be accepted. Hybrid designs, for example where external controls will supplement an internal control arm with randomization occurring at a three-to-one ratio for active to comparator treatment, can still keep some randomization while addressing challenges with trial recruitment, participant retention, etc. However, these types of trial design are not yet in any guidance, and it can be even more challenging in terms of how you compare patients, especially when the ECA is very small.

Pierre Net: From a manufacturer perspective, it is important to anticipate what are the requirements from health authorities and to get early advice from both regulators and HTA bodies as soon as possible, maybe even sooner than compared to traditional RCT. This means training teams on ECA and establishing solid communication and cooperation across departments.

"This point is even more important in Europe in the context of joint scientific consultations (i.e., parallel EMA/HTA evaluation), which might be applied more regularly with the introduction of the European HTA regulation in 2025.”

Amr Makady: From an industry perspective, there are several things that need to happen. An important one is having a collaborative approach to identify where it has been going well, when is 'good' good enough and when it is not. This provides the foundation to build further because it's such a technical topic, but also the one that ties into a lot of current policy discussions. There needs to be a general consensus around what approaches work and in what contexts; for example, between regulators and HTA agencies. Joint scientific consultations may offer a fitting solution to this since it would entail consistent discussions between the regulator (e.g., EMA) and HTA agency. Currently, companies seek advice from each agency separately, be it the EMA and/or national HTA agencies, who can each offer different advice that then needs to be reconciled. This is something we need to work on in the future: to ensure we do this in a joint manner to be able to streamline these discussions in one or several meetings.

From a methodology perspective, a lot of methods are already available to address potential biases; however, the main issue is the acceptability of such methodologies amongst decision-makers. A constructive dialogue is needed to be able to discuss which methods work in which contexts and we haven't been able to bridge that gap well yet. What different methodologies are available and what is the rationale for choosing these?

Jasdeep Hayre: This is a really good point. Methodologically, we have many different ‘recipes’ to choose from. There’s always been a tendency to suggest an alternative method but is this really the best thing to do? As Carla said, from the HTA perspective we look at the evidence and assess if there has been a robust analysis of the data and whether all biases have been considered. We’re looking for a complete and thorough analysis of potential biases, and how this translates into cost effectiveness from a NICE perspective.

Another point to consider is that there is still a lot more to do in translating uncertainties in relation to rare diseases into what effect it has on value. At NICE we can flex our cost effectiveness threshold depending on how certain a committee may feel about the evidence provided. But this isn’t uniform at all. Different organizations in Europe or across the world can have a totally different view. They may even reject outright SATs but NICE may accept it with caveats. There's a lot of international collaboration that needs to happen. The quality-adjusted life year (QALY), the cornerstone of NICE evaluations, isn't accepted everywhere so some of these areas can't be reconciled. But we are making progress.

“There's consensus that ECAs are a valid option for many rare diseases. And we've seen a lot more of these in in recent years. But I really do feel on the HTA side there's more to be done to reach consensus.”

Patrice Verpillat: From a regulatory perspective, as I mentioned, RCTs will also be the standard. But I do agree with what has been discussed previously. We need a strong justification for a different methodology and, again, it depends on the clinical and regulatory context. We need more use cases to understand where it does or does not work.

Could more convergence and guidance from regulatory and HTA bodies help advance the field?

Patrice Verpillat: In terms of internationalization and guidance, it's always challenging because from a regulatory perspective we have a lot of experience with safety. This is why the first International Council for Harmonisation (ICH) guidance on RWD focused on safety. When it comes to effectiveness/efficacy, it's quite different. We have initiated some discussions with our colleagues from the FDA and Health Canada on related topics, but not specifically on ECAs, and it will take time to come to a convergence.

The EMA has a new Methodology Working Party and has published a draft three-year work plan that was open for public consultation until the end of November 2023. Whilst the EMA does not have a reflection paper on ECAs, this could be one of the topics to be considered if suggested during the consultation process.

Jasdeep Hayre: At NICE, we’re moving in the same direction by carrying out modular updates of our processes. We’re looking at any methodological innovations and will update our manuals if these new methods are acceptable. This means a company may come to us in 3 years’ time, for example, and the methods may be different, which will be reflected in our guidance.

Amr Makady: We also have to acknowledge that some agencies have not revisited their methodologies, particularly when it comes to ECAs. But we can’t ignore the scientific discussion, particularly over the past few years, into ECAs. That's one thing we need to work on; translating this scientific discussion to a level which we all understand and use.

About the interviewees

Carla Vossen
Senior Director Epidemiology, RWE, Syneos Health

Carla has over 20 years of experience in epidemiological research as a clinical epidemiologist in academia and as an HEOR/epi/HTA consultant for the pharmaceutical industry. She has experience in observational studies from start to end, including design of externally controlled studies. Therapeutic and methodological topics of interest include innovative observational study designs, rare disease, oncology, and genetics/genomics. Carla holds a PhD in Clinical Epidemiology and a MSc in Biomedical Sciences.

Pierre Net
Associate Project Manager, HEOR, Syneos Health

Pierre holds a PharmD and has 6+ years of experience in HEOR spent in pharmaceutical industry, public health agency and consulting. Pierre is specialized in HTA topics and evidence synthesis, with a proficiency in infectious diseases and vaccines.

Amr Makady
Compound Market Access Lead [Early Products] – Europe, Middle-East & Africa [EMEA], The Janssen Pharmaceutical Companies of Johnson & Johnson)

Amr is a compound market access lead for the Europe, Middle-East and Africa (EMEA) region at J&J Innovative Medicines, leading and executing the market access strategies for selected, priority early products. Additionally, he supports the market access assessment of new business development opportunities for the EMEA region. Amr previously worked at the ZIN (Dutch HTA agency) for 6 years where he specialized in RWE use (IMI Get Real project), health economics assessments and accelerated access. Amr holds a PhD in RWE use for HTA.

Jasdeep Hayre
Associate Director, NICE

Jasdeep Hayre currently provides leadership for the HTA of new medicines at NICE. Within his role he is supporting the development of new methods and processes for a new proportionate approach to technology appraisals (PATT). Jasdeep has worked at NICE since 2011 and previously worked on the methodology and quality assurance of economic evaluations in guidelines and health care staffing levels. He holds a post-graduate degree in Health Economics from the University of York, and a degree in Economics from the University of Nottingham.

Patrice Verpillat
Head of Real World Evidence, EMA

Patrice Verpillat is the Head of the RWE Workstream at the EMA. He is a medical doctor, specialist in epidemiology. He has worked during 20 years in the pharmaceutical industry where he had positions in several international companies, always dealing with real world data (RWD) and non-interventional studies (NIS) in order to bring RWE into research, access and life-cycle product management. Patrice has published over 70 articles in Medline-indexed journals. He has been involved in many organizations such as ENCePP, ICH M14 working group, European pharma association (EFPIA) and ISPE.

Disclaimers:

The opinions expressed in this feature are those of the interviewee/author and do not necessarily reflect the views of The Evidence Base® or Becaris Publishing Ltd.

Disclaimer for Patrice Verpillat: The views expressed in this interview are the personal views of the interviewee, and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.

Disclaimer for Amr Makady: The views expressed in this interview are the personal views of the interviewee, and may not be understood or quoted as being made on behalf of or reflecting the position of Johnson & Johnson Innovative Medicines.