Letter to the Editor – Qualitative evidence submitted by patients to NICE: need for more quality or unrealistic and unfair requirements?

The evaluation of elosulfase alfa for the treatment of mucopolysaccharidosis type 4A (MPS4A) with the appraisal no.: HST19 (conducted between 2019 and 2022) is cited as a best practice example of qualitative evidence provided by a patient organization [2]. Ayyar Gupta et al. state that the submission “was a source of evidence used to support claims of clinical effectiveness” such as “experienced benefits of improved endurance and increased energy for most patients taking this treatment”, “improved life expectations, better sleep, disease stability and ability to plan for the future”. The appraisal resulted in a positive recommendation for funding of elosulfase alfa in England and Wales.

The second example concerns the appraisal of afamelanotide for treating erythropoietic protoporphyria (EPP; appraisal no.: HST27), an ultra-rare inborn error of metabolism that causes painful phototoxic burn injuries in the blood vessels after short exposure to visible light [3]. Our patient organization, the International Porphyria Patient Network, was a stakeholder in this appraisal procedure. The evaluation of afamelanotide had started in 2016, with a consultation phase in 2017 during which patients with EPP and carers submitted testimonies. In 2018, NICE issued a negative recommendation for funding. However, appeals against this recommendation submitted by the stakeholders including our patient organization, were upheld and the evaluation was remitted to NICE. In 2023, NICE issued a final negative recommendation and patients with EPP living in England and Wales do not have access to the only therapy for their condition. According to Ayyar Gupta et al., “the need for qualitative evidence to be collected systematically and analyzed using standard techniques was highlighted as potentially limiting the influence of qualitative data in this case”.

We strongly disagree with the use of these two examples to illustrate how patient evidence should and should not be provided in NICE’s appraisal process. In fact, there are several reasons preventing a direct comparison of the qualitative patient evidence submitted in these two appraisals:

First, HST19 is a reassessment. The first appraisal of elosulfase alfa for treating MPS4A at NICE was conducted in 2015 (HST2) and had resulted in a positive recommendation for funding. During the consultation, patients and caregivers submitted written testimonies and participated in the committee meetings. The positive recommendation was issued under the condition that further data are collected during a managed access agreement and that the recommendation for funding will be reassessed based on this data [2]. Therefore, the current submission of qualitative evidence on patient and carer perspectives is built on an already agreed on protocol and supplements the information collected during the managed access agreement.

Second, Ayyar Gupta et al. fail to mention that the qualitative evidence in the reassessment of elosulfase alfa (HST19) was collected and prepared by a professional research company: while founded and led by patients and caregivers, Rare Disease Research Partners (RDRP) is a contract research organization that provides services for and works together with manufacturers of drugs for patients with a form of mucopolysaccharidosis, including the manufacturer of elosulfase alfa, which also provided funding for the qualitative patient data collection that supported the reassessment [2,4–6]. Evidently, the expertise and financial resources of a contract research organization differ from those available to most patient organizations for ultra-rare diseases. If “qualitative evidence to be collected systematically and analyzed using standard techniques” is a condition for patient evidence to be considered by NICE, this requirement will effectively silence the patient voice in most of the appraisals.

According to a systematic review by Majid, one indicator for tokenistic patient engagement is when decision makers use “a subset of patient perspectives that fit within a particular narrative” as a way “to achieve their personal or professional objectives” [7]. In 2018, NICE had issued a negative recommendation for funding of afamelanotide, with the justification that the “clinical trial results suggest small benefits with afamelanotide” [8]. The stakeholders in this evaluation appealed against the recommendation. The independent panel upheld the appeals in all three possible grounds and concluded that the NICE committee had failed to act fairly (ground 1a), had exceeded it powers (ground 1b) and that the recommendation is unreasonable in light of the evidence submitted to NICE (ground 2) [9,10]. The panel further elaborated “that it was unreasonable for the committee to state that the trial results show small benefits with afamelanotide”. Moreover, the members of the panel in their appeal decision document openly expressed concerns that NICE had discriminated against patients with EPP: during the hearing, the Chair of the NICE committee publicly stated that “the committee did not consider EPP as a disability” because “they had interpreted “disability” as referring to a patently visible disability”.

When reading the 33 testimonies submitted by patients and carers to NICE and the oral inputs provided by patient representatives and clinical experts during the evaluation of afamelanotide, it becomes clear that EPP is a debilitating condition and that the patients struggle with all aspects of life [11]. It also becomes clear that afamelanotide is a life-changing treatment. Therefore, we do not agree that in the case of afamelanotide the “qualitative information from patient testimonies provided important insights” which had “impacted interpretation of trial evidence”, as stated by Gupta et al. In contrast, the example involuntarily demonstrates unfair and tokenistic patient engagement at NICE. The experience with NICE is particularly disappointing, as in other countries patients with EPP have been meaningfully involved in discussions with regulatory authorities, HTA bodies and payor organizations [12,13].

On the one hand, Ayyar Gupta et al. in their editorial criticize that qualitative evidence in the case of afamelanotide has not been systematically collected and analyzed. On the other hand, two paragraphs later, they admit that “rare disease organizations in the UK are often small with a limited number of staff and resources” and might not be able to conduct systematic data collection according to NICE’s real-world evidence framework. Instead of adjusting their requirements accordingly, Ayyar Gupta et al. provides a normative statement that “organizations need to have confidence that any qualitative evidence produced will be taken into consideration by NICE committees and impact the decision-making processes”. As evidenced by the evaluation of afamelanotide at NICE, this is not always the case: for example, the existing evidence from qualitative and quality of life research such as the groundbreaking interview study on psychosocial aspects in EPP by Rufener in 1987 and the more recent analysis on the clinical features, psychosocial impact and the effect of afamelanotide in the Dutch cohort conducted by Wensink et al. in 2023, both of which were available to NICE during the evaluation, could have been used to supplement the insights provided by the patients during the evaluation of afamelanotide at NICE [14,15]. Moreover, as outlined by Vanstone et al., it is not clear whether aggregated qualitative data are indeed superior to direct testimonies of lived patient experience, which could lead to a more “rich and useful HTA process” [16]. Instead of hollow normative statements, adjusting the expectations and requirements to what patient organization can contribute and external oversight in case of clear evidence for inconsistencies would provide more confidence in a fair and just evaluation process at NICE [17,18].

The call for more systematically collected and analyzed data from patient organizations is unrealistic because it is neither within the normal competencies nor the tasks of most patient organization. Moreover, the requirement creates a need for (expensive) commissioned studies, for example from contract research organizations, which can only be afforded by patient organization with the sufficient financial resources. Rather than shifting the burden of evidence generation onto patients, we urge NICE to consider how it can improve the requirements for qualitative evidence to be achievable for patients and ensure consistency and fairness of its decision-making by putting accountability front and center.

R Falchetto and J Barman-Aksözen together developed the concept and drafting of the manuscript. J Barman-Aksözen was responsible for drafting the manuscript and revising the manuscript. Both authors agreed with the final version and take responsibility for the accuracy of the content of the letter. Both authors are patients with EPP and are currently under treatment with afamelanotide.

The authors received no financial and/or material support for this research or the creation of this work.

The authors have no competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

No funded writing assistance was utilized in the production of this manuscript.

The documents on HST2 are no longer available from NICE`s website but can be shared by the authors of this letter upon reasonable request.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/