Expert panel agreement with large language model responses to uro-oncology guideline-based questions: insights from seven Delphi consensus panels across major urological cancers

The integration of artificial intelligence (AI) into clinical practice represents a paradigm shift in medicine, offering novel tools that may facilitate access to and interpretation of evidence-based information [1,2]. Among these emergent technologies, large language models (LLMs) have demonstrated remarkable aptitude for processing and synthesizing complex textual information, such as clinical practice guidelines [3–5]. The field of uro-oncology, with its diverse tumor types, rapidly evolving standards, and nuanced therapeutic pathways, is particularly poised to benefit from such computational advances.

This supplement presents coordinated Delphi studies that rigorously assess how well three leading LLMs, Claude [6], Gemini [7] and ChatGPT [8], interpret the 2025 European Association of Urology (EAU) guidelines and respond to related clinical questions across seven major urological cancers [9–15]. The work is organized into four complementary articles, each focusing on a distinct group of cancers: penile and testicular cancers, renal cell carcinoma, prostate cancer and muscle-invasive and nonmuscle-invasive bladder cancer together with upper tract urothelial carcinoma. For each cancer type, the studies assessed expert panel agreement with LLM responses to 25 structured clinical questions using disease-specific multidisciplinary panels of 7 to 11 specialists.

This multidisciplinary initiative was developed under the coordination of the Spanish Association of Urology (AEU), with technical implementation carried out by the Aragón Institute of Health Research (IIS-A) and the Aragón Institute of Engineering Research (I3A). The EAU signed a collaboration agreement with the AEU, granting permission to use the 2025 EAU Guidelines within the project. This collaboration supports the evaluation and future development of guideline-conditioned AI tools.

The project’s multidisciplinary nature is reflected in the ongoing involvement of key Spanish scientific societies across all seven tumor-specific panels. These include the AEU, Spanish Society of Radiation Oncology (SEOR), Genitourinary Alliance of Research and Development (GUARD), Spanish Society of Nuclear Medicine and Molecular Imaging (SEMNIM), Spanish Society of Anatomical Pathology (SEAP), Spanish Society of Hospital Pharmacy (SEFH) and Spanish Society of Medical Radiology (SERAM). Their participation ensures comprehensive expert input and oversight throughout the project.

A foundational step for this project was a comprehensive comparative evaluation of six LLM configurations, using both simple and rephrased prompts of Claude, Gemini and ChatGPT, across seven major uro-oncological cancers [16]. For each tumor type, 25 carefully designed clinical questions based on the 2025 EAU guidelines were presented to the LLMs, which generated corresponding responses. These responses were independently assessed by 9 to 11 experienced specialists who were blinded to the model identities. Ratings were assigned using a 5-point Likert scale, where 1 indicated ‘incorrect/unacceptable’ and 5 indicated ‘optimal.’ Across a total of 10,500 expert evaluations, Claude’s simple prompt configuration achieved the highest proportion of top scores, with 45.9% rated as ‘optimal’ (score 5) and 87% rated as either ‘optimal’ or ‘acceptable’ (scores 4 or 5). Gemini and ChatGPT demonstrated lower frequencies of optimal responses but maintained acceptable performance, with 68–70% of responses rated in the combined optimal or acceptable categories. Interestingly, rephrased prompts did not consistently enhance performance; in some cases, simple prompts yielded better results. This rigorous evaluation provided insight into both the strengths and limitations of these models, highlighting the critical importance of thorough expert validation and meticulous data preparation for the rigorous evaluation of guideline-conditioned LLM responses.

A crucial step in the project’s development was the meticulous adaptation of complex, multimodal guideline content, including all figures, tables and flowcharts from the EAU guidelines on seven major uro-oncological cancers, into clear, structured, plain-language textual summaries. The guidelines were organized into thematic sections and the transformation into plain language was independently reviewed by members of the specialized tumor boards. Any discrepancies or uncertainties arising during this process were resolved by consensus among the guideline coordinators. This process was designed to reduce information loss and ensure that the transformed guideline content remained faithful to the original materials.

In the four studies presented in this supplement, three specific model versions were evaluated: Claude 3.5 Haiku (Anthropic; version claude-3-5-haiku-20241022, launched 22 October 2024) [6], Gemini 2.5 Flash (Google; version gemini-2.5-flash-preview-05-20, launched 20 May 2025) [7], and GPT-4.1-mini (OpenAI; version gpt-4.1-mini-2025-04-14, launched 14 April 2025) [8]. The original English-language 2025 EAU guideline content was provided to each model, with figures and tables converted into structured, plain-language textual summaries. The same guideline materials were supplied to all three models. For each of the seven clinical entities, 25 structured questions were submitted in Spanish without modification, independently and one at a time, and each model was evaluated separately rather than within a continuous conversation. One response was generated per question and model, resulting in 525 model responses. Disease-specific multidisciplinary panels of 7 to 11 specialists evaluated the responses using a 9-point scale and a two-round modified Delphi process. Panelists were blinded to model identity and could abstain from questions outside their area of expertise; only submitted ratings were included in the calculations. In the first round, all responses were rated. Responses not achieving consensus were reassessed in a second round after panelists reviewed anonymized, aggregated feedback summarizing the group ratings. The LLM response text remained unchanged between rounds, and only panelist ratings could be revised. Consensus was defined by three criteria: a median score within the agreement (7–9) or disagreement (1–3) range; fewer than a third of panelists rating outside these ranges and an interquartile range below 4.

Across the seven clinical entities, consensus in agreement ranged from 56% to 96% for Claude, 92% to 100% for Gemini and 80% to 96% for ChatGPT. These results describe expert panel agreement under the specific study conditions and do not establish factual accuracy, clinical safety or independent clinical decision-making ability. Comparisons apply only to the specific model versions tested. Because these versions were released at different times, the observed differences cannot be attributed solely to vendors or interpreted as stable rankings of model families.

Across the studies, lower panel agreement or the need for further discussion was more frequent in scenarios involving advanced staging, prognostic nuances, limited clinical evidence or complex clinical decisions. These findings reinforce the need for specialist review, particularly in complex or atypical scenarios. Similar caution has been recommended in other studies of LLMs in oncology and urology [4,5,17,18].

The frequent expert panel agreement observed supports further evaluation of guideline-conditioned LLMs. However, the studies used limited sets of predefined questions, variable panel participation, and one response per question and model. They did not assess factual accuracy against an independent reference standard, patient outcomes, workflow integration, clinical safety or performance in real clinical settings. Therefore, the findings do not establish real-world utility or readiness for autonomous decision support. The structured adaptation of guideline materials may nevertheless provide a useful basis for developing more standardized approaches to evaluating LLM-generated responses.

The development of the urooncoguias.aeu.es chatbot represents a potential future application of this initiative. Its clinical use will require broader validation, repeated testing, assessment in real clinical settings and continued expert oversight. Together, the studies in this supplement provide a coordinated foundation for further research into the interpretation of uro-oncology guideline information by LLMs.