Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study

Eleven expert panelists initially participated in the Delphi process. Each of the 25 clinical questions was evaluated by at least 6 panelists; specifically, 1 question was assessed by 9 panelists, 8 questions by 8 panelists, 9 questions by 7 panelists and 7 questions by 6 panelists. The questions, details of the consensus results, and the main summarized panelist comments are shown in Supplementary Table 1. Median scores assigned by panelists for each response and LLM are shown in Figure 1. The proportion of panelists scoring in each category (1–3 = disagreement, 4–6 = no consensus, 7–9 = agreement) per response and LLM is shown in Supplementary Figure 1.

In the first round, consensus in agreement with the responses generated by the LLMs was reached in 76% of questions for Claude, 56% for Gemini and 88% for ChatGPT. During the second round, additional consensus was achieved for 16% of questions for Claude, 40% for Gemini and 4% for ChatGPT. No question reached consensus in disagreement for any of the evaluated LLMs. Overall, after two rounds of assessment, consensus in agreement was achieved for 24 out of 25 Gemini responses (96%) and 23 out of 25 responses (92%) for both Claude and ChatGPT (Figure 2A & Supplementary Table 1).

Comparative analysis of consensus versus non-consensus items showed no statistically significant differences among the three models (p = 0.779) (Figure 2A). When evaluating responses based on median scores on a 9-point Likert scale, with scores ≥7 versus <7, Gemini achieved a median score ≥7 in all 25 responses (100%), Claude in 24 responses (96%) and ChatGPT in 23 responses (92%), again with no significant differences among models (p = 0.368) (Figure 2B). In the pairwise comparisons, no significant differences were observed between the models in either the proportion of consensus items or the proportion of items with scores ≥7 (all Bonferroni-adjusted p = 1.000) (Figure 2).

Questions requiring additional Delphi rounds or that ultimately failed to reach consensus predominantly focused on clinical scenarios marked by uncertainty and the need for individualized patient management. In particular, consensus was not reached for at least one LLM regarding the following: the appropriateness of active surveillance in metastatic RCC (question 4), the incidence of recurrence and survival outcomes in chromophobe RCC (question 10), the optimal ablative technique for localized renal cancer (question 17) and possible alternatives to renal biopsy for the differential diagnosis of small renal masses (question 19) (Supplementary Table 1).

Qualitative comments provided by expert panelists revealed some recurring limitations across the LLM-generated responses. These included occasional minor inaccuracies, omission of clinically relevant options and insufficient detail in more nuanced clinical scenarios. In some cases, the answers were perceived as overly general or not fully aligned with guideline recommendations. Such observations were more frequent in questions that underwent a second Delphi round or failed to reach consensus (Supplementary Table 1).

This study investigated the performance of three selected LLMs, Claude, Gemini and ChatGPT, in generating guideline-based responses to clinical questions related to RCC. Using a Delphi methodology, a multidisciplinary expert panel systematically rated their agreement with each model's output. A consensus in agreement was reached in 92% to 96% of cases across the models, with median scores of 7 or higher observed in 92–100% of responses. No statistically significant differences were found between the models. However, given the limited number of matched questions and the variable number of raters across items, this finding should not be interpreted as demonstrating equivalent performance among the models. Several questions, particularly those addressing areas of clinical uncertainty or requiring nuanced, individualized decision-making, either necessitated a second Delphi round or failed to reach consensus. These results indicate that, while LLMs frequently generated guideline-concordant information and may support guideline interpretation in routine scenarios, their performance varied in more complex settings, and their outputs require expert oversight.

LLMs are increasingly being adopted as tools to support evidence-based decision-making in oncology [20,21]. These advanced AI systems are trained on extensive biomedical literature and clinical guidelines, enabling them to generate detailed, contextually relevant responses to complex clinical queries [22,23]. LLMs and other AI-driven tools can synthesize large volumes of clinical information, assist with treatment planning, and predict outcomes in selected scenarios, potentially bridging the gap between evolving guidelines and real-world clinical practice [23,24]. Nevertheless, the reliability and safety of LLM-generated recommendations require careful validation before clinical integration, as variations in content accuracy and clinical completeness persist, particularly in ambiguous or high-stakes cases [22,25].

A detailed examination of our results reveals variability in panelist agreement across the 25 clinical questions related to RCC. While several questions involving standard management achieved high levels of consensus, others addressing clinically complex scenarios were also consistently rated in agreement. However, questions that required a second Delphi round or failed to reach consensus with one or more models tended to reflect areas of greater uncertainty or scenarios requiring individualized decisions. The non-consensus scenarios involved active surveillance in metastatic RCC, prognosis in chromophobe RCC, selection of ablative techniques for localized disease and alternatives to renal biopsy. Panelists’ comments additionally identified unsupported references to CARMENA and SURTIME in the active-surveillance context and omission of imaging-based alternatives to renal biopsy, including sestamibi SPECT/CT and Zr-girentuximab PET/CT (Supplementary Table 1). These findings illustrate recurring limitations such as omission of clinically relevant options, insufficient clinical nuance, and occasional factual inaccuracies. Importantly, no question resulted in consensus in disagreement, suggesting overall alignment with clinical expectations, albeit with limitations. Overall, these observations underline the potential of LLMs to support guideline interpretation, while reinforcing the need for expert validation, especially in areas where established guidance is limited.

Recent evidence in RCC supports this caution [21]. In a pilot study by Bertolo et al., AI-generated treatment suggestions were compared against decisions made by a multidisciplinary tumor board across 103 RCC cases. While an overall agreement of 62% was reached, concordance varied widely depending on the disease stage, with the lowest alignment observed in advanced metastatic settings [21]. These findings suggest that although LLMs can assist clinicians in standard RCC scenarios, their integration into routine practice is limited by their current inability to fully interpret complex or individualized clinical contexts. Therefore, continuous oversight, model refinement and external validation will be crucial to ensure their safe use in urologic oncology.

Several recent studies in general and urological oncology further support and contextualize these findings. Longwell et al. evaluated the performance of a leading proprietary LLM using clinical questions standardized for oncology certification exams from the American Society of Clinical Oncology (ASCO) and the European Society for Medical Oncology (ESMO). These questions, similar to those used to assess the knowledge of practicing clinical oncologists, tested the model’s ability to provide high-level, context-specific responses. The study reported an overall accuracy of 85.0%, with most explanations containing no or only minor errors. However, 82% of the incorrect responses were judged to have the potential to cause moderate or severe clinical harm if applied in practice, highlighting the need for expert oversight even in the presence of high accuracy [23]. In urological oncology, Eckrich et al. compared several LLMs with practicing urology consultants using case-based clinical questions. Although the models produced fluent and comprehensible answers, their medical adequacy was consistently rated lower than that of human experts. Importantly, potential misinformation was identified in between 2.8% and 18.9% of the LLM-generated responses, compared with less than 1% for consultant answers [26]. Similarly, Hoen et al. evaluated the ability of LLMs to assist in updating pediatric urology guidelines and found that, although the models were often able to generate scientifically accurate outputs, they lacked the consistency and reliability required to replace expert panel review, particularly when integrating new or emerging evidence [27]. Together with our findings in RCC, these studies reinforce that while LLMs hold promise as supportive tools in clinical decision-making, their current performance remains variable and insufficiently reliable for unsupervised use. Rigorous validation and expert oversight are crucial for ensuring their safe and effective integration into clinical urological oncology practice.

A notable challenge in evaluating LLMs in oncology is the lack of standardized methodology across studies. As noted by Carl et al., existing research employs diverse strategies, ranging from multiple-choice formats and open-ended clinical prompts to comparisons with expert decisions, which complicates the direct comparison of results [22]. In this study, we employed a Delphi process involving a multidisciplinary panel of experts from urology, oncology, and other specialties, offering the advantages of structured consensus, iterative review and detailed qualitative feedback. However, other evaluation approaches may also provide valuable insights, and the current heterogeneity in methodology underscores the need for greater harmonization in future research on medical applications of LLMs.

This issue is particularly relevant when interpreting the differences between our current results and those of a previous study, which used a different evaluation framework. In a previous study evaluating LLM performance across major urological cancers, we applied a direct expert rating approach, in which 9–11 multidisciplinary uro-oncology specialists independently scored each response using a standardized 5-point Likert scale [14]. That evaluation suggested Claude’s superior performance. In contrast, the present study, which focused exclusively on RCC, employed a Delphi methodology that incorporated expert feedback and iterative consensus rounds. Under these conditions, Gemini attained the highest numerical level of agreement among panelists, although no statistically significant differences were observed among the three models. While methodological differences may account for part of the variation, the most likely explanation lies in the rapid and ongoing updates of LLM architectures and training data, which can substantially alter model behavior within short periods. These findings underscore the importance of continuous validation, as performance cannot be assumed stable over time or comparable across versions, even within the same model family.

Taken together, these findings highlight the potential of LLMs as supportive tools in RCC, particularly in standard settings or resource-limited environments, where they can facilitate quick access to guideline-based recommendations and help reduce care disparities [22,23]. By enabling rapid and consistent access to clinical recommendations, they may help reduce variability in care. However, their integration into practice should be gradual, focusing first on low-risk tasks such as education, documentation or triage, and expanding only with robust evidence of safety and clinical utility [23,24]. Transparent reporting of model characteristics and independent validation remain essential, as does close collaboration between clinicians, AI developers and patients to ensure alignment with real-world needs [22,24]. To support safe implementation, clinical guidelines, especially in complex areas, should also adopt clear and structured language interpretable by both humans and AI systems.

This study has several limitations that should be considered when interpreting the results. First, it focused exclusively on RCC and included a relatively small number of predefined clinical scenarios, which may limit the generalizability of the findings to other tumor types or more diverse clinical contexts. Second, the evaluation was based on expert consensus through a Delphi process, a robust method, but one that remains inherently subjective and may be influenced by individual interpretation or the composition of the panel. Moreover, no predefined score-specific rubric was established for each individual point on the 9-point scale, which may have contributed to variability in panelists’ interpretation of intermediate scores. In addition, only one response was generated per question and model. As the models were accessed through their user-facing platforms, temperature, top-p, top-k and other sampling parameters were not user-configurable or available for reporting and were left at the platform defaults. Repeated runs were not performed; therefore, response stability and variability across generations could not be assessed. Third, in some questions, especially those with greater specialty specificity, only a subset of the panel voted, and consensus was calculated exclusively among those who responded. While this increases content expertise for each item, it may introduce potential bias due to variable sample sizes across questions, which should be taken into account when interpreting the results. Moreover, the comparison was based on only 25 matched questions, with variable numbers of expert raters across items. Accordingly, the absence of statistically significant differences should not be interpreted as evidence of equivalent performance among the models. Furthermore, the rapid pace of development and frequent updates in LLMs mean that their performance is dynamic and may evolve significantly after the evaluation period. Finally, only proprietary, user-facing LLMs were evaluated, and open-weight models such as Llama, Qwen and DeepSeek were not included. Because the performance of open-weight models may vary according to model size, quantization, hardware, inference framework and implementation parameters, the present findings may not be generalizable to these architectures. Future studies should directly compare proprietary and open-weight models under standardized deployment and prompting conditions.

Despite these limitations, the study also has important strengths. In particular, figures and tables from the clinical guideline were converted into structured, plain-language textual summaries to facilitate consistent interpretation by the AI models while preserving the original guideline content. The use of a multidisciplinary Delphi methodology further strengthened the clinical relevance of the evaluation, allowing for both quantitative consensus assessment and qualitative analysis of model responses.

Our results indicate that selected LLMs, when provided with structured, guideline-derived information, can generate responses that frequently align with multidisciplinary expert consensus. However, this controlled evaluation assessed their ability to interpret supplied guideline content and should not be considered evidence of independent clinical reasoning or real-world decision-making performance. Persistent limitations were observed in more complex settings where clinical judgment remains indispensable. While LLMs show potential as tools to support guideline interpretation, their outputs should be interpreted with appropriate caution and under expert oversight. To enhance their safe integration into clinical workflows, we recommend that urological oncology guidelines adopt clear and structured formats that are readily usable by both clinicians and AI systems, and that future research continues to refine evaluation methodologies and assess performance consistency as LLMs evolve.

A Borque-Fernando led the project and contributed to the study’s conceptualization, methodology, project administration and supervision. All authors contributed jointly to the investigation, formal analysis, data curation, validation, resources, visualization, writing of the original draft and review and editing of the manuscript, and approved the final version for publication.

The authors thank Dr Pablo Rivas for his assistance with medical writing on behalf of Content Ed Net.

This project was funded by the Spanish Association of Urology. The authors received support from Instituto de Investigación Sanitaria Aragón (Spain) for the project “Uro-Onco GuÍAs: Desarrollo de un Chatbot basado en IA Generativa para la Interpretación de Guías Clínicas en Uro-Oncología”.

The authors have no competing interests or relevant affiliations with any organization or entity with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Medical writing of the manuscript through Content Ed Net was funded by the Spanish Association of Urology.

This study involved only expert panelists who participated voluntarily in the Delphi process. No patient data were used. Participation implied informed consent, and formal ethical approval was not required.

Participation by expert panelists was voluntary and implied informed consent.

The results of the Delphi process are presented within the manuscript. The complete set of responses generated by each large language model to all 25 questions are available for consultation at https://www.contentednet.com/proyecto-urooncoguias-aeu/.

This work is licensed under the Attribution-NonCommercial-NoDerivatives 4.0 Unported License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/