Scientists have created and successfully tested a new chatbot powered by artificial intelligence named ‘UroBot.’ This innovative chatbot was able to accurately respond to urology specialty exam questions, outdoing both other language models and seasoned urologists. Notably, it provides detailed justifications for its answers, referencing established guidelines.
Researchers from the German Cancer Research Center (DKFZ), in collaboration with doctors from the Urological Clinic at Mannheim University Hospital, have introduced and evaluated an advanced AI-based chatbot known as “UroBot.” This chatbot demonstrated a remarkable ability to answer inquiries from the urology specialist examination with a significant level of accuracy, exceeding the capabilities of other language models and the expertise of experienced urologists. It also supports its responses with detailed explanations based on the relevant guidelines.
As personalized oncology continues to evolve, urological guidelines are becoming increasingly intricate. In settings such as tumor boards, hospital wards, or clinics, a precise second-opinion system could provide essential support for medical decision-making in urology, particularly when time or resources are constrained.
Large language models (LLMs), like GPT-4, possess the ability to access medical information and tackle complicated medical inquiries without needing further training. However, their effectiveness in clinical environments is often hindered by outdated training data and a lack of clear explanations. To address these limitations, a team led by Titus Brinker at DKFZ developed “UroBot,” a urology-specific chatbot enhanced with the latest guidelines from the European Society of Urology.
UroBot utilizes OpenAI’s most advanced language model, GPT-4o. It employs a tailored retrieval-augmented generation (RAG) approach, effectively extracting relevant information from numerous documents in response to specific questions, thereby delivering precise and explainable answers. The modified version underwent testing with 200 specialized questions from the European Board of Urology and was assessed through several rounds of evaluation.
UroBot-4o achieved a correct response rate of 88.4 percent on specialist exam questions, which is an improvement of 10.8 percentage points over the current GPT-4o model. This indicates that UroBot not only surpasses other language models but also exceeds the average performance of urologists in specialty examinations, which has been reported at 68.7 percent in prior studies. Furthermore, UroBot demonstrated a high degree of reliability and consistency in its answers.
Clinical experts can verify UroBot’s responses, as it cites the key sources and relevant text sections. “The study highlights the potential of integrating large language models with evidence-based guidelines to enhance performance in specialized areas of medicine. The ability to verify and the high accuracy achieved make UroBot a valuable assistance tool in patient care,” states Brinker. “In the coming years, the use of intelligible language models like UroBot will become increasingly crucial in patient care, ensuring adherence to guidelines even as treatment decisions grow more complex.”
The research team has made the code and user instructions for UroBot available, paving the way for further advancements in urology and potentially other medical fields.
