Osaka Metropolitan University researchers evaluated how well ChatGPT and radiologists perform in diagnosing brain tumors using MRI reports from 150 cases. Their results could be surprising.
Leveraging authentic Japanese language data, this large language model demonstrates an accuracy comparable to that of neuroradiologists.
With the rapid evolution of artificial intelligence, its applications and functionalities in practical scenarios are expanding, potentially outpacing human skills. In radiology, where accurate diagnoses are essential for effective patient treatment, models like ChatGPT could enhance precision or serve as a reliable second opinion.
To evaluate this potential, graduate student Yasuhito Mitsuyama and Associate Professor Daiju Ueda from Osaka Metropolitan University’s Graduate School of Medicine spearheaded a project comparing the diagnostic accuracy between ChatGPT, powered by GPT-4, and radiologists concerning 150 MRI reports of brain tumors prior to surgery. The study assessed ChatGPT alongside two board-certified neuroradiologists and three general radiologists, all tasked with presenting differential diagnoses and final diagnoses based on clinical notes written in Japanese.
Afterward, their accuracy rates were determined by the actual diagnosis confirmed post-surgery. The findings revealed that ChatGPT achieved an accuracy of 73%, closely followed by neuroradiologists at 72% and general radiologists at 68%. Interestingly, ChatGPT’s accuracy for final diagnoses differed based on the origin of the clinical report: it registered an 80% accuracy with reports from neuroradiologists, whereas it dropped to 60% with those from general radiologists.
“These findings indicate that ChatGPT can be a valuable tool for the preoperative imaging assessment of brain tumors,” remarked graduate student Mitsuyama. “Moving forward, we plan to investigate the application of large language models across various diagnostic imaging areas, aiming to alleviate the workload on healthcare professionals, enhance diagnostic precision, and utilize AI to foster educational settings.”
