A recent study has uncovered methods for implementing large language models (LLMs), a form of artificial intelligence (AI), in healthcare systems, all while keeping costs in check and ensuring effective performance. This research highlights how health organizations can utilize these advanced AI technologies to automate processes efficiently, conserving time and lowering operational expenses, all while ensuring reliability even during peak task demands.
A recent study from researchers at the Icahn School of Medicine at Mount Sinai has unveiled strategies for deploying large language models (LLMs)—a kind of artificial intelligence (AI)—within healthcare systems while preserving cost-effectiveness and performance levels.
Published in the online edition of npj Digital Medicine on November 18, these findings shed light on how health systems can tap into cutting-edge AI tools to effectively automate tasks, thereby saving time and reducing operational expenses while ensuring the reliability of these models, even when under significant workload stress.
“Our results provide a guideline for healthcare systems to effectively integrate advanced AI tools for task automation, potentially reducing costs related to application programming interface (API) calls for LLMs by as much as 17 times while maintaining consistent performance during heavy usage,” says co-senior author Girish N. Nadkarni, MD, MPH. He holds the Irene and Dr. Arthur M. Fishberg Professorship of Medicine at Icahn Mount Sinai and serves as the Director of The Charles Bronfman Institute of Personalized Medicine and Chief of the Division of Data-Driven and Digital Medicine (D3M) at Mount Sinai Health System.
Healthcare providers generate extensive amounts of data daily. LLMs, such as OpenAI’s GPT-4, present promising opportunities for automating and streamlining workflows across a range of tasks. Nonetheless, the ongoing operational expenses of using these AI models can be a significant financial hurdle for widespread application, the researchers point out.
“The motivation behind our study was to discover practical solutions to minimize expenses without compromising performance, enabling health systems to confidently adopt LLMs on a larger scale. We aimed to ‘stress test’ these models to assess their effectiveness in managing multiple simultaneous tasks and identify methods that could sustain high performance while keeping costs manageable,” explains first author Eyal Klang, MD, Director of the Generative AI Research Program in the D3M at Icahn Mount Sinai.
This study examined 10 LLMs using actual patient data, focusing on their responses to various clinical queries. The team conducted over 300,000 tests, progressively increasing the workload to determine how efficiently the models coped with escalating demands.
In addition to checking accuracy, the researchers also measured how well the models adhered to clinical guidelines, followed by an economic analysis. They discovered that consolidating tasks could help healthcare institutions minimize AI-related expenses while maintaining strong model performance.
The research revealed that by specifically grouping up to 50 clinical tasks—such as matching patients for clinical trials, designing research cohorts, extracting epidemiological study data, assessing medication safety, and identifying candidates for preventive health screenings—LLMs can effectively manage them simultaneously with minimal reduction in accuracy. This approach suggests that hospitals can enhance their workflows and potentially cut API expenses by up to 17 times, translating to significant savings—potentially millions of dollars annually for larger health systems—making the use of advanced AI tools more financially viable.
“It is critical to identify the threshold at which these models start to experience difficulties under heavy cognitive loads to ensure reliability and operational stability. Our findings lay out a realistic approach for integrating generative AI into healthcare settings and encourage further exploration of LLMs’ capabilities within actual constraints,” highlights Dr. Nadkarni.
One of the surprising discoveries, according to the researchers, was that even sophisticated models like GPT-4 exhibited signs of strain when tested to their cognitive limits. Rather than showing minor errors, their performance would occasionally degrade unpredictably under high stress.
“This study holds crucial implications for the integration of AI into healthcare systems. By grouping tasks for LLMs, not only do we cut costs, but we also safeguard resources for more direct patient care,” states co-author David L. Reich, MD, Chief Clinical Officer of the Mount Sinai Health System; President of The Mount Sinai Hospital and Mount Sinai Queens; Horace W. Goldsmith Professor of Anesthesiology; and Professor of Artificial Intelligence and Human Health as well as Pathology, Molecular and Cell-Based Medicine at Icahn Mount Sinai. “Moreover, by understanding the cognitive limitations of these models, healthcare providers can enhance the utility of AI while reducing risks, thereby ensuring these tools function as dependable support within critical healthcare scenarios.”
Looking ahead, the research team intends to investigate how these models perform in real-time clinical situations, managing actual patient workloads and directly engaging with healthcare teams. They also plan to assess emerging models to determine if cognitive thresholds change with technological advancements. Their overarching aim is to establish a dependable framework for integrating AI in healthcare, providing systems with tools that balance efficiency, accuracy, and cost-effectiveness to improve patient care without introducing additional risks.
