A recent study led by UCSF has found that artificial intelligence could potentially assist in prioritizing patients in overcrowded and overtaxed emergency departments.
Artificial intelligence (AI) is being considered as a potential solution to help determine which patients require urgent treatment in overcrowded and overburdened emergency departments. The study analyzed anonymized records of 251,000 adult emergency department (ED) visits and found AI to be as effective as a physician in prioritizing patient needs.
Researchers at UC San Francisco conducted a study to assess an AI model’s ability to extract symptoms from patients’ clinical notes and determine the urgency of their need for treatment. They compared the AI analysis with the patients’ scores on the Emergency Severity Index, a scale used by ED nurses to prioritize care and resources for arriving patients. The study, which will be published on May 7, 2024, in JAMA Network Open, involved de-identified patient data. The researchers used the ChatGPT-4 large language model (LLM) to analyze the data.
UCSF’s secure generative AI platform was used to analyze patient data, with a focus on protecting privacy. The LLM’s performance was tested using a sample of 10,000 matched pairs, totaling 20,000 patients. Each pair included one patient with a serious condition, such as a stroke, and another with a less urgent condition, such as a broken wrist. Using only the patients’ symptoms, the AI accurately identified the ED patient with the more serious condition 89% of the time.
In a sub-sample of 500 pairs, both the AI and a physician evaluated the patients. The AI’s accuracy was 88%, compared to 86% for the physician.
AI could assist in the triage process to free up critical physician time for treating patients with the most serious conditions. It also provides backup decision-making tools for clinicians managing multiple urgent requests.
Lead author Christopher Williams, MB, BChir, a UCSF postdoctoral scholar at the Bakar Computational Health Sciences Institute, gave an example of a scenario where two patients need to be transported to the hospital with only one ambulance available. Another scenario is when a physician is on call and has three people paging her simultaneously, and she needs to prioritize who to respond to first.
Â
The research is one of the few that has looked at the effectiveness of an LLM using real clinical data instead of simulated situations. It is also the first to use over 1,000 clinical cases for this purpose and to use data from visits to the emergency department, where there are many different medical conditions. Despite the success in this study, Williams warned that AI is not yet ready to be used responsibly in the ED without further validation and clinical trials. Showing that AI can do impressive things is great, but it’s essential to consider who is being helped.
and who is being hindered by this technology,” said Williams. “Is just being able to do something the standard for using AI, or is it being able to do something well, for all types of patients?”
One important issue to untangle is how to eliminate bias from the model. Previous research has shown these models may perpetuate racial and gender biases in health care, due to biases within the data used to train them. Williams said that before these models can be utilized, they will need to be adjusted to remove that bias.
“First we need to determine if it works and comprehend how it works, and then be cautious and intentional inWilliams explained, “The next step is to determine the best way to implement this technology in a clinical environment.” The study is published in JAMA Network Open.
