Researchers evaluated an advanced genAI model known as GPT-4 to assess its capability in answering questions related to five crucial aspects of patient safety. This evaluation was conducted using a 50-question self-assessment format that is part of the Certified Professional in Patient Safety (CPPS) certification exam, which is a standardized multiple-choice test designed for patient safety experts. GPT-4 managed to answer 88% of the inquiries accurately, showcasing its impressive performance.
Generative artificial intelligence (genAI) learns from a vast array of data—often ranging from hundreds of millions to billions of data points—to generate outputs that closely resemble those created by humans. Its uses span from personalized shopping recommendations to producing audio and visual media, as well as expediting engineering designs. Within healthcare, genAI has potential applications such as improving imaging technologies, forecasting disease progression for individual patients, and aiding in the development of new vaccines.
Researchers at BU performed tests on GPT-4, a publicly available advanced genAI model, to evaluate its ability to respond accurately to questions covering five vital domains of patient safety in the CPPS self-assessment exam. The model succeeded in answering 88% of the questions correctly, which indicates a strong performance level.
“While previous studies have examined genAI’s effectiveness in evaluating exams from various healthcare fields over the past year, our research marks the first thorough assessment of its capabilities specifically concerning patient safety,” stated Nicholas Cordella, MD, MSc, the study’s lead author and an assistant professor of medicine at BU Chobanian & Avedisian School of Medicine.
James Moses, MD, MPH, once an associate professor of pediatrics at the school and now the chief of quality, safety, and patient experience at Corewell Health in Michigan, contributed as a co-author to this study.
The researchers presented GPT-4 with questions from the CPPS self-assessment exam without any prior training or medical updates. They then analyzed the model’s responses in various categories of the exam. Notably, GPT-4 excelled in the areas of Patient Safety and Solutions, Measuring and Improving Performance, and Systems Thinking and Design/Human Factors. These results prompted the researchers to highlight potential areas where patient safety experts could explore the strengths and weaknesses of AI in real-world applications.
“Our study indicates that AI could assist physicians in better identifying, addressing, and preventing errors in hospitals and clinics. Although further research is essential to grasp what current AI can accomplish regarding patient safety, this investigation demonstrates AI’s potential to enhance healthcare by supporting clinicians in reducing avoidable injuries,” remarked Cordella, who also serves as the medical director for quality and patient safety at Boston Medical Center.
Cordella emphasized the possibility of AI improving patient safety systems to address the persistent challenge of medical errors, which are estimated to result in around 400,000 fatalities annually.
Cordella expressed that this study reflects a broader perspective that AI can aid professionals, including physicians, in refining their work. By integrating AI to support their responsibilities, healthcare providers may enhance the safety and efficiency of services, akin to how other professionals are leveraging AI to elevate their performance.
The research also indicated limitations in extant AI technologies and cautioned that users should remain alert to issues of bias, undue confidence, false data, or fabrication in responses generated by large language models like GPT-4.
“Our findings suggest a significant potential for AI to enhance patient safety, representing a crucial step towards using this technology to minimize avoidable injuries and achieve improved healthcare results. However, this should be seen as a preliminary move, necessitating thorough testing and refinement of AI applications to genuinely benefit patient care,” emphasized Cordella.
The results of this study were published online in the Joint Commission Journal on Quality and Patient Safety.
