A medically grounded LLM agent–based tool to detect patient safety events in medical records

Abstract

Large language models (LLMs) have shown incredible promise in medicine. While LLMs may be particularly useful in areas requiring extensive review of clinical records, their use remains limited due to their tendency to hallucinate and fabricate information. Hallucination issues, as well as their consequences, are exacerbated in low–probability, high–stakes scenarios such as rare adverse safety events or medical errors. We present SAFE–AI (Structured and Automated Framework for Explainable AI), a novel method for clinical decision making that combines the strengths of clinical expert knowledge with LLMs in an ontology–driven model that minimizes hallucinations using strict rules. 

Introduction

It is estimated that over 250,000 adverse safety events (ASEs) occur annually in the United States among patients receiving medical care, resulting in more than 100,000 patient deaths [1–3]. ASEs, such as medication, diagnostic, or procedural errors; miscommunications; and healthcare associated infections, pose significant risks to patient safety in healthcare settings. The current standard for detecting ASEs is for expert clinicians to manually review clinical records [4,5]. Manual chart reviews are labor-intensive, slow, and costly. Consequently, ASEs are undetected and under-reported, with over 95% of documented ASEs in medical records not being included in the official records, which hampers quality improvement and patient safety efforts [6].

Methods

Overview

SAFE–AI is structured in four major steps (Fig 1). Firstly, an ontology of epinephrine errors is defined by clinician experts using established clinical guidelines, and this ontology is converted into a directed graph. Secondly, an LLM is used to retrieve initial nodes in the graph - such as patient age and weight from an unstructured patient chart. Thirdly, a coding LLM translates the guidelines defined in the ontology into executable code. Lastly, a final prediction is made by executing the computational graph of all nodes from the initial inputs retrieved by the LLM to the final output node. 

Results

Clinician annotation and inter-rater agreement

A final sample of 300 pediatric emergency medical charts was reviewed by clinician experts. Gwet’s AC1 showed excellent inter–rater agreement among clinical reviewers for evaluating epinephrine ASEs, with values of 0.85 for delays of ≥ 10 minutes to administer epinephrine and 0.96 for incorrect epinephrine doses.

Discussion

In this study, we proposed SAFE–AI, a framework that combines flexible language understanding with explicit sym- bolic reasoning grounded in clinical guidelines. We demonstrated the capabilities of SAFE–AI in detecting overdoses and delays in medication administration in clinical charts, and found that SAFE–AI shows strong overall performance and consistently competitive or superior F1-scores relative to the evaluated baselines. While LLM-only approaches often achieved high recall, they were more prone to false-positive predictions and inconsistent reasoning behavior, particularly in complex cases.
Citation: Trujillo D, Wang D, Bahr N, Yi-Jin Hsieh T, Cho B, Meckler G, et al. (2026) A medically grounded LLM agent–based tool to detect patient safety events in medical records. PLOS Digit Health 5(9): e0001174. https://doi.org/10.1371/journal.pdig.0001174

Editor: Xiaoli Liu, Chinese PLA General Hospital, CHINA

Received: December 15, 2025; Accepted: July 23, 2026; Published: September 17, 2026

Copyright: © 2026 Trujillo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The code repository for the proposed work is publicly available at: https://github.com/dietrujillo/medically_grounded_llm. The authors had permission to use records from AMR under a data use agreement for the study but do not have authority to publicly share or redistribute them. The records contain health information and are subject to applicable privacy, ethical, institutional, and/or legal restrictions. The authors did not receive special access privileges beyond those available to qualified researchers who obtain the required approvals. Contact information for AMR AMR Corporate Headquarters 6501 S Fiddlers Green Cir, Suite 100 Greenwood Village, CO 80111AMR Corporate Headquarters 6501 S Fiddlers Green Cir, Suite 100 Greenwood Village, CO 80111 303.495.1200 https://www.amr.net/patients/medical-records.

Funding: Authors DT, DW, NB, GM, MH, CE, SB, XJ, and JMG were supported through a grant by the National Heart, Lung, and Blood Institute (NHLBI) grant number R01HL161385. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.