Isa Spiero, Merijn H. Rijk, Matthew A. Scheeres, Frans H. Rutten, Geert-Jan Geersing, Tamara N. Platteel, Karel G.M. Moons, Lotty Hooft, Johanna A.A. Damen, Roderick P. Venekamp, Artuur M. Leeuwenberg
Abstract
Electronic health records (EHRs) provide a large source of data that can be used for research purposes. Extraction of information from unstructured clinical notes in EHRs can be automated by large language models (LLMs). Although LLMs are promising for this task, challenges remain in reliable application of LLMs to EHR, including the lack of development and validation for languages other than English.
Introduction
Electronic health records (EHRs) are increasingly used in clinical research as they form a valuable source of information on patients and health care use [1]. Generally, EHRs contain structured data, such as disease codes and demographics, and unstructured data, such as clinical notes [2].
Materials & methods
We evaluated the performance of Dutch LLMs in a case study aiming to extract nine signs and symptoms from EHR data of adult LRTI patients presenting to their general practitioner (GP).
Results
As direct classifiers, the MedRoBERTa.nl and RobBERT models performed similarly in classifying the presence of LRTI signs and symptoms from the Dutch clinical notes (Fig 1).
Discussion
We evaluated the performance of two LLMs for the Dutch language, RobBERT and MedRoBERTa.nl, for extracting signs and symptoms of LRTI patients from Dutch EHR data. Overall, the models performed similarly, but variation was observed across implementation methods (direct classification outperformed prompt-based classification), across signs and symptoms that were extracted (higher performance for signs or symptoms with limited missingness and class-imbalance), and across the sample size of the training data.
Conclusion
After a thorough search for currently available LLMs for the extraction of signs and symptoms from Dutch EHR data, we selected the MedRoBERTa.nl and RobBERT models and compared their performance in a case study.
Citation: Spiero I, Rijk MH, Scheeres MA, Rutten FH, Geersing G-J, Platteel TN, et al. (2026) Comparison of local large language models for extraction of signs and symptoms data from electronic health records. PLoS One 21(6): e0350625.
https://doi.org/10.1371/journal.pone.0350625
Editor: Ardashir Mohammadzadeh, University of Bonab, IRAN, ISLAMIC REPUBLIC OF
Received: November 20, 2025; Accepted: May 17, 2026; Published: June 10, 2026
Copyright: © 2026 Spiero et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The code used to train and evaluate the models used in this study is shared on: https://github.com/isa-sp/Local-LLM-comparison-LRTI. Data access is restricted to protect patient privacy and has been granted under license of the current study. Data are therefore only available from the authors or the Julius General Practitioners Network (JGPN) upon reasonable request and after formal permission of the JGPN (Website: https://www.juliushuisartsennetwerk.nl/en/, Email: SecretariaatJHN-3@umcutrecht.nl).
Funding: This research was funded by ZonMw, as part of the ELEMENT project (grant number 08391052110003) and the Dutch Research Council, as part of the project “RAISE: Responsible AI Science Explorations” (Grant Number NWA.1418.22.008). The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Competing interests: The authors have declared that no competing interests exist.
