A case demonstration of the Open Health Natural Language Processing Toolkit from the National COVID-19 Cohort Collaborative and the Researching COVID to Enhance Recovery programs for a natural language processing system for COVID-19 or postacute sequelae

Wen, A; Wang, L; He, H; et al., JMIR Medical Informatics, September 2024

View Publication on PubMed

September 2024
JMIR Medical Informatics

Short Summary

RECOVER researchers know that there is a lot of important information about COVID-19 and Long COVID in electronic health records (EHRs), especially in the notes that doctors write during care. The researchers used a computer tool called Natural Language Processing (NLP) to find signs and symptoms of different conditions. However, NLP doesn’t always work well, especially with new health problems like COVID-19 or Long COVID, which are always changing. To improve this, researchers created a new NLP system that could collect detailed information about Long COVID signs and symptoms from EHRs. After testing the system at other medical sites, they found it worked well across different locations. The study showed that the new NLP system could successfully find the information they were looking for about COVID-19 and Long COVID in EHRs. The NLP is now available to other researchers and is being used to collect information that can support additional studies on COVID-19 and Long COVID.

This summary was prepared by the RECOVER Initiative.

Publication Details

DOI: 10.2196/49997

Abstract

Background: A wealth of clinically relevant information is only obtainable within unstructured clinical narratives, leading to great interest in clinical natural language processing (NLP). While a multitude of approaches to NLP exist, current algorithm development approaches have limitations that can slow the development process. These limitations are exacerbated when the task is emergent, as is the case currently for NLP extraction of signs and symptoms of COVID-19 and postacute sequelae of SARS-CoV-2 infection (PASC).

Objective: This study aims to highlight the current limitations of existing NLP algorithm development approaches that are exacerbated by NLP tasks surrounding emergent clinical concepts and to illustrate our approach to addressing these issues through the use case of developing an NLP system for the signs and symptoms of COVID-19 and PASC.

Methods: We used 2 preexisting studies on PASC as a baseline to determine a set of concepts that should be extracted by NLP. This concept list was then used in conjunction with the Unified Medical Language System to autonomously generate an expanded lexicon to weakly annotate a training set, which was then reviewed by a human expert to generate a fine-tuned NLP algorithm. The annotations from a fully human-annotated test set were then compared with NLP results from the fine-tuned algorithm. The NLP algorithm was then deployed to 10 additional sites that were also running our NLP infrastructure. Of these 10 sites, 5 were used to conduct a federated evaluation of the NLP algorithm.

Results: An NLP algorithm consisting of 12,234 unique normalized text strings corresponding to 2366 unique concepts was developed to extract COVID-19 or PASC signs and symptoms. An unweighted mean dictionary coverage of 77.8% was found for the 5 sites.

Conclusions: The evolutionary and time-critical nature of the PASC NLP task significantly complicates existing approaches to NLP algorithm development. In this work, we present a hybrid approach using the Open Health Natural Language Processing Toolkit aimed at addressing these needs with a dictionary-based weak labeling step that minimizes the need for additional expert annotation while still preserving the fine-tuning capabilities of expert involvement.

Authors

Andrew Wen, Liwei Wang, Huan He, Sunyang Fu, Sijia Liu, David A Hanauer, Daniel R Harris, Ramakanth Kavuluru, Rui Zhang, Karthik Natarajan, Nishanth P Pavinkurve, Janos Hajagos, Sritha Rajupet, Veena Lingam, Mary Saltz, Corey Elowsky, Richard A Moffitt, Farrukh M Koraishy, Matvey B Palchuk, Jordan Donovan, Lora Lingrey, Garo Stone-DerHagopian, Robert T Miller, Andrew E Williams, Peter J Leese, Paul I Kovach, Emily R Pfaff, Mikhail Zemmel, Robert D Pates, Nick Guthe, Melissa A Haendel, Christopher G Chute, Hongfang Liu, National COVID Cohort Collaborative, RECOVER Initiative

Keywords

COVID; COVID-19; NLP; OHNLP; Open Health Natural Language Processing; PASC; SARS-CoV-2; clinical information extraction; clinical phenotyping; extract; extraction; narratives; natural language processing; phenotype; phenotyping; unstructured