Using Natural Language Processing Methods to Predict Topics Included in 2019 Ohio Syphilis Disease Intervention...

Free-text notes in disease intervention specialist (DIS) records may contain relevant information for sexual transmitted infection control. In their current form, the notes are not analyzable without manual reading, which is labor-intensive and prone to error.

We used natural language processing methods to analyze 2019 Ohio DIS syphilis records with nonmissing notes (n = 1987). We identified 21 topics relevant for transmission and case investigations. We manually coded these records to create "gold standard" labels for each topic (0 = topic not present, 1 = topic present), then trained machine learning models to identify the topics in the text. For models to analyze text data, the text must be converted to numbers. We explored 2 approaches to numerically represent words: (1) term frequency, inverse document frequency, which measures importance of words based on how many times they appear in a record and in the dataset as a whole, and (2) GloVe embeddings, which are numerical vectors that were developed by researchers for each word in the English language to encode its semantic meaning. We explored 3 types of statistical models (naive Bayes, support vector machine, and logistic regression) using term frequency, inverse document frequency, and 1 type of neural network model (long short-term memory [LSTM] model) using GloVe. All models were used for binary prediction (i.e., topic not present, topic present).

For most topics, the LSTM model performed the best overall in identifying topics, and the support vector machine model performed the best among the statistical models. For example, the LSTM model predicted the topic "substance use" with high accuracy (97%), sensitivity (92%), and specificity (98%). No model performed well for uncommon topics (e.g., "alcohol use" or "delays in care").

Machine learning models performed well in identifying some topics in 2019 Ohio syphilis records. This analysis is a first step in applying natural language processing methods to making DIS notes more accessible for analysis.