Using natural language processing to facilitate the harmonisation of mental health questionnaires: a validation study...

Area of Science:

• Computational linguistics
• Mental health research
• Psychometrics

Background:

• Pooling data from diverse sources enhances mental health research through larger sample sizes and cross-study comparisons.
• Data heterogeneity in variable measurement across studies presents a significant challenge to effective data pooling.
• Natural Language Processing (NLP) offers a potential solution for harmonizing disparate data sources in mental health research.

Purpose of the Study:

• To explore the use of NLP for harmonizing mental health questionnaires by matching questions based on semantic content.
• To assess the accuracy of NLP-derived semantic similarity scores in predicting real-world correlations between questionnaire items.
• To investigate the utility of NLP in identifying semantically similar items for cross-study data pooling.

Main Methods:

• Utilized the Sentence-BERT model to compute semantic similarity (cosine index) for 741 question pairs from five mental health questionnaires.
• Calculated Spearman rank correlations for corresponding item pairs using data from a UK adult sample (N=2,058).
• Estimated the correlation between NLP-derived cosine values and empirical Spearman coefficients, employing network analysis for data structure exploration.

Main Results:

• A moderate correlation (r=0.48, p<0.001) was observed between semantic similarity (cosine index) and empirical correlations (Spearman coefficient).
• NLP-derived cosine scores demonstrated a small error margin in predicting real-world correlations in a holdout sample (MAE=0.05, MedAE=0.04, RMSE=0.064).
• The NLP model identified complex data patterns but required manual rules for network edge inclusion.

Conclusions:

• Quantified semantic similarity between questionnaire items using NLP, showing correlation with empirical responses.
• Demonstrated the potential of NLP to facilitate cross-study data pooling in mental health research by identifying semantically similar items.
• Recommended researchers verify the psychometric equivalence of NLP-matched items before pooling data.