Preprocessing Large-Scale Conversational Datasets: A Framework and Its Application to Behavioral Health Transcripts

Area of Science:

• Computational Linguistics
• Health Informatics
• Artificial Intelligence

Background:

• Automatic transcription of conversations generates noisy datasets with errors and unintended recordings.
• Preprocessing and filtering are crucial for the research utility of large conversational transcript datasets.
• Accurate conversation representation is vital for deriving insights in behavioral health contexts.

Purpose of the Study:

• To present a framework for preprocessing and filtering large conversational transcript datasets.
• To remove non-session transcripts unrelated to behavioral treatment sessions.
• To enhance the utility of behavioral health transcripts for research.

Main Methods:

• Integrated feature extraction, human annotation, and large language models (LLMs).
• Utilized LLM perplexity to measure transcript noise and zero-shot prompting for classification.
• Prioritized data security and anonymity throughout the process.

Main Results:

• Approximately one-third of transcripts contained errors, including incomprehensible segments and speaker diarization issues.
• LLM perplexity showed higher scores in non-sessions, but moderate classification performance alone.
• Zero-shot LLM prompting achieved high agreement with expert ratings (κ=0.71) in distinguishing sessions from non-sessions.

Conclusions:

• The hybrid approach effectively characterizes errors and distinguishes text types in conversational datasets.
• Provides a foundation for ensuring data quality and usability in mental health research.
• Emphasizes integrating clinical experts with AI tools while prioritizing data security.