Toward scalable detection of clinical reasoning in supervision: A machine learning feasibility study

Clinical reasoning shapes every clinical decision but remains largely invisible in therapy. Supervision, where supervisors and therapists deliberate together through dialogue, is a natural context for studying clinical reasoning, yet such reasoning is difficult to measure at scale. This study examined whether machine learning could detect clinical reasoning in supervision transcripts. We analyzed 631 supervision transcripts from the Reaching Families trial, in which supervisor-therapist dyads discussed youth cases with treatment engagement challenges, coded by human raters across three categories: reasoning activities (e.g., considering, selecting, monitoring), content focus (problems or practices), and referent (youth or caregiver). We first compared a bag-of-words model (Complement Naïve Bayes) and a transformer-based model (DistilBERT) to assess signal detectability. We then examined whether DistilBERT's performance varied by condition: a coordinated knowledge systems intervention, which structures supervisory dialogue around a specific reasoning cycle, versus a practice guidelines control, and benchmarked DistilBERT's agreement against human coders. DistilBERT outperformed Complement Naïve Bayes across all metrics, achieving moderate to substantial agreement with human ratings across code categories (κ > 0.58), though McNemar's tests indicated systematic underlabeling. Agreement was stronger for coordinated knowledge systems than practice guidelines transcripts. These findings support the feasibility of using machine learning to detect clinical reasoning in supervision transcripts, an early step toward scalable labeling methods that capture what reasoning was about and how it unfolded. With continued development, this approach could advance understanding of how supervisory reasoning relates to clinical practice and support data-informed feedback for supervisors. (PsycInfo Database Record (c) 2026 APA, all rights reserved).