Validity and Reliability of Speech Data in the Norwegian Registry of Cleft Lip and Palate

Cleft lip and palate (CLP) treatment in Norway is centralized and multidisciplinary, with long-term follow-up from birth to adulthood. The Norwegian Registry of Cleft Lip and Palate was established to ensure high-quality care and enable systematic data collection. Speech data are a key component, assessed by speech-language therapists (SLTs) using standardized tools. Ensuring the validity and reliability of these data is essential for clinical decision-making and quality assurance.

The study aimed to (1) assess the validity of speech data obtained from routine assessments recorded in the Norwegian Registry of Cleft Lip and Palate and (2) propose and evaluate speech-related quality indicators for use in the Norwegian Registry of Cleft Lip and Palate.

To assess the validity of speech data obtained from routine assessments, comparisons were made between registry entries and blinded re-assessments. The registry data consisted of speech data from 143 children born in 2011-2012 with cleft palate ± lip. For the re-assessments, four experienced SLTs independently re-assessed anonymized audio recordings. Inter- and intra-rater reliability were analysed using Gwet's agreement coefficients and intra-class correlation coefficients (ICCs). Variables assessed included hypernasality, hyponasality, audible nasal air emission, weak pressure consonants, perceived velopharyngeal competence (VPC), intelligibility of speech, percent non-oral articulation errors, percent consonants correct (PCC) and percent oral articulation errors. Three speech-related quality indicators were proposed and evaluated as follows: (1) proportion of children without non-oral articulation errors, (2) proportion of children with competent or marginally incompetent VPC and (3) proportion of children with good intelligibility of speech.

Participants: 143 children (47% with cleft palate only, 36% unilateral, 17% bilateral cleft lip), mean age 6 years. Comparisons between registry entries and blinded re-assessments revealed almost perfect agreement (≥0.81) for hypernasality, hyponasality, audible nasal air emission, weak pressure consonants and perceived VPC. Agreement ranged from substantial (0.80) to almost perfect (≥0.81) for intelligibility of speech. Agreement was substantial (0.61-0.75) for percent non-oral articulation errors and mainly also for PCC (0.58-0.71). Agreement for percent oral articulation errors ranged from fair to moderate (0.25-0.50). The quality indicators, proportion of children without non-oral articulation errors and proportion of children with competent or marginally incompetent VPC, revealed substantial to almost perfect agreement (0.80-0.91). For good intelligibility of speech, agreement ranged from moderate to almost perfect (0.60-0.81). Inter-rater agreement was almost perfect (≥0.81) for hypernasality, hyponasality, audible nasal air emission, weak pressure consonants, perceived VPC and intelligibility of speech. Agreement was substantial for percent non-oral articulation errors (0.61-0.75), moderate to substantial for PCC (0.58-0.71), but only fair to moderate for percent oral articulation errors (0.25-0.50). The quality indicators proportion of children without non-oral articulation errors and the proportion of children with competent or marginally incompetent VPC showed almost perfect agreement (0.82, 0.92). Agreement was substantial (0.67) for the proportion of children with good intelligibility of speech. SLTs showed almost perfect intra-rater agreement (≥0.81) for most speech characteristics, including hypernasality, hyponasality, audible nasal air emission, weak pressure consonants, perceived VPC and intelligibility of speech. For percent non-oral articulation errors, agreement was almost perfect for three SLTs and substantial for one. For PCC, three SLTs had almost perfect agreement, while one had substantial agreement. Percent oral articulation errors showed more variation, ranging from fair to almost perfect across SLTs. Agreement ranged from substantial to almost perfect for the speech-related quality indicators.

The speech data in the Norwegian Registry of Cleft Lip and Palate can be considered valid, as the degree of coverage and reporting is high, and the variables are assessed for reliability. The reliability of the speech data is generally strong, both between registry data and blinded assessments, and in terms of inter- and intra-rater agreement among assessors. However, lower reliability in the rating of articulation errors, particularly oral errors, indicates a need for regular calibration among SLTs to ensure the validity of the registry's speech data. The three proposed quality indicators are reliable and suitable for inclusion in national reporting.

What is already known on this subject Quality registries for cleft patients have been established in several countries and are recognized as valuable tools for systematic data collection. The use of standardized protocols and national benchmarks for speech outcomes enhances the meaningfulness of reporting and facilitates comparisons across treatment centres. Previous studies on the validity and reliability of speech data indicate variability across different parameters, underscoring the importance of careful interpretation of registry data to ensure optimal patient care. These findings highlight the need for ongoing efforts to improve consistency in assessment procedures and to strengthen the quality of data used in clinical decision-making and research. What this paper adds to existing knowledge The Norwegian Registry of Cleft Lip and Palate has systematically collected data since 2011, maintaining a high inclusion rate. Speech data are gathered in accordance with a national protocol and assessed by an individual speech-language therapist (SLT). To ensure the validity and reliability of these data, achieving consistent inter- and intra-rater agreement among SLTs nationwide is essential. This study investigates the validity of speech data obtained from routine assessments recorded in the Norwegian Registry of Cleft Lip and Palate. The study also proposes and evaluates speech-related quality indicators for use in the Norwegian Registry of Cleft Lip and Palate. The study adds a significant contribution to research on quality registries and the development of national standards for speech outcomes in cleft care. What are the potential or actual clinical implications of this work? The speech data in the Norwegian Registry of Cleft Lip and Palate can be considered valid, as the degree of coverage and reporting is high, and the variables are assessed for reliability. The reliability of the speech data is generally strong, both between registry data and blinded assessments, and in terms of inter- and intra-rater agreement among assessors. However, findings related to articulation variables underscore the need for systematic calibration among SLTs within the cleft teams. Three proposed speech-related quality indicators have shown high reliability and can be implemented as clear, accessible and meaningful measures within the Norwegian Registry of Cleft Lip and Palate.