Agreement, Calibration, and Exploratory Performance of AI-Based Ultrasound in Thyroid Nodule Assessment

Background/Objective: Ultrasound is the first-line imaging modality for thyroid nodule assessment; however, it remains highly operator-dependent and subject to interobserver variability. Artificial intelligence (AI)-based systems have been proposed to improve reproducibility, yet evidence regarding their agreement with clinician assessment-particularly at the level of individual sonographic features-remains limited. Importantly, most available studies evaluate concordance rather than true diagnostic accuracy against an independent reference standard. To evaluate agreement between an AI-based ultrasound system and expert clinician assessment in thyroid nodule evaluation, focusing on concordance of size measurements, agreement in sonographic feature classification, and exploratory diagnostic performance relative to cytological outcomes. Methods: This retrospective single-center study included 74 thyroid nodules from adult patients undergoing routine ultrasound examination. Archived ultrasound images were independently assessed by an experienced clinician and an AI-based system. Agreement for quantitative measurements was evaluated using Bland-Altman analysis, while categorical features were assessed using percent agreement and Cohen's kappa coefficients. Calibration was examined using scatter plots with the line of identity. Cytological results, when available, were used as a non-uniform exploratory reference standard for diagnostic analyses. Exploratory diagnostic performance was assessed using receiver operating characteristic (ROC) curves and area under the curve (AUROC) estimates. Given the study design, analyses primarily reflect agreement and measurement concordance rather than true diagnostic accuracy. Results: AI-derived and clinician measurements demonstrated strong agreement across all dimensions, with minimal systematic bias and stable calibration patterns. A small but consistent underestimation of one measurement axis by approximately 1 mm was observed. For categorical features, agreement ranged from fair to moderate (κ = 0.196-0.368), with the highest concordance for echogenic foci and lowest for echogenicity. Exploratory analyses showed variable diagnostic discrimination, with the best performance observed for size measurements and selected sonographic features. Conclusions: AI-based ultrasound analysis demonstrates robust agreement with clinician assessment for quantitative thyroid nodule measurements, while agreement for categorical feature classification remains moderate and variable. The findings highlight that the present study evaluates concordance rather than definitive diagnostic accuracy, particularly given the lack of a uniform independent reference standard. These results support the role of AI as an assistive tool in thyroid ultrasound practice, improving measurement reproducibility while requiring ongoing clinician oversight for qualitative interpretation.