Use of Critical Items in Determining Point-of-Care Ultrasound Competence

Current methods for evaluating clinical ultrasound skills rely on global ratings, which may miss specific errors. Prior research has shown that global assessments often lack precision in identifying critical skill gaps. This gap motivated the development of a more focused approach to competence evaluation. No prior work had resolved how to balance detailed checklists with global ratings effectively. The need for a standardized method to identify essential skill components remains unmet. Existing tools struggle to align checklist ratings with overall competence judgments. This uncertainty drove the investigation into critical item checklists. The study aimed to clarify whether focusing on essential items improves assessment accuracy.

The goal was to compare the effectiveness of a full checklist versus a critical item checklist in assessing POCUS competence. The specific problem is the lack of a reliable method to identify essential skill components in ultrasound training. The motivation stems from the need for a more precise and standardized evaluation tool. Experts were asked to rate video performances using both checklists and a global rating. The study sought to determine which method better aligns with overall competence judgments. The focus was on identifying whether critical items improve rater agreement and accuracy. The researchers proposed that focusing on essential items could enhance assessment reliability. The aim was to inform future training and evaluation practices in POCUS.

Twenty-four experts evaluated three video performances of POCUS exams using two checklists and a global rating. The videos were designed to show varying levels of competence and included potential critical errors. A modified Delphi process was used to define critical items for the checklist. Interrater reliability was calculated using intraclass correlation coefficients. Agreement with global ratings was measured using kappa statistics. Sensitivity and specificity were compared between the two checklists. The study design included a standard setting procedure for cut scores. The approach focused on comparing reliability and agreement metrics between tools.

The critical item checklist showed higher interrater reliability (ICC = 0.84) compared to the full checklist (ICC = 0.78). Agreement with global ratings was stronger for the critical item tool (κ = 0.71) than the full tool (κ = 0.48). Sensitivity was slightly higher for the full checklist (85.4%) than the critical item checklist (81.3%). Specificity was significantly higher for the critical item checklist (70.8%) compared to the full checklist (29.2%). The critical item tool detected fewer false positives than the full checklist. The study found that focusing on essential items improved agreement among raters. The results suggest that critical item checklists better align with global competence judgments. The findings support the use of focused checklists for POCUS assessment.

The authors propose that using a critical item checklist improves rater agreement and alignment with global competence ratings. They suggest that focusing on essential items enhances assessment reliability. The study found that the critical item tool outperformed the full checklist in specificity and agreement. The results indicate that critical item checklists may be more effective for POCUS evaluation. The authors recommend adopting critical item checklists for competence assessment. The findings support the use of focused tools in clinical skills evaluation. The study highlights the importance of identifying essential skill components. The authors conclude that critical item checklists better reflect overall competence judgments.