Computer-Based Competency Assessment in Point-Of-Care Ultrasound: A Systematic Review

Point of care ultrasound (POCUS) is a critical skill for physicians across multiple medical specialties, yet substantial heterogeneity exists in how competency is assessed. Computer-based approaches can be used to deliver, grade, and analyze learner performance, and may be more objective and reliable than traditional approaches using expert assessments. This study aimed to systematically review and summarize the existing literature surrounding computer-based approaches to assessing POCUS competency.

We searched six online databases (MEDLINE, IEEE Xplore Digital Library, Association for Computing Machinery Digital Library, PsycINFO (Ovid), EMBASE, Web of Science Core Collection). We included original peer-reviewed studies that assessed computer-based metrics of POCUS competence among any learner group performing POCUS. We also reviewed reference lists of all included studies. We extracted data elements that included the specialty of participants, POCUS experience, POCUS modality used, and type and results of computer-based competency assessments. At least two authors conducted title and abstract screening, full text review, and data extraction, with discrepancies adjudicated by a third author. We present a qualitative synthesis of study findings.

Of 7375 identified studies, we included 28 in our final analysis. Computer-based metrics were used to assess knowledge (n = 10), skills (n = 25), and cognitive load (n = 1) using hand tracking (n = 14), eye tracking (n = 7), image analysis (n = 6), and simulation scores (n = 1). In general, hand tracking analysis showed that experts had shorter probe path lengths, took less time to identify areas of interest, and had fewer discrete movements compared with novices. Eye tracking assessment showed increased dwell time was associated with successful completion of procedures and increased accuracy in interpreting images.

We identified four computer-based metrics for assessing POCUS competence, many of which demonstrated consistent performance in distinguishing skill level. Further work is needed to standardize and validate those approaches.