Development of and Gathering Validity Evidence for a Theoretical Test in Thoracic Ultrasound

Thoracic ultrasound (TUS) has a high diagnostic accuracy for many common pulmonary diseases, but theoretic knowledge in sonographic physics, thoracic anatomy and physiology, and sonopathologic patterns is required to develop competence.

The aims of the study were to develop and gather validity evidence for a theoretical test in TUS and to establish a pass/fail standard.

Content was provided based on expert interviews, leading to the creation of 113 initial multiple-choice question (MCQ) items. Consensus was reached on 92 proceeding items through a Delphi process, and items were presented to physicians with different knowledge and experience in TUS. Answers were used for item statistics in order to select the items with the most optimal item discrimination and difficulty (i.e., level I items) to be included in the final test. Mean scores of the novice, intermediate and experienced groups were compared, and a pass/fail score was established using the contrasting groups' standard setting method.

Item statistics revealed 38 level I items, of which 30 were selected to be included in the final test. The internal consistency was high (Cronbach's alpha = 0.88). Differences in mean scores were 8.6 points (p < 0.001), 6.3 points (p = 0.01), and 14.9 points (p < 0.001) between novices and intermediates, intermediates and experienced, and novices and experienced, respectively. A pass/fail standard of 20 points was established.

The established MCQ test can distinguish between physicians with different levels of competence in TUS and enables an objective, evidence-based approach for assessing the theoretical knowledge of trainees undergoing an educational programme in TUS.