Designing a web-based factorial survey experiment on diagnostic evaluation of cardiovascular symptoms in emergency...

Guidelines describe evidence-based diagnostic testing strategies for patients with suspected pulmonary embolism (PE), yet there is sparse empirical literature on how clinicians decide whether to suspect PE.

Our goal was to design a valid and reliable approach to assess how patients' demographics, symptoms, and risk factors impact clinicians' decisions to pursue PE testing.

We designed and conducted early-stage validation of an online factorial survey experiment featuring 7 vignettes of hypothetical adult emergency department (ED) patients with cardiopulmonary symptoms. Using a 2 × 2 × 2 factorial design, we generated 8 versions of each vignette that differed by patient gender (woman/man), race (Black/White), and a case-specific clinical factor. In each vignette, participants rated their suspicion for PE and other cardiovascular conditions using a visual analog scale and indicated diagnostic tests they would order. Participants were not informed of the study design. We measured test-retest reliability in 16 physicians from 4 EDs and conducted pilot testing at a large national conference of ED clinicians.

Visual analog scale scores for perceived PE risk demonstrated high test-retest reliability (Pearson r = 0.85). Pilot testing by 61 ED physicians from 18 states had a response rate of 68%, a completion rate of 95%, and a mean completion time of 8.1 minutes (SD, 3.1 minutes). This cohort reported higher perceived PE risk in vignettes with intermediate-risk rather than low-risk revised Geneva scores, providing early support for this tool's criterion validity. Cognitive debriefing interviews suggest good face validity and response process validity.

These pilot data provide preliminary support for the validity and reliability of using an online factorial survey with clinical vignettes to experimentally assess ED clinicians' perceptions of PE risk and decisions for PE diagnostic testing.