Consensus-Based Marking Schemes to Assess Competence Using Free-Text Echocardiogram Summary Statements

statements of transthoracic echocardiography (TTE) reports can be used to assess trainees' competence in TTE interpretation. However, the inherent variability in wording and composition of summary statements creates challenges for developing marking schemes. This study aims to develop consensus-based marking schemes for TTE summary statements.

A modified Delphi approach with 2 rounds of surveys and a consensus-finalizing meeting was conducted to establish items that should be included in the summary statement and items critical to fulfill competence. Via consensus (defined as > 80% agreement), experts determined what items always should be included, and what items should be included in specific indications or cardiac diagnoses.

The first and second round surveys were completed by 27 experts. Eleven experts participated in the consensus-finalizing meeting. This process established that 79 items should be included in the summary statement. Five general items (left ventricular [LV] size, LV systolic function, right ventricular size and right ventricular function, and a statement about valves) should be included in all summary statements, and the remaining 74 items should be included in the presence of abnormalities, specific indications, or relevant diagnoses. Sixteen items were deemed critical for competence. Only LV systolic function was deemed critical to fulfill competence in all summary statements, with the remaining 15 items critical in specific indications.

These findings inform consensus-based marking schemes for TTE summary statements. When integrated into competence assessments, these findings will inform the training of TTE interpretation for cardiology residents.