Development, Usability, and Validity Evidence of a Rheumatology Telehealth Feedback Form

Rheumatology telehealth is widespread, making it essential that rheumatology fellows-in-training (FITs) achieve competence delivering telehealth care before entering the workforce. Feedback enhances telehealth skill development. This study develops a Rheumatology Telehealth Feedback Form (RTFF) that incorporates existing data and expertise as well as gathers validity evidence supporting its use.

The American College of Rheumatology Workforce Solutions Committee formed a working group with expertise in rheumatic diseases, telehealth, and medical education. The working group conducted a modified Delphi and focus groups to define content grounded in Rheumatology Telehealth Competencies and evidence-based practice as well as develop the RTFF. Rheumatology educators and FITs piloted the RTFF during patient care and simulated telehealth encounters, rating the helpfulness of the RTFF and providing data to calculate intraclass correlation coefficients (ICCs).

The modified Delphi identified 16 skills essential to conducting rheumatology telehealth encounters, which were converted into 17 items on the RTFF. Comment boxes prompted supplementary narrative. Five educators and 10 FITs piloted the materials. All educators (5 of 5) "agreed" or "strongly agreed" that the RTFF helped teach rheumatology telehealth skills. ICCs calculated from two simulated telehealth encounters demonstrated moderately to strongly supportive interrater reliability (ICC = 0.857 and 0.632, respectively).

The RTFF incorporates an evidence-based, expert consensus of the skills most essential for rheumatology telehealth encounters. Educators can use the RTFF as a guide for observing FITs delivering telehealth care, providing formative feedback, and helping FITs develop their skills before entering the workforce. The RTFF is the first direct observation formative assessment tool designed for teaching rheumatology telehealth skills.