Survey Authentication Challenges During a Geriatrics Knowledge Assessment Pilot

Area of Science:

• Medical Education
• Geriatrics
• Survey Methodology

Background:

• A pilot study of the Age-Friendly Knowledge Assessment Tool (AF-KAT), a geriatrics assessment for medical students, encountered significant survey authentication issues.
• The AF-KAT is designed to evaluate geriatrics knowledge based on the 5Ms framework and American Geriatrics Society competencies.

Purpose of the Study:

• To describe a multi-institutional survey authentication challenge during the AF-KAT pilot deployment.
• To highlight the impact of inauthentic participation on assessment results.

Main Methods:

• The AF-KAT was electronically distributed to graduating medical students at three US medical schools with a financial incentive.
• A structured multi-institutional authentication process, including developing exclusion criteria and multi-reviewer adjudication, was implemented to address inconsistent responses.
• Exclusion criteria targeted issues like invalid reimbursement codes, incorrect graduation years, duplicate responses, unrealistic specialty choices, IP address duplication, and rapid completion times.

Main Results:

• Out of 1029 initial responses, a substantial number were excluded based on authentication criteria.
• After adjudication, only 88 responses remained for the final analytic sample.
• The median AF-KAT score significantly increased from 5 (pre-authentication) to 12 (post-authentication), indicating the impact of inauthentic data.

Conclusions:

• The study identified likely inauthentic participation that skewed the results of a geriatrics knowledge assessment.
• Proactive implementation of survey security measures is strongly recommended for financially incentivized geriatrics education research to ensure data integrity.