The precision of practice analysis results in the professions

Area of Science:

• Psychometrics
• Professional Assessment
• Test Development

Background:

• Practice analysis is crucial for content validity in professional licensure and certification.
• Estimating the frequency of professional activities informs test development and content relevance.

Purpose of the Study:

• To examine the precision of mean frequency estimates for activity categories in practice analyses.
• To explore the impact of respondent sampling, activity statement selection, and occasion on estimate variability.

Main Methods:

• Administered an activity inventory to a sample of professionals on two separate occasions (6 weeks apart).
• Analyzed the variability in mean frequency estimates for different activity categories.
• Investigated sources of variability including respondent sampling, activity statements, and measurement occasions.

Main Results:

• Identified sources of variability in mean frequency estimates for professional activities.
• Determined that samples of 200 respondents are adequate for precise estimation.
• Found that 10-20 activity statements per category and a single occasion are sufficient.

Conclusions:

• Established guidelines for sample size and design in practice analysis for psychometric assessments.
• Concluded that current practices for estimating activity frequencies are sufficiently precise with optimized sampling strategies.
• Provided evidence supporting the reliability of practice analysis data for licensure and certification test validity.