Exploratory study of residents' conceptual framework for critical appraisal of the literature

Limited success of efforts to improve residents' critical appraisal skills may be due to poor understanding of their preintervention conceptual framework.

To describe internal medicine residents' a priori conceptual approach to critical appraisal of intervention trials, we performed an exploratory content analysis of 41 residents' unprompted descriptions of strengths and weaknesses of a randomized trial of spa therapy for back pain.

Eighty-eight percent of responses were assigned independently by 2 reviewers to 23 categories ( kappa .65). Residents agreed on some important characteristics (e.g., blinding), disagreed on others (e.g., similarity of treatment and control groups), frequently mentioned some irrelevant characteristics (e.g., "objective" vs. "subjective" outcomes), and rarely mentioned some important criteria (e.g., intention to treat).

An open-ended questionnaire reliably revealed both expected and unexpected conceptions and misconceptions among residents who had received minimal or no formal instruction in critical appraisal. We propose a classification model for residents' critical appraisal concepts that can be tested in future studies and used to identify areas for focusing interventions to improve resident physicians' critical appraisal skills.