A method for assessing reflective journal writing

Reflection is widely accepted as a learning tool and is considered integral to professional practice. Journal writing is advocated in facilitating reflection, yet little is written about how to assess reflection in journals. The purpose of this study was to develop and test a method of assessing the elements of reflection in journals and to determine whether, and to what level, reflection occurs in journals. Twenty-seven physical therapy students maintained written reflective journals throughout three of their four eight-week clinical affiliations. The students were introduced to concepts of reflective practice with definitions of terms and reflective questions before their second affiliation. A coding schema was developed to assess the journals. Three raters assessed forty-three journals. The text of each journal was analyzed for evidence of nine elements of reflection, and each journal was categorized as showing no evidence of reflection, evidence of reflection, or evidence of critical reflection. Descriptive statistics were used to demonstrate evidence of reflection. Reliability between each pair of raters was assessed using percent agreement, phi coefficients, and gamma statistics. Interrater reliability of all raters was assessed using intraclass correlation coefficients (ICC[2,1]). Results showed that the raters assessed 95.3%-100% of the journals as showing at least one element of reflection. The percent agreement between rater pairs for the nine elements of reflection ranged from 65.1% to 93.0%, the phi coefficient ranged from 0.08 to 0.81, and the ICC(2,1) values used to assess reliability among the three raters on each element ranged from 0.03 to 0.72. Averaging the assessment of the three raters for the overall journal, 14.7% of the journals were assessed as showing no evidence of reflection, 43.4% as showing evidence of reflection, and 41.9% as showing evidence of critical reflection. The percent agreement between rater pairs for the overall assessment of the journals ranged from 67.4% to 85.7%, the gamma statistic ranged from 0.88 to 0.98, and the ICC(2,1) among all raters was 0.74 (95% confidence interval, 0.61-0.84). These results represent an acceptable level of agreement for use of this method of assessment for educational purposes. The coding schema developed provides a mechanism to assess evidence of reflection in written journals, which will enable instructors to evaluate student competency, obtain a baseline for facilitating reflective practice, and assess their own efficacy in facilitating reflection among students.