The use of reflection as an assessment of practice. Can you mark learning contracts?

Learning contracts are used by many educational institutions as a means of assessing the practice of their students. However, there have been no studies into the reliability of marking learning contracts. A statistical analysis of the marks from the assi gnments of a practice module, which included a marked learning contract and a more traditional assessment strategy, namely, a clinical study, was made which measured the reliability of the marking process. This was undertaken in 3 ways: an analysis of th e clinical study as a measure of its reliability, secondly the learning contract marking itself was tested for reliability and finally a comparison between the two assessment methods. The statistical analysis was undertaken by means oft -test, intra-class coefficient calculation, weighted Kappa calculation, regression analysis and by a method outlined by Bland and Altman (1986, Lancet: 307-310) in which the set of means of two values is plotted against the differences. The clinical stud ies were found to be marked consistently, the maximum difference between the markers being only 8.3%, this value was then adopted as a measure of educational significance The inter-rater reliability of the learning contract marking was measured and was f ound to be not as high as the clinical studies. The t -test revealing a statistically significant (but not educationally) difference of 1.03%. The Bland and Altman plot revealed an maximum difference of 13.2%. However, a regression analysis revealed that 99.3% should fall within the 8.3% significance bandin g The clinical studies comparison with the learning contracts revealed that the difference as measured by t -test was significantly (not educationally) different at 2.3%. The Bland and Altman plot revealed that learning contracts may be marked up to 18.2% higher than the clinical studies but this may reflect the difference in what is being assessed by the two methods. However, it was noted that there was a greater agreement when the marks were higher. The validity of reflective learning contracts as an assessment tool has not been measured directly by this study, this would seem to be a natural next phase of analysis. This study supports the use of the assessment of learning contracts as a means of rigorous assessment of a student's practice.