Reliabilities of human judgements: measurements from photographic CT scan images

Studies of brain lesions are generally dependent on human judgement for identification and possibly for measurement, but estimates of reliability are frequently neglected. The present study involves three investigation based on X-ray CT scans into reliabilities of human judgements: (1) the areas of brain lesions identified over two occasions by a single judge, (2) brain areas based on the projection of scans by a second judge over two occasions, and (3) brain areas computed from brain outlines by two independent judges. Errors decreased geometrically over procedures in the order listed, reflecting the decreasing complexity of judgement involved. Nevertheless, all three reliabilities proved satisfactory, showing that these procedures may be applied consistently over occasions and between raters. This was reassuring since computerization is currently practicable only in (2) and (3), where errors were least. Although not always performed, reliability checks are important, as indicated by the outlier, Case 10. Where there is a large discrepancy, seeking the reason(s), with a view to standardizing the criteria of judgement, is preferable to automatic averaging, both as a safeguard in individual cases and also to estimate error of measurement in group studies. To assist decision in any particular instance as to whether averaging is an acceptable solution, a statistical rule of thumb is proposed for testing the significance of the difference between two judgements.