Inter-rater reliability of digital MoCA administration with advanced scoring instructions

The objective of the present study was to assess the inter-rater reliability of MoCA subtests with automated administration and retrospective manual scoring and to evaluate if advanced scoring instructions improved inter-rater reliability.

MoCAs were administered through the MoCA Electronic Data Capture platform using a digital version of the MoCA, the MoCA Solo, and were independently scored manually after administration by three raters. Raters were blind to the scores from other raters, were trained in the administration and scoring of the MoCA and had several years of clinical experience.

The study shows good to excellent inter-rater reliability for the total MoCA (0.91-0.96) and for most of the subtests and individual test items under both the original and advanced scoring instructions. Only for the Cube (0.58), Clock numbers (0.50) and Clock hands (0.58), the inter-rater reliabilities are fair under the original scoring instructions. Using the advanced scoring instructions increased the inter-rater reliabilities for these three tests from fair to good for the Cube (0.65) and Clock hands (0.66) and from fair to excellent for the Clock numbers (0.78), while slightly decreasing total MoCA scores (-0.32). Small modifications to the advanced scoring instructions reduced the difference in total MoCA scores (-0.22) while preserving good inter-rater reliability.

Given the small difference in total MoCA score and weighting this difference against the increases in inter-rater reliability overall, we conclude that the advanced scoring instructions represent a meaningful improvement, providing greater consistency between raters with only a minimal impact on total scores.