Defining measurement precision for effort dependent tests: the case of three neurobehavioural tests

For effort dependent tests, the estimation of measurement precision (reproducibility) is complicated by learning effects and submaximal efforts which inflate the variance of repeated trials. To illustrate an approach to the estimation problem, precision was evaluated for three neurobehavioural tests based on the responses of 76 boatbuilders tested on four separate test occasions within a one-week period. The average Coefficient of Variation (CV) for repeated trials within a test session was 6%, 16% and 13% for the Continuous Performance Test (CPT), Symbol-Digit substitution test (SDS) and the Hand-Eye motor-coordination test (HEM), respectively. In order to adjust for the effects of learning, the first trial(s) of a session were excluded from the calculation of performance level and its precision. This adjustment for learning significantly improved the precision for SDS and HEM to a CV of 8%. Inspection of the distributions of best efforts by trial number indicated that dropping the early trial(s) eliminated the best efforts of 34%, 22% and 7% of the subjects on the three tests respectively. When the worst two trials were excluded regardless of order, precision improved significantly to less than 5% for all three tests. On the basis of these results, a 5% precision rule for CPT and a 10% precision rule for SDS and HEM are provisionally recommended. The test results of subjects unable to meet this criterion should be identified, but in order to avoid selection bias, they should be analysed separately rather than excluded.