Mystery shopping in health service evaluation

Over the last 5 years, primary care telephone triage systems have been introduced in the United Kingdom, United States, Australia, and most recently in New Zealand. Evaluation of the clinical safety of such systems poses a challenge for health planners and researchers.

To evaluate the use of simulated patients in the assessment of aspects of clinical safety in a pilot New Zealand primary care telephone triage service.

'Mystery shopping', an evaluation strategy commonly used in market research, was adapted by using simulated patients for telephone triage service evaluation.

New Zealand.

Four scripted clinical scenarios were developed by academic general practitioners, validated in student teaching situations, and then used by simulated patients to make 101 telephone calls. The scenarios were designed to necessitate a referral to a medical practitioner for further investigation. The documentation kept by the callers was compared with the call records from the telephone triage company, and both were analysed for capture and handling of the clinical safety features of each scenario. In cases where the endpoint was not a medical assessment, possible reasons for this were explored.

Records were retrieved for 85 telephone calls. Considerable triage variability was discovered. There were discrepancies between expected and actual triage outcomes with 51% of analysed calls resulting in a self-care recommendation. A number of reasons were identified both for the triage variability and the unpredicted outcomes. Audiotaping of consultations would have enhanced the credibility of the evaluation but it would have carried ethical constraints.

Simulated patients can be used to evaluate the limitations of health services and to identify areas that could be addressed to improve patient safety. Evaluation of patient satisfaction with services is not sufficient alone to evaluate safety.