OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases

Area of Science:

• Clinical informatics
• Machine learning in healthcare
• Electronic health records

Background:

• Clinical decision support systems aim to improve healthcare quality.
• Automated order recommenders can streamline clinical workflows.
• Assessing usability and usefulness is crucial before system deployment.

Purpose of the Study:

• To evaluate the usability and usefulness of a machine learning-based order recommender system.
• To compare clinical appropriateness of orders with and without the recommender system.
• To assess physician acceptance and workflow integration.

Main Methods:

• 43 physicians used a clinical order entry interface with or without an automated order recommender system for simulated cases.
• Clinical appropriateness of orders was scored by a panel.
• Secondary outcomes included order count, case time, and user surveys.

Main Results:

• Clinical appropriateness scores were comparable between groups (mean difference -0.11).
• Physicians using the recommender placed more orders (1.09 incidence rate ratio).
• Physicians found suggestions useful (98% usage) and agreed the system would be beneficial (95% agreement).

Conclusions:

• Clinicians can effectively use and accept machine learning-based order recommendations in simulated settings.
• Automated recommendations did not compromise clinical appropriateness.
• Simulated testing is valuable for assessing clinical decision support tools.