Reflection Machines: Supporting Effective Human Oversight Over Medical Decision Support Systems

Area of Science:

• Human-Computer Interaction
• Medical Informatics
• Artificial Intelligence Ethics

Background:

• Decision support systems (DSS) are prevalent in modern decision-making, requiring human oversight.
• Overreliance on DSS can lead to reduced human vigilance and automation complacency.
• Maintaining meaningful human control is crucial, especially in high-stakes fields like medicine.

Purpose of the Study:

• To introduce Reflection Machines (RMs) as a novel approach to enhance human control in DSS.
• To explore the derivation of RM design requirements from EU regulations.
• To investigate the application of RMs in the medical domain, specifically for chronic low back pain.

Main Methods:

• Proposing RMs that stimulate human reflection through critical questions and counterarguments.
• Referencing preliminary research and outlining a potential RM prototype.
• Connecting RM design to existing regulatory frameworks, such as EU regulations.

Main Results:

• RMs are designed to foster critical thinking by challenging proposed decisions.
• The paper outlines a framework for integrating RMs into decision-making workflows.
• A specific application in chronic low back pain management is conceptualized.

Conclusions:

• Reflection Machines offer a promising method to increase meaningful human control over DSS.
• Further research is needed to explore RM functionality and user experience.
• RMs can potentially mitigate risks associated with automation complacency in expert decision-making.