Evaluation framework for automatic privacy auditing tools for hospital data breach detections: A case study.

Area of Science:

• Health Informatics
• Computer Science
• Information Security

Background:

• Automated privacy auditing tools are increasingly adopted in hospitals.
• Accurate auditing is crucial for protecting sensitive patient data.
• Existing tools require rigorous evaluation for performance validation.

Purpose of the Study:

• To design an evaluation framework for automatic privacy auditing tools.
• To validate a machine learning-based auditing tool's performance in a hospital setting.
• To identify areas for improvement in hospital privacy auditing processes.

Main Methods:

• Utilized a black box method with user acceptance testing.
• Developed appropriate and inappropriate behavior scenarios based on expert input and real-world examples.
• Tested the framework using data from Mackenzie Richmond Hill Hospital.

Main Results:

• The auditing tool correctly identified 98.09% of total hospital accesses as appropriate.
• A statistically significant increasing trend in categorizing appropriate accesses was observed (P < 0.05).
• Unexplained accesses were linked to hospital workflow inefficiencies and data quality issues, such as missing staff role and department information.

Conclusions:

• The developed framework provides empirical evidence for the effectiveness of automated privacy auditing.
• The study highlights the need for addressing workflow and data quality issues for optimal tool performance.
• This research offers valuable insights for hospitals implementing machine learning for privacy auditing.