Foundations for Studying Clinical Workflow: Development of a Composite Inter-Observer Reliability Assessment for...

Area of Science:

• Healthcare Management
• Clinical Operations Research
• Health Informatics

Background:

• Clinical workflow complexity is critical for healthcare management and research.
• Workflow time studies significantly influence decision-making in healthcare.
• Current methods for assessing observer reliability in these studies lack standardization.

Purpose of the Study:

• To address methodological inconsistencies in workflow time studies.
• To propose a novel composite score for Inter-Observer Reliability Assessments (IORA).
• To establish a standardized protocol for reporting observer reliability in workflow studies.

Main Methods:

• Developed a composite score integrating multiple reliability metrics.
• Included overall agreement (Kappa), naming agreement (Kappa with time-overlap), duration agreement (concordance correlation coefficient), timing agreement (descriptive statistics), and sequence agreement (Needleman-Wunsch algorithm).
• Applied the protocol to virtually created one-second tasks for evaluation.

Main Results:

• Highlighted limitations of current Inter-Observer Reliability Assessments.
• Proposed a comprehensive composite score for systematic reliability evaluation.
• Provided a foundational step towards standardized reliability reporting in workflow time studies.

Conclusions:

• The novel composite IORA protocol offers a standardized and comprehensive method for validating observer reliability.
• This standardization improves the validity of data and results in workflow time studies.
• Empowers researchers with a robust tool for ensuring data integrity in clinical workflow analysis.