OPERAS decision support system versus manual job coding: a quantitative analysis on coding time and inter-coder...

Area of Science:

• Occupational Health Research
• Data Science
• Sociology

Background:

• Manual coding of job descriptions is resource-intensive, requiring significant time, cost, and specialized expertise.
• Decision support systems (DSS) offer automated coding, potentially enhancing efficiency and consistency, but their effectiveness with expert coders needed validation.
• The OPERAS DSS was developed to address these challenges in occupational coding.

Purpose of the Study:

• To compare the efficiency and reliability of manual job description coding against coding assisted by the new DSS, OPERAS.
• To evaluate the performance of OPERAS with expert coders using established French classification systems (PCS2003, NAF2008).
• To assess the usability of the OPERAS system.

Main Methods:

• Five expert coders performed job description coding both manually and using OPERAS.
• Coding time and inter-coder reliability (using Cohen's kappa) for occupation (PCS) and activity sector (NAF) codes were measured.
• System Usability Scale (SUS) was used to evaluate the perceived usability of OPERAS.

Main Results:

• OPERAS significantly reduced coding time, with a median of 38 seconds per description compared to 60.8 seconds for manual coding.
• Inter-coder reliability was higher with OPERAS (PCS: 0.61-0.70; NAF: 0.38-0.61) versus manual coding (PCS: 0.56-0.61; NAF: 0.34-0.61).
• The average usability score for OPERAS was 75.5, indicating good user acceptance.

Conclusions:

• The OPERAS DSS demonstrably improves both coding time and inter-coder reliability compared to traditional manual coding methods.
• OPERAS is a valuable tool for facilitating large-scale, harmonized job coding in occupational health research.
• Future research could explore OPERAS with international classification systems like ISCO-88 and ISCO-68.