Related Experiment Videos
Evaluation of using the SOCcer automated occupation auto-coding algorithm to assist expert coding of job descriptions
Pabitra R Josse1, Daniel E Russ2, Leila Orszag2
1Occupational and Environmental Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD 20850, United States.
Background/Objective:
Systems to automatically code job descriptions to standardized occupations, such as SOCcer, reduce the burden of expert coding in occupational studies, but expert coding remains needed. We evaluated how frequently experts select a SOCcer-suggested code when using the SOCAssign platform, which shows SOCcer's top 10 scoring codes (Si, with i denoting results ranking, compared to independent coding).
Methods:
We evaluated 2 protocols for SOCcer-assisted coding using the SOCAssign platform that we undertook when we had insufficient resources to employ the best practice of independent double-coding, with discordances resolved by consensus. Protocol 1 used a single coder, with a supervisor providing additional review for a subset based on features we identified that were more likely to indicate potential discordance between coders. Protocol 2 allocated jobs to be coded by 1 or 2 coders based on the SOCcer score (the probability an expert would have selected that code) for the highest scoring code (S1 score) and the score difference between the top 2 scoring codes. For comparison purposes, we included a third set of jobs that had previously undergone independent expert coding. For each job, we identified if the expert selected the top-scoring (S1) code, or one of the top 3 (S1-3) or top 10 (S1-10) scoring codes from SOCcer and compared findings across coding protocols, overall and by S1 score.
Results:
In Protocol 1, 25% of 6,824 jobs underwent supervisory review. In Protocol 2, jobs with an S1 score <0.2 or S1 - S2 score difference of <0.2 underwent coding by 2 experts (45% of 7,381 jobs); the remainder were coded by one expert. In both protocols and in independent coding, experts selected S1, one of S1-3 or one of S1-10 with similar prevalences (48% to 50%, 62% to 64%, and 73% to 74%, respectively). Expert coders were more likely to deviate from the SOCcer-suggested list for jobs with low score/score differences (39% to 44%) than high score/score differences (11% to 13%). In all evaluations, the frequency of selecting a SOCcer-suggested code increased with score, with some attenuation observed when the S1 - S2 score difference was <0.2.
Conclusions:
Our evaluations provide support for SOCcer-assisted expert coding. We provide recommendations for identifying jobs where multiple reviewers may be most needed.
