Field testing a new ICD coding system: methods and early experiences with ICD-11 Beta Version 2018

A beta version (2018) of International Classification of Diseases, 11th Revision for MMS (ICD-11), needed testing. Field-testing involves real-world application of the new codes to examine usability. We describe creating a dataset and characterizing the usability of ICD-11 code set by coders. We compare ICD-11 against ICD-10-CA (Canadian modification) and a reference standard dataset of diagnoses. Real-world usability encompasses code selection and time to code a complete inpatient chart using ICD-11 compared with ICD-10-CA.

A random sample of inpatient records previously coded using ICD-10-CA was selected from hospitals in Calgary, Alberta (N = 2896). Nurses examined these charts for conditions and healthcare-related harms. Clinical coders re-coded the same charts using ICD-11 codes. Inter-rater reliability (IRR) and coding time improved with ICD-11 coding experience (23.6 to 9.9 min average per chart). Code structure comparisons and challenges encountered are described. Overall, 86.3% of main condition codes matched. Coder comments regarding duplicate codes, missing codes, code finding issues enabled improvements to the ICD-11 Browser, Coding Tool, and Reference Guide. Training is essential for solid IRR with 17,000 diagnostic categories in the new ICD-11. As countries transition to ICD-11, our coding experiences and methods can inform users for implementation or field testing.