Exploring the consistency, quality and challenges in manual and automated coding of free-text diagnoses from hospital outpatient letters

Clinical coding is the process of extracting key information contained within clinical free-text and representing this information using standardised clinical terminologies. In doing so, unstructured text is transformed into structured data that can be retrieved and analysed more effectively. This process is essential to improving direct care, supporting communication between clinicians and enabling clinical research. However, manual clinical coding is difficult and time consuming, motivating the development and use of natural language processing for automated coding. This work evaluates the quality and consistency of both manual and automated coding of diagnoses from hospital outpatient letters. Using 100 randomly selected letters, two human clinicians performed coding of diagnosis lists to SNOMED CT. Automated coding was also performed using IMO's Concept Tagger. A gold standard was constructed by a panel of clinicians from a subset of the annotated diagnoses. This was used to evaluate the quality and consistency of manual and automated coding via (1) a distance-based metric, treating SNOMED CT as a graph, and (2) a qualitative metric agreed upon by the panel of clinicians. Correlation between the two metrics was also evaluated. Comparing human and computer-generated codes to the gold standard, the results indicate that humans slightly out-performed automated coding, while both performed notably better when there was only a single diagnosis contained in the free-text description. Automated coding was considered acceptable by the panel of clinicians in approximately 90% of cases.