Machine learning tools match physician accuracy in multilingual text annotation

In the medical field, text annotation involves categorizing clinical and biomedical texts with specific medical categories, enhancing the organization and interpretation of large volumes of unstructured data. This process is crucial for developing tools such as speech recognition systems, which help medical professionals reduce their paperwork. It addresses a significant cause of burnout reported by up to 60% of medical staff. However, annotating medical texts in languages other than English poses unique challenges and necessitates using advanced models. In our research, conducted in collaboration with Gdańsk University of Technology and the Medical University of Gdańsk, we explore strategies to tackle these challenges. We evaluated the performance of various tools and models in recognizing medical terms within a comprehensive vocabulary, comparing these tools' outcomes with annotations made by medical experts. Our study specifically examined categories such as 'Drugs', 'Diseases and Symptoms', 'Procedures', and 'Other Medical Terms', contrasting human expert annotations with the performance of popular multilingual chatbots and natural language processing (NLP) tools on translated texts. The conclusion drawn from our statistical analysis reveals that no significant differences were detected between the groups we examined. This suggests that the tools and models we tested are, on average, similarly effective-or ineffective-at recognizing medical terms as categorized by our specific criteria. Our findings highlight the challenges in bridging the gap between human and machine accuracy in medical text annotation, especially in non-English contexts, and emphasize the need for further refinement of these technologies.