Evaluation of Generative Artificial Intelligence Implementation Impacts in Social and Health Care Language...

Area of Science:

• Health Informatics
• Natural Language Processing
• Public Health Administration

Background:

• Generative artificial intelligence (GAI) shows promise for enhancing public social and health care productivity.
• Empirical evidence on GAI's real-world impact in these sectors is currently limited.

Purpose of the Study:

• To investigate the impacts of the GPT-4 language model on productivity, translation quality, and user experience.
• Focus on an in-house language translation team within a Finnish public social and health care organization.

Main Methods:

• A mixed-methods study involving 4 translators over 3 months (March-June 2024).
• Quantitative analysis of 908 translation segments using Trados software for productivity.
• Quality assessment of 1373 GAI-human translated segments via 4 automatic metrics.
• Qualitative data from 5 semistructured interviews exploring user experience.

Main Results:

• Post-editing machine translation was 14% faster than translating from scratch (P=.03), with individual gains ranging from -2% to 102%.
• Translation quality showed promise: 11-16% of GAI outputs required no edits, with good Bilingual Evaluation Understudy scores (38-43).
• User feedback was mixed; GAI was valued for repetitive content, but challenges included terminology consistency and context limitations.

Conclusions:

• GPT-4 GAI demonstrates potential for improving translation productivity and quality in public health translation services.
• Effectiveness is contingent on translator skills, workflow design, and organizational readiness.
• Recommendations include investing in training, optimizing technical integration, and redesigning workflows for similar contexts.