Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Diagnostic errors are a major cause of medical errors, necessitating improved clinical decision support.
• Generative artificial intelligence (AI) shows potential for enhancing diagnostic accuracy and reducing errors.
• The Diagnostic Reasoning Benchmark (DR.BENCH) was developed as a framework for evaluating clinical AI reasoning.

Purpose of the Study:

• To compare the performance of in-domain versus out-of-domain language models for clinical diagnostic reasoning.
• To evaluate the effectiveness of multi-task versus single-task training for clinical AI systems.
• To establish a new state-of-the-art performance on the problem summarization task within DR.BENCH.

Main Methods:

• Comparative analysis of language models trained on general versus clinical domains.
• Evaluation of multi-task learning versus single-task learning approaches.
• Focus on the problem summarization task within the DR.BENCH framework.
• Utilized ROUGE-L score for performance measurement.

Main Results:

• A multi-task, clinically-trained language model significantly outperformed general domain models.
• The proposed model achieved a state-of-the-art ROUGE-L score of 28.55 on the problem summarization task.
• Domain-specific training is crucial for optimizing clinical AI performance.

Conclusions:

• Multi-task, clinically-trained generative AI models represent a significant advancement in diagnostic decision support.
• Domain-specific AI training is essential for achieving high performance in clinical reasoning tasks.
• This work establishes a new benchmark for generative AI in clinical diagnostics.