Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation.

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Informatics

Background:

• Generative AI models like GPT-4 show immense potential across applications.
• Healthcare, particularly clinical note generation, faces challenges with time-consuming documentation.
• Physician burnout is exacerbated by the arduous task of creating electronic medical records.

Purpose of the Study:

• To address the lack of large, ethically sourced datasets for benchmarking AI-assisted clinical note generation.
• To introduce the Ambient Clinical Intelligence Benchmark (ACI-BENCH) corpus.
• To evaluate the performance of current state-of-the-art generative models on this task.

Main Methods:

• Creation of the ACI-BENCH corpus, the largest dataset for dialogue-to-note generation from clinical encounters.
• Inclusion of benchmark performances for several leading generative AI models.
• Focus on overcoming patient confidentiality barriers for ethical data sharing.

Main Results:

• The ACI-BENCH corpus represents a significant advancement in dataset size for this specific AI task.
• Benchmark results provide a baseline for current state-of-the-art model capabilities.
• The dataset facilitates further research into model weaknesses and evaluation metrics.

Conclusions:

• The ACI-BENCH corpus is essential for the rigorous evaluation and advancement of AI in clinical note generation.
• Availability of this dataset will accelerate the development of more effective AI tools for physicians.
• Standardized benchmarking is critical for understanding and improving AI performance in healthcare documentation.