How to Design, Create, and Evaluate an Instruction-Tuning Dataset for Large Language Model Training in Health Care:...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Data Science

Background:

• High-quality data is crucial for healthcare decisions and the performance of large language models (LLMs).
• Instruction-tuning datasets (ITDs) are vital for enhancing LLM capabilities in diverse tasks.
• There is a need for accessible guidance on creating ITDs specifically for healthcare applications.

Purpose of the Study:

• To provide a comprehensive tutorial on designing, creating, and evaluating ITDs for healthcare.
• To make complex ITD concepts understandable for medical practitioners.
• To bridge the gap between clinical and technical domains in AI development.

Main Methods:

• Exploration of data sources and characteristics of well-designed datasets.
• Examination of three primary dataset construction methods: manual, synthetic (AI-generated), and hybrid.
• Discussion of metadata selection and human evaluation strategies for ITD quality.

Main Results:

• A structured framework for establishing ITDs in healthcare is presented.
• Practical approaches and considerations for different dataset preparation methods are detailed.
• The tutorial emphasizes the critical role of ITDs in advancing AI in medicine.

Conclusions:

• ITDs are essential for developing effective AI in healthcare, even with the advent of artificial general intelligence (AGI).
• A unified global framework for ITDs is needed for continued interdisciplinary advancement.
• Human-curated ITDs will remain indispensable for AI to process and apply medical knowledge.