Automation of Trainable Datasets Generation for Medical-Specific Language Model: Using MIMIC-IV Discharge Notes

Area of Science:

• Medical Informatics
• Natural Language Processing
• Artificial Intelligence

Background:

• Fine-tuning specialized language models requires large, high-quality datasets.
• Manual dataset creation for medical NLP is time-consuming and expensive.
• Existing datasets may not capture the nuances of clinical documentation.

Purpose of the Study:

• To introduce a novel, automated approach for generating instruction datasets for medical language models.
• To leverage MIMIC-IV discharge records for creating a large-scale, machine-generated dataset.
• To evaluate the quality and validity of the generated dataset.

Main Methods:

• Utilized MIMIC-IV discharge records to generate instruction-following datasets.
• Employed a three-stage process: seed task generation, instruction/output creation, and data filtering.
• Developed a dataset in JSONL format comprising instructions, input notes, and outputs.

Main Results:

• Generated a dataset of 51,385 instruction sets.
• Achieved a mean ROUGE score of 0.185 between seed tasks.
• Demonstrated high validity rates: 88.0% by GPT-3.5 and 88.5% by human annotators.

Conclusions:

• Automated dataset generation is feasible and effective for medical NLP tasks.
• The proposed method offers a scalable solution for creating specialized instruction datasets.
• This approach has the potential to accelerate the development of advanced medical language models.