A Reliable and Accessible Caregiving Language Model (CaLM) to Support Tools for Caregivers: Development and...

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing for Caregiver Support
• Digital Health Interventions

Background:

• 1 in 5 US adults are family caregivers, often lacking formal training.
• Technology, particularly AI, offers potential to enhance caregiver capabilities.
• Foundation models (FMs) show promise but suffer from "hallucination," impacting information reliability.

Purpose of the Study:

• To develop a reliable and accessible caregiving language model (CaLM).
• To utilize foundation models (FMs) and a specific knowledge base for CaLM development.
• To evaluate CaLM performance against larger models, focusing on accuracy and resource efficiency.

Main Methods:

• Employed retrieval augmented generation (RAG) with FM fine-tuning on a caregiving knowledge base.
• Utilized small FMs (LLaMA 2, Falcon) and a large FM (GPT-3.5) for comparison.
• Focused knowledge base on Alzheimer disease and related dementias caregiving.
• Evaluated models on benchmark metrics and reference accuracy.

Main Results:

• RAG framework improved all FM performances.
• Small, fine-tuned FMs with RAG outperformed the large FM (GPT-3.5) on all metrics.
• Fine-tuned LLaMA 2 demonstrated superior reference accuracy compared to GPT-3.5.

Conclusions:

• A reliable and accessible CaLM can be developed using small FMs.
• Domain-specific knowledge bases are crucial for enhancing FM reliability in caregiving.
• This approach offers a practical AI solution for supporting family caregivers.