Augmenting interpretable models with large language models during training.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Neuroscience

Background:

• Large language models (LLMs) show high performance but lack interpretability and efficiency.
• High-stakes domains and compute-limited settings necessitate interpretable and efficient AI models.

Purpose of the Study:

• Propose Aug-models, a framework to create efficient and interpretable prediction models using knowledge from LLMs.
• Enable LLM knowledge transfer for inference-only, ensuring transparency and speed.

Main Methods:

• Developed Aug-models framework, using LLMs during training but not inference.
• Instantiated Aug-models in NLP: Aug-Linear (linear model + LLM embeddings) and Aug-Tree (decision tree + LLM feature expansions).
• Applied Aug-models to text classification and natural language fMRI data analysis.

Main Results:

• Aug-models significantly improve inference speed and memory efficiency (over 1000x) compared to LLMs.
• Aug-Linear and Aug-Tree outperform non-augmented interpretable models in text classification.
• Aug-Linear, with 10,000x fewer parameters, surpasses a 6-billion parameter GPT-J model while remaining transparent.

Conclusions:

• Aug-models offer a viable solution for interpretable and efficient AI in resource-constrained environments.
• The framework successfully transfers LLM capabilities to smaller, transparent models.
• Aug-models provide valuable scientific interpretations from complex data, such as fMRI.
• This approach democratizes the use of advanced AI capabilities across diverse scientific fields.