Large Language Models are In-context Teachers for Knowledge Reasoning

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• In-context teaching (ICT) relies on human-crafted examples, which are costly and variable.
• Large language models (LLMs) offer a potential alternative for creating in-context demonstrations.

Purpose of the Study:

• To investigate if LLMs can serve as more effective teachers than humans in ICT.
• To develop novel methods for improving LLM-based ICT.

Main Methods:

• Proposing Self-Explain: using an LLM's self-generated explanations as in-context demonstrations.
• Validating the Encoding Specificity Hypothesis: teacher exemplars should match student training data.
• Introducing Teach-Back: aligning teacher and student LLMs to enhance ICT performance.

Main Results:

• Self-Explain significantly outperforms human-crafted exemplars and other baselines.
• Explanations resembling the student LLM's self-explanations serve as better demonstrations.
• Teach-Back enables a smaller LLM to teach a larger one, surpassing human teachers in accuracy.

Conclusions:

• LLMs can be superior in-context teachers compared to humans.
• The proposed Self-Explain and Teach-Back methods enhance LLM-based ICT.
• Alignment between teacher and student LLM explanations is crucial for effective ICT.