Open- and closed-source LLMs in medical and engineering education.

Area of Science:

• Artificial Intelligence in Education
• Open-Source Large Language Models (LLMs)
• STEM Education

Background:

• Large language models (LLMs) like GPT-4 are transforming education.
• Open-source LLMs offer accessibility and customization but are underused in medical and engineering fields.
• A systematic evaluation is needed to compare open-source and closed-source LLMs in these domains.

Purpose of the Study:

• To evaluate the performance of open-source LLMs (DeepSeek, GLM-4, Kimi) against GPT-4 for medical and engineering education.
• To investigate the impact of prompt engineering strategies on LLM accuracy.
• To highlight the potential of open-source LLMs in academic settings.

Main Methods:

• Comparative performance analysis of DeepSeek, GLM-4, Kimi, and GPT-4 across diverse question types.
• Implementation and testing of prompt engineering techniques (role-playing, generated knowledge prompting, chain-of-thought, few-shot, output style).
• Post-training accuracy assessment of LLM responses.

Main Results:

• DeepSeek demonstrated superior performance across all question types compared to other models.
• Prompt engineering significantly improved LLM accuracy, with DeepSeek achieving over 95% accuracy.
• Short-answer questions yielded the highest accuracy (up to 97%) across all four LLMs, underscoring prompt engineering's role.

Conclusions:

• Open-source LLMs, particularly DeepSeek, are effective tools for medical and engineering education.
• Prompt engineering is crucial for maximizing the utility of LLMs in academic problem-solving.
• Wider integration of open-source LLMs is recommended to bridge current evaluation gaps and enhance educational support.