Evaluating a Large Language Model's Ability to Synthesize a Health Science Master's Thesis: Case Study

Area of Science:

• Health Sciences
• Artificial Intelligence
• Academic Integrity

Background:

• Large language models (LLMs) pose a challenge to educational institutions by mimicking academic writing, making it difficult to distinguish student work from AI-generated content.
• The traditional master's thesis, used as proof of higher-level learning, is threatened by LLMs' ability to produce expert-level academic writing.
• The widespread availability of LLMs raises concerns about the trustworthiness of theses in verifying subject comprehension and academic competencies.

Purpose of the Study:

• To assess the quality of machine-generated papers produced by a large language model (LLM) against the standards of a health science master's program.
• To explore the capabilities of LLMs in synthesizing research data and generating academic papers suitable for master's level graduation.

Main Methods:

• An exploratory case study utilized ChatGPT (OpenAI) to generate two research papers simulating qualitative and quantitative health science projects.
• The process involved prompting ChatGPT to synthesize credible datasets and then generate papers, requiring iterative dialogue and prompt optimization.
• Time investment included developing synthetic datasets (1.5 hours for qualitative, 16 hours for quantitative) and generating papers (2.5 hours for qualitative, 3.5 hours for quantitative).

Main Results:

• ChatGPT successfully generated two credible research papers and underlying datasets within a short timeframe, simulating medium-quality master's theses.
• The LLM demonstrated proficiency in synthesizing data, conducting analyses, and producing well-written academic papers that could pass program standards.
• The study highlights the ease and speed with which LLMs can produce academic work comparable to that of master's students.

Conclusions:

• A master's thesis generated by an LLM can no longer be considered definitive proof of a student's extensive study or subject mastery.
• The integrity of academic standards and the value of university diplomas are challenged by the sophisticated output of LLMs.
• Master's programs should prioritize alternative assessment methods, such as oral examinations and school exams, to ensure a rigorous evaluation of higher-order learning.