Related Experiment Video
Updated: Jan 11, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1.0K
Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large
Jake Linardon1, Hannah K Jarman1, Zoe McClure1
1School of Psychology, Faculty of Health, Deakin University, Geelong, Victoria, Australia.
JMIR Mental Health
|November 12, 2025
Summary
Large language models (LLMs) like GPT-4o frequently fabricate citations in mental health reviews, with nearly two-thirds of references being inaccurate. Reliability varies by topic visibility and prompt specificity, emphasizing the need for human verification.
Area of Science:
- Artificial Intelligence in Mental Health Research
- Natural Language Processing Applications
- Scholarly Communication Integrity
Background:
- Large language models (LLMs) are increasingly used in mental health research for efficiency.
- LLMs can generate plausible but fabricated content, including non-existent bibliographic citations.
- Previous research on citation fabrication lacks a systematic analysis across mental health topics with varying public visibility and scientific maturity.
Purpose of the Study:
- To assess citation fabrication and bibliographic errors in GPT-4o generated literature reviews on mental health topics.
- To investigate how public familiarity and scientific maturity of mental health disorders influence citation accuracy.
- To determine if prompt specificity (general vs. specialized) impacts fabrication and accuracy rates.
Main Methods:
- GPT-4o generated six literature reviews (~2000 words, ≥20 citations) on major depressive disorder (high visibility), binge eating disorder (moderate), and body dysmorphic disorder (low).
- Reviews were conducted at general (symptoms, impacts, treatments) and specialized (digital interventions) levels.
- A total of 176 citations were extracted and verified using multiple academic databases; fabrication and accuracy rates were compared using chi-square tests.
Main Results:
- GPT-4o generated 176 citations, with 35 (19.9%) being fabricated and 64 (45.4%) of the real citations containing errors.
- Fabrication rates significantly differed by disorder (P=.001), being higher for binge eating disorder (28%) and body dysmorphic disorder (29%) compared to major depressive disorder (6%).
- Specialized reviews showed higher fabrication rates for binge eating disorder (46% vs. 17%; P=.01), and accuracy varied by disorder and review type.
Conclusions:
- Citation fabrication and bibliographic errors are prevalent in GPT-4o outputs, affecting nearly two-thirds of references.
- The reliability of LLM-generated citations is influenced by disorder familiarity and prompt specificity, posing greater risks for less visible or specialized topics.
- Rigorous human verification, careful prompt design, and institutional safeguards are crucial for maintaining research integrity with LLM integration.
Keywords:
AIacademic researchartificial intelligencecitationslarge language modelsmental healthpsychiatryMore Related Videos
Related Concept Videos
01:10Higher Mental Functions of the Brain: Language
3.4K
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
3.4K
02:16Stereotype Content Model
15.3K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
15.3K
01:14Fundamental Attribution Error
13.7K
According to some social psychologists, people tend to overemphasize internal factors as explanations—or attributions—for the behavior of other people. They tend to assume that the behavior of another person is a trait of that person, and to underestimate the power of the situation on the behavior of others. They tend to fail to recognize when the behavior of another is due to situational variables, and thus to the person’s state. This erroneous assumption is...
13.7K
01:12Hindsight Biases
4.2K
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
4.2K
02:09Stereotype Threat and Self-fulfilling Prophecies
41.9K
When we hold a stereotype about a person, we have expectations that he or she will fulfill that stereotype. A self-fulfilling prophecy is an expectation held by a person that alters his or her behavior in a way that tends to make it true. When we hold stereotypes about a person, we tend to treat the person according to our expectations. This treatment can influence the person to act according to our stereotypic expectations, thus confirming our stereotypic beliefs. Research by Rosenthal and...
41.9K
01:25The Anchoring-and-Adjustment Heuristic
7.7K
In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. However, sometimes, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the...
7.7K