大型言語モデルによる自動スコア付けの一貫性について
Mingfeng Xue1, Xingyao Xiao2, Yunting Liu3
1University of North Carolina Greensboro, USA.
Educational and psychological measurement
|February 19, 2026
まとめ
大型言語モデル (LLM) は,LLM内でのスコア付けの一貫性が高いが,LLM内での一貫性は適度である. LLMの出力を組み合わせた投票戦略は,得点の精度を向上させます.
科学分野:
- 人工知能 (AI) とは,人工知能 (AI) のことです.
- 教育的な測定方法について
- 自然言語処理 (Natural Language Processing) とは,自然言語処理で処理される言語のことです.
背景:
- 大型言語モデル (LLM) は,自動スコア付けタスクの可能性を示しています.
- LLMのスコア付けの不一致は,モデルの変動とトレーニングデータの違いから生じることがあります.
- LLMのスコアの一貫性を理解することは,信頼性の高い自動評価に不可欠です.
研究 の 目的:
- 5つのLLMでLLM内およびLLM間スコアの一貫性を調査する.
- LLMのスコアの一貫性に対する温度設定の影響を調べる.
- スコアの一貫性と正確性の関係を評価する.
- LLMの得点を改善するための投票戦略を提案し,評価する.
主な方法:
- 5つのLLM (Claude, DeepSeek, Gemini, GPT, Qwen) のスコアの一貫性を評価しました.
- 異なる温度設定下での評価一致性.
- 科学教育とASAPデータセットから構築された応答項目を利用しました.
- LLM全体で過半数投票戦略を実施しました.
主要な成果:
- LLMは,温度に関係なく,LLM内のほぼ完璧な一貫性を示しました.
- LLM間の一貫性は中等で,より簡単な項目では高かった.
- LLM内での一貫性は,LLM間の一貫性を上回りました.
- LLM内の一貫性は精度と相関していないが,LLM内の一貫性は正の相関を示している.
- 多数投票により,さまざまなLLMの強みを組み合わせることで,得点の精度が向上しました.
結論:
- LLMは,高い内部スコア信頼性を提供しますが,外部合意は異なります.
- Inter-LLMの一貫性は,LLM内の一貫性よりも,得点の精度を予測する上で優れている.
- 多数投票のようなアンサンブル・メソッドは,LLMの得点不一致を軽減し,教育評価の正確性を向上させることができます.
関連する概念動画
Improving Translational Accuracy
15.2K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
15.2K
Improving Translational Accuracy
3.7K
3.7K
Automatic Processing and Automatic Social Behavior
276
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
276
Language and Cognition
836
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
836
Language Development
973
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
973
Accuracy, limits, and approximation
1.3K
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
1.3K
