在大学级编码课程中对人类,GPT-3.5和GPT-4的表现进行比较
Will Yeadon1, Alex Peach2, Craig Testrow2
1Department of Physics, Durham University, Durham, DH1 3LB, UK. will.yeadon@durham.ac.uk.
Scientific reports
|October 7, 2024
概括
大学的学生大学生.
科学领域:
- 物理教育 物理教育
- 教育中的人工智能
背景情况:
- 大学级物理编码作业越来越容易受到人工智能生成的提交.
- 在学术环境中评估高级人工智能模型 (如GPT-3.5和GPT-4) 的性能至关重要.
研究的目的:
- 将学生编写的物理编码作业的表现与来自GPT-3.5和GPT-4.5的AI生成提交结果进行比较.
- 评估快速工程对AI性能的影响.
- 为了确定人工智能产生的工作的人类评估者的可检测性.
主要方法:
- 在Python物理编码任务中对50个学生提交和50个人工智能生成的提交 (GPT-3.5和GPT-4,有或没有提示工程) 进行盲目评估.
- 分析了三个独立评分机构颁发的分数.
- 人类评价者试图使用利克特级别来识别作者身份,后来简化为二进制的人工智能/人类分类.
主要成果:
- 学生的平均得分为91.9%,显著超过最佳AI类别 (GPT-4与快速工程) 的81.1%.
- 快速工程显著提高了GPT-4和GPT-3.5.5的分数.
- 人类评估者在提交的二元AI/人类分类中获得了85.3%的准确性,对于"绝对人类"的工作,准确性为92.1%.
结论:
- 大学学生的物理学编码工作仍然优于当前的人工智能能力,即使使用先进的模型和快速工程.
- 人工智能生成的提交通常可以被人类评估者检测到,这表明人工智能完全复制人类学术表现的能力目前存在局限性.
- 快速工程提高了人工智能性能,但在这种情况下并不等同于人类水平的成就.
更多相关视频
09:34A Virtual Machine Platform for Non-Computer Professionals for Using Deep Learning to Classify Biological Sequences of Metagenomic Data
Published on: September 25, 2021
3.9K
06:09P300-Based Brain-Computer Interface Speller Performance Estimation with Classifier-Based Latency Estimation
Published on: September 8, 2023
515
相关概念视频
Comparing Experimental Results: Student's t-Test
1.5K
The t-test is a statistical method used to compare the sample mean with a population mean or compare two means from two data sets. The test statistic is calculated from the standard deviation, mean, and number of measurements in the data set at a selected confidence interval and then compared to a table of critical values at this confidence level. If the test statistic is smaller than the critical value, the null hypothesis is accepted. In this case, we state that the difference between the...
1.5K
Multiple Comparison Tests
3.9K
Multiple comparison test, abbreviated as MCT, is a post hoc analysis generally performed after comparing multiple samples with one or more tests. An MCT will help identify a significantly different sample among multiple samples or a factor among multiple factors.
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
It would be easy to compare two samples using a significance alpha level of 0.05. In other words, there is only one sample pair to be compared. However, it would be difficult to identify a significantly different sample if the number...
3.9K
Improving Translational Accuracy
9.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.3K
Reliability and Validity
12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
Evolutionary Relationships through Genome Comparisons
5.7K
Genome comparison is one of the excellent ways to interpret the evolutionary relationships between organisms. The basic principle of genome comparison is that if two species share a common feature, it is likely encoded by the DNA sequence conserved between both species. The advent of genome sequencing technologies in the late 20th century enabled scientists to understand the concept of conservation of domains between species and helped them to deduce evolutionary relationships across diverse...
5.7K
Machines: Problem Solving II
299
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. Consider a lifting tong carrying a 100 kg load. It comprises movable sections DAF and CBG linked together with member AB.
299
