一个天文问题的答案数据集用于评估大型语言模型
Jie Li1, Fuyong Zhao1, Panfeng Chen1
1State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China.
Scientific data
|March 19, 2025
概括
研究人员开发了Astro-QA,这是一个新的基准数据集,用于评估天文学问题答案 (QA) 中的大型语言模型 (LLM). 这一数据集有助于评估在天体物理学和相关领域的LLM能力.
科学领域:
- 天文学和天体物理学
- 人工智能的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 大型语言模型 (LLM) 在问答 (QA) 中表现有前途,但由于缺乏专门的基准数据集,评估它们的天文知识受到阻碍.
- 评估在天文学中的LLM绩效需要一个数据集,涵盖各种各样的子领域和问题类型.
研究的目的:
- 引入Astro-QA,这是天文QA的第一个全面的基准数据集.
- 为了提供一个标准化工具来评估在天文学LLM的表现.
- 为了促进未来的研究和开发的LLMs用于天文学应用.
主要方法:
- 构建 Astro-QA 数据集,包含3082个英语和中文问题,涵盖天体物理学,天体测量,天体力学,天文学史和天文学技术.
- 开发了DGscore,这是一个新的指标,将客观和主观问题评估与难度加权结合在一起.
- 通过对27个开源和商业LLM进行广泛的实验来验证数据集.
主要成果:
- 阿斯特罗-QA数据集有效地作为评估天文QA中的LLMs的基准.
- 实验证明了数据集在评估LLM指令遵循,知识推理和自然语言生成方面的实用性.
- 该DGscore提供了一个准确的衡量LLM质量保证在天文领域的表现.
结论:
- 阿斯特罗-QA是评估士天文学能力的可靠基准.
- 数据集和DGscore可以指导开发用于天文学研究的专门LLM的进展.
- 这项工作弥合了对天文学等科学领域的LLM评估的差距.
相关概念视频
Improving Translational Accuracy
2.5K
2.5K
Aggregates Classification
295
Aggregate classification is generally based on its size, petrographic characteristics, weight, and source. Size classification ranges from coarse to fine aggregates, defined by the size of the particles. Coarse aggregates are particles that do not pass through ASTM sieve No. 4, and aggregates that pass through the sieve are fine aggregates.
Petrographic classification groups aggregates based on common mineralogical characteristics. Some of the common mineral groups found in aggregates are...
Petrographic classification groups aggregates based on common mineralogical characteristics. Some of the common mineral groups found in aggregates are...
295
Quantifying and Rejecting Outliers: The Grubbs Test
1.4K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.4K
Distributions to Estimate Population Parameter
4.0K
The accurate values of population parameters such as population proportion, population mean, and population standard deviation (or variance) are usually unknown. These are fixed values that can only be estimated from the data collected from the samples. The estimates of each of these parameters are sample proportion, the sample mean, and sample standard deviation (or variance). To obtain the values of these sample statistics, data are required that have particular distribution and central...
4.0K


