使用锦标赛来计算AUROC,用于LLMs的零射击分类.
WonJin Yoon1,2, Ian Bulovic1, Timothy A Miller1,2
1Boston Children's Hospital.
概括
本研究引入了一种新的方法,通过将它们转化为对对比,在分类任务中评估大型语言模型 (LLM). 使用Elo评级系统的这种方法提高了分类性能,并提供了比传统的零射击方法更丰富的见解.
科学领域:
- 自然语言处理自然语言处理.
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 大型语言模型 (LLM) 在零射击分类中表现有希望,但由于它们的静态决策边界,缺乏公平的比较方法.
- 现有的方法很难直接将LLM与监督分类器进行比较.
研究的目的:
- 提出和评估一种新的方法,以公平地比较对二进制分类任务的LLM.
- 将分类转化为一个可以通过LLMs解决的对对比任务.
- 为了利用Elo评级系统,例如评分和信心排序.
主要方法:
- 二元分类任务被重新定义为数据集实例之间的对对比.
- 使用LLM来生成实例的相对排名.
- 使用Elo评级系统,根据重复的对对比,对实例进行评分.
- 调度算法被评估为比较最小化.
主要成果:
- 建议使用LLMs和Elo评级的对对比方法,证明了更好的分类性能.
- 该方法提供了对数据集实例的信任顺序.
- 对调度算法的评估显示,它们有效地减少了比较.
- 与传统的零射击分类相比,这种方法提供了更全面的信息.
结论:
- 双对比方法提供了一种可靠和公平的方式来评估LLM的分类.
- 由LLM驱动的对对比与Elo评级提高了分类准确性,并提供了有价值的实例级见解.
- 这种方法推进了LLM评估和零射击学习领域.
相关概念视频
Calculating the Equilibrium Constant
37.9K
The equilibrium constant for a reaction is calculated from the equilibrium concentrations (or pressures) of its reactants and products. If these concentrations are known, the calculation simply involves their substitution into the Kc expression.
For example, gaseous nitrogen dioxide forms dinitrogen tetroxide according to this equation:
For example, gaseous nitrogen dioxide forms dinitrogen tetroxide according to this equation:
37.9K
Calculating Standard Free Energy Changes
24.9K
The free energy change for a reaction that occurs under the standard conditions of 1 bar pressure and at 298 K is called the standard free energy change. Since free energy is a state function, its value depends only on the conditions of the initial and final states of the system. A convenient and common approach to the calculation of free energy changes for physical and chemical reactions is by use of widely available compilations of standard state thermodynamic data. One method involves the...
24.9K
Calculating pH Changes in a Buffer Solution
58.4K
A buffer can prevent a sudden drop or increase in the pH of a solution after the addition of a strong acid or base up to its buffering capacity; however, such addition of a strong acid or base does result in the slight pH change of the solution. The small pH change can be calculated by determining the resulting change in the concentration of buffer components, i.e., a weak acid and its conjugate base or vice versa. The concentrations obtained using these stoichiometric calculations can be used...
58.4K
Numerical Calculations
1.2K
In engineering applications, the representation of the numerical value is critical. Presenting or reporting the answer is one of the essential parts of engineering practices. Numerical calculations are performed using handheld calculators or computers since numerically accurate answers are always preferred.
The solution to a problem is obtained using different methods. While manually solving algebraic symbols is one of the most common methods, the graphical method is often preferred. Computers...
The solution to a problem is obtained using different methods. While manually solving algebraic symbols is one of the most common methods, the graphical method is often preferred. Computers...
1.2K
Net Torque Calculations
11.4K
When a mechanic tries to remove a hex nut with a wrench, it is easier if the force is applied at the farthest end of the wrench handle. The lever arm is the distance from the pivot point (the hex nut in this case) to the person’s hand. If this distance is large, the torque is higher. Only the component of the force perpendicular to the lever arm contributes to the torque. Therefore, pushing the wrench perpendicular to the lever arm is more advantageous. If multiple people apply force to...
11.4K
Calculating Equilibrium Concentrations
53.1K
Being able to calculate equilibrium concentrations is essential to many areas of science and technology—for example, in the formulation and dosing of pharmaceutical products. After a drug is ingested or injected, it is typically involved in several chemical equilibria that affect its ultimate concentration in the body system of interest. Knowledge of the quantitative aspects of these equilibria is required to compute a dosage amount that will solicit the desired therapeutic effect.
A more...
A more...
53.1K


