美联储-HeLLo:高效的联邦基金会模型微调与异质的LoRA分配
概括
联合学习 (FL) 与异质的LoRA分配 (Fed-HeLLo) 改善了基础模型对不同客户资源的微调. 新型策略适应LoRA层分布,以获得更好的性能和稳定性.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 分布式系统 分布式系统
背景情况:
- 联合学习 (FL) 允许跨客户端的基础模型 (FMs) 的协作微调.
- 联合低级适应 (LoRA) 方法允许在更少的参数下进行高效的局部微调.
- 现有的方法往往忽视了客户资源异质性和最佳的本地培训策略.
研究的目的:
- 提出Fed-HeLLo,一个基于LoRA的联合微调框架,解决客户资源异质性问题.
- 开发适应性异质LoRA分配 (HLA) 策略,以优化全球微调性能.
主要方法:
- 通过Fed-HeLLo,客户可以通过不同的本地可训练的LoRA层微调FM.
- 开发的HLA策略包括基于动态层重要性的费舍尔信息矩阵 (FIM-HLA) 和对稳定性的几何定义 (GD-HLA) 策略.
- 引入随机的GD-HLA (RGD-HLA) 以提高准确性.
主要成果:
- 联邦-HeLLo有效地根据客户资源和层次重要性分配了LoRA层.
- FIM-HLA和GD-HLA策略在联合微调方面取得了显著的改进.
- 跨不同数据集和非i.i.d.的评估. 分发证实了框架的有效性和效率.
结论:
- Fed-HeLLo为基于LoRA的异质联合微调提供了有效的解决方案.
- 拟议的HLA策略提高了资源有限的FL设置中的性能,稳定性和准确性.
- 这项工作为不同环境中的协作FM微调提供了一个实用的框架.
相关概念视频
Load-frequency control
253
Load-frequency control (LFC) is vital for maintaining power system stability, ensuring that frequency and power flows remain within acceptable limits during load changes. Turbine-governor control eliminates rotor accelerations and decelerations following load changes. However, a steady-state frequency error persists when the change in the turbine-governor reference setting is zero. In an interconnected power system, each area agrees to export or import a scheduled amount of power through...
253
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
100
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
100
Fineness Modulus
769
The fineness modulus (FM) of aggregate is a numerical index that measures the coarseness or fineness of the particles. It is calculated by adding the cumulative percentages of aggregate retained on each of a specified series of sieves and dividing the sum by 100.
Consider performing sieve analysis on sand through a set of ASTM sieves. The weight of aggregate retained in each sieve and pan placed at the bottom is recorded, as given in Column B of Table 1.
To determine the fineness modulus of...
Consider performing sieve analysis on sand through a set of ASTM sieves. The weight of aggregate retained in each sieve and pan placed at the bottom is recorded, as given in Column B of Table 1.
To determine the fineness modulus of...
769
Linear Approximation in Frequency Domain
131
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
131
Association Areas of the Cortex
6.2K
Association areas are regions of the cerebral cortex that do not have a specific sensory or motor function. Instead, they integrate and interpret information from various sources to enable higher cognitive processes such as memory, learning, and decision-making. Some key association areas include the following:
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
Prefrontal Association Area: This area is located in the frontal lobe and is involved in planning, decision-making, and moderating social behavior. It connects with primary motor areas,...
6.2K
Frequency-dependent Selection
22.2K
When the fitness of a trait is influenced by how common it is (i.e., its frequency) relative to different traits within a population, this is referred to as frequency-dependent selection. Frequency-dependent selection may occur between species or within a single species. This type of selection can either be positive—with more common phenotypes having higher fitness—or negative, with rarer phenotypes conferring increased fitness.
22.2K


