A Universal Approximation Result for Difference of Log-Sum-Exp Neural Networks

Area of Science:

• * Computational mathematics
• * Machine learning
• * Optimization

Background:

• * Classical feedforward neural networks can approximate continuous functions.
• * Designing systems with specific mathematical properties, like difference-of-convex functions, presents optimization challenges.

Purpose of the Study:

• * To introduce and analyze a novel neural network architecture: the log-sum-exp (LSE) network.
• * To demonstrate the universal approximation capabilities of LSE networks for continuous and positive functions.
• * To establish LSE networks as effective surrogate models for optimization-based design.

Main Methods:

• * Development of a neural network architecture using feedforward networks with exponential and logarithmic activation functions.
• * Application of a logarithmic transform to map LSE networks to generalized posynomials (GPOS).
• * Adaptation of a difference-of-convex algorithm for optimization using LSE-derived surrogate models.

Main Results:

• * LSE networks are proven to be smooth universal approximators of continuous functions over convex, compact sets.
• * The network class maps to subtraction-free ratios of generalized posynomials (GPOS), which are universal approximators of positive functions.
• * The proposed approach facilitates effective optimization-based design through efficient numerical methods.

Conclusions:

• * LSE networks offer a powerful tool for creating optimizable surrogate models with a difference-of-convex structure.
• * The methodology is validated through applications in data-driven diet design for type-2 diabetes and nonconvex optimization problems.
• * This work bridges neural network approximation theory with practical optimization challenges.