Interpretable vocal tract and respiratory inversion via physics-informed neural operators.

Area of Science:

• Acoustics and Signal Processing
• Biomedical Engineering
• Machine Learning

Background:

• Physiological variations in vocal tracts and respiratory systems challenge accurate timbre modeling and real-time vocal analysis.
• Current data-driven methods often lack physical interpretability and speaker robustness.
• Accurate vocal tract geometry and respiratory dynamics reconstruction from acoustic data is crucial for advanced vocal analysis.

Purpose of the Study:

• To propose a physics-informed multimodal inversion framework using Kolmogorov-Arnold (KAN) operators for interpretable reconstruction of vocal tract geometry and respiratory dynamics.
• To achieve high timbre fidelity and low latency in vocal analysis.
• To provide a foundation for fine-grained timbre reconstruction and personalized vocal analysis.

Main Methods:

• A nested three-layer KAN inverts audio spectrum into vocal tract cross-sectional areas.
• A gated recursive module constrains pressure evolution using mass-momentum conservation.
• Super-resolution prediction head optimized with fractional-order temporal regularization and wave-equation residuals enhances timbre fidelity.

Main Results:

• Achieved log-spectral distortion of 1.83 ± 0.32 dB and sub-band error rate of 6.4 ± 1.1% (1.2-2.4 kHz).
• Demonstrated low end-to-end latency (14.2-18.3 ms) and compact memory usage (108-121 MB) on edge devices.
• Maintained minimal vocal tract geometry error (MAE-CSA ≤ 0.23) and respiratory estimation bias (RMSE-P ≤ 0.52) across unseen voice types.

Conclusions:

• Integrating neural operators with physical constraints enables accurate, interpretable, and real-time inversion of vocal physiology.
• The proposed framework offers a principled technical foundation for advanced timbre reconstruction.
• This approach paves the way for personalized vocal analysis and improved speech technology.