A JSBSim Sensor-Interface Protocol for Selecting Learned Fixed-Wing Flight-Dynamics Surrogates

Area of Science:

• Aerospace Engineering
• Artificial Intelligence
• Robotics

Background:

• Autonomous flight planners rely on estimated states, not ground truth, making learned dynamics surrogates crucial for predicting future states.
• Short-term accuracy of these surrogates does not guarantee long-term trajectory reliability, posing a challenge for planning.
• Existing evaluation methods often overlook the complexities of real-world sensor data and estimation processes.

Purpose of the Study:

• To develop and present a novel evaluation and selection methodology for learned flight dynamics surrogates used in autonomous planning.
• To assess surrogate performance under realistic sensor-interface and estimator conditions, including delays, dropouts, and filtering.
• To guide the selection of appropriate surrogates based on specific planning tasks and performance criteria.

Main Methods:

• Developed a JSBSim sensor-interface evaluation protocol to test surrogate predictions using simulated onboard sensing data (IMU, GNSS, air-data, etc.).
• Evaluated five families of learned surrogates (including LSTM and Transformer networks) over a 20-second prediction horizon.
• Assessed surrogates using various criteria such as Root-Mean-Square Error (RMSE), shared-threshold failure, self-scaled divergence, and capped-risk.

Main Results:

• All evaluated surrogates diverged significantly under strict physical tolerances, indicating none are currently field-ready; results are comparative stress-test evidence.
• Long Short-Term Memory (LSTM) networks excelled in RMSE and absolute fidelity metrics, while Transformer networks performed better under self-scaled divergence and capped-risk criteria.
• The choice of evaluation metric and threshold family critically influences surrogate selection, highlighting the task-dependent nature of performance.

Conclusions:

• Learned flight dynamics surrogates must be evaluated using sensing-aware, estimator-conditioned, and task-specific long-horizon criteria, not solely short-horizon accuracy.
• The developed protocol provides a transparent and auditable method for assessing surrogates, crucial for reliable autonomous flight planning.
• No single surrogate architecture is universally superior; performance is contingent on the specific application, sensing conditions, and chosen evaluation benchmarks.