Making Bipedal Robot Experiments Reproducible and Comparable: The Eurobench Software Approach

Area of Science:

• Robotics
• Biomechanics
• Software Engineering

Background:

• Systematic benchmarking of bipedal systems (humanoids, exoskeletons, prosthetics) is crucial for performance evaluation.
• Existing methods lack standardization in data handling, algorithm definition, and implementation, hindering reproducibility.
• The European project Eurobench has gathered numerous protocols and testbeds for characterizing bipedal system performance.

Purpose of the Study:

• To propose a software methodology for the systematic benchmarking of bipedal systems.
• To address the challenge of standardizing the scoring process for experimental data.
• To ensure reproducibility and replicability of performance indicator computations.

Main Methods:

• Developed a methodology constraining software interfaces for scoring algorithms.
• Utilized continuous integration and deployment tools for software verification.
• Employed dockerization to create language-independent, executable instances of algorithms.

Main Results:

• A standardized methodology for preparing and implementing scoring algorithms has been established.
• The approach ensures reproducibility and replicability of benchmarking results.
• Metrics and documentation repositories adhering to this policy are available within the Eurobench project.

Conclusions:

• The proposed methodology simplifies the development and execution of benchmarking algorithms for bipedal systems.
• This standardization facilitates easier reproduction, replication, and comparison of experimental results across different studies.
• The approach is adaptable for application to other protocols and metrics, enhancing the field of bipedal system evaluation.