Off-Line Evaluation of Mobile-Centric Indoor Positioning Systems: The Experiences from the 2017 IPIN Competition

Area of Science:

• Computer Science
• Robotics
• Geomatics

Background:

• Smartphone-based indoor positioning is a rapidly advancing field with significant application potential.
• Current research often lacks standardized evaluation metrics, hindering direct comparison of different positioning solutions.
• Fair evaluation frameworks are crucial for assessing the real-world performance and adaptability of these technologies.

Purpose of the Study:

• To detail the 2017 IPIN (Indoor Positioning and Indoor Navigation) indoor localization competition.
• To present the datasets, participating teams, and obtained results from the competition.
• To analyze the competition's effectiveness in generating open data for comparing indoor positioning solutions.

Main Methods:

• Organized the 2017 IPIN competition featuring extensive pre-recorded datasets.
• Collected and analyzed results from participating research teams.
• Compared competition outcomes with other evaluation approaches (Microsoft, Perf-loc) and online platforms.

Main Results:

• The competition successfully generated comparable data for evaluating diverse indoor positioning solutions.
• Results provided insights into the performance of various smartphone-based localization techniques.
• Lessons learned from organizing the competition offer valuable guidance for future standardization efforts.

Conclusions:

• Competition-based evaluations using standardized datasets are effective for advancing indoor positioning research.
• The 2017 IPIN competition demonstrated the benefits of open data and community-driven benchmarking.
• This work advocates for standardized evaluation frameworks and widely adopted benchmark strategies in the field.