Measuring progress in multirobot research with rating methods--the RoboCup example

Rating the intelligence of artificially made systems is important for measuring progress in scientific and engineering methods. Unfortunately, there is currently no universal agreement about what is considered an intelligent system, and how to measure its intelligence. This research focus on measuring the progress in the robotic technologies deployed for the RoboCup competitions, since one of the original premises of those competitions was to advance the development of intelligent robotic systems. A method used for rating the competence of human chess players is adapted for measuring the advancement in the competence of robotic teams. The results indicate significant yearly improvements in the capabilities of the robotic teams. The same method can be used to indirectly quantify the benefits in specific technology choices.