Apply computer vision in GUI automation for industrial applications.

Area of Science:

• Computer Science
• Human-Computer Interaction
• Industrial Automation

Background:

• Industry 4.0 is transforming workplaces by automating manual labor with smart machines and robots.
• Human roles are shifting towards desktop software usage for tasks like testing, quality inspection, and data monitoring.
• Repetitive and monotonous desktop tasks are prime candidates for graphical user interface (GUI) automation.

Purpose of the Study:

• To introduce Korat, a novel tool utilizing computer vision for GUI automation.
• To explore the challenges and solutions in applying computer vision for industrial GUI automation.
• To enhance the feasibility of GUI automation through improved recognition rates.

Main Methods:

• Development of Korat, a cross-platform GUI automation tool leveraging computer vision.
• Application of open-source Optical Character Recognition (OCR) for GUI automation using color screenshots.
• Implementation of critical pre-processing stages and algorithms to improve recognition accuracy.

Main Results:

• Korat demonstrates successful adoption in several industrial customer applications.
• Pre-processing techniques significantly increase the recognition rate of computer vision-based GUI automation.
• The enhanced recognition rate makes computer vision feasible for practical industrial usage.

Conclusions:

• Computer vision, particularly with enhanced OCR via pre-processing, offers a viable solution for GUI automation.
• Korat provides a robust platform for industrial test automation and robotic process automation.
• Further research is needed to address remaining challenges in computer vision for GUI automation.