Multitasking vision language models for vehicle plate recognition with VehiclePaliGemma

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• License Plate Recognition (LPR) traditionally uses Optical Character Recognition (OCR) but struggles with image distortions like noise, blurring, and close characters.
• Existing LPR methods require substantial improvements for accurate recognition, particularly with challenging image quality.

Purpose of the Study:

• To evaluate the efficacy of various Visual Language Models (VLMs) in overcoming LPR challenges.
• To introduce and validate "VehiclePaliGemma", a specialized VLM for robust license plate recognition.

Main Methods:

• Evaluated multiple VLMs (GPT-4o, Gemini 1.5, PaliGemma, Llama 3.2, Claude 3.5 Sonnet, LLaVA, VILA, moondream2) for license plate recognition.
• Developed and fine-tuned "VehiclePaliGemma" using a dataset of Malaysian license plates under complex conditions.
• Compared VehiclePaliGemma against state-of-the-art methods and other VLMs.

Main Results:

• VehiclePaliGemma achieved a superior accuracy of 87.6% on a challenging dataset.
• The model demonstrated efficient processing at 7 frames per second on an A100-80GB GPU.
• Explored VehiclePaliGemma's multitasking ability in identifying plates from multiple vehicles with varied orientations and conditions.

Conclusions:

• VehiclePaliGemma significantly enhances license plate recognition accuracy, especially for distorted and complex images.
• VLMs offer a promising advancement over traditional OCR-based LPR systems.
• The fine-tuned VehiclePaliGemma model shows potential for real-world applications requiring high-accuracy LPR.