Visual Food Ingredient Prediction Using Deep Learning with Direct F-Score Optimization

Food ingredient prediction from images is a challenging multi-label classification task with significant applications in dietary assessment and automated recipe recommendation systems. This task is particularly difficult due to highly imbalanced classes in real-world datasets, where most ingredients appear infrequently while several common ingredients dominate. In such imbalanced scenarios, the F-score metric is often used to provide a balanced evaluation measure. However, existing methods for training artificial neural networks to directly optimize for the F-score typically rely on computationally expensive hyperparameter optimization. This paper presents a novel approach for direct F-score optimization by reformulating the problem as cost-sensitive classifier optimization. We propose a computationally efficient algorithm for estimating the optimal relative cost parameters. When evaluated on the Recipe1M dataset, our approach achieved a micro F1 score of 0.5616. This represents a substantial improvement from the state-of-the-art method's score of 0.4927. Our F-score optimization framework offers a principled and generalizable solution to class imbalance problems. It can be extended to other imbalanced binary and multi-label classification tasks beyond food analysis.