What's a good prediction? Challenges in evaluating an agent's knowledge

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• General knowledge acquisition through task-independent world models is crucial for AI agents.
• Current evaluation methods often rely on estimator accuracy, which may not reflect true usefulness.
• Assessing the utility of learned knowledge remains a significant challenge in AI research.

Purpose of the Study:

• To highlight the limitations of using estimator accuracy for evaluating AI knowledge.
• To propose a novel evaluation approach for AI models in continual learning settings.
• To introduce a method for assessing the usefulness of predictive knowledge.

Main Methods:

• Demonstrated the conflict between accuracy and usefulness using thought experiments and empirical examples in Minecraft.
• Utilized the General Value Function (GVF) framework for analysis.
• Proposed evaluating AI knowledge by examining internal learning processes, specifically feature relevance.

Main Results:

• Accuracy of a model does not always correlate with its practical usefulness for an agent.
• A new evaluation metric based on feature relevance to the prediction task was proposed.
• The study provides a framework for assessing knowledge utility in online continual learning.

Conclusions:

• Rethinking AI model evaluation beyond simple accuracy is essential.
• Evaluating predictions based on their utility and internal learning processes offers a more robust assessment.
• This work pioneers the exploration of 'evaluation by use' for predictive knowledge in AI.