How Visually Literate Are Large Language Models? Reflections on Recent Advances and Future Directions

Large language models (LLMs) are now being applied to the tasks of visualization generation and understanding, demonstrating these models' ability to be "visually literate." On the generation side, LLMs have shown promise in powering natural languages' interfaces for visualization authoring while also suffering from usability and inconsistency issues. On the interpretation side, models (especially vision-language models) can answer basic questions about visualizations, synthesize visual and textual information, and detect misleading visual designs. However, models also tend to struggle with certain analytic tasks, and their takeaways from reading visualizations often differ from those of humans. We aim to both illuminate the state of the art in LLMs' visualization literacy and speculate on where such work may, and perhaps ought to, take us next.