pxGPT: A schema-constrained multimodal LLM framework for plant phenotyping

Deep learning can scale image-based plant phenotyping but typically targets predefined traits, whereas manual phenotypic description is more flexible but difficult to scale. We present Phenotype eXplorer Generative Plant Traits (pxGPT), a Python framework that converts multi-view plant images into structured phenotype records. The workflow combines free-text description, dataset-specific schema generation with human review, and schema-constrained information extraction. We evaluated the extraction stage on three Brassica datasets using 78,973 annotations from ten annotators who had no access to pxGPT predictions. For 38-45 qualitative traits per dataset, pxGPT-human agreement ranged from 0.572 to 0.712, compared with 0.620 to 0.739 agreement among human annotators. All 32,736 required trait fields were present; 93 values were empty, and no qualitative descriptor violations were recorded. For selected morphological structures, pooled false-positive rates were 0.5-8.0%, while sensitivity varied across structure types in one mature-plant dataset (37.5-100%). The curated schemas were applied unchanged with two commercial models and one locally hosted open-weight model, with differences in mean accuracy of 0.03 or less. In the evaluated Brassica datasets, these results support schema-guided extraction of visible plant traits without separate model training for each trait. Human-guided schema development offers a path to extend pxGPT to other crops and imaging conditions, with accuracy requiring validation in each new setting.

The software is available at https://github.com/YaoChengLab/pxgpt and an archival version of the source code is available on Zenodo (https://doi.org/10.5281/zenodo.22931745). Curated schemas used in this study, annotation guides, phenotype records and human annotations are available at https://doi.org/10.5281/zenodo.22907554.

Supplementary data are available at Bioinformatics online.