Topic classification of synchrotron experiment proposals using the OpenAlex model: enhancing metadata granularity and...

Experiment proposals at synchrotron facilities serve as the primary gateway for instrument access. They currently lack the standardized and granular topic metadata necessary for tasks such as classification and review, and, broadly speaking, reuse. This paper defines and tests the feasibility of a real-time topic classification service for experiment proposals using an open-source machine-learning model and domain experts for the evaluation phase. We applied the OpenAlex topic classification model to 5384 experiment proposals and selected 209 of them to each be independently evaluated by three domain experts to assess the performance and utility of the model. Analysis of the evaluations reveals a general consensus among the reviewers regarding the model's predictions, with a Krippendorff's alpha of 0.572. We also find that 74.2% of the proposals had at least one topic that was unanimously deemed relevant, which suggests that the model performs well enough to be used in a live setting with real-time verification. However, we do not recommend using it in automated environments without human oversight, given the proposal-based precision score of 56.0%. By aligning the data infrastructure of photon and neutron facilities with the OpenAlex ecosystem, we also lay the groundwork for the eventual inclusion of proposals and experiment reports into OpenAlex, which is necessary for a complete record of a research activity.