A hybrid LSTM-Transformer model for simulation-to-experiment transfer in radionuclide identification using NaI(Tl) gamma spectroscopy

Radionuclide identification is central to nuclear safety, security, and environmental monitoring, yet reliable identification with low-resolution NaI(Tl) gamma spectrometers remains difficult in mixed-source measurements because broadened photopeaks, Compton continua, counting fluctuations, and background contributions reduce spectral separability. Although deep learning has improved automated spectral interpretation, models trained mainly on simulated spectra often lose accuracy when applied to experimentally measured data because detector response, spectral shape, and acquisition conditions are not fully reproduced in simulation. This study develops a hybrid Long Short-Term Memory-Transformer model, denoted as LT_R for mixed-radionuclide identification under a train-on-simulated and test-on-experimental framework. The model first uses recurrent sequence encoding to preserve ordered spectral continuity along the energy axis and then applies self-attention to capture long-range dependencies among separated peak and continuum regions. Trained on simulated NaI(Tl) spectra and evaluated on an independent experimental dataset containing single- and mixed-source combinations of ⁶⁰Co, ¹³⁷Cs, ¹⁵²Eu, and ¹³³Ba, LT_R outperformed Convolutional Neural Network, Long Short-Term Memory, and Transformer models. On experimental spectra, it achieved a total nuclide accuracy of 0.930 and an average accuracy of 0.953, with only a 3.8-percentage-point decrease from simulated to experimental testing. The model also maintained stable performance across different simulated training-set sizes, source-detector distances, acquisition times, and mixture complexities. These results indicate that combining sequential spectral encoding with global attention improves simulation-to-experiment robustness for radionuclide identification in experimentally measured NaI(Tl) spectra.