Transformer-based classification and interpretability of NR3C1 expression patterns in OSCC: Metabolic adaptation...

Oral squamous cell carcinoma (OSCC) involves several oncogenic proteins for survival. Protein sequence classification is a fundamental challenge in computational biology, complicated by the complex, non-linear relationships within sequences. Recent advances in transformer-based language models have yielded promising results on biological sequence tasks. The study involves a comprehensive evaluation of four transformer models, compared with two deep learning models and two traditional machine learning classifiers (Random Forest and SVM), for protein sequence classification of NR3C1 peptide sequences.

All models were trained for 100 epochs on 5 UniProt sequences, split into medium (200-500 aa) and long (>500 aa) classes. Sequences were tokenized, padded, or truncated to 512 tokens, and converted for BERT, RoBERTa, DistilBERT, and ALBERT. The dataset was split into 80% for training and 20% for validation, with stratified class balance.

Among the four transformer models, RoBERTa performed best with an F1-score of 0.8574, followed by ALBERT and BERT with scores of 0.8509 and 0.8378, respectively. These models performed far better than the deep learning models, which had an F1-score of approximately 0.763, and the traditional methods, which had an F1-score of 0.693. ALBERT achieved approximately 99.2% of RoBERTa's performance while using only about 9.6% of its parameters. Overall, RoBERTa and other transformers yield the best-performing models for protein sequence classification.

Transformer models, especially RoBERTa, outperform conventional methods for NR3C1 protein sequence classification, achieving higher accuracy and efficiency.