When multimodal fusion fails: contrastive alignment as a necessary stabilizer for TCR-peptide binding prediction

Multimodal learning is often assumed to improve predictive performance by combining complementary views, yet in biological applications auxiliary modalities are frequently imperfect, incomplete, or derived from upstream predictors and heuristics. We study this issue in TCR-peptide binding prediction, where sequence embeddings from pretrained protein language models are strong and transferable, but structure-derived residue graphs must be built from predicted folds and discretized contacts. These structural views can therefore be noisy, inconsistent across proteins, and sensitive to modeling choices, making them difficult to optimize jointly with sequence features. In this setting, naive sequence + graph fusion can destabilize training and degrade generalization, falling below a sequence-only baseline when supervision is scarce or contacts are noisy. This motivates a practical goal: use imperfect structural information when it helps, without sacrificing stability when it does not.

We introduce TRACE, a lightweight framework that encodes each entity (TCR and peptide) with parallel sequence (frozen ESM-2) and residue-graph (GNN) towers, then applies CLIP-style intra-entity contrastive alignment before interaction modeling. The alignment encourages modality-consistent representations for the same biological entity, preventing noisy graph signals from dominating fusion. We evaluate under a leakage-controlled TCHard RN protocol with pair-disjoint splits, training-only model selection, and exclusion of negative-sampling metadata that otherwise trivially inflates AUROC. In this setting the task is near chance for all methods, and we do not claim state-of-the-art accuracy; our contribution is the failure-mode analysis, the alignment stabilizer, and the audited protocol itself. Among matched baselines TRACE attains the best mean AUROC (0.578±0.033 over five folds), and an ablation shows that intra-entity alignment acts as a stabilizer: it gives a small but consistent full-data gain (better on 4 of 5 folds), stays robust under substantial graph-edge corruption, and prevents collapse toward chance under limited supervision (+0.05 AUROC at 10%-20% of labels), the regime where unconstrained fusion fails. How modalities are integrated, and how carefully they are evaluated, matters more than how many are used.

Code is available at https://github.com/MineSelf2016/TRACE and archived at https://doi.org/10.5281/zenodo.20635593. Data are available at https://doi.org/10.6084/m9.figshare.31991007.