Beyond performance metrics: evaluating the unique value of generative AI in hybrid cybersecurity threat detection

This study examines the role of generative artificial intelligence (GenAI) in cybersecurity threat detection, focusing on its usefulness in workflows that support human decision-making.

Experiments were performed on the BODMAS dataset (134,435 samples) and a smaller exploratory subset of UNSW-NB15. State-of-the-art machine learning (ML) classifiers were compared with a zero-shot large language model (LLM) using standard classification metrics, while also considering latency, cost, and hallucination risk.

ML classifiers consistently outperformed the LLM-based system on standard detection metrics. However, the LLM showed value in cases of ambiguity, where it could provide short plain-language explanations, organize alert-related context, and generate initial interpretations for instances that did not match learned classes.

GenAI is unlikely to replace ML-based detection methods, but it can provide useful interpretive support for ambiguous or unfamiliar alerts. A hybrid pipeline is therefore proposed, in which ML handles high-confidence and time-sensitive decisions, while the LLM is used selectively for low-confidence cases or when explanatory support is needed. Human oversight remains necessary to address hallucination risk and ensure reliability.