Enhanced lung cancer detection using BiLSTM attention and transformer models on clinical text data

Lung cancer continues to be a major cause of death globally, making early and precise detection crucial for improving patient outcomes. In this research, we introduce an enhanced method for predicting and detecting lung cancer by combining Bidirectional Long Short-Term Memory (BiLSTM) networks with an attention mechanism and transformer-based models. This hybrid model effectively tackles common issues such as overfitting, data imbalance, and long-sequence dependencies in textual clinical data. A well-curated dataset of patient medical records, including demographics, symptoms, radiology reports, and other clinical information, was used to assess the model's performance. The experimental results showed a marked improvement in prediction accuracy, with the proposed model achieving 99.8% accuracy and significantly lowering validation loss to 0.3112, outperforming existing systems. The incorporation of attention mechanisms with BiLSTM and transformer models improved the model's ability to focus on critical clinical features, thus enhancing its generalization capabilities. Additionally, the model's faster training time (13.99 s) demonstrates its practicality for clinical use, particularly in resource-constrained settings. While the results are promising, potential overfitting due to the small dataset suggests further research with larger datasets and advanced regularization methods is needed. This research highlights the potential of advanced deep learning techniques in lung cancer detection, providing a robust, efficient, and accurate tool that can be integrated into clinical decision-making processes to enable timely and accurate diagnoses.