Lung Cancer Management: Revolutionizing Patient Outcomes Through Machine Learning and Artificial Intelligence

Lung cancer remains a leading cause of cancer-related deaths worldwide, with early detection critical for improving prognosis. Traditional machine learning (ML) models have shown limited generalizability in clinical settings. This study proposes a deep learning-based approach using transfer learning to accurately segment lung tumor regions from CT scans and classify images as cancerous or noncancerous, aiming to overcome the limitations of conventional ML models.

We developed a two-stage model utilizing a ResNet50 backbone within a U-Net architecture for lesion segmentation, followed by a multi-layer perceptron (MLP) for binary classification. The model was trained on publicly available CT scan datasets and evaluated on an independent clinical dataset from Hazrat Rasool Hospital, Iran. Training employed binary cross-entropy and Dice loss functions. Data augmentation, dropout, and regularization were used to enhance model generalizability and prevent overfitting.

The model achieved 94% accuracy on the real-world clinical test set. Evaluation metrics, including F1 score, Matthews correlation coefficient (MCC), Cohen's kappa, and Dice index, confirmed the model's robustness and diagnostic reliability. In comparison, traditional ML models performed poorly on external test data despite high training accuracy, highlighting a significant generalization gap.

This research presents a reliable deep learning framework for lung cancer detection that outperforms traditional ML approaches on external validation. The results demonstrate its potential for clinical deployment. Future work will focus on prospective validation, interpretability techniques, and integration into hospital workflows to support real-time decision making and regulatory compliance.