Automated identification of keratinocyte cancers in pathology reports using large language models

Keratinocyte cancers (KCs) are the most prevalent cancers in white-skinned individuals, yet remain underrepresented in cancer registries because reporting requirements differ greatly across jurisdictions. Manual extraction of KC subtypes from medical reports is labor-intensive and time-consuming, particularly as reports often document multiple co-excised skin lesions. Artificial intelligence offers automated solutions for disease phenotyping from unstructured clinical text. We fine-tuned the open-source large language model Meta-AI (LLaMA) 3.1-8B-Instruct on 26,179 manually reviewed pathology reports from 10,326 Australian QSkin Sun and health study participants who had histologically confirmed KCs. Independently validation was performed on 217 pathology reports from the Skin Tumors in Allograft Recipients cohort. The model achieved F1-scores above 0.90 for the four KC subtypes of interest: squamous cell carcinoma, basal cell carcinoma, keratoacanthoma, and intraepidermal carcinoma. Frequency-weighted mean F1-scores reached 0.84 (95% confidence interval: 0.843-0.846) for diagnosis classification and 0.83 (95% confidence interval: 0.826-0.828) for anatomical site. External validation demonstrated robust performance with F1-scores between 0.73-0.86 for the four KCs of interest. Our fine-tuned model QSkin-llama-3.1-8b works locally. It accurately classifies lesion counts, diagnoses, and anatomical sites. It processes 24 pathology reports per minute with minimal preprocessing, enabling scalable automated disease phenotyping for large health datasets.