End-to-End Pipeline Integrating Local Small Language Models and Machine Learning for Data Extraction and Stroke...

Background: A substantial proportion of clinical data is stored in unstructured text, limiting its utility for evidence generation. Small language models (SLMs) can extract information but face hallucination risks and privacy concerns. Locally deployable SLMs are needed for secure and reliable healthcare use. Methods: We designed an end-to-end pipeline integrating clinical text extraction with stroke outcome prediction. Records of 1,398 patients screened for ischemic stroke were reviewed, with 1,166 included. A Llama 3 8B SLM was fine-tuned using low-rank adaptation and 4-bit quantization for local feasibility, guided by few-shot prompting. Multi-tiered validation-rule-based checks, retrieval-augmented generation, cosine similarity flagging, and human-in-the-loop review-was implemented. Structured data from 767 patients were used to train models predicting poor outcomes at 3 months. Results: Baseline extraction accuracy was 64.9% (95% CI, 62.0% to 67.8%), improving to 86.0% after fully automated multi-tiered validation, and further to 97.0% (95% CI, 95.7% to 98.3%) following human-in-the-loop review. Template-based variables achieved F1 > 0.90 (95% CI, 0.88 to 0.96). Narrative extraction reached F1 = 0.87 (95% CI, 0.84 to 0.90). NIHSS scores were extracted with a mean absolute error of 0.853 (95% CI, 0.791 to 0.915). TabPFN achieved an AUROC of 0.816 (95% CI, 0.784 to 0.847) with good calibration, confirming reliable risk stratification. Conclusion: This study demonstrates a privacy-preserving, efficient pipeline for clinical text processing. By combining an SLM with multi-tiered validation and predictive modeling, it offers a proof-of-concept solution with potential for broader deployment to transform unstructured records into structured data suitable for stroke outcome research and decision-support modeling.