Leveraging ChatGPT and explainable AI for enhancing clinical decision support

Large language models (LLMs) excel in many natural language processing tasks. However, their direct application to tabular, domain-specific clinical data remains challenging, as they lack innate mechanisms for reasoning over structured numerical features. This paper presents HealthAI-Prompt, a novel framework that systematically adapts LLMs for tabular clinical decision-making-specifically, predicting diabetes risk-through contextual prompts that combine detailed task descriptions with domain knowledge. Our domain knowledge integration leverages insights from high-performing machine learning models optimized via automated machine learning (AutoML) technique, together with local explanations for representative examples. These explanations, generated by multiple methods, are rigorously evaluated using fidelity, stability, and monotonicity metrics, ensuring reliability and clinical validity. The most accurate explanations are then embedded into the prompts, enabling the LLM to interpret structured features in a clinically meaningful way without fine-tuning. This approach uniquely bridges AutoML-driven predictive modeling with LLM-based reasoning over tabular inputs, improving predictive accuracy, and transparency in healthcare AI. Through comparative analysis of HealthAI-Prompt and various AutoML techniques under diverse data conditions, we offer insights into the impact of different prompt engineering strategies on model performance. As part of our evaluation, we also include Chain-of-Thought prompting as a baseline to contextualize the gains of our proposed method.