Effects of education level on natural language processing in cardiovascular health communication

Cardiovascular disease (CVD) remains the leading cause of mortality worldwide, underscoring the importance of accessible health communication. Artificial intelligence (AI) tools such as ChatGPT and MediSearch have potential to bridge knowledge gaps, but their effectiveness depends on both accuracy and readability. This study evaluated how natural language processing (NLP) models respond to CVD-related questions across different education levels.

Thirty-five frequently asked questions from reputable sources were reformatted into prompts representing lower secondary, higher secondary, and college graduate levels, and entered into ChatGPT Free (GPT-4o mini), ChatGPT Premium (GPT-4o), and MediSearch (v1.1.4). Readability was assessed using Flesch-Kincaid Ease and Grade Level scores, and response similarity was evaluated with BERT-based cosine similarity. Statistical analyses included ANOVA, Kruskal-Wallis, and Pearson correlation.

Readability decreased significantly with increasing education level across all models (p < 0.001). ChatGPT Free responses were more readable than MediSearch (p < 0.001), while ChatGPT Free and Premium demonstrated higher similarity to each other than to MediSearch. ChatGPT Premium explained the greatest variance in readability (r = 0.350; p < 0.001), suggesting stronger adaptability to user education levels compared to ChatGPT Free (r = 0.530; p < 0.001) and MediSearch (r = 0.227; p < 0.001).

These findings indicate that while NLP models adjust readability by education level, output complexity often exceeds average literacy, highlighting the need for refinement to optimize AI-driven patient education.