Sarcouncil Journal of Medical Series

Abstract: Background: Heart failure (HF) remains a significant global health burden, necessitating effective patient education and accessible medical information. Artificial intelligence (AI), particularly large language models (LLMs) like ChatGPT and Gemini, has emerged as a potential tool for providing on-demand medical explanations. However, the linguistic complexity and readability of these AI-generated responses in the context of HF require systematic evaluation. Objective: This study aimed to comparatively evaluate the readability levels of responses generated by ChatGPT and Gemini regarding heart failure–related queries. Methods: A comparative cross-sectional study was conducted using 40 standardized questions based on established HF guidelines, covering domains such as pathophysiology, symptoms, treatment, and self-care. Responses were collected over 20 repetitions. Readability was assessed using Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level, Gunning Fog Index, and SMOG Index. Statistical analyses were performed using JASP 0.96, with a significance threshold of p < 0.05. Results: No statistically significant differences were found between Gemini and ChatGPT in FRE scores (p=0.542), Flesch-Kincaid Grade Level (p=0.101), or Gunning Fog Index (p=0.094). However, Gemini (Median: 11.00) demonstrated a statistically significantly higher SMOG Index score compared to GPT (Median: 6.00) (p=0.046). Overall, both models produced content at an academic or "medium difficulty" level. Conclusion: While both models exhibit similar readability in most metrics, Gemini produces more complex text according to the SMOG Index, suggesting a higher density of technical terminology. The academic level of these responses may pose barriers for patients with low health literacy. Therefore, AI-generated HF information should be supervised by healthcare professionals to ensure clarity and accessibility for patient education.