General Medicine · Medical Literature
Impact of Large Language Model-Based AI Tools on Physician-Patient Communication: Systematic Review and Meta-Analysis.
Abstract
Recent advances in large language models (LLMs) such as GPT-3/4 have spurred the development of artificial intelligence (AI) chatbots and advisory tools in medicine. These systems are posited to assist or augment physician-patient communication, potentially improving empathy, clarity, and responsiveness. However, their actual impact on communication outcomes remains uncertain. This study aimed to systematically review and meta-analyze peer-reviewed studies (2020-2025) evaluating how LLM-based interventions affect physician-patient communication, including empathy, clarity, trust, and patient understanding. Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines, we searched PubMed/MEDLINE, Embase, Scopus, and Web of Science for studies published from 2020 to 2025 examining LLM or chatbot applications in clinical communication contexts. Eligible designs included randomized, observational, cross-sectional, and qualitative studies. Two reviewers (WHP and SR) independently screened titles or abstracts, assessed full texts, and extracted data on study design, population, LLM type, communication measures, and outcomes. We conducted a qualitative synthesis and random-effects meta-analysis, reporting pooled standardized mean differences or odds ratios with 95% CIs. From 312 records, 10 studies were included, all quantitative and predominantly cross-sectional. Populations ranged from patients with chronic conditions to health care professionals and laypersons. Outcomes assessed included empathy (8 studies), clarity or information quality (6 studies), satisfaction or usefulness (4 studies), and trust perceptions (2 studies). In 6 direct comparisons of AI- versus physician-generated responses, LLMs were rated significantly higher in empathy in 5 studies. One large study found that chatbot replies were judged empathetic in 45.1% of cases versus 4.6% for physician replies (odds ratio approximately 9.8, P<.001). Similarly, ChatGPT-4 answers scored higher in empathy on a 5-point scale than human-written responses (mean 4.18 vs 2.70, P<.001). One neurology study showed higher empathy scores (Consultation and Relational Empathy Scale +1.38, P<.01) for ChatGPT answers. Only 1 study found no significant empathy difference. LLM content was also longer and more information-rich, improving patient-perceived clarity and understanding. On the other hand, GPT-4 simplified pathology reports, increasing patient comprehension scores (7.98 vs 5.23/10, P<.001) and reducing consultation time by 70%. However, AI replies were sometimes less concise or less readable for low-literacy patients. In pooled analyses (k=4 studies; total evaluations N=2604), LLM assistance showed a large positive effect on empathy (standardized mean difference 1.02, 95% CI 0.44-1.60; random-effects model). Patient satisfaction results were mixed. No study directly assessed long-term trust. Current evidence suggests that LLM-based chatbots can enhance physician-patient communication by producing more empathetic, detailed, and understandable responses. These improvements may positively influence patient experience and engagement. However, LLMs may also generate overly lengthy or occasionally inaccurate advice, emphasizing the need for physician oversight. While meta-analytic findings are promising, robust randomized controlled trials, real-world and longitudinal studies are needed to confirm benefits, assess trust outcomes, and define optimal clinical integration strategies.
Clinical relevance
This systematic review and meta-analysis addresses a critical and timely gap in evidence regarding whether LLM-based AI tools (e.g., GPT-3/4, ChatGPT) meaningfully improve physician-patient communication across domains of empathy, clarity, trust, and patient understanding. It is directly relevant to any clinician interacting with patients who may use or encounter AI-generated medical communication, spanning chronic disease management, neurology, and pathology contexts. As LLM adoption accelerates across health systems, clinicians need rigorous, pooled evidence — not anecdote — to guide responsible integration and oversight protocols.
Key findings
1. In 6 direct AI-versus-physician comparisons, LLM-generated responses were rated significantly higher in empathy in 5 of 6 studies, establishing a consistent directional advantage for AI-generated communication. 2. In the largest individual study, chatbot replies were judged empathetic in 45.1% of cases versus only 4.6% for physician replies (OR ≈ 9.8, P<.001), representing a clinically striking disparity in perceived empathic quality. 3. ChatGPT-4 responses scored higher in empathy on a 5-point Likert scale compared to human-written responses (mean 4.18 vs. 2.70, P<.001), a difference of 1.48 points indicating a large magnitude effect. 4. A neurology-specific study demonstrated superior empathy scores for ChatGPT answers using the validated Consultation and Relational Empathy (CARE) Scale (net difference +1.38 points, P<.01). 5. GPT-4 simplification of pathology reports significantly improved patient comprehension scores (7.98 vs. 5.23 out of 10, P<.001) and reduced associated consultation time by 70%, suggesting substantial efficiency gains. 6. Pooled random-effects meta-analysis across 4 studies (N=2,604 total evaluations) demonstrated a large positive effect of LLM assistance on empathy outcomes (standardized mean difference [SMD] 1.02, 95% CI 0.44–1.60), though heterogeneity across designs warrants caution in generalizability.
Practice implications
Clinicians and health systems considering LLM integration into patient communication workflows should recognize that current evidence supports a meaningful enhancement in perceived empathy and information clarity, particularly for complex or distressing communications such as pathology report disclosure, where AI-assisted simplification reduced consultation burden by 70% and significantly improved comprehension. However, because AI replies were sometimes excessively lengthy or less readable for low-health-literacy patients, and no study assessed long-term trust outcomes, deployment should include mandatory physician review and editing of AI-generated content before patient delivery, with particular attention to readability calibration for vulnerable populations. The evidence base currently rests on predominantly cross-sectional, non-randomized studies (k=10), and robust RCTs with longitudinal follow-up are urgently needed before LLM tools can be recommended as standard augmentation to physician-patient communication across clinical settings.