In a masked cross-sectional study of 4 major LLMs answering 45 expert-level anterior segment ophthalmology questions, 11 subspecialists completed 9,900 ratings and found better perceived quality for Claude and Grok than for ChatGPT and Gemini. But in the same study, mean safety did not significantly differ across models, while post hoc analysis found a larger severe-safety-rating tail for Claude and Grok — a hypothesis-generating warning for clinicians using AI in specialty care.

Why It Matters To Your Practice

• For clinicians testing AI in consult support, patient messaging, or education, average scores may miss the rare but consequential unsafe answer.

• This study suggests the highest-rated models on quality and usability may still produce more severe safety concerns in a subset of responses.

• The paper assessed expert-rated perception rather than verified correctness, which mirrors real-world first impressions but not ground-truth clinical accuracy.

Clinical Implications

• Do not equate smoother, more complete answers with lower risk — especially in subspecialty decision support.

• If your team pilots LLMs, track outlier failures and escalation-worthy responses, not just average satisfaction or readability.

• Use human review for high-stakes ophthalmic recommendations, since the study was zero-shot and disabled browsing, retrieval, and memory.

Insights

• The design was masked and randomized in display order, reducing obvious brand bias across ChatGPT, Gemini, Claude, and Grok.

• Questions spanned 6 anterior segment domains and were rated on accuracy, completeness/clinical utility, clarity/readability, overall safety, and potential for harm.

• The authors note these findings reflect a single time point and expert preference, not validated clinical correctness; the severe-safety signal was post hoc.

The Bottom Line

• For practice adoption, mean performance alone is no longer enough for AI risk assessment.

• Claude and Grok were judged stronger on perceived quality, but 2 models also showed more severe safety flags despite tied average safety.

• Clinicians should evaluate both typical performance and worst-case behavior before trusting LLMs in specialty workflows.

Reference: Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions.