⚠️ More than 1 in 4 LLM CTS answers missed AAOS CPGs
Embedded Files
In a study of 4 large language models answering standardized carpal tunnel syndrome management prompts, more than 1 in 4 recommendations were not concordant with American Academy of Orthopaedic Surgeons clinical practice guidelines. Across 252 recommendations, 72.6% were concordant, 11.5% discordant, and 15.9% indeterminate; ChatGPT-4 performed best at 92.1% concordance, while Mistral-7B trailed at 52.4%.
Why It Matters To Your Practice
Patients and clinicians are already using LLMs for CTS advice, but this study suggests those answers are not reliably guideline-aligned.
Even when outputs sound confident, about 27.4% were either discordant or too unclear to confirm against AAOS guidance.
Performance varied widely by model, so one strong result should not be generalized to all AI tools.
Clinical Implications
Do not treat LLM output as a substitute for AAOS CTS guideline review, especially for management recommendations.
If AI is used in workflow, pair it with clinician verification before applying advice to diagnosis, counseling, or treatment planning.
Model selection matters: ChatGPT-4 had the highest concordance and lowest discordance, but still requires oversight.
Insights
Researchers queried ChatGPT-4, Gemini, Claude-3, and Mistral-7B using standardized prompts derived from AAOS CTS CPGs, with each prompt asked 3 times per model.
Two blinded reviewers graded responses as concordant, discordant, or indeterminate, with excellent inter-rater reliability (Cohen's κ = 0.91).
The findings highlight that inconsistency is not just a technical issue — it can directly affect whether AI advice matches evidence-based orthopedic care.
The Bottom Line
LLMs can support information retrieval in CTS, but they are not yet dependable enough to use without guideline cross-checking.
For now, AI may be best used as an adjunct drafting or education tool — not an autonomous source of CTS management recommendations.
Page updated
Report abuse