A retrospective study of 5,000 colonoscopies found 97.5% agreement between a random forest model and a large language model for colorectal neoplasia classification, with the LLM more often matching expert review when the models disagreed. The study, presented at the 2026 American College of Gastroenterology Annual Meeting, suggests AI may be enough for scalable colonoscopy quality monitoring, though cost and false-positive tradeoffs remain important.

Why It Matters To Your Practice

• Automated neoplasia classification could reduce the manual burden of tracking colonoscopy quality endpoints across large volumes of reports.

• High concordance suggests AI may be practical for routine quality surveillance, especially in systems seeking scalable reporting workflows.

• The main clinical caveat: strong overall agreement can still mask meaningful differences in false positives and missed lesions.

Clinical Implications

• The models agreed in 4,877 of 5,000 cases.

• Among 123 discordant cases, the LLM more often matched expert assessment for adenomas (116 cases vs 29 for random forest) and serrated lesions (113 vs 108).

• The random forest model had slightly lower omission rates, but a substantially higher false-positive rate.

Insights

• This was a retrospective comparison of 2 independently developed AI approaches applied to colonoscopies performed from 2023 to 2025.

• Experts manually reviewed conflicting classifications to determine which model aligned better with clinical judgment.

• The authors flagged limitations of NLP-based systems across larger datasets with variable reporting patterns and called for cost-effectiveness studies before wider deployment.

The Bottom Line

• For clinicians, the takeaway is not just that AI can agree with itself, but that model choice may affect downstream quality metrics, workload, and trust.

• If your practice is considering AI for colonoscopy reporting, prioritize validation against expert review and pay close attention to false-positive performance, not just headline agreement.