📈 ChatGPT ICC 0.90 vs faculty 0.39 grading DI
📈 ChatGPT ICC 0.90 vs faculty 0.39 grading DI
In a retrospective study of 55 deidentified drug information (DI) responses written by second-year pharmacy students, ChatGPT 5.2 graded with much higher reliability than faculty (ICC 0.90 vs 0.39), while Gemini 2.5 Flash also outperformed faculty (ICC 0.72). Using mean faculty score as the benchmark, both AI models had a mean absolute error of 2.1 points and showed no proportional bias, according to the study.
Why It Matters To Your Practice
AI may be reliable enough for first-pass grading or quality checks when human raters vary widely, even after calibration.
The gap here was notable: faculty agreement was low, while ChatGPT approached excellent reliability on the same 32-point rubric.
For clinician-educators, that raises a practical use case for AI in standardized assessment of written DI work.
Clinical Implications
AI could support rubric-based scoring of student, trainee, or staff written responses before final human review.
Both models graded slightly harsher than faculty, by a median 1.2 to 1.7 points lower, about a 5% difference on the overall rubric.
Because neither model showed proportional bias, the harsher scoring did not worsen for stronger or weaker essays.
Insights
The study used 55 deidentified student DI responses, a 32-point grading rubric, and five faculty graders who completed two calibration sessions.
Each response was graded by three randomly assigned faculty plus two AI models: OpenAI ChatGPT 5.2 and Google Gemini 2.5 Flash.
Median total scores were 25.9 for faculty, 24.2 for ChatGPT, and 24.7 for Gemini; the two AI models did not differ significantly from each other.
The Bottom Line
Even with faculty consensus-building and rubric calibration, AI delivered more consistent grading than human evaluators in this DI assessment study.
For practices involved in education, competency review, or written knowledge assessment, AI looks more useful as a standardizer than as a replacement for final judgment.
The near-term opportunity is not autonomy, but dependable augmentation.
Reference: The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses.