In an implementation study of NLP on 454,127 CAMHS EHR notes from 5,906 autistic young people and those with ADHD, agreement was highest for medication labels, with held-out macro F1 of 0.87 (95% CI 0.86-0.89) across 4 medication labels versus 0.71 (95% CI 0.68-0.73) across problem, therapy, and outcome labels. The study, conducted in the Oxford Health NHS Foundation Trust CAMHS research setting, found NLP integration was feasible but less reliable for cognitive problems and treatment outcomes, underscoring that free-text extraction is not yet enough for broader clinical inference.

Why It Matters To Your Practice

• Medication names, doses, and frequencies were the most consistently captured elements, suggesting AI may reach practice value first in medication reconciliation, audit, and service evaluation rather than nuanced symptom interpretation.

• For clinicians working in child and adolescent mental health, the study shows that secure, locally deployed NLP can be integrated into NHS-style research workflows without major reported harms.

• Performance gaps across labels matter: what works well for medication data may not translate to outcomes, therapy details, or cognitive symptoms.

Clinical Implications

• If your organization is considering AI on EHR text, start with narrower use cases where documentation is more standardized, such as medication extraction.

• Be cautious about using NLP outputs to support decisions on treatment response or complex presenting problems, where agreement and model performance were lower.

• Human annotation quality remains a limiting factor: Krippendorff α ranged from 0.665 to 0.967 for medication spans but only 0.325 to 0.579 for problem, therapy, and outcome spans.

Insights

• The development set included 697 manually annotated notes from 302 patients, drawn from a source cohort of 5,906 patients and 454,127 notes.

• Label-specific F1 ranged from 0.60 to 0.98, reinforcing that AI performance depends heavily on the exact clinical concept being extracted.

• Implementation was helped by multidisciplinary schema design and clear protocols, but slowed by annotation burden, variable documentation, data access delays, and computing constraints.

The Bottom Line

• For clinicians, the near-term effect of AI on practice is likely to be strongest in structuring medication information from notes, not in reliably interpreting outcomes or complex mental health presentations.

• This CAMHS study supports feasibility in research environments, but external validation and refinement are still needed before broader operational or clinical use.

Reference: Integrating Natural Language Processing in Child and Adolescent Mental Health Services Research Using Normalization Process Theory: Implementation Report