(PS10-A11) Advancing Depression Assessment: Machine Learning Model Predicts Self-report and Aligns with Clinician Judgment When Discrepant

Intro: Asking people to fill out “just one more form” can exacerbate existing inequities in mental health care screening. For example, patients with low literacy report having to fill out forms as a barrier to receiving mental health care. Shifting the form-filling responsibilities to clinicians is often not a practical solution. Clinicians in settings where screening for mental health difficulties is critical (e.g., primary care) already report spending most of their time with patients filling out forms. Developing a brief machine learning (ML) screening tool that relies on natural language conversation could help decrease form burden and corresponding inequities in screening.

Methods: The training data consisted of a demographically diverse sample of 1324 English-speaking USA residents recruited from social media ads: 66.9% women, 29.7% men, and 3.4% other gender between the ages of 12 and 86 years old (median age = 34, IQR = 26-46); 60.3% European American or White, 17.3% African American or Black, 15.4% Asian, 1.4% American Indian or Alaska Native, and 1.3% Native Hawaiian or Other Pacific Islander. Participants completed a brief (~ 5 min) interview over Zoom in which they were asked open-ended analogues of depression scale items. For example, participants were asked, “What has your mood been like the past couple of weeks?” without being explicitly directed to talk about how often they felt down, depressed, or hopeless. Following the interview, participants completed the Quick Inventory of Depressive Symptomatology (QIDS SR-16). Recordings of interviews were processed using natural language and speech processing algorithms to automatically transcribe and semantically match question-answer pairs to specific symptom domains. For each symptom-relevant utterance, a set of sentiment and paralinguistic features was extracted and used to train a ML ensemble to predict the corresponding QIDS item response. The hold-out predictions from the individual symptom models using 10-fold cross-validation were then used as inputs to train a meta-model using elastic-net regression to predict overall depression severity on the QIDS-SR-16 scale, and validated on a subsample (N = 367) with minimal or no (N = 87), mild (N = 104), moderate (N = 111), or severe (N = 65) depression.

Results: The model predicted a severity level that matched to within one QIDS severity level for 92.1% of cases. For the remaining 7.9% (N = 29), videos of the interviews were reviewed by a clinician (MM) who, in a masked forced-choice paradigm, selected which severity rating was more likely. The clinician agreed more with the model prediction over the QIDS for 25/29 cases (86.2%).

Conclusions: ML could infer self-reported depression with high fidelity, and its largest “errors” were in fact highly consistent with clinician opinion, suggesting a natural-language screener can provide an incremental improvement in accuracy over standard self-report instruments. Further integration of this screener into a self-administered framework would help realize its potential for reducing form burden and corresponding inequities