(PS10-A14) Evaluating the Potential of Conversational Artificial Intelligence to Change the Behavioral Health Training Game

Background: Conventional models of professional development have not been successful in scaling or sustaining Evidence-Based Psychotherapeutic Interventions (EBPIs) in real-world settings--particularly for individuals with Serious Mental Illness (SMI). CBTpro is an integrated training and quality assurance platform that leverages conversational Artificial Intelligence to provide automated performance-based feedback on cognitive behavioral techniques for psychosis (CBTp). CBTpro uses a deliberate practice framework to provide learners with multiple opportunities to practice CBTp on simulated patients presenting psychotic and mood symptoms. AI-enabled practice with feedback can be accessed continually to train and evaluate CBT practitioners.

Methods: We conducted a single arm 2-week field trial among mental health practitioners who treat outpatients with psychosis. Ten CBTp-naive and 11 CBTp-trained practitioners were asked to use CBTpro over the 2-week period. Use patterns were observed naturalistically. We assessed feasibility [Feasibility of Intervention Measure (FIM)], acceptability [Acceptability of Intervention Measure (AIM)], appropriateness [Intervention Appropriateness Measure (IAM)], perceived usability [System Usability Survey (SUS)], and overall satisfaction [Net Promoter Score (NPS)].

Results: On average, 58.11% of all participants engaged with the tool on a daily basis; participants averaged 6.57 logins per participant (SD=4.24). Most participants (44.7%) engaged for 15–30-minute increments. SUS scores exceeded industry-standard by 7 points [M(SD)= 74.88(11.47)]. The total NPS score was 28.57, indicating good promotion of the tool. Average scores for each implementation measure were all over 4 points on a 5-point Likert scale: AIM (M(SD) = 4.07(0.59), IAM (M(SD)=4.18(0.61), and FIM (M(SD) = 4.08(0.57).

Discussion: CBTpro was used and found to be feasible, acceptable, and appropriate among practitioners in a community mental health setting. CBTpro is a highly usable AI training tool that can support CBTp training efforts and enhance access to CBT-informed care.