(PS5-C61) Cultivating a Vignette-based Knowledge Measure for Practical Use in a Public Mental Health Service System

Research on mental health science and practice has gradually developed more formal and explicit specifications of the concepts of treatment targets (TTs) and practice elements (PEs; e.g., CAMHD, 2019; Hogue et al., 2019). However, the extent to which therapists can accurately identify and demonstrate knowledge of these metrics in day-to-day clinical practice remains an open question. Practical, reliable, and validated knowledge measures are needed to inform large scale system and individual level training challenges with the goal of increased understanding and application of a shared ontology (NASEM, 2022). Along these lines, vignette-based knowledge measures (Miller, 1990; Muse & McManus, 2013) are less time intensive than observational methods and are highly applicable to actual clinical practice (Hill, 2020). As one example, the Treatment Targets and Practice Elements Reliability Check (TTPERC) is a vignette-based knowledge measure developed to assess therapists’ practical understanding of TTs and PEs (Carreira Ching et al., 2022). The aim of the current study is to develop and test two, short form versions of the original 50-item measure to support implementation within a state mental health system of care.

Participants in this study were from two pilot sites: (1) undergraduate- and graduate-level psychology students and faculty situated in a local, university setting (N = 13) and (2) public mental health psychologists and psychiatrists (N = 20). The overall sample was primarily female (72.7%), multiethnic (42.4%), and had an average age of 33 (SD = 10.57). The initial TTPERC consisted of 50 vignettes describing a typical treatment session for a youth and/or caregiver within the service system. Following each vignette, respondents were asked to select the TT and PE described. Initial content validity ratings (Carreira Ching et al., 2022) were used to adjust vignettes for improved clarity. To reduce completion time and allow the test to be retaken if needed, the initial 50-vignette TTPERC was split into two 25-vignette, short form versions. Forms were balanced on client demographic factors (e.g., age, diagnosis, gender), and PE and TT response options, including distractors. Participants at each site were randomly assigned to receive form A (N = 15) or form B (N = 18). Results indicated, across both forms, participants received a mean score of 44.72 (SD = 2.08; range = 40 to 48) out of 50 possible points. Further, the average time to complete the check across forms was 30.35 minutes (SD = 12.16; range = 10.07 to 63.33). Findings from this study, conducted with scholars and on-the-ground change agents within a public mental health context, have implications for informing therapist trainings and improving state credentialing practices. Such efforts further ontologies within mental health to support efficient knowledge generation and overcome large-scale challenges within systems of care.