3 - (SYM 31) Supporting Universal Prevention Programs in Schools: Centering School Leaders' Expertise to Evaluate Usability of a Leadership-focused Implementation Strategy

Introduction: In the absence of stakeholder involvement in their design, school-based implementation strategies may be complex in ways that interfere with their widespread application to improve the use of evidence-based prevention practices. Usability – extent to which a product can be used by specified individuals to achieve specified goals in specified context – evaluation of implementation strategies is critical to ensure that they align with user needs, resources, expectations and contextual constraints.

Methods: The Cognitive Walkthrough for Implementation Strategies (CWIS) method was used to iteratively adapt the organizationally-focused strategy Leadership and Organizational Change for Implementation (LOCI) to the school setting to create Helping Educational Leaders Mobilize Evidence (HELM). This mixed methods approach to evaluating implementation strategy usability began with determining preconditions, hierarchically analyzing tasks, prioritizing tasks and converting tasks to scenarios. Facilitators conducted three group testing sessions, guiding elementary school principals through the intervention in a Leader (n=10; 4 scenarios and 13 subtasks) or Coach role (n=5; 3 scenarios and 8 subtasks). Users reported their anticipated success of completing each subtask and provided qualitative rationales, serving as data to identify usability issues using a highly structured common usability issue framework. Following sessions, users completed a quantitative assessment of strategy usability.

Results: To identify, classify and prioritize usability issues, users’ subtask ratings (scale 1-4) were calculated. Strategy usability ratings (scale 0-100), interpreted using worst imaginable/poor/ok/good/excellent/best imaginable categorizations, indicated that Leaders found the strategy usability to be excellent (M=77.8, SD=15.5) and Coaches found it to be the best imaginable (M=87.5, SD=7.9). Usability issues were identified as: (1) communication barrier in online training, (2) Coaches’ lack of knowledge about how specific school building climate impacts ability to advise, (3) inability to track goals and barriers in provided materials, (4) overwhelming volume of Feedback Report data and (5) Leaders’ struggle to interpret Feedback Report data.

Conclusion: This application of the CWIS methodology is expected to yield a more usable HELM strategy and offers more broadly applicable information surrounding the development of psychosocial implementation strategies in real-world settings, such as schools.