(PS1-D94) Investigating the Concurrent Validity of the School Functioning Survey with and Without Aggregation of Data

Teacher rating scales are an important part of the identification of impairment for students in secondary school. The currently available measures fail to adequately address the challenges and obstacles inherent to measurement in a secondary school setting, resulting in a lack of strong empirical evidence of their psychometric soundness, possibly due to the methods used to evaluate concurrent validity such as aggregation of data across multiple teachers. Recent publications have highlighted that aggregating data to resolve informant discrepancies can lose the often valuable and unique information that informant discrepancies provide (De Los Reyes et al., 2023). The School Functioning Survey (SFS) was designed to be used in a secondary school setting and attempted to address those obstacles and challenges. However, the SFS did not show expected levels of treatment sensitivity following a recent treatment study. This suggests that there are other factors underlying the difficulty in obtaining accurate and useful information about the school functioning of adolescents. In this study, we evaluated the concurrent validity of the SFS by comparing SFS teacher ratings to teacher-assigned grades from the previously mentioned treatment study without aggregation to address one possible explanation of why a teacher rating scale, tailored to the secondary school setting, failed to reflect grade-related change.

Our sample consisted of 121 students from Ohio and Pennsylvania in the 9^th, 10^th, or 11^th grades with ADHD. Teachers completed the School Functioning Survey at the mid-point and end of year. Transcript grades were collected simultaneously.

Results from the correlation analyses and subsequent z-tests show that the School Functioning Survey demonstrated significantly greater concurrent validity with class grades when evaluated within each teacher as opposed to when grades were aggregated across teachers. Further analyses indicated that within the non-aggregated sample, the ratings provided by teachers of certain subjects correlated significantly higher with class-assigned grades than the ratings provided by other teachers. These results suggest that the widely used method of aggregating teacher report data in secondary school treatment outcome studies may interfere with obtaining valid indices of changes that may occur within some of the classrooms. Further research should be conducted to understand if the aggregation effect can be replicated in other measures of school functioning in adolescents.