Graduate student University of Pennsylvania Merion Station, Pennsylvania
Introduction: Complex algorithms (i.e., machine-learning) have renewed interest in clinical prediction models. They have outperformed simple rule-based algorithms such as linear regression models with pretreatment severity as a predictor and even a linear regression model with predictors identified in the extant literature (Kim et al., 2019). While these algorithms present potential in predicting a patient’s prognosis, the models created by these algorithms have primarily only been evaluated on the samples that they had been built on, or the development sample. The predictive accuracy of models built from complex algorithms could be worse in forecasting outcomes of patients receiving treatment at a different setting. The aim of the current study is to assess the ability of a model to predict outcomes for patients who received treatment at several Improving Access to Psychological Therapies (IAPT) services across the United Kingdom. The second aim of the current study is to compare the performance of the model that was developed from the IAPT site used in a data tournament (SMART) with a model developed from another IAPT site. The performances were compared at each model’s respective IAPT site, except the IAPT site from the SMART tournament. This aim attempts to address whether a model developed from site-specific data can outperform a model built from another site.
Methods: IAPT services are part of the United Kingdom’s National Health Service (NHS). Each service delivers systematic treatment for depression and anxiety and collects a standard set of predictor and outcome variables (Clark, 2018). Representatives from multiple IAPT services formed a Practice Research Network (PRN). Data from the SMART tournament and the other five IAPT sites from the PRN were used to answer the aims of the current study.
Results: The predictions from the model did not outperform those from the Simple severity model (a simple logistic regression with pretreatment severity as the independent variable) at Site 4 (p=0.22); however, at all the other sites, they did outperform those from the Simple severity model at the adjusted significance level (p=0.01). The models built from their respective sites did not outperform the model built from the SMART tournament.
