(PS10-A25) Evaluating the Feasibility and Acceptability of Behavioral Weight Loss Coaching Driven by a Large Language Model AI System

It is estimated that 75 million U.S. adults are overweight and seeking to lose weight. Previous research shows that extended contact with clinician-drafted text messages may attenuate weight regain, induce weight loss and promote weight maintenance. However, this approach requires clinician time and may be costly. Therefore, many studies have investigated fully automated rule-based systems with pre-drafted messages that require no clinician involvement. However, these messages can be repetitive and generic, which might be unhelpful in promoting effective behavioral changes. Compared to a conventional rule-based automated text messaging system, artificial intelligence (AI) conversational chatbots using large language models (LLM) (e.g., ChatGPT) may be superior due to their access to a larger body of information and a better understanding of natural language and context. Recent research has explored the potential for LLM AI to promote a healthy diet and exercise and demonstrated initial acceptability to individuals interested in increasing physical activity. However, whether the system can engage in coaching that is perceived as similarly helpful and effective to a human weight loss clinician is unknown. In the current study, 47 weight loss-seeking adults (BMI ≥ 27) rated the helpfulness of 10 coaching messages (on a 5-point scale), 5 of which were written by a Master’s-level human weight loss coach and 5 by an LLM AI (ChatGPT). Participants also indicated which messages they believed were written by an AI. Both human coach and AI messages were generated using real weight and behavioral data from participants enrolled in a behavioral weight-loss trial. Both the coach and the AI were then directed in plain English to write a short message summarizing the participant’s data pattern and suggesting strategies to address specific aspects of behavioral adherence (e.g., calorie intake and physical activities) that participants should improve. Results show that ratings of helpfulness for human-written messages (M = 3.64) were higher than AI-written ones (M = 2.91, t = -6.20, p < .001). However, 66% of the AI-written messages were rated as somewhat helpful to extremely helpful. Qualitatively, participants reported that AI-written messages had a less interpersonal tone. Nevertheless, 30% of the AI-written messages were misidentified as human-written, suggesting that many messages successfully imitated a human coach. The current study demonstrates the preliminary feasibility and acceptability of using an LLM AI to create potentially effective weight control coaching messages. Future work should investigate methods for improving message acceptability and testing the clinical effectiveness of AI-generated messages in controlled trials.