Using reinforcement learning to design just-in-time adaptive interventions. The DIAMANTE pilot study (Invited Talk @ Young Researchers Meeting, Bernoulli-IMS2021)

Abstract

The increasing progress in mobile technologies has led to a recent surge of interest among both researchers and practitioners in developing just-in-time adaptive interventions (JITAIs). Application domains include, for example, education and mobile health (mHealth). Here, JITAIs represent special types of sequences of decision rules that prespecify when and which components of instructional or behavioral interventions, respectively, should be delivered, and how these should be adjusted in real-time, based on individuals’ personal progress and time-varying preferences and/or needs. A major methodological challenge for developing effective JITAIs arise in its design, particularly in: (i) the choice of the adaptive algorithm for optimising and personalising an intervention delivery in real time, and (ii) necessary adjustments of the adaptive strategy to target the real-world setting it has to operate in. Reinforcement learning (RL), and more specifically, multi-armed bandit (MAB) techniques, hold great promise for solving such sequential decisionmaking problems. Their underlying idea is indeed based on a continuous interaction between an agent (e.g., a mobile app) and an unknown environment (e.g., a user), in which the agent has to take and adjust its actions (e.g., deliver text messages), in order to maximise a cumulative reward (i.e., an outcome of interest) over time. In this work we illustrate a mHealth setting where this framework can be successfully applied, namely for promoting healthy (in our case, physically active) behaviors among users. We discuss our real-world implementation (i.e., the micro-randomised DIAMANTE pilot trial), providing: (1) evidence of existing effects of several active intervention components (i.e., motivational text messages) on improving physical activity, and (2) challenges encountered, lessons learned and guidelines in designing JITAIs. We also describe how knowledge about the mHealth domain guided our choice and design of the adaptive MAB algorithm for optimising the JITAI.