On the finite-sample and asymptotic validity of an allocation-probability test for adaptively-collected data (Invited Talk @ StaTalk2023)

Abstract

Response-adaptive designs, either based on simple rules, urn models, or bandit problems, are of increasing interest among both theoretical and practical communities. In particular, regret-optimising bandit algorithms like Thompson sampling hold the great promise to optimise an experimental design by limiting the exposure to inferior arms and enhancing participants’ outcomes. However, their practical potential in biomedical settings is yet to be realised. In fact, the analysis of adaptively-collected data has exposed the inadequacy of traditional statistics for robust and reliable conclusions, with considerable issues in hypothesis testing. Attempts to develop valid testing procedures are predominantly focused on type-I error control, are very inefficient in terms of power–especially in small samples–and their applicability is limited to specific design characteristics. This work addresses this gap by illustrating an alternative approach to statistical testing that can be in principle applied to any randomised algorithm for which the randomisation probabilities are available. Finite-sample and asymptotic properties of this solution are discussed both in a general setting and under the specific Thompson sampling design. The empirical advantages of the proposed approach–able to preserve regret optimalities, while offering inferential benefits–are finally illustrated in empirical examples.