Dr. Jia (Kevin) Liu, 'Incentivized Bandit Learning under Self-Reinforcing User Preference for Online Recommender Systems', TADS Lunch-n-Learn

Incentivized Bandit Learning under Self-Reinforcing User Preference for Online Recommender Systems

Abstract: 

In this research, we investigate a new multi-armed bandit (MAB) online learning model that considers two real-world phenomena that exist many online recommender systems: (i) the learning agent cannot pull the arms by itself and thus has to offer payments to users to incentivize arm-pulling indirectly; and (ii) if users with specific arm preferences are well rewarded, they induce a “self-reinforcing” effect in the sense that they will attract more users of similar arm preferences. Besides addressing the conventional tradeoff between exploration and exploitation, another key feature of this new MAB model is to balance reward and incentivizing payment. The goal of the agent is to minimize the accumulative regret over a fixed time horizon with a low total payment. Our contributions in this research are two-fold: (i) We propose a new MAB model with random arm selection that considers the relationship of users' self-reinforcing preferences and incentives; and (ii) We leverage the properties of a multi-color Polya urn with nonlinear feedback models to propose two MAB policies termed “At-Least-n Explore-Then-Commit” and “UCB-List.” We prove that both policies achieve an expected regret that grows logarithmically in the time horizon duration, with an expected payment that also grows logarithmically in the time horizon duration. Our numerical experiments demonstrate and verify the performances of these two policies and show their robustness under various settings. Finally, we will discuss several interesting future directions of this research, including the impacts of feedback delay, impacts of limited arm presentation budget, and fairness assurance in user diversity and arm exposure, all of which are highly relevant to online recommender systems.

Bio: 

Jia (Kevin) Liu is an Assistant Professor in the Dept. of Electrical and Computer Engineering at The Ohio State University, where he joined in Aug. 2020. He has also been an Amazon Visiting Academics (AVA) since Nov. 2021. He received his Ph.D. degree from the Dept. of Electrical and Computer Engineering at Virginia Tech in 2010. From Aug. 2017 to Aug. 2020, he was an Assistant Professor in the Dept. of Computer Science at Iowa State University. His research areas include theoretical machine learning, control and optimization for stochastic networks, and optimization for data analytics infrastructure and cyber-physical systems. Dr. Liu is a senior member of IEEE and a member of ACM. He has received numerous awards at top venues, including IEEE INFOCOM'19 Best Paper Award, IEEE INFOCOM'16 Best Paper Award, IEEE INFOCOM'13 Best Paper Runner-up Award, IEEE INFOCOM'11 Best Paper Runner-up Award, and IEEE ICC'08 Best Paper Award. Dr. Liu is a recipient of the NSF CAREER Award in 2020. He is a recipient of the Google Faculty Research Award in 2020. He is also a winner of the LAS Award for Early Achievement in Research from the College of Liberal Arts and Sciences at Iowa State University in 2020, and the Bell Labs President Gold Award in 2001. His research is supported by NSF, AFOSR, AFRL, and ONR.