Longitudinal Impact of Preference Biases on Recommender Systems' Performance
COM1 Level 2
SR3, COM1-02-12
Abstract:
Research studies have shown that recommender systems' predictions that are observed by users can cause biases in users' post-consumption preference ratings. Because users' preference ratings are typically fed back to the system as training data for future predictions, this process is likely to influence the performance of the system in the long run. We use a simulation approach to study the longitudinal impact of preference biases (and their magnitude) on the dynamics of recommender systems' performance. We look at the influence of preference biases in two conditions: (i) during the normal system use, where biases are typically caused by the system's inherent prediction errors, and (ii) in the presence of external (deliberate) recommendation perturbations. Our simulation results show that preference biases significantly impair the system's prediction performance (i.e., prediction accuracy) as well as users' consumption outcomes (i.e., consumption relevance and diversity) over time. The impact is non-linear to the size of the bias, i.e., large bias causes disproportionately large negative effects. Also, items that are less popular and less distinctive (in terms of their content) are affected more by preference biases. Additionally, intentional recommendation perturbations, even on a small number of items for a short time, substantially amplify the negative impact of preference bias on a system's longitudinal dynamics and cause long-lasting effects on users' consumption. Finally, given the impact of preference bias on the recommender systems' performance, we explore the problem of debiasing user-submitted ratings. We empirically demonstrate that relying solely on historical rating data is unlikely to be effective in debiasing. We also propose and evaluate two debiasing approaches that take into account additional relevant information that can be collected by recommendation platforms. Our findings provide important implications for the design of recommender systems.
Bio:
Prof. Jingjing Zhang is an Associate Professor of Information Systems and holds the Judith Norman Davis and Kim G. Davis Professorship of Business Analytics at the Kelley School of Business, Indiana University. She earned her Ph.D. in Business Administration with a concentration in Information Systems from the University of Minnesota in 2012. Her research interests include personalization techniques, recommender systems, human-computer interactions, mobile app marketplace, and emerging digital platforms. Jingjing's work has been published in leading academic journals such as MIS Quarterly, Information Systems Research, INFORMS Journal on Computing, IEEE Transactions on Knowledge and Data Engineering, ACM Transactions on Information Systems, and ACM Transactions on Computer-Human Interaction. Her research contribution has been recognized with numerous awards, including the 2020 ISR Best Published Paper Award, the 2023 Women in RecSys Journal Paper of the Year Award, the INFORMS ISS Sandra A. Slaughter Early Career Award, the Nunamaker-Chen Dissertation Award, and multiple Best Paper accolades at academic conferences. Currently, Jingjing serves as an Associate Editor for Information Systems Research. She has previously served as a Guest Associate Editor for esteemed journals such as MIS Quarterly, Information Systems Research, Decision Support Systems, and Journal of the Association for Information Systems. Her editorial contribution was acknowledged with the AE of the Year Award from ISR in 2022 and the Best AE Award at ICIS in 2019. Furthermore, Jingjing is an AIS Distinguished Member and has played a major role in co-organizing many major academic conferences. These include the Workshop on Information Technologies and Systems in 2022, 2020, and 2016, the INFORMS Workshop on Data Science in both 2022 and 2021, and the Americas Conference on Information Systems in 2022.