I Introduction

Fairness Methods for Cold Start problem (video)

A well-researched topic in recommender systems is the cold start problem, where new items are recommended prior to gathering sufficient information about them. Various exploration methods were proposed to collect such information, but they commonly focus on minimizing the overall regret of the system and are less attentive to the success of the new items. In multi-stakeholder systems (e.g display advertising) this could cause unsatisfactory results to some of the stakeholders. In this talk I will suggest a new perspective on the cold start problem by formulating it as a fairness problem - considering new items as a population that might suffer from algorithmic bias. I will share our experience with using standard fairness measures and methods for correcting this bias in order to help new advertisers in Taboola’s display ads system.

Inbar Naor is an Algorithms Team Lead at Taboola, where she works on content and ads recommendations. Her main interests are in Machine Learning for Recommender Systems. Most recently, she focused on exploration and fairness in the domain of display ads.

Conversational Recommendation with Natural Language

Conversational interaction with recommender systems has become increasingly common, yet the language of the interaction is often still constrained. In this talk, I will describe recent work aimed at increasing the sophistication and naturalness of conversations with state of the art recommenders. Considering three aspects of conversational recommendation, I will describe methods improving the vocabulary understood by systems, as well as the reliability of interpretation of users’ answers to questions asked by a system. Further, I will show how a system employing richer natural language can provide users more control over, and improved understanding of the recommendation process.

Filip Radlinski is a Research Scientist at Google, UK. His research focuses on improvements to conversational search and recommendation through better understanding and modeling user interests through natural language, improved transparency of conversational systems, as well as human-centered evaluation and personalization of information retrieval and recommendation tasks. He received his PhD from Cornell University.

How Recommendation System Feedback Loops Disproportionately Hurt Users with Minority Preferences (video)

Algorithmic recommendation systems impact the choices of millions of consumers daily; these systems exist for a wide variety of markets, including both consumable and durable goods, as well as digital and physical goods. After a recommendation system is in place, it will need to be periodically updated to incorporate new users, new items, and new observed interactions between users and items. These observed data, however, are algorithmically confounded: they are the result of a feedback loop between human choices and the existing algorithmic recommendation system. Using simulations, we explore the impact of updating a recommendation system. We find that the choices surrounding system updates have the greatest impact on users belonging to minority preference segments.

Allison Chaney is an Assistant Professor of Marketing in the Fuqua School of Business at Duke University. Her research is at the intersection of machine learning and marketing, focusing on developing scalable and interpretable machine learning methods and understanding the impacts of these methods on individuals and society when they are deployed in real-world markets. She received her Ph.D. in Computer Science at Princeton University and holds a B.A. in Computer Science and a B.S. in Engineering from Swarthmore College. She has worked for Pixar Animation Studios and the Yorba Foundation for open-source software and has collaborated with many research teams in industry including eBay/Hunch, Etsy, and Microsoft Research.

1. Aggregated Customer Engagement Model, Priya Gupta and Cuize Han, Amazon

2. Embedding-based Retrieval with Elasticsearch, Div Dasani, Discovery

3. User Taste-Aware Image Search, Jiyun Luo, Pak Ming Cheung, Wenyu Huo, Ying Huang, and Rajat Raina, Pinterest

4. Language Agnostic Representations for e-commerce Product Search, Nikhil Rao and Karthik Subbian, Amazon

5. Multitask learning for Recommendation Systems, Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, Ed Chi, Google

6. Modeling Dynamic Attributes for Next Basket Recommendation, Yongjun Chen, Jia Li, Chenghao Liu, Chenxi Li, Markus Anderle, Julian McAuley and Caiming Xiong, Salesforce

7. Contrastive Self-supervised Sequential Recommendation with Robust Augmentations, Yongjun Chen, Zhiwei Liu, Jia Li, Julian McAuley and Caiming Xiong, Salesforce

8. Shaping Recommendations in Marketplaces via User & Content Understanding, Rishabh Mehrotra, Spotify

9. Shifting Consumption towards Diverse content via Reinforcement Learning, Christian Hansen, Rishabh Mehrotra, Casper Hansen, Brian Brost, Lucas Maystre and Mounia Lalmas, Spotify

10. Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer, Zhiwei Liu, Ziwei Fan, Yu Wang, Philip S. Yu, Salesforce

11. Deep Retrieval: An End-to-End Learnable Structure Model for Large-Scale Recommendations, Weihao Gao, Xiangjun Fan, Chong Wang, Jiankai Sun, Kai Jia, Wenzi Xiao, Ruofan Ding, Xingyan Bin, Hui Yang and Xiaobing Liu, Bytedance

12. Bias-Variance Decomposition For Ranking, Pannaga Shivaswamy and Ashok Chandrashekar, Netflix

13. COFFEE : Completeness-Constrained Faithful Explanations, Ehtsham Elahi, Claudia Roberts and Ashok Chandrashekar, Netflix

14. What Users Want? Generative Models for Recommendation, Flavian Vasile, Jules Samaran, Ugo Tanielian and Romain Beaumont, Criteo

15. Recommendation with Pricing Incentives, Flavian Vasile, Benjamin Heyman, Amine Benhalloum and David Rohde, Criteo

Fair Recommendations with Biased Data (video)

Search engines and recommender systems have become the dominant matchmaker for a wide range of human endeavors -- from online retail to finding romantic partners. Consequently, they carry substantial power in shaping markets and allocating opportunity to the participants. In this talk, I will discuss how the machine learning algorithms underlying these system can produce unfair ranking policies for both exogenous and endogenous reasons. Exogenous reasons often manifest themselves as biases in the training data, which then get reflected in the learned ranking policy and lead to rich-get-richer dynamics. But even when trained with unbiased data, reasons endogenous to the algorithms can lead to unfair or undesirable allocation of opportunity. To overcome these challenges, I will present new machine learning algorithms that directly address both endogenous and exogenous unfairness.

Thorsten Joachims is a Professor in the Department of Computer Science and in the Department of Information Science at Cornell University, and he is an Amazon Scholar. His research interests center on a synthesis of theory and system building in machine learning, with applications in information access, language technology, and recommendation. His past research focused on counterfactual and causal inference, learning to rank, structured output prediction, support vector machines, text classification, learning with preferences, and learning from implicit feedback. He is an ACM Fellow, AAAI Fellow, KDD Innovations Award recipient, and member of the SIGIR Academy.

Messaging at Netflix: Causal Modeling in Practice

At Netflix, we message our members about relevant shows and movies through a wide array of messages via channels such as email, push notifications, and within-product notifications. We strive to reach our members with the right message at the right time to help them find something great to watch,  whilst weighing the negative impacts of messaging fatigue and opt-out. We achieve this through causal machine learning algorithms that are supported by carefully designed engineering services. In this talk, we will motivate building such a system and cover many of the practical challenges in designing it and how we overcame them. We will discuss our motivation behind causal personalization algorithms, how we balance the positive impact a great recommendation sent at the perfect time can have, and the annoyance an extra message can lead to, how we select models for causal messaging policies, and discuss some of the engineering challenges in training, selecting models, deploying, and maintaining such a system.

Mariak Dimakopoulou is a Senior Research Scientist at Netflix and her research focuses on bandit learning, reinforcement learning and causal inference. She leads the Adaptive Experimentation research agenda to improve A/B testing at Netflix and she collaborates closely with personalization teams on causal machine learning methods that improve personalized recommendations. Prior to that, she worked at Google Research and she obtained her PhD from Stanford.

Christopher Alvino is a Senior Machine Learning Researcher. At Netflix, he has worked on row ordering, evidence and image personalization, and member messaging. He is currently primarily interested in applied causal inference. Prior to Netflix, Chris worked in computer vision applied to medical imaging and security imaging. Chris holds an engineering Ph.D. from Georgia Tech and engineering degrees from Rutgers University, New Brunswick.

Devesh Parekh is a Senior Research Engineer who has worked on marketing, signup optimization, and messaging at Netflix applying causal inference and bandit learning.

Counterfactuals and Offline Reinforcement Learning (video)

There is a huge opportunity for enhancing evidence-driven decision making by leveraging the increasing amount of data collected about decisions made, and their outcomes. Reinforcement learning is a natural framework to capture this setting, but online reinforcement learning may always be feasible for higher stakes domains like healthcare or education. In this talk I will discuss our work on offline, batch reinforcement learning, and the progress we have made in techniques that can work efficiently with limited data, and under limited assumptions about the domain.

Emma Brunskill is an associate professor in the Computer Science Department at Stanford University. Her goal is to create AI systems that learn from few samples to robustly make good decisions, motivated by applications to healthcare and education. Her lab is part of the Stanford AI Lab, the Stanford Statistical ML group, and AI Safety @Stanford. She was previously an assistant professor at Carnegie Mellon University. Her and her group's work has been honored by early faculty career awards (National Science Foundation, Office of Naval Research, Microsoft Research (1 of 7 worldwide)) and several best research paper nominations (CHI, EDMx3) and awards (UAI, RLDM).

Near real time systems to build active communities on LinkedIn

At LinkedIn our mission is to use AI to connect every member of the global workforce to make them more productive and successful. The social network is the backbone for professionals to engage with each other at every stage of their career. There are millions of professionally relevant conversations taking place on LinkedIn every second. Notifications are how we help our members find out about these conversations or how they can stay connected to the people they know, and the topics they care about. With so much activity across the platform, our focus is to notify the right members, about the right content, at the right time, with the right frequency, and through the right channel.  In this talk we will give an overview of the notifications relevance architecture that processes a stream of activity on the social network. We will present the problems of volume optimization, channel selection and delivery time optimization that make notifications relevance different from standard ranking and recommendations problems. We will go deeper into the objective function, and present it as a multi-objective optimization problem and talk about the AI technology we use to power notifications at LinkedIn.

Hema Raghavan heads the team that builds AI solutions for fueling the professional Linkedin’s growth. Her team is responsible for all of the intelligent decision making systems that enable a member to find the right network and be heard. These systems include People You May Know recommendations and AI for Linkedin's Air Traffic Controller (ATC) which controls all of the notifications and email that members receive. Prior to that, she was a Research Staff Member at IBM T.J Watson and Yahoo Labs. Her interests span the broad area of applications of AI and her experience spans a spectrum of products she has built in the areas of Search, Advertising, Question Answering and Recommendations. She has published in several conferences like WWW, SIGIR, ACL and COLING and serves on several program committees.

See 10.50 AM session for full poster list

Off-policy ranking for personalized recommendation

A common challenge faced by streaming media services is recommending personalized content to present to their users from a collection of potentially millions of items. Online contextual bandits that learn from implicit user feedback (e.g., clicks) are useful for this task because they can adapt to user behavior in real time. However, repeated online experiments are expensive and can potentially degrade the user experience during the early phases of learning. This talk will discuss the advantages and challenges of supplementing online experimentation with off-policy learning and evaluation using logged user behavioral data. An important consideration is how to account for biases present in logged data, both selection bias caused by the choices made by the logging policy and presentation bias incurred by the visual layout of recommendations.

Kat Ellis is a Senior Applied Scientist at Amazon Music. Her work focuses on optimizing personalized music recommendations using learning to rank methods, deep learning, and offline experimentation. She did her Ph.D. study at the University of California, San Diego, where she researched physical activity recognition from wearable sensor data and audio classification of musical instruments and genres. She received her B.S. in Electrical Engineering from the University of Southern California.

Accordion: a Trainable Simulator for Long-Term Interactive Systems

As machine learning methods are increasingly used in interactive systems it becomes common for the user experience to be the result of an ecosystem of machine learning systems in aggregate. Simulation offers a way to deal with the resulting complexity by approximating the real system in a tractable and interpretable manner, but existing methods do not fully incorporate the interaction between user history and subsequent visits. We develop Accordion, a trainable simulator based on Poisson processes that can model visit patterns to an interactive system over time from large-scale data. New methods for training and simulation are developed and tested on two datasets of real world interactive systems. Accordion shows greater sensitivity to hyperparameter tuning and offline A/B testing than comparison methods, an important step in building realistic task-oriented simulators for recommender systems.

James McInerney is a senior research scientist at Netflix focusing on Bayesian modeling and inference with applications to recommendation and experimentation. He was previously a researcher at Spotify and has held postdoctoral positions at Columbia University and Princeton University. He has a PhD in Computer Science from the University of Southampton, an MSc in Artificial Intelligence from Imperial College London, and a BA in Computer Science from Oxford University.

Come join us in gather.town for discussions & networking!