NIPS '17 Workshop: Criteo Ad Placement Challenge

Welcome to the Criteo Ad Placement challenge, accompanying the NIPS’17 Workshop on Causal Inference and Machine Learning.

Consider a display advertising scenario: a user arrives at a website where we have an advertisement slot (“an impression”). We have a pool of potential products to advertise during that impression (“a candidate set”). A “policy” chooses which product to display so as to maximize the number of clicks by the users.

The goal of the challenge is to find a good policy that, knowing the candidate set and impression context, chooses products to display such that the aggregate click-through-rate (“CTR”) from deploying the policy is maximized.

To enable you to find a good policy, Criteo has generously donated several gigabytes of <user impression, candidate set, selected product, click/no-click> logs from a randomized version of their in-production policy! Go to the Dataset section of the challenge to access the dataset and visit the GitHub repo to access a starter-kit for counterfactual policy learning with the dataset.

Prizes worth $4500 sponsored by Criteo!

This policy learning challenge is subtly different from the 2014 Criteo challenge to compare CTR prediction algorithms. CTR prediction is a standard supervised regression problem: for any ad in the training data, the correct regression target is given (click/no-click). For policy learning, when the in-production policy chose a product that was not clicked, we do not know “what-if” (counterfactual) we chose a different product to display. These differences and some baseline approaches are detailed in the dataset companion paper.

The objective of this challenge is to spur participants to explore several issues on a real-world benchmark dataset, and share their findings during the NIPS’17 workshop, such as:

• Discover new training objectives, learning algorithms and regularization mechanisms for counterfactual learning scenarios.
• Find appropriate ways to perform model selection (analogous to cross-validation for supervised learning) using large amounts of logged interaction data.
• Develop algorithms that can scale to massive datasets (typically orders of magnitude larger than labeled datasets).

Evaluation criteria

In the dataset, each impression is represented by M lines where M is the number of candidate ads. Each line has feature information for every other candidate ad. In addition, the first line corresponds to the candidate that was displayed by the logging policy, an indicator whether the displayed product was clicked by the user (“click” encoded as 0.001, “no-click” encoded as 0.999), and the inverse propensity of the stochastic logging policy to pick that specific candidate (see the companion paper . for details). Each <user context-candidate product> pair is described using 33 categorical (multi-set) features and 2 numeric features. Of these, 10 features are only-context-dependent while the remaining 25 features depend on both the context and the candidate product. These categorical feature representations have been post-processed to a 74000-dimensional vector with sparse one-hot encoding for each categorical feature. The semantics behind the features will not be disclosed.

These post-processed dataset files are available here .

Your task is to build a function _policy which takes M candidates, each represented by a 74000-dimensional sparse vector, and outputs scores for each of the candidates between 1 and M.

The reward for an individual impression is, did the selected candidate get clicked (reward = 1) or not (reward = 0)? The reward for function f is the aggregate reward over all impressions on a held out test set of impressions.

We will be using an unbiased estimate of the aggregate reward using inverse propensity scoring (see the companion paper for details).

For further details on evaluation please refer to the Getting Started guide in the challenge starter kit.

Resources

Contact:

• Technical issues : https://gitter.im/crowdAI/NIPS17-Criteo-Ad-Placement-Challenge
• Discussion Forum : https://www.crowdai.org/challenges/nips-17-workshop-criteo-ad-placement-challenge/topics

We strongly encourage you to use the public channels mentioned above for communications between the participants and the organizers. In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at :

• Adith Swaminathan adswamin@microsoft.com
• Sharada Prasanna Mohanty sharada.mohanty@epfl.ch

Prizes

• 1st - $2000
• 2nd - $1500
• 3rd - $1000

Additionally:

• Invitation to NIPS’17 Causal Inference and Machine Learning Workshop

• Invitation to the 2nd Applied Machine Learning Days at EPFL in Switzerland on January 29 & 30, 2018.

Datasets License