There are so many distinct odors in everything we see or interact with. Our reactions to different smells are almost always instant and instinctual, not cultivated. A particular smell can sometimes trigger a specific memory too. Still, most of us would not know how our brain categorizes different smells from different sensory inputs.
What happens when particles responsible for smell enter our nose?
Our noses have more than 400 types of olfactory receptors expressed in 1 million+ olfactory sensory neurons, which are all on a small tissue - olfactory epithelium. The olfactory sensory neurons send signals to the olfactory bulb in the brain and then to more structures from there, to understand the smell.
We are turning this process digital!
In our noses, what finally goes in are particles that have odorant molecules responsible for the smell. These molecules are the actual building blocks of all fragrances. For this challenge, we take these molecular compounds as an input, parse them through, and predict what multitude of fragrances they contain out of 100+ different ones.
The dataset contains the description of molecules (as its SMILES string), and the odors it possesses. The challenge is a multiclassification problem, each molecule has multiple odors written in a form of a sentence with a single
, between each odor. Following are the columns in the dataset with their description:
SMILES: Simplified molecular-input line-entry system (SMILES) is a specification in the form of a line notation for describing the structure of chemical species using short ASCII strings.
SENTENCE (target): Its a combination of the odors of the molecules. Each odor is separated by a
,to form an (odor) sentence.
Following files are available in the
4316molecules) : This csv file contains the attributes describing the molecules along with their "Sentence" .
1079molecules) (Round-1) : File that will be used for actual evaluation for the leaderboard score but does not have the "Sentence" for molecules.
vocabulary.txt: A file containing the list of all odors present in the dataset
🖊 Evaluation Criteria
The evaluation of the submissions is done using the Jaccard Index / Tanimoto Similarity Score.
Description of odour can be heteregenous based on personal experience, perfumer, company, so it is hard to expect to get an unique and perfect description. In this case, we can evaluate the best sentence matching in proposed Top 5 sentences.
For example, if for a single molecule, the ground truth is :
rose and the top-5 proposed sentences are :
Then the Jaccard Index is computed for all the top-5 sentences in comparison to the ground truth, and the best score across all the 5 predictions is considered for the said molecule. When comparison the individual sentences with the ground-truth sentences, only the first 3 words from the ground truth sentence are considered.
The overall score is computed by taking the mean of the said score across all the molecules in the test set.
Round-2 Evaluation Criteria
For Round 2, you can choose a subset of the whole vocabulary(composed of 109 smell words) and create your own - if you believe that it improves your accuracy.
Read on to understand how it works 👇
Lets define :
voc_gt: (the ground truth vocabulary) as the set of smell words in the actual challenge dataset (ground truth).
109distinct smell words as present in the training set and test set of Round-1.
voc_x: (submission vocabulary) as a subset of
voc_gt, on which participants choose to train their models on, and sample their predictions from.
voc_xhas to be composed of atleast 60 distinct smell words. This is estimated as the set of all distinct smell words used across all the predictions made by the model.
model_compression: We define the model compression as :
1 - [len(voc_x) / len(voc_gt)].
- For every
1%model compression, we expect to have an improvement in accuracy of atleast
top_2_TSS_voc_x: This refers to the
top_2_TSScomputed using the vocabulary used by the participants. When computing this metric, any smell word which is not present in
voc_xis removed from the ground truth sentences.
top_5_TSS: The Jaccard Index computed using the top-5 sentences in comparison to the ground truth (as described for Round 1 above)
top_2_TSS: The Jaccard Index computed using the top-2 sentences in comparison to the ground truth (as opposed to top 5 for top_5_tss)
top_2_TSS_voc_gt: This refers to the
top_2_TSScomputed using the vocabulary present in the ground truth data. Here, this is exactly the same as
- The adjusted scores are computed like this 👇
if (top_5_TSS_voc_x - top_5_TSS_voc_gt) >= 0.5 * model_compression : adjusted_top_5_TSS = top_5_TSS_voc_x adjusted_top_2_TSS = top_2_TSS_voc_x else: adjusted_top_5_TSS = top_5_TSS_voc_gt adjusted_top_2_TSS = top_2_TSS_voc_gt
So, if the improvement in accuracy between
voc_gt is greater than the expected
0.5 * model_compression, then we use the improved
voc_x accuracy, else we use the original
The leaderboard is sorted based on
adjusted_top_5_TSS as the primary score, and the
adjusted_top_2_TSS as the secondary score.
During the course of Round-2, all the scores are based on 60% of the whole test data, and the final leaderboards on the whole test data will be released at the end of Round-2.
Round 2 submissions are code-based as compared to csv-based submissions in Round 1. More on that below.
Round - 1
Prepare a CSV file containing header as
SMILEScolumn has to contain the
SMILESvalues as mentioned in the test set
PREDICTIONScolumn has to contain the the top-5 predictions of your model separated by
;where each of the odors in each sentence is separated by
For example, if the value of the
PREDICTIONScolumn for a particular row is :
Then, the top-5 predictions of your model are :
Note: If any of the sentences contain more than 3 words, then only the first 3 words will be considered for evaluation.
Sample submission format available at
sample_submission.csvin the Resources section.
Round-2 requires participants to submit their code which will be evaluated on our evaluation infrastructure. Each submission will have access to the following resources during evaluation :
- 4 CPU cores
- 16 GB RAM
- 1 NVIDIA K80 (optional, needs to be enabled in
All submissions will have a 10 minute setup time for loading their models, any preprocessing that they need, and then they are expected to make a single prediction in less than 1 second (per smile string).
🚀 For more instructions on how to make a submission, check out this getting_started_kit
The competiton consists of 3 separate Rounds.
- Round-1 : September 8th, 2020 - October 27th, 2020
- Round-2 : November 23rd - Jan 10th, 2021
- Round-3 : Jan 15th, 2021 - Feb 15th, 2021
2 participants of the
Round-3 will be awarded a cash prize of:
- 1st Prize :
- 2nd Prize :
- 💪 Challenge Page: https://www.aicrowd.com/challenges/learning-to-smell
- 🗣️ Discussion Forum: https://www.aicrowd.com/challenges/learning-to-smell/discussion
- 🏆 Leaderboard: https://www.aicrowd.com/challenges/learning-to-smell/leaderboards
We have the permission to use Olfactive descriptions and Molecules from "PMP database" authored by Mans Boelens and distributed by Leffingwell & Associates for this challenge.