Loading
2 Follower
0 Following
dipam
Dipam Chakraborty

Organization

ML Engineer at AIcrowd

Location

Kolkata, IN

Badges

7
4
3

Connect

Activity

May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Mon
Wed
Fri

Ratings Progression

Loading...

Challenge Categories

Loading...

Challenges Entered

Behavioral Representation Learning from Animal Poses.

Latest submissions

See All
graded 182254
graded 182252
graded 179065

What data should you label to get the most value for your money?

Latest submissions

No submissions made in this challenge.

ASCII-rendered single-player dungeon crawl game

Latest submissions

See All
graded 158823
failed 158209
failed 158208

Airborne Object Tracking Challenge

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 152892
graded 152891
failed 152884

Machine Learning for detection of early onset of Alzheimers

Latest submissions

No submissions made in this challenge.

Measure sample efficiency and generalization in reinforcement learning using procedurally generated environments

Latest submissions

No submissions made in this challenge.

Self-driving RL on DeepRacer cars - From simulation to real world

Latest submissions

See All
graded 165209
failed 165208
failed 165206

Robustness and teamwork in a massively multiagent environment

Latest submissions

No submissions made in this challenge.

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Multi-Agent Reinforcement Learning on Trains

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 143804
graded 125756
graded 125751

Learn to Recognise New Behaviors from limited training examples.

Latest submissions

See All
graded 125756
graded 125589

Reinforcement Learning, IIT-M, assignment 1

Latest submissions

See All
graded 125767
submitted 125747
graded 125006

IIT-M, Reinforcement Learning, DP, Taxi Problem

Latest submissions

See All
graded 125767
graded 125006
graded 124921

Latest submissions

See All
graded 128400
submitted 128365

Latest submissions

See All
failed 131869
graded 130090
graded 128401

Latest submissions

See All
failed 131869
graded 130090
graded 128401

Latest submissions

See All
graded 135842
graded 130545

Latest submissions

No submissions made in this challenge.

Round 1 - Completed

Latest submissions

No submissions made in this challenge.

Identify Words from silent video inputs.

Latest submissions

No submissions made in this challenge.

Round 2 - Active | Claim AWS Credits by beating the baseline

Latest submissions

See All
graded 182252
graded 178951
graded 178941

Round 2 - Active | Claim AWS Credits by beating the baseline

Latest submissions

See All
graded 182254
graded 179065
Participant Rating
cadabullos 0
nachiket_dev_me18b017 0
Participant Rating

Multi Agent Behavior Challenge 2022

Scores of all tasks made public 💡

20 days ago

Hi Everyone,

We’ve decided to make the scores of all the tasks and borda ranks of each submission public. Now you can view the scores of all the tasks. Our original motivation was to prevent overfitting on the tasks in the spirit of the challenge being unsupervised representation learning. But we understand that its frustrating for participants to not have any source of feedback on the tasks and why one of their submissions does better than another.

P.S: The scores visible are on the public split of the data. (Ignore that the names say private). The scores on the private split will be available after the competition ends, and will be used for selecting the winners (No changes here)

P.P.S: Negative scores indicate MSE scores, these are made negative because the borda system requires “higher is better”. So you can ignore the negative sign when looking at the MSE scores.


For those interested, here’s a short description on how the selection system works (taken from this discussion from AIcrowd Discord)

The selection system we have works like this:
First your own submissions are borda ranked, the best among these is selected, then your submission is borda ranked against other top submissions from each team. This is done to prevent any team from having multiple entries to increase their rank gap based on a single task.

As a consequence of this, you are also competing against your submissions in the borda system. In this case, your new submissions may perform slightly better in average borda rank against your own submissions, because it can significantly outperform on a few of the hidden tasks.

Note that the above case may cause the a submission with lower average f1 score to be selected.

Round 2 Deadlines updated 🚀

21 days ago

Hi Everyone,

Happy to announce that we’ve extended Round 2 by 1 and a half months after receiving your feedback that the current timeline was too short to work with video data. Looking forward to your solutions!

Round 2 now ends on 3rd July 2022.

Data Purchasing Challenge 2022

🚀 Share your solutions! 🚀

30 days ago

Hi Everyone,

Thank you for participating in the Data Purchasing Challenge, it has been a unique journey for this one-of-a-kind challenge. We’re excited to know your ideas and solutions. Regardless of whether you won, any ideas you share on the discussion forum are highly appreciated. :smiley:

Sharing your solutions also helps you reflect upon your learnings throughout the competition. :rocket:

I’ll summarize all the solutions that are shared into this post, as I’m sure they’ll be tremendously useful for participants of future Data Purchasing Challanges (yes there will be more :wink:)

Please also share failed ideas, as all machine learning afficianados know, negative samples are just as important.

Looking forward to your solutions! :raised_hands:

[Announcement] Leaderboard Winners

30 days ago

Hi @xiaozhou_wang, glad you’ll be sharing your solutions. Congratulations on bagging the top spot. As @Camaro suggested, please make a discourse post, and you can also make your submissions repository on AIcrowd Gitlab public if you want to share your full submissions journey. :innocent:

:rotating_light: Select submissions for final evaluation

About 1 month ago

Hi @chuifeng

No, all the runs are completed. The leaderboard is final. The best scoring submission of each participant from the full runs was used. The “successful entries” column is generic feature of the leaderboard, since leaderboard was calculated offline and the final results added separately, it shows 1.

Please check this sheet with the full scores of all the runs.

:rotating_light: Select submissions for final evaluation

About 2 months ago

Hi @leocd

The highest scoring among the selected submissions will be used.

:rotating_light: Select submissions for final evaluation

About 2 months ago

Hi everyone,

Thanks for participating in the Data Purchasing Challenge! :rocket:

We hope you had fun with the problem and the dataset. In this next phase, we will be evaluating your selected submissions on the full hidden dataset.

‎ℹ️ Know more about it here:

:alarm_clock: Deadline for filling this form is 08 April 2022, 23:59 UTC (earlier the better, thanks :dancer:)

:point_right: Select Submission ID for Full Evaluation

Code for End of Competition Training pipelines

About 2 months ago

Hi @sergey_zlobin , all the compute budget pairs will be run with the purchase phase first. Then the post purchase evaluations will be run on each of the purchased sets. As is already the case, there is no time limit on the post purchase training as its entirely controlled by us.

Code for End of Competition Training pipelines

About 2 months ago

Hi Everyone,

Below are the links to the end of competition training pipelines that will be used for the final winner selection. Each pipeline will be run for 2 seeds. The code release is for your feedback, so please feel free to go through them. :pray:

In case you have no idea what I’m talking about, please check this post.

If there are any critical issues we’ll fix them.

I tried many different combinations and rejected some if the gap between random and all label purchase was too low, apologies if your favourite combination didn’t make it. :upside_down_face:

Check the commit history easily navigating the changes.

IMPORTANT: Details about end of competition evaluations 🎯

About 2 months ago

@Camaro Probably no extension as far as I know. Will inform in case of any changes.

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

About 2 months ago

@Camaro

Yes what you say makes sense, although on the other end a very strong model was getting nearly as good as “all label purchase” scores with just random purchase, so the dataset needed more difficulty, important lessons learnt. In any case, I agree with your definition of useful, for now we’ve come up with the end of competition evaluations scheme. Please check this recent post.

IMPORTANT: Details about end of competition evaluations 🎯

About 2 months ago

Hi Everyone,

I’ve seen multiple people’s posts suggesting changes in the purchase phase training pipeline. This post is to clarify some details about the end of competition evaluations and explain some its implications.

First of all, thanks a lot to everyone who have provided feedback for the training pipeline. It has been very valuable to us for the challenge design. :heart:


TL;Dr :yawning_face: - Please submit your best purchase strategies, you’ll need to select them for end of competition evaluations which will run 5 post purchase training pipelines.


Current problems

Many have pointed out that the training pipeline does not provide good scores. I’ll break these down some categories.

  • Concerns about the best purchases not being incentivised correctly :control_knobs:

    1. Model is too weak and cannot learn hard examples - This is a legitimate concern, though I do not currently know the scale to which it is applicable. We’re investigating this.
    2. Not using GaussianBlur in testing changes the optimal labels - Currently I have not found any strong evidence of this, but would love to discuss further.
    3. Model does not converge due to low epochs hence too stochastic - I did not find the stochasticity in scores to be too much and feel it is in expected practical limits. However, in the later part of this post I’ll be addressing this further.

    These are the most important concerns we’re looking to resolve. Some of them are also difficult to measure, and the discussion becomes qualitative and opinionated. We want to resolve it in the most quantitative and fair way possible.

  • Concerns that the score is too low :arrow_down:

    1. The feature layers are frozen
    2. The model is too small and plateaus
    3. GaussianBlur is not used in test

    For these, I’d like to reiterate that the score is not important, rather, maximising the score by making the best purchase strategies is the goal of the competition.


End of competition Evaluations :rocket:

In addition to changing the dataset for end of competition evaluations. We will run select submissions on multiple training pipelines in the post purchase phase.

The detailed steps are given below:

  1. Eligible teams will select two of their submissions to evaluate - Eligibility criteria to be announced soon, it will be based on Round 2 leaderboard.
  2. Each submission will run through the pre-train and the purchase phase on the end of competition dataset.
  3. The same purchased labels will be put through 5 training pipelines - Details to be released soon.
  4. Each training pipeline will be run for 2 seeds and scores averaged, to address any stochasticity in scores.
  5. To avoid issues due to difference of average scores from different training pipelines, a Borda ranking system will be used.

We hope this will inceltivize participants to select the best purchase strategies and not optimize for the current training pipeline. We’re unable to incorporate this setup during the live round due to the prohibitive cost of compute for each run.


Training pipeline survey :receipt:

Please vote here for your favoured schemes. Note that is not a vote to select the training pipelines, just a survey of participant’s preferences, but we’ll take the results very seriously.

  • Unfreeze feature layers in base model
  • Use Gaussian Blur During Test
  • Train for more epochs
  • Use bigger model - Efficientnet-B7
  • Train more epochs + Unfreeze feature layers in base model
  • Train more epochs + Use Gaussian Blur During Test
  • Remove Gaussian Blur from training
  • Unfreeze feature layers in base model + Remove Gaussian Blur from training

0 voters

Please feel free to suggest any other changes if I have missed them. :pray:

Title for the post

2 months ago

Body for the post, which will be edited soon.

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

2 months ago

Hi @tfriedel

Seems I missed your message. The point on ruling out the most useful images, do you feel its a huge issue with the current training pipeline?

We do want the solutions to be as agnostic to the training pipeline while buying the best images possible. Yes its not completely possible to make things training agnostic, but that is the spirit of the competition we’d like to promote. If you’re finding that you’re deliberately having to remove too much of the useful images, please let me know.

Baseline Released :gift:

2 months ago

We hope you have been following the development around the challenge.

Today, we are finally releasing a baseline, which was promised in the last townhall.

Did you miss the townhall?
You can check about it here: 📹 Town Hall Recording & Resources from top participants

:books: About the Baseline

The baseline contains fast heuristic implementations of some simple ideas.

  • Purchase images with more labels - For multilabel datasets, often having images with more than one label gives a boost for deep learning models.
  • Purchase uncertain images - Purchase images which have the most uncertainty in their predictions. While many methods exists to measure uncertainty, a simple output probability based heuristic method is used here.
  • Purchase images to balance labels - Well balanced datasets can improve model performance in deep learning. We set a uniform target distribution and try to purchase labels to get closer to that distribution. The provided code can try to purchase labels to any target distribution.

:student: Learn more about the implementation here.
:robot: Jump to the codebase directly here.
:sparkles: Looking for more resources to explore? Check out this thread.

:mag: Ok, talk is easy, how does it perform?

Well, as it stands, the baseline is currently on 5th position on the whole leaderboard.

:thinking: Do you have additional questions?

Feel free to drop them in this thread and we can reply them asap.
What are you waiting for? Let’s get started with the submissions! :sparkles:

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

2 months ago

Hi @Camaro

Thanks for the clarification and useful explanation. I’ll consider this and try some experiments. One issue with designing the challenge is it needs balance between good training pipeline, score gap for better data, and compute constraints. All these need to be satisfied while iterating our synthetic data to match these constraints. We’ll try to improve the training pipeline accordingly if we’re able to match them in a reasonable way.

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

2 months ago

Hi @Camaro

I understand your concern about underfitting. However, this challenge is a bit non-traditional, the data is completely synthetic, and the purpose is solely the methods used for research only. The final model trained is not of importance to any real-world setting, only the algorithms you develop.

Also, the purpose of the training pipeline is to give a level playing field to all participants to that they focus the purchase strategies. Whether the final loss value is reached is not important as long as better data purchased produces better scores using the same training pipeline, which is what we’ve tried to setup with Round 2.

Your goal is to improve the score by purchasing better labels, the score may be limited by the training setup once the best labels are purchased, but that is not yet the case in my opinion.

Why there is no GaussianBlur in test transform?

2 months ago

Hi @chuifeng

Yes the GaussianBlur in training should have been applied with a probability value during training to get higher scores. Or as you suggest, could be used during test which will also increase the score.

Though I’d still like to stress that the gap in scores between random purchase and the all-label purchase is the criteria that is more important that the raw scores.

You can refer to the discussion here:

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

3 months ago

Hi @tfriedel

Currently, I do not see this much spread in scores when testing on the private set which is used for leaderboard. Nevertheless, thanks for pointing this out, will check further on the score spread across the budgets used.

Also note that the final end of competition evaluations will not use the dataset used on the leaderboard now, but a different dataset sampled from the same distribution. The end of competition evalutions will also feature more exhaustive evaluations with many more models. Hence overfitting the leaderboard is likely to hurt participants when the final evaluations take place. We’ll make communications about this clearer in case this isn’t properly explained.

MABe 2022: Mouse Triplets

Evaluations are being re-run with balanced class weights

About 2 months ago

Hi Everyone,

As you might have recently got a e-mail for, the leaderboard is currently being re-run with balanced class weights. The scores might be unstable until all the runs finish. Please do not worry if you suddenly see your ranks jumping around too much. They should stabilize once all the runs are done.

dipam has not provided any information yet.

Notebooks

Create Notebook