Loading

mohanty ###

Name

Sharada Mohanty

Organization

AIcrowd

Location

Geneva, CH

Ratings Progression

Loading...

Challenge Categories

Loading...

Challenges Entered

Sample-efficient reinforcement learning in Minecraft

10 Travel Grants
Misc Prizes : 1x Titan RTX GPU

Latest submissions

See All
graded 18575
graded 15152
graded 11232

Multi Agent Reinforcement Learning on Trains.

30'000 Prize Money
5 Travel Grants
Misc Prizes : To Be Announced

Latest submissions

See All
failed 18455
failed 18370
failed 18105

A benchmark for image-based food recognition

1 Travel Grants
1 Authorship/Co-Authorship
Misc Prizes : Various Prizes

Latest submissions

See All
graded 58585

Reinforcement Learning on Musculoskeletal Models

3 Travel Grants
2 Authorship/Co-Authorship

Latest submissions

See All
graded 18446
graded 10385
graded 10099

Disentanglement: from simulation to real-world

20,000 EUR Prize Money
Misc Prizes : Brilliancy Prizes

Latest submissions

See All
graded 21050
graded 13148
graded 13117

Classify images of snake species from around the world

1 Travel Grants
1 Authorship/Co-Authorship

Latest submissions

See All
failed 5795
graded 392
failed 391

Robots that learn to interact with the environment autonomously

Latest submissions

See All
graded 11208
failed 11198
failed 11193

A new benchmark for Artificial Intelligence (AI) research in Reinforcement Learning

$25000 Prize Money
3 Travel Grants
Misc Prizes : 3 x $5000 GCP credits

Latest submissions

See All
graded 7057
graded 664
graded 663

Latest submissions

See All
graded 1137
graded 1135
graded 1134

2000 Prize Money
0 Travel Grants
Misc Prizes : SIRIUS and IBM Research sponsors the prize for the best systems and best student systems, respectively.

Latest submissions

See All
graded 9169
graded 9164

2000 Prize Money
0 Travel Grants
Misc Prizes : SIRIUS and IBM Research sponsors the prize for the best systems and best student systems, respectively.

Latest submissions

See All
graded 9168
graded 9163

2000 Prize Money
0 Travel Grants
Misc Prizes : SIRIUS and IBM Research sponsors the prize for the best systems and best student systems, respectively.

Latest submissions

See All
graded 9167
failed 9165
failed 9162

CRDSM -Crowdsourced Map Land Cover Prediction

About the CRDSM -Crowdsourced Map Land Cover Prediction category

5 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

pandemic-rl

About the pandemic-rl category

6 days ago

Note :

This forum is currently in private mode.
The permissions have to be manually changed for this to be visible to the final users.

OCPDT - Occupancy Detection-c704a6

About the OCPDT - Occupancy Detection-c704a6 category

7 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

OCPDT - Occupancy Detection-d5215a

About the OCPDT - Occupancy Detection-d5215a category

7 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

OCPDT - Occupancy Detection

About the OCPDT - Occupancy Detection category

7 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

EMSPM - Email Spam Prediction

About the EMSPM - Email Spam Prediction category

7 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

DBSRA - Diabettes Readmission Prediction

About the DBSRA - Diabettes Readmission Prediction category

8 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

DOTAW - Dota 2 Prediction

About the DOTAW - Dota 2 Prediction category

8 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

DCRCL - Default of Credit Card Clients

About the DCRCL - Default of Credit Card Clients category

8 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ICMPR - Income Prediction

About the ICMPR - Income Prediction category

9 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

YPMSD - Music year prediction

About the YPMSD - Music year prediction category

13 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

SPCRT - Superconductor critical temperature

About the SPCRT - Superconductor critical temperature category

13 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

OLNWP- Online News Prediction

About the OLNWP- Online News Prediction category

14 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

BKMKT - Bank Marketing Stategy Analysis

About the BKMKT - Bank Marketing Stategy Analysis category

15 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

PKHND - Poker Hand Recognition

About the PKHND - Poker Hand Recognition category

16 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

MNIST - Recognise Handwritten Digits

About the MNIST - Recognise Handwritten Digits category

18 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

QM energy challenge

About the QM energy challenge category

18 days ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Poker AI Challenge

About the Poker AI Challenge category

About 1 month ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

HECKTOR

About the HECKTOR category

About 2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

droneRL

About the droneRL category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

AWS DeepRacer Sim2Real Challenge #2

About the AWS DeepRacer Sim2Real Challenge #2 category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 VQA-Med - VQG

About the ImageCLEF 2020 VQA-Med - VQG category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Snake Species Identification Challenge

Can I have an example of a code which is working to make a submission on gitlab?

2 months ago

@amapic: If you built the conda env drom the initial environment.yml file, then conda env export --no-build will export the updated state of the environment.

Can I have an example of a code which is working to make a submission on gitlab?

2 months ago

Dear @ValAn,

Our sincere apologies for the inconveniences faced by you.

Regarding the slow evaluation speeds, given that we have to execute your code (and models etc) on a large number of test images, the evaluations are indeed slow. Your model has to make predictions for a large number of images. We are trying to improve this experience by providing better feedback in terms of progress etc, and will definitely address this in the upcoming version of the challenges.

Regarding the competition, we are providing all updates on this forum here, and we would be happy to answer any and all questions you have here. We are also working on better notification systems so that you get relevant updates from the challenge over emails and other notification channels on the platform that you subscribe to.

In the meantime, we really appreciate your feedback. Your feedback helps us make the platform much better for thousands of other users on the platform, and under no circumstances we take it as an offense.

Thank You,
Mohanty
(on behalf of the organizing team)

NVCC not found error

8 months ago

@EarthAI-Gramener: Why do you need nvcc in your submission ? The base image already has cuda-10 available, and you should easily be able to access the GPU via pytorch, tf and the likes if you install their appropriate versions and specify gpu:true in aicrowd.json

Instructions to include non-standard packages

8 months ago

You can include a file called as apt.txt and include “git” as its content.

Please try using aicrowd-repo2docker to build the images before submitting.

Submission Errors

8 months ago

As mentioned in the starter kit, the tag names have to begin with “submission-“ , so pushing a tag called as say : submission-v010
should do the trick.

This was a loosely held rule and has only been enforced recently.

Submission Errors

8 months ago

The error is :

Traceback (most recent call last):
  File "run.py", line 258, in <module>
    run()
  File "run.py", line 146, in run
    os.remove(sRemove)
OSError: [Errno 30] Read-only file system: '/test_data/2e23ade63c4e32b728a423ff19e52ef1.jpg'

Your code is trying to remove images from the test set here

Instead of trying to delete the corrupt files, please try to just append a random prediction to the final prediction CSV file.

Instructions to include non-standard packages

8 months ago

For packages installed directly from git, you should be able to manually replace the relevant line (from the pip subsection in the environment.yml), with something like :

- git+https://github.com/google-research/disentanglement_lib.git

Also, do remember to build the image locally and see if it worked.
This is unfortunately a bug with anaconda, and we will see if we can do anything about it in the short term.

Instructions to include non-standard packages

8 months ago

The container would not have network access, hence cloning the repository would not work.
But you can always install everything you need in an Anaconda environment, and dump the environment as specified here

Submission Errors

8 months ago

@nilabha: Some images in the test set are corrupt unfortunately. And we have not removed them to stay consistent. But if you cannot read an image with PIL, then please use a random prediction for the same.

Access Logs for submission

8 months ago

Pasted the logs on the relevant issue.

Submission Errors

8 months ago

Looks like this is because the Dataloader is trying to spawn way too many workers. Can you set num_workers=0 in the DataLoader so that it does all the data loading in the main thread.

Submission Errors

8 months ago

I pulled down the image built from your submission, and the permissions for run.sh indeed do not have any execution permission :

root@408dfe6f4a7c:~# ls -al
total 112
drwxr-xr-x 1 aicrowd aicrowd 4096 Jul 24 16:54 .
drwxr-xr-x 1 root    root    4096 Jun 14 08:09 ..
-rw-rw-r-- 1 aicrowd aicrowd 3033 Jul 24 16:46 aicrowd_helpers.py
-rw-rw-r-- 1 aicrowd aicrowd  194 Jul 24 16:46 aicrowd.json
-rw-r--r-- 1 aicrowd aicrowd  220 Jun 14 08:09 .bash_logout
-rw-r--r-- 1 aicrowd aicrowd 3771 Jun 14 08:09 .bashrc
-rw-rw-r-- 1 aicrowd aicrowd  339 Jul 24 16:46 build.sh
drwx------ 3 aicrowd aicrowd 4096 Jul 24 16:54 .cache
drwxrwsr-x 2 aicrowd aicrowd 4096 Jul 24 16:53 .conda
drwx------ 3 aicrowd aicrowd 4096 Jun 19 08:45 .config
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 data
-rw-rw-r-- 1 aicrowd aicrowd  349 Jul 24 16:46 debug.sh
drwxr-xr-x 2 aicrowd aicrowd 4096 Jul 24 16:53 .empty
-rw-rw-r-- 1 aicrowd aicrowd 2123 Jul 24 16:46 environment.yml
-rw-rw-r-- 1 aicrowd aicrowd   94 Jul 24 16:46 environ.sh
-rw-rw-r-- 1 aicrowd aicrowd   94 Jul 24 16:46 .gitattributes
-rw-rw-r-- 1 aicrowd aicrowd   57 Jul 24 16:46 .gitignore
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 models
-rw-r--r-- 1 aicrowd aicrowd  807 Jun 14 08:09 .profile
-rw-rw-r-- 1 aicrowd aicrowd 8559 Jul 24 16:46 README.md
-rw-rw-r-- 1 aicrowd aicrowd 8461 Jul 24 16:46 run.py
-rw-rw-r-- 1 aicrowd aicrowd   28 Jul 24 16:46 run.sh
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 sample
-rw-rw-r-- 1 aicrowd aicrowd  132 Jul 24 16:46 trainnew.csv

What happens if we re-submit?

8 months ago

@gloria_macia_munoz: 1) Yes you can interrupt a submission by simply deleting the note created by aicrowd-bot where it provides you feedback.
2) the ranking is based on your highest score.
3) order of the filenames does not matter.

Round 2: Understand the submissions

8 months ago

@gloria_macia_munoz: No, the order does not matter. As long as all the files are represented, the evaluator sorts them based on the filename.

What happens if we re-submit?

8 months ago

@gloria_macia_munoz: I interrupted your evaluation. Your evaluation was not using GPUs.
I enabled gpus in this commit : https://gitlab.aicrowd.com/gmacia/snake-species-identification-challenge/commit/3a7635043771384775c0f7d50b8fc7bce5ff9d23

Where I simply add "gpu":true to aicrowd.json.

Please feel free to resubmit now with this change included.

Submission Errors

8 months ago

@nilabha: Had posted earlier on the relevant issue. The problem was the run.sh does not have execution permissions, so a chmod +x run.sh (and a subsequent commit) should fix the problem.

Submission Errors

8 months ago

Looking at the logs, it seems that your aicrowd.json is checked into LFS, while the evaluator expects it to be checked in directly in the repository (this is a bug, and should be fixed at our end, and we will), but in the meantime, if you move your aicrowd.json from LFS to a direct check in into the git repository, the evaluation should go through !

Submission Errors

8 months ago

The build logs are here : https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/snippets/7288

From the look of it, you can solve it by adding a apt.txt at the root of your repository with gcc in it.

Can't make a submission

8 months ago

@EarthAI-Gramener: Can you please confirm that all the pending submissions were evaluated ?

Can't make a submission

8 months ago

@gokuleloop: We did have a small outage last Sunday, and some evaluations were affected.
But, looking at the stream of submissions, I do see that the submissions of the snake challenge are being picked up now.
Can you please try making a new submission now ?

Submission Errors

9 months ago

@nilabha: We had a small outage yesterday, and some of the evaluations were affected including yours. The same has been resolved now.

Your submission was re queued, and there was still an error in the image build, where certain packages were not found on Conda for linux :

ResolvePackageNotFound: 
  - m2w64-gcc-libs-core=5.3.0
  - m2w64-gcc-libgfortran=5.3.0
  - win_inet_pton=1.1.0
  - m2w64-gcc-libs=5.3.0
  - vs2015_runtime=14.15.26706
  - m2w64-gmp=6.1.0
  - winpty=0.4.3
  - msys2-conda-epoch=20160418
  - icc_rt=2019.0.0
  - m2w64-libwinpthread-git=5.0.0.4634.697f757
  - vc=14.1
  - pywinpty=0.5.5
  - pyreadline=2.1
  - wincertstore=0.2

Can't make a submission

9 months ago

Dear @shah_jamal,
We had a small outage on the evaluator because of a spike in usage due to one of the challenges ending. We have fixed the same, and have increased the capacity of the evaluators. Please do try submitting again.

Round 2: Understand the submissions

9 months ago

Hi @EarthAI-Gramener,

The aicrowd.json file is required, and so is a file for specifying your software runtime (requirements.txt or environment.yml). The environ.sh, build.sh and debug.sh etc are for local use only.
The test images folder page is provided by the AICROWD_TEST_IMAGES_PATH environment variables (see here )
And you will have to check in the class_idx_mapping.csv file into the repository to be able to access it when your code is being evaluated.

Submission Errors

9 months ago

@nilabha: can you please pull in the latest changes from the main starter kit

Also, if you are using environment.yaml for packaging your software runtime, please delete the Dockerfile at the root of your repo. That is the main cause of your submission failing at the moment. Sorry for the confusion.

Issue in Test Images

9 months ago

Please use random predictions for the listed filenames which are corrupt, the evaluator expects that you provide predictions for all the filenames in the test set.

Regarding evaluation metric

9 months ago

@gloria_macia_munoz: Whoops !! Thanks for pointing that out. I realised I had a typo in my previous message, which I just corrected.

Here is what is being used in the evaluator :

f1 = skm.f1_score(ground_truth, submission_idx_max, average='macro')

PS: the prob distributions etc are passed through a separate softmax
(before computing the submission_idx_max), so please dont even try to try with randomly large values for the prob values for individual classes :angel: !!!

Best of Luck !!

How to show image batch with real class name?

9 months ago

@gloria_macia_munoz: You can use the class_idx_mapping.csv file from here to find the mapping of the class names used here and the real class names.

Regarding evaluation metric

9 months ago

@gokuleloop: We use macro averaged f1_score from sklearn.metrics.

Cannot upload trained model to gitlab.aicrowd

9 months ago

Well, if you use git-lfs correctly, then the debugging of your submission would be the same as any other submission ? If you have a concrete pain point where you are stuck, we would be happy to help !

Cannot upload trained model to gitlab.aicrowd

9 months ago

@liberifatali: You can use git-lfs to add large files to your repository. Here are some instructions : https://dzone.com/articles/git-lfs-why-and-how-to-use

Best of luck

https://discourse.aicrowd.com/t/

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

https://discourse.aicrowd.com/t/

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

https://discourse.aicrowd.com/t/

5 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Please send me evaluation error log on gitlab

5 months ago

Can you please reach out to help@aicrowd.com for the same ?

A post in "Using open-sourced networks" requires staff attention

7 months ago

Thanks for letting us know. We agree there is an issue and we’ve removed the post.

Profile Pic

7 months ago

Hey,

Given you are one of the official organizers of the competition. It might be good to have a profile picture :wink:

As your account is linked with your github account, you will probably have to first put up a profile picture on github. And then sign out and sign into AIcrowd again.

Cheers,
Mohanty

Submission in-queue for a long time

8 months ago

@weyshi: Can you please send a link to help@aicrowd.com with appropriate links to the issue ?

Interview

8 months ago

Dear @gloria_macia_munoz,
Thank you for your interest in our platform, and for the encouraging words.

I would be happy to chat with you, and answer any questions you have.
My email is : mohanty@aicrowd.com , and please feel free to reach out with some time slots that work for you the best, and we can find a good overlap that works for both of us the best.

Cheers,
Mohanty

Submission in-queue for a long time

8 months ago

@weyshi: There are many submissions in the queue, which are slowly being processed. They will eventually be processed.

and by tag the aicrowd-bot we basically mean, that you tag it like @devops .Sorry if that was confusing. Some people from our team should eventually catch up with all the failed submissions and paste the logs if they haveent already.

Regarding reproducing results

8 months ago

Can you please send this an email at : mohanty@aicrowd.com with devops@aicrowd.com in cc.

Cheers,
Mohanty

Potential way to cheat in OTC?

9 months ago

@Leckofunny: You are right ! Winners of Round 2 are selected based on the leaderboard and if they qualified after Round 1.

It might be important to clearly separate out the stochasticity because of the environment and because of the agents. The environments use the same fixed seeds across all the evaluations, just to minimise the stochasticity across evaluations (from the environment). But if you are convinced that the environment still contributes a lot to the stochasticity, it might make sense to take it up with the Unity team (Arthur Juliani), and we can have an open conversation about that together with them on a good plan of action.

Cheers,
Mohanty

Potential way to cheat in OTC?

9 months ago

Ahhh, you are right !!
That was a weird artifact that we had to implement, because our legal teams messed up in terms of writing the Rules. And yes, the latest submissions are being used on the leaderboard, but I believe, thats still not a major issue. As people know the exact tag that created their best score, and they simple make the same submission again (with a small change if its still cached)

But the whole idea of the latest submission on the leaderboard is not something we plan to use in any other challenges on the platform.

Potential way to cheat in OTC?

9 months ago

Hey,

Yes, we do know about it. The scores on the leaderboard are always the highest score you have across all your submission. But if a participant wants to not have his score reflected on the leaderboard, he could very well intentionally crash the code upon some condition. But this does not affect the max number of submissions a participant can make.

If we choose not to render any scores until a successful evaluation (and update on the leaderboard), then the user experience will be bad for all the participants who are waiting for their evaluations to be completed (in terms of the realtime feedback), and hence we are okay with this tradeoff for now.

Regarding evaluation

9 months ago

Hi @gokuleloop,

Yes, the output path has to be picked up from :

AICROWD_PREDICTIONS_OUTPUT_PATH = os.getenv('AICROWD_PREDICTIONS_OUTPUT_PATH',False)

And please find the individual logs on the individual issue pages.

Cheerrs,
Mohanty

https://discourse.aicrowd.com/t/

9 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Caption - Concept Detection

About the ImageCLEF 2020 Caption - Concept Detection category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 VQA-Med - VQA

About the ImageCLEF 2020 VQA-Med - VQA category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 DrawnUI

About the ImageCLEF 2020 DrawnUI category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Coral - Pixel-wise parsing

About the ImageCLEF 2020 Coral - Pixel-wise parsing category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Coral - Annotation and Localisation

About the ImageCLEF 2020 Coral - Annotation and Localisation category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Lifelog - SPLL

About the ImageCLEF 2020 Lifelog - SPLL category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Lifelog - LMRT

About the ImageCLEF 2020 Lifelog - LMRT category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

LifeCLEF 2020 Snake

About the LifeCLEF 2020 Snake category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

LifeCLEF 2020 Plant

About the LifeCLEF 2020 Plant category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

LifeCLEF 2020 Geo

About the LifeCLEF 2020 Geo category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

LifeCLEF 2020 Bird - Monophone

About the LifeCLEF 2020 Bird - Monophone category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

LifeCLEF 2020 Bird - Stereo

About the LifeCLEF 2020 Bird - Stereo category

2 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Flatland Challenge

Submission issue

3 months ago

@eli_meirom: The git remote add step should be :

git remote add aicrowd git@gitlab.aicrowd.com:<YOUR_AICROWD_USER_NAME>/flatland-challenge-starter-kit.git

Note the lack of / after the git@gitlab.aicrowd.com

Submissions counting

3 months ago

@vitaly_bondar: Yes, failed submissions are indeed counted towards the daily submission limit.
And the submission limits are enforced in the “last 24 hours” period ( moving window of 24 hours), and hence are agnostic to the timezone you are in.

Evaluation server hardware specifications

4 months ago

@sungbinchoi: Agree. We would be happy to provide GPUs if you make a case for the same. Please reach out to us over email, and share any code which requires GPU for evaluation, and then we should be able to make GPUs available for valid requests.

[ANNOUNCEMENT] Start Round 2

5 months ago

Update

We are currently working on a few more bug fixes, and performance and stability related issues with flatland. These fixes will be made available as another patch to the current latest release of 2.1.6.

Much of the delay in reliably accepting submissions has been due to tuning the complexity of the test environments on which your submissions will be evaluated. Please expect a further update from us soon.

But please be assured that no key changes in the features or the environment interfaces will be introduced at this stage, hence you can reliably continue experimenting with the flatland library at your end before we start accepting submissions again.

And our apologies for not being more communicative about the updates and announcements related to the competition. But rest assured, the whole team is working really hard to ensure you all can have a great experience taking part in the competition.

Thanks,
Mohanty
(on behalf of the organizing team)

Local evaluator on Windows not working

7 months ago

I have a feeling that the pickle files you are using are corrupt somehow :confused: Or the dataset directory paths are not properly set.

Are you correctly specifying the /path/to/test-env-data/ ?

Test case 532 evaluation problem

7 months ago

@ryznefil: Look at your score on the leaderboard, looks like you managed to get past that test case :wink: ! Anything you would want to share with the rest of the participants ? :angel: !

Baseline installation

8 months ago

Thanks all for the inputs.
Just wanted to add that, the baseline repository can evolve much faster with community contributions.
Please feel free to send in pull requests with changes that worked for you.

Cheers,
Mohanty

Repo2Docker question

8 months ago

Hi @ryznefil,

You can actually include any anaconda package in your environment.yml file (as long as they exist for linux, which is where they are eventually installed in the docker container). So if you can find a conda package for your dependencies, then you can just include them like this one is.

And you can set the priority of different conda channels here and even include some new ones if you wish.

Cheers,
Mohanty

[ANNOUNCEMENT] Submissions Open

8 months ago

@mugurelionut: We had pasted some logs, and investigating why your latest submissions are failing. The 15minutes timeout one is a good direction, and yes we do have that constraint, where the submissions have to send some signal every 15minutes. (You could in principle even send a no-op :wink:).

[ANNOUNCEMENT] Submissions Open

8 months ago

Hi @mugurelionut,

All good questions.
I just went through your submission, and it looks like you are using an older version of the starter kit ?
https://gitlab.aicrowd.com/mugurelionut/flatland-challenge-starter-kit ?

And coming to your question of why does the remote_client need access to the test files locally, that was indeed a non intuitive design decision that we had to take, because we want to lets participants have access to a whole RailEnv object to do whatever they want with it, but we did not want participants to arbitrarily change the state of the environment (for instance, a participant could simply loop over all the agents, and set their position to the target position, and the env would be done in a single step :wink: ), which would affect our ability to reliably measure the score for all the participants.

So what we instead do now, is run two copies of the same environment, one inside the container running the user-submitted code, and one inside a separate trusted container. When you do an env_step on the remote_client, we apply that step on both the local instance of the environment, and the “remote” instance of the environment in the trusted container.

We compute things like the total rewards, when an env is done, etc from the env instance in the trusted container (remote), and we compute observation from the env instance in the container running user code (which also allows us to safely let users design their own observations).

Now, to make this happen, we can ofcourse expose the RailEnv (and all its interfaces) in the remote container as an RPC, adding enough safe guardings to ensure it cant be abused. But that will cause a lot of data flowing back and forth between the containers over the network on all interactions. For instance, the env state objects for larger grids (100x100 or more) can be in hundreds of megabytes, and that would hugely affect the performance during the evaluation.

He we just securely mount a copy of all the test envs in both the containers, and when you do a env_create call, the remote service sends you the “key” of the test-env that its loading at its end, and the remote_client also locally instantiates a copy of the same env, which you of course can access by : remote_client.env (ref) .

In your case, the only reason you could be gettting this error, is either if you have not downloaded the tests, or placed them incorrectly, or passing a wrong value to the AICROWD_TESTS_FOLDER env var.

The AICROWD_TESTS_FOLDER env variable should point to the root of the directory which has the individual test sets. If say, my directory containing the tests is called as test-env, then its structure should look like :

└── test-envs
    ├── Test_0
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_1
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_2
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_3
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_4
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_5
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_6
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_7
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_8
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    └── Test_9
        ├── Level_0.pkl
        └── Level_1.pkl

And in your case, lets say, you set the AICROWD_TESTS_FOLDER as /your/path/to/tests/folder,
then can you ensure that you can get this following snippet of code to run :

import os
TESTS_FOLDER = os.getenv("AICROWD_TESTS_FOLDER", "/tmp")
test_path = os.path.join(TESTS_FOLDER, "Test_0/Level_0.pkl")
if not os.path.exists(test_path):
  raise Exception("Could not access test file at the right location :( ")
else:
  raise "File structure all correct !!  Env variables set correctly !! Finally :D !"

Do note that on the evaluator, this env variable will be automatically set, so please do not try to overrride that env variable in your run.sh file.

Cheers,
Mohanty

Deadline for first round

8 months ago

@ryznefil: We have an announcement scheduled for later today, which will clarify all the details, including the submission instructions, the updated timelines, etc.
Sorry for all the delays, and thank you so much for your patience.

Submission format

8 months ago

@PerInDisguise: Thats correct, and that’s why we haven’t publicly announced the link to the starter kit.
I will work on polishing the instructions tonight and hope to have the updated instructions ready tomorrow morning with something that you can use to make a submission. We are almost there. Thank you so much for your interest in the competition and the patience !

Cheers,
Mohanty

Submission format

8 months ago

@PerInDisguise: Lol !! Yeah, that is the eventual starter-kit. But I am adding some final touches there, and trying to simplify it a bit before making the public release.

Can we get observations before the first step?

8 months ago

@mugurelionut: Sorry if the examples are a bit misleading. We will try to make the examples consistent.

But when you do the env.reset() call, it should return you an observation. So basically :

first_observation = env.reset()

ANNOUNCEMENT: Beta Prolonged

8 months ago

Hi @PerInDisguise,

Sorry for the delay. We are setting up and testing the final production clusters for the evaluator as we speak. Please expect an update on how to make submissions later today.

Unable to install Flatland

8 months ago

@pmousavi: You are right. The flatland-demo script hasnt been added to the library yet. It will be shipped in the next release (due later today).

For now, you can safely ignore the fact that flatland-demo script doesnt exist in your path.

Will the worlds our submissions will be evaluated against be known or unknown to participants

8 months ago

@marcoliver_gewaltig: Yes that is correct. In steps 4 and 5, we will be running your submitted code against a series of test environments of different levels of difficulty, and your cumulative score will be computed based on the cumulative performance of your code across all these test environments. More details about this should be released latest by this weekend.

Cheers,
Mohanty

The problem with the round duration

9 months ago

@RomanChernenko: We will start accepting the submissions next week. There have been a few delays internally, and sorry about the same. We will make an announcement as soon as we are ready with the evaluators !

Submission format

9 months ago

@RomanChernenko: We will be posting instructions for submission sometime this week. Thank you for your patience.

Mutli Agent Setup

9 months ago

@RomanChernenko: Not yet, but in one of the future versions of the competition, definitely !!!

Novartis DSAI Challenge

How is F1 score calculated?

4 months ago

@carlos.cortes: Yes, F1 Score computation works around a 0.5 threshold of the probability of approval.

How to use conda-forge or CRAN for packages in evaluation?

4 months ago

@bjoern.holzhauer: We are on it. We will send you a working version with tidyverse soon. Thanks for your patience.

Test Post Please Ignore

5 months ago

Test Post. Please Ignore. Test Post. Please Ignore.

Food Recognition Challenge

Is GPU available?

4 months ago

@shivam: Yes, please lets start allocating GPUs for evaluations on this challenge.

External datasets

4 months ago

@joao_schapke: Yes you are allowed to use external datasets, pre-trained models, etc.

About the Food Recognition Challenge

9 months ago

A discussion forum for the Food Recognition Challenge on AIcrowd.

ImageCLEF 2020 Security - Forged File Discovery

About the ImageCLEF 2020 Security - Forged File Discovery category

5 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ImageCLEF 2020 Tuberculosis - Severity Scoring

About the ImageCLEF 2020 Tuberculosis - Severity Scoring category

5 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

AMLD 2020 - Transfer Learning for International...

About the AMLD 2020 - Transfer Learning for International Crisis Response

5 months ago

AMLD 2020 - Transfer Learning for International Crisis Response

AMLD 2020 - D'Avatar Challenge

About the AMLD 2020 - D'Avatar Challenge category

6 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

NeurIPS 2019 : MineRL Competition

Environment variables

7 months ago

@kashin: That is an artifact left from the transition from crowdAI to AIcrowd. And on all evaluations on AIcrowd, we set both the variables CROWDAI_IS_GRADING and AICROWD_IS_GRADING to True.

Using open-sourced networks

7 months ago

@ijunggeon_isaac_lee: It is not quite clear what you mean here, but if you mean pre-trained networks, then yes.

Submission Entries and Challenge Rules

7 months ago

@hrim: All submissions, irrespective of if they passed or failed, will count towards the submission limit.

[Announcement] Competition Launch Date Moved & Timeline [Update Jun 8th]

8 months ago

@jazzpiano3 : If you are referring to this page : https://www.aicrowd.com/participants/petros_christodoulou

Then, you are right, the message is misleading. The MineRL challenge will show up there as soon as you make a single submission to the competition. But your account has been registered for the competition (and hence the Participate button doesnt show up)

Cheers,
Mohanty

No registration material received

9 months ago

@arpit_garg: We will send out all the resources once the evaluator starts accepting submissions. That is scheduled to happen sometime this week.

Cheers,
Mohanty

Staff

Test Topic for demo

7 months ago

Test Topic

AIcrowd-Logo

This is a test topic.
Please have a look at example_evaluator.py for a sample implementation.

Steps

  • Installation
git clone https://github.com/AIcrowd/aicrowd-example-evaluator
cd aicrowd-example-evaluator
pip install -r requirements.txt

Novartis Pharma Data Science Hackathon

Novartis Pharma Data Science Hackathon category

7 months ago

A discussion forum for the Novartis Hackathon Participants.

EPFL ML Recommender System 2019

About the EPFL ML Recommender System 2019 category

7 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

EPFL ML Road Segmentation 2019

About the EPFL ML Road Segmentation 2019 category

7 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

EPFL ML Text Classification 2019

About the EPFL ML Text Classification 2019 category

7 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

EPFL Machine Learning Higgs 2019

About the EPFL Machine Learning Higgs 2019 category

7 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

NeurIPS 2019 : Disentanglement Challenge

[Announcement] Start of Round-2

7 months ago

@amirabdi: The current computation actually chooses the best submission for you automatically. I havent closely looked at how well they agree with the nominated ones, but if they do not match, your rank would only go down :smiley:

[Announcement] Start of Round-2

7 months ago

@amirabdi: Yes, the ranking algorithm used is the same one as discussed here

[Announcement] Start of Round-2

7 months ago

Dear @all,

Confirming that we are accepting submissions now for Round-2.
The leaderboard for the new submissions is available here

Cheers,
Mohanty

The competition is halted?

7 months ago

@jozerozero @amirabdi: Sorry for the delay. Please refer to this post by Stefan for more details : [Announcement] Start of Round-2

The summary is that we are starting to accept submissions for Round-2 starting 20:00 CET today (September 2nd)

Best of Luck.

Cheers,
Mohanty

I have submitted many time, but nothing happened

8 months ago

Your tags name should begin with submission- for it to be considered by the evaluator as a valid submission. This was an unsaid rule from the beginning, but has only now been enforced.

[Announcement] Leaderboard Computation

8 months ago

[Update]
The leaderboard is now updated with the following process:

  • For all submitted models of the participant we evaluate the average rank across all metrics on the leaderboard
  • We select the model with the highest average rank across all metrics on the leaderboard and call it the selected model.
  • We then remove all the submissions of this participant from the leaderboard, calculate the rankings on the leaderboard without him and repeat the procedure for the n-1 participants.
  • We keep doing this till we have the list of best submissions
  • We then calculate the final ranks for each row on the leaderboard

More details will follow soon as an annoucement.

[End Update]

Dear participants,

The Leaderboard is now being computed as described in the rules of the competition, (instead of just sorting based on FactorVAE as was being done until now.

Now you position on the leaderboard is being computed by the following approach :

  • For all the metrics individually

    • Compute the rank of all the submissions
    • The rank of the submission is defined by the index of the said submission, if all the submissions are sorted in a descending order
  • Then for all the submissions :

    • compute the mean_rank of each submission across all the metrics
  • Generate the leaderboard by :

    • Sorting all the submissions by their mean_rank (in an ascending order), and grouping by the individual participants : meaning only the “best submission” of a participant shows up on the leaderboard.

You can very well check the ranks of the said submission across all the metrics, by going into the “submission details” page (which you can access by clicking on the View button), and checking all the variables of the form _rank.

If there are any questions, or confusions, please feel free to reach out to us on this thread, or you can also optionally reach out to me directly at mohanty@aicrowd.com .

Cheers,
Mohanty

Should the dataset name be kept as cars3d during submission

8 months ago

@sourabh_balgi: Your account would not have write access to push to a new branch on the starter kit repository.
You will have to first fork the repository, then push your changes to a new branch there, and then create a Pull Request to the main starter kit repository.

I have submitted many time, but nothing happened

8 months ago

@lapr_hanabi: Looking at your failure logs, it seems you are not including torch as a dependency. Please follow the instructions here to specify your software runtime correctly.

Should the dataset name be kept as cars3d during submission

8 months ago

@sourabh_balgi: Sure, please do send across a PR. and we can move the discussion to the relevant PR when its ready.

There is a bug in the leaderboard

8 months ago

@jozerozero: We have rolled out the leaderboard computation as described in the rules : The rank of a submission is the average rank of the said submission across all metrics. We are looking closely to ensure that there are no obvious bugs there, and thanks for pointing out this entry, we will review the code again, and get back to you soon.
@arjun_nemani is the point person on this one.

We will also make an official announcement once this has been properly reviewed.

Should the dataset name be kept as cars3d during submission

8 months ago

@sourabh_balgi: No, the aicrowd_helpers.submit() function will just send a signal that the evaluation can begin, and another container calls the local_evaluation.py separately. You will not have to manually call the local_evaluation.py.

And regarding the torch example, as far as I remember, the included code in the repository wasnt using anything fancy, and we will be using a similar local_evaluation.py as included in the repository, but not the exact one. If you need any custom changes in the evaluation script, you will have to send in a pull request, and we can consider it.

What's the exact daily limit on the number of submissions?

8 months ago

@sourabh_balgi: At the moment, this is per person. But we are releasing the Teams feature soon, and details about how the Teams feature affects submission quotas would be announced then.

Should the dataset name be kept as cars3d during submission

8 months ago

@sourabh_balgi: You can change all the files as you please. The evaluator just expects that you do the aicrowd_helpers.submit() call after you are done with dumping the models and the representations in the required format.

I had a look at your latest failues, it seems thats because you have not included disentanglement_lib as a dependency. I pasted the exact error log on the relevant issue.

Should the dataset name be kept as cars3d during submission

8 months ago

@sourabh_balgi : The evaluator sets these environment variables. So your code should not internally override these variables, and can reliably expect the evaluator to set them.

During debugging you could very well check it these variables exist, and accordingly set them so you can use the same pipeline for debugging too.

What's the exact daily limit on the number of submissions?

8 months ago

@sourabh_balgi: You are allowed 5 submissions per day !

Problem with submission in gitlab repo

8 months ago

@sourabh_balgi : Thats correct, we were increasing the capacity of the cluster to allow more parallel submissions, but the Google Kubernetes Service clusters crashed.

We have however fixed the problem and have requeued all the affected submissions. Someone from our team must have posted a message on all the affected issues.

We are currently evaluating 15 submissions in parallel, and the capacity will be increased even more later in the day.

Wrt the count of the quota, we will unfortunately have to count the failed submissions too, else it opens up the system for abuse by participants.

We are keeping a close eye on the evaluations and all the submissions you referenced which failed will eventually be re-evaluated by EOD today.

Which tag in git will you choose for final ranking?

8 months ago

@jozerozero : The leaderboard has a messgae on top at the moment, which says that the current rankings are tentative. And the approach described above will indeed be used to compute the final rankings.
The tentative leaderboard is computed using FactorVAE, but we will soon release the actual leaderboard based on the mean-ranks as described above.
Thank you for your patience.

Cheers,
Mohanty

Which tag in git will you choose for final ranking?

8 months ago

By rank here, what I meant was :

  • For a single metric :
  • take the best submission of each participant
  • Sort all these submissions
  • rank of a participant is defined as the index of the participant in this sorted list
  • Repeat the above for all the metrics
  • Compute the mean of the participant’s rank across all the metrics, and assign it as say mean-R for each participant.
  • Now sort the list of participants based on the mean-R values, and then the order generated (lower the better), is the final leaderboard.

Problems with disentanglement-lib

8 months ago

@esidorovics: Thats because disentanglement-lib==1.2 is not available on PyPi :frowning:

I had checked in with the collaborators from google-research who have access, and it should soon be available on PyPi.
In the meantime, you can change the line in your environment.yml to :

- git+https://github.com/google-research/disentanglement_lib.git

in the pip: section.

Which tag in git will you choose for final ranking?

8 months ago

The final rankings will be computed by computing the rank of each participant across all the individual metrics. And then taking the mean of the rank of the participant across all the metrics. Across all the valid submissions you have made.

[Announcement] Regarding Transfer Learning!

8 months ago

@jriseley : You can include pre-trained weights in your submission, you will just have to ensure to use git-lfs to check in the model weights.

and the second part of the question is more complicated. As long as you use standard pre-trained models to get started in your training phase, I think its still fair.

Resource restrictions for training the submissions

8 months ago

@rauf_kurbanov: The total timelimit for the whole training + evaluation is now 8 hours. This could potentially be increased. But have to check in with the rest of the team.

What changes should one make to let evaluation work?

8 months ago

@siyuzhou: Sorry for the inconvenience. We just pushed a fix for a bug which was potentially the reason for these errors. Also requeued your submission.

What changes should one make to let evaluation work?

8 months ago

@siyuzhou: Weird !! Can you build your code locally and run the built image by following the instructions here : https://github.com/AIcrowd/neurips2019_disentanglement_challenge_starter_kit/blob/master/FAQ.md

It might be much faster to debug that way.

What changes should one make to let evaluation work?

8 months ago

@siyuzhou: The error seems to be tied to the tensorflow version.
I found something https://github.com/tensorflow/datasets/issues/127

The starter kit uses tensorflow-gpu==1.13.1, can you confirm you are using the same ?

Failed builds are being considered in the daily submission limit

8 months ago

@amirabdi: Agree ! But the decision is a site wide policy decision to prevent abuse of the setup by participants (either to game the system to squeeze our more probes, or also in some cases to overload the evaluation system just before the deadline). At the moment, the site wide policy decision will unfortunately stay the same.

But, if you make a case with the organizers, the number of daily submissions allowed could very well be increased if the organizers (who eventually are the compute providers) are onboard !

We are also working on releasing a few features which will allow you to better debug your submission before making the actual submission. Please expect a few updates on the same soon.

Cheers,
Mohanty

What changes should one make to let evaluation work?

8 months ago

@siyuzhou: I pasted the whole error on the relevant issue, but the line of interest seems to be :

2019-07-15T18:30:44.306963517Z AttributeError: 'GFile' object has no attribute 'seekable'

And you should not be adding local_evaluation.py in run.sh that would probably only cause conflicts. If you add the aicrowd_helpers.submit() call, that should trigger the actual evaluation code at our end.

The key idea being, if we trust the local_evaluation.py included, then anyone could very simply modify it to register arbitrary scores. Hence we have the actual evaluation score running in a separate container which computes the score after the training has been done, and the mean representation has been dumped.

Pytorch fail during the evaluation

8 months ago

Also @amirabdi, if you are around now, I am hanging out on the gitter channel here : https://gitter.im/AIcrowd-HQ/disentanglement_challenge

I will be online there for another 2-3 hours, and it might be easier to sort out the exact issue on realtime chat there.

Pytorch fail during the evaluation

8 months ago

@amirabdi: I just pasted the logs from multiple failed issues. It seems you are either not including the dependencies correctly (like tqdm), or you are not using the environment variables to pick up the correct dataset name, etc.

And regarding the point about not counting failed submissions towards maximum daily allowed submissions, that opens up the case where participants would intentionally crash their submissions after receiving the relevant feedback to increase the number of probes they can do on the production evaluator. So that wouldnt be in the interest of all the participants to not count failed submissions in the max daily submissions.

What changes should one make to let evaluation work?

8 months ago

@siyuzhou: Can you please point us to the relevant issue with the failed evaluation ?

Pytorch fail during the evaluation

8 months ago

@amirabdi: I see you have a successful submission here : https://gitlab.aicrowd.com/amirabdi/disentanglement/issues/12 ?
Are you still facing issues making the submission ?

And sorry for the delay in the response. We are adding a few internal processes in place which will ensure you guys have much faster feedback from the evaluation.

Where should the model be saved to be loaded by aicrowd_helpers.submit()

9 months ago

@amirabdi: We do provide the path to save the models here :

And in case of pytorch, they are being picked up from here :

Run.sh is unclear; should it include a call to evlauate as well?

9 months ago

@amirabdi: The run.sh should include a call only to the train_* files.
But the evaluation is triggered by a aicrowd_helpers.submit() call inside the individual train_* files. So please do ensure that after your training is complete, models+representations dumped at the appropriate location, you do include the call to aicrowd_helpers.submit().

Pytorch fail during the evaluation

9 months ago

@Jie-Qiao: That is because you havent pulled in the latest changes in the starter kit, sorry about that.
You can have a look at the latest two commits here : https://github.com/AIcrowd/neurips2019_disentanglement_challenge_starter_kit/commits/master

which add the necessary aicrowd_helpers.submit() calls at the end of the training.

For others facing the same problem, when the evaluation fails after the completion of the training, please do ensure that the aicrowd_helpers.submit() call is present at the end of the training, which triggers the final evaluation of the dumped representations.

Is the final average ranking base on the current leader board?

9 months ago

@Jie-Qiao: Yes, currently the leaderboard is sorted based on factorVAE, but we are soon adding the proper ranking as described in the rules.
The final prizes etc are determined based on the average rank of a submission across all the metrics.

Run.sh is unclear; should it include a call to evlauate as well?

9 months ago

@amirabdi: No just the final aicrowd_helpers.submit() call should trigger the final evalution (which would happen in a separate trusted container).

You will just have to ensure that PYTHON_PATH contains the path of the location where aicrowd_helpers.py is located.

Unable to listen to messages intended for the Oracle. Please contact administrators

9 months ago

@namvi: That is usually because of stray errors during the evaluation. And usually we just re queue the said submission. Please do point us towards the submission, and we would re queue the same.

Is there any way to pass custom Dockerfile to `aicrowd-repo2docker`?

9 months ago

@rauf_kurbanov: You can indeed drop in a Dockerfile at the root, and aicrowd-repo2docker will use it. You will just have to ensure that the default working directory is /home/aicrowd, and the entrypoint of the code is at /home/aicrowd/run.sh, you will also have to ensure the necessary cuda etc packages are available.

Alternatively, to install gcc, you can add a line to apt.txt as done here : https://github.com/AIcrowd/neurips2019_disentanglement_challenge_starter_kit/blob/master/apt.txt#L2

Model error many times

9 months ago

@namvi: I pasted the logs on the issue. But the problem there is you do not have torch included as a dependency in your environment.yml

Lower or higher of metric is good?

9 months ago

@golovanov: This is an interim solution until we have a proper rank computation in place, which takes the mean of the rank of the submission across all the individual metrics.

Using the sum-of-all-metrics to rank the submissions is a bit misleading too, as the distributions of all the individual metrics are quite different, and some of the metrics can heavily dominate the whole sum-of-all-metrics.

Lower or higher of metric is good?

9 months ago

@namvi: That was actually a typo I believe, and it has been fixed in the description. If there are other instances of the same mis-information, please do point us to it, and we will have it fixed.

But as @waleedgondal mentions, in case of all scores, the range of all the metrics is [0,1], and the higher score is better !

Troubles with submitting starter kit as is

9 months ago

@rauf_kurbanov : We noticed a small bug at our end, but have requeued your submission after fixing the bug.

NeurIPS 2019: Learn to Move - Walk Around

The promotion code you entered is not valid - Google Cloud

8 months ago

@Ryan_Amaral: I suspect the solution is to try to redeem the credits at this link : https://console.cloud.google.com/edu
(Instead of the usual /billing/redeem link)

Can you let us know if that works ?
Cheers,
Mohanty

Issue with project = True and obs_as_dict = True

8 months ago

@yhamidullah: This has been fixed. And I see one of your submission has already been evaluated.

Reinforcement Learning

About the Reinforcement Learning category

8 months ago

A familiar place for all Reinforcement Learning enthusiasts to hang out, share stuff, discuss whatever they love.

NeurIPS 2019 : Disentanglement Challenge | Private

About the NeurIPS 2019 : Disentanglement Challenge | Private category

8 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

ICCV 2019: Learning-to-Drive Challenge

About the ICCV 2019: Learning-to-Drive Challenge

8 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Unity Obstacle Tower Challenge

Submissions are stuck

8 months ago

@youngalou: Acknowleding the same. Will check in with the Unity team and decide accordingly.
Cheers,
Mohanty

Submissions are stuck

8 months ago

@youngalou: The submission you reference in your comment above was made at July 16, 2019 09:22 UTC, and according to the rules (and the configuration of the said challenge), the challenge officially ended at July 15, 2019 11:59 p.m. PST which is July 16 6:59am UTC.

Hence according to our records, your submission was made after the official end of the competition.
We will send across the actual submission scores and times to the organizers, and will let Unity Legal make the final determinations according to the Rules of the competition set by them.

Best of luck,
Cheers,
Mohanty

Submissions are stuck

8 months ago

@youngalou: Thats unusual o.O. We had checked that all the submissions during the outage were eventually evaluated (some of them failed because of unrelated errors in the code).

Can you point us to the relevant issues of the submissions which have not been evaluated ?

Submissions are stuck

9 months ago

Update: Some of the evaluations which were broken due to the clog in the image build service have been requeued. Confirming that the rest of the submissions are still safe in the queue, and will slowly be evaluated. Thank you for your patience.
For any specific queries regarding your submissions, you can also reach out to us at : mohanty@aicrowd.com (with devops@aicrowd.com) in cc.

Submissions are stuck

9 months ago

Acknowledging that the Image Building service has been baked. But all the submissions have been received, and we will requeue them manually and ensure that all submissions which were submitted in time do get evaluated.

cc. @shivam

Submissions are stuck

9 months ago

The evaluation system is up and running again. Sorry for the inconvenience.

Submissions are stuck

9 months ago

Acknowledging the outage of the evaluation server. And we are working on having that fixed, and will post an update here as soon as we have more information.

Docker Image Not Building/Running Correctly

9 months ago

@HappySlice: It seems like you did not pass the --no-build parameter, which is pinning particular builds of the packages you are using which are not available on linux.
The correct way to do this would be :

conda env export --no-build > environment.yml

which should result in the dependencies being listed as :

  - python==3.6.8
  - sqlite==3.27.2
  - readline==7.0
  - libffi==3.2.1
  - tk==8.6.8
.... and so on

Why I was blocked from uploading new models for 3 days

9 months ago

@joe_booth: The evalutor internally uses a custom binary which is not public, but @arthurj might share more information about test seeds locally in the publicly available environment.

Aicrowd-repo2docker error

9 months ago

@hanschoi86: But if you can deterministically replicate this, I think I would be curious to figure out exactly whats happening. We can the discussion to PM, and when (if ? :wink:) we can post an update here.

UnityTimeOutException in evaluation

9 months ago

Maybe we should add a FAQ section to the starterkit which could begin as a separate markdown file (linked from the README) as a pull request from you :wink: !

Cheers,
Mohanty

Evaluation perspective config?

9 months ago

@harperj will have to jump in to answer the question. As the perspective etc are controlled by the evaluation binaries internally.
(and if you get a 0 reward, please do ensure that you are not submitting in debug mode)

Docker Image Not Building/Running Correctly

9 months ago

Did you remember to export your conda environment into the environment.yaml file ?
And if so, was PIL installed there ?

UnityTimeOutException in evaluation

9 months ago

Looks like that other submission is on its way towards a timeout, as the agent and the evaluation binary donot seem to be able to communicate. Both are waiting for the first communication. Did you change the default ports of communication etc by any chance ?

UnityTimeOutException in evaluation

9 months ago

@Miffyli: I just requeued the submission and it seems to work : https://gitlab.aicrowd.com/Miffyli/obstacletower-2019/issues/26

This also seems to be some instability in the evaluation binary thats being used. I will follow up with @harperj and @arthurj to see if we can pin point the exact cause.

Thanks,
Mohanty

Why I was blocked from uploading new models for 3 days

9 months ago

Hi @joe_booth,

Apologies for the trouble you had to go through and for the lost time.
We did announce the updates to the aicrowd-repo2docker and the update to the run.sh on the AIcrowd forums, but you are right we could have posted is uniformly across all the communication channels : the github issue trackers and the forums.

Regarding better ways to provide logs back to the participants, we do understand the need for that, and are working on figuring out a better solution. From a technical point of view, it is actually straight forward for us to give you access to the whole build and evaluation logs, which would make the debugging process much easier for you, but that also opens the possibility of participants trying to game the system by leaking information out of the evaluation setup. It might be argued that in a simple reinforcement learning setup like this, it might not be a huge risk, but in some other competitions, it opens up the risk of participants intentionally or unintentionally leaking out the ground truth. With these constraints in mind, we are still working on coming up with a unified solution which finds a good balance between limiting the risk of information leak, while still making debugging easier for users, and are open to your feedback and suggesstions regarding the same.

The question about the mid-competition changes to the evaluation system, those changes are scheduled incremental changes that are site wide and are not tied to the timeline of an individual changes.
Given reproducibility of the solutions is a key goal for us, and we depend on docker base images like nvidia/cuda, it is important to keep incrementally accepting the upstream changes to the stable base image(s), else sooner or later the individual submissions would anyway break and not be reproducible.

If it helps, we might let participants pin their aicrowd-repo2docker version to their submissions, which would allow retrospective builds even when you are not using the most updated version of aicrowd-repo2docker.

Thank you for your patience and your active interest and participation in the competition, and apologies again for the troubles you faced.

Cheers,
Mohanty

Aicrowd-repo2docker error

9 months ago

@hanschoi86: Did you do pip install -U aicrowd-repo2docker ?

Evaluation error:Page not found

9 months ago

@tatsuyaogawa: Please refer to this post to use the debug mode : Announcement: Debug your submissions

[Solved] No module named numpy (or similar)?

9 months ago

@shivam: The source activate base like was a remnant from other competitions where specific env variables needed to be loaded. In this case, I believe removing it should just be fine.

NeurIPS 2019 - Robot open-Ended Autonomous Lear...

Welcome to the NeurIPS 2019 - Robot open-Ended Autonomous Learning!

9 months ago

Hi All,

Because of a few unexpected circumstances, the launch has been delayed a bit. Please expect some communication from our team on this over the next couple of days.

Cheers,
Mohanty

Mapping Challenge

About the Mapping Challenge category

9 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

PlantVillage Disease Classification Challenge

About the PlantVillage Disease Classification Challenge category

9 months ago

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

  • Why should people use this category? What is it for?

  • How exactly is this different than the other categories we already have?

  • What should topics in this category generally contain?

  • Do we need this category? Can we merge with another category, or subcategory?

Research Problems, Datasets, Miscellaneous

About the Research Problems, Datasets, Miscellaneous category

9 months ago

A place to post details about interesting research problems you are working on, or new datasets that you are gathering. A place to reach out to a community of Machine Learning experts and enthusiasts who might help you with your problems and datasets.

mohanty has not provided any information yet.