Badges
Activity
Challenge Categories
Challenges Entered
Completed
MineRL Labs - Carnegie Mellon University
Sample-efficient reinforcement learning in Minecraft
Latest submissions
Unity Technologies
A new benchmark for Artificial Intelligence (AI) research in Reinforcement Learning
Latest submissions
See All| graded | 6976 | ||
| failed | 6923 | ||
| failed | 6913 |
| Participant | Rating |
|---|
| Participant | Rating |
|---|
Unity Obstacle Tower Challenge
Evaluation is not triggered
Over 6 years agoThanks for looking into it! Issues are being created but unfortunately they seems to be stuck in the queue, e.g https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/32
Evaluation is not triggered
Over 6 years agoI just made a change and push a tag again but still no new issues show up
New commit: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/commit/c3d21d8b726bf871b0462de145076724d6b52a41
New tag: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/tags/submission-20190610-01
Evaluation is not triggered
Over 6 years agoHi @mohanty, the evaluation is not being triggered after pushing a new commit with a tag associated with that new commit.
- https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/tags/submission-v0.1
- https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/tags/submission-20190608-02
Is there anything I am missing? Can you please help manually triggering the evaluation for the latest tag? Thank you!
[Admin] Good News for Dopamine Lovers!
Almost 7 years agoAnother trick I found is to override agent._replay_load like below to prevent Dopamine from loading the replay buffer. The buffer isn’t necessary for evaluation anyway and you don’t have to commit huge files such as $store$_observation_ckpt.*.gz
agent._replay.load = lambda _1, _2: True
Starter kit stuck "pending" state for a day
Almost 7 years ago@mohanty So I have finally successfully made my first submission (that is not a random agent)! 
The one important change I made is to defer importing RainbowAgent and tensorflow. See this commit here https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/commit/4979d7e65de6012e92405542a1ab73ac6ea16cb4. The evaluation ran successfully after this change.
I suspect that you guys might have a race condition when the agent (run.sh) and the environment (env.sh) are launched in docker. Importing RainbowAgent and tensorflow usually a bit of time (like a few seconds) and they might cause env.sh to try to listen to the port first before the env is ready in run.py.
I can replicate this issue locally as I always have to wait for Start training by pressing the Play button in the Unity Editor message to show up in run.py before I launch env.sh. Otherwise the environment will time out.
You can probably replicate this issue by trying to put time.sleep(10) in the very beginning for run.py.
While deferring import works for me for now, it is not ideal. So it will be great if you guys could look into this issue. And please let me know if there is any further information you’d like me to provide.
Thank you!
Number of available submissions
Almost 7 years agoThank you for raising the daily submission limit to 20.
Can the admin please clarify if there is a limit of total number of submissions for round 1 and round 2? Thanks!
Starter kit stuck "pending" state for a day
Almost 7 years agoHi @mohanty unfortunately my last submission from last week was still failing. Can you please take a look at the issue https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/18?
It’s working in my local machine but the submission somehow failed. Thank you!
Starter kit stuck "pending" state for a day
Almost 7 years agoI just tried now. I am still getting this error:
Submission failed : The participant has no submission slots remaining for today.
Starter kit stuck "pending" state for a day
Almost 7 years ago@mohanty That’s good news. But it seems I have used up my quota today just to test out the submission 
Is it possible to reset my quota for today? Thanks!
Starter kit stuck "pending" state for a day
Almost 7 years agoHi @mohanty I think I have isolated the problem to enabling GPU during evaluation. I made a new submission (issue #10) based off the obstacle-tower-challenge master from GitHub and the only modification is in aicrowd.json having gpu: true.
Is GPU not supported during evaluation or is this a bug? Let me know if you need more details in reproducing this issue.
Thank you!
Starter kit stuck "pending" state for a day
Almost 7 years agoHi @mohanty, it appears my new submission it’s stuck at pending evaluation. It seems the error message this time was something to do with timeout with Gitlab.
Another question do you know if failed evaluation counts towards the 100 submission quotas?
Starter kit stuck "pending" state for a day
Almost 7 years agoHi there, I am having trouble with my pod being stuck the in “pending” state. My submission was just the starter kit without any modifications.
Here’s a link to the issue: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/1 and here’s aicrowd-bot says
---------------------------------------------
##### Pod Health :heart:
---------------------------------------------
- *Pod State* : `Pending`
- *Pod Scheduling Time* :clock1: : `26150 secs`
- *Containers Waiting* :large_orange_diamond: : `redis-server`,`aicrowd-subcontractor`
Thank you!
Evaluation is not triggered
Over 6 years agoHi @mohanty, I am running into a different problem which my submission is stuck at the “Generating Video” stage for more than 5 hours. https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/41 is there anything I am doing incorrectly?