Has filled their profile page
Hi @mohanty, I am running into a different problem which my submission is stuck at the “Generating Video” stage for more than 5 hours. https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/41 is there anything I am doing incorrectly?
Thanks for looking into it! Issues are being created but unfortunately they seems to be stuck in the queue, e.g https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/32
I just made a change and push a tag again but still no new issues show up
New commit: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/commit/c3d21d8b726bf871b0462de145076724d6b52a41
New tag: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/tags/submission-20190610-01
Hi @mohanty, the evaluation is not being triggered after pushing a new commit with a tag associated with that new commit.
Is there anything I am missing? Can you please help manually triggering the evaluation for the latest tag? Thank you!
Another trick I found is to override
agent._replay_load like below to prevent Dopamine from loading the replay buffer. The buffer isn’t necessary for evaluation anyway and you don’t have to commit huge files such as
agent._replay.load = lambda _1, _2: True
@mohanty So I have finally successfully made my first submission (that is not a random agent)!
The one important change I made is to defer importing
tensorflow. See this commit here https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/commit/4979d7e65de6012e92405542a1ab73ac6ea16cb4. The evaluation ran successfully after this change.
I suspect that you guys might have a race condition when the agent (run.sh) and the environment (env.sh) are launched in docker. Importing
tensorflow usually a bit of time (like a few seconds) and they might cause
env.sh to try to listen to the port first before the env is ready in run.py.
I can replicate this issue locally as I always have to wait for
Start training by pressing the Play button in the Unity Editor message to show up in
run.py before I launch
env.sh. Otherwise the environment will time out.
You can probably replicate this issue by trying to put
time.sleep(10) in the very beginning for
While deferring import works for me for now, it is not ideal. So it will be great if you guys could look into this issue. And please let me know if there is any further information you’d like me to provide.
Thank you for raising the daily submission limit to 20.
Can the admin please clarify if there is a limit of total number of submissions for round 1 and round 2? Thanks!
Hi @mohanty unfortunately my last submission from last week was still failing. Can you please take a look at the issue https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/18?
It’s working in my local machine but the submission somehow failed. Thank you!
I just tried now. I am still getting this error:
Submission failed : The participant has no submission slots remaining for today.
Hi @mohanty I think I have isolated the problem to enabling GPU during evaluation. I made a new submission (issue #10) based off the
obstacle-tower-challenge master from GitHub and the only modification is in
Is GPU not supported during evaluation or is this a bug? Let me know if you need more details in reproducing this issue.
Hi @mohanty, it appears my new submission it’s stuck at pending evaluation. It seems the error message this time was something to do with timeout with Gitlab.
Another question do you know if failed evaluation counts towards the 100 submission quotas?
Hi there, I am having trouble with my pod being stuck the in “pending” state. My submission was just the starter kit without any modifications.
Here’s a link to the issue: https://gitlab.aicrowd.com/felixlaumon/obstacle-tower-challenge/issues/1 and here’s
--------------------------------------------- ##### Pod Health :heart: --------------------------------------------- - *Pod State* : `Pending` - *Pod Scheduling Time* :clock1: : `26150 secs` - *Containers Waiting* :large_orange_diamond: : `redis-server`,`aicrowd-subcontractor`