Submissions are stuck

Both submissions failed after 9hrs with the message Task exceeded maximum timeout value (32400 seconds)

I tried submitting again, it has not gone into image_build mode - https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/147

I looks like submissions are totally down

View post 0

End time / date is wrong

In the rules/competition overview it states that the Round 2 end date is: 11:59 pm, Monday, July 15th

However, on the competition ticker, it states Jul 15th, 7:59 UTC which would be 11:59 pm, Sunday, July 14th. - it should be Jul 16th, 7:59 am UTC

image.png1354×1114 98.8 KB

image.jpg780×486 112 KB

View post 0

Submissions are stuck

I submitted earlier today and they have been stuck in image build mode for 3hrs:

image.png1340×456 61.4 KB

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/145

and

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/146

View post 0

Why I was blocked from uploading new models for 3 days

@Miffyli - which platform are you running on? I can not get it working in MacOS and from what I read, docker does not support --network=host on Mac

View post 0

Why I was blocked from uploading new models for 3 days

it’s when I try and run it local per these steps: https://github.com/Unity-Technologies/obstacle-tower-challenge#run-docker-image

it used to work, but not since the update of aicrowd-repo2docker - @arthurj, does it work for you guys on Mac?

View post 0

Why I was blocked from uploading new models for 3 days

Thanks @mohanty! One problem I’m still having is running the test seeds locally (100-105) - when I invoke the environment docker it ends. I’m trying on MacOS

View post 0

Why I was blocked from uploading new models for 3 days

I’ve spent countless hours over the past 3 days trying to figure out why I could not upload/evaluate a new model nor reproduce the problem locally.

Basically, there is a breaking change whereby existing code will no longer run server-side - It would have saved me hours had there been a better error and/or someplace to look for notifications (I don’t think it helps to have 2 unity repros with issue trackers as well as the ai-crowd message board)

• June 13th, was my last successful upload of a model.
• On July 5th I tried to upload a new model - the only change to my code base the addition of the model and a reference to that model.
• When aicrowd-bot posted the log: it just said this:

2019-07-06T07:13:55.1056876Z root
2019-07-06T07:13:55.129380334Z Traceback (most recent call last):
2019-07-06T07:13:55.12943787Z   File "run_evaluation.py", line 7, in <module>
2019-07-06T07:13:55.129471922Z     import gym
2019-07-06T07:13:55.129478407Z ModuleNotFoundError: No module named 'gym'

• I learned that we can now do debug submits: Announcement: Debug your submissions however, all it gave was the same log
• I tried to reproduce locally, however, the build.sh script was giving me the error: AttributeError: /srv/conda/bin/python: undefined symbol: archive_errno
• I thought maybe a conda or pip package update may have broken something so I manually tied each one to the valid version from June 13th
• I thought there may be some local issue with my docker, so I cleaned, deleted, reset
• I ran pip install --upgrade aicrowd-repo2docker and saw that it updated. This solved my local issue and was able to reproduce the server side error.
• Given that aicrowd-repo2docker had been updated, i thought to look at the commit logs and found that this https://github.com/Unity-Technologies/obstacle-tower-challenge/commit/99c68faf2ed0f01ee8bc3e411bbdd4e85484a733 removed source activate base from run.sh

Note: I still can not test locally - the agent code runs, but the environment docker immediately drops out. I also have to manually delete the docker image to force it to rebuilt (this was not the case prior to the aicrowd-repo2docker upgrade

View post 0

Is the new v2.2 used for scoring?

Looks good! It would be great to have the videos fixed so we can see @unixpickle 's agent through level 16!!

View post 0

Is the new v2.2 used for scoring?

oh strange - it recorded an average floor of 6, then removed it when it restarted training

View post 0

Is the new v2.2 used for scoring?

It looks like the video worked and it updated the leaderboard but then it started scoring again.

View post 0

Is the new v2.2 used for scoring?

@mohanty - it failed - it looks like it gets stuck when recording the video and then times out

View post 0

Is the new v2.2 used for scoring?

I tried again - this time i get ‘Unable to orchestrate submission, please contact Administrators.’

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/124

View post 0

Is the new v2.2 used for scoring?

@mohanty - Have you been able to update v2.2? I resubmitted my agent but got errors:

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/123

View post 0

UnityTimeOutException in evaluation

@mohanty - would you post the logs on my submission here - https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/121 - I’m not sure if it is the same problem. thanks

View post 0

Possible bug with tower-seed in env 2.0

could it be related to this issue: https://github.com/Unity-Technologies/obstacle-tower-env/issues/91#issuecomment-494218846

they pushed a fix for this

View post 0

Has the submission process changed for round 2?

I tried re-submitting my agent from round #1 to see how it did on round #2 - but get the following error

did something change? (all i did was push a text change and re-tag my repro)

   The following containers terminated prematurely. : agent
    Please contact administrators, or refer to the execution logs.

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/119

View post 0

Multiple dead submissions

Hi @mohanty - thanks for pointing this out - sorry I didn’t see this until this evening and so had continued to push but will back off per you points.

I think it’s great that you extended the timeline - it gives us the time to take more risks with experimental approaches.

View post 0

Multiple dead submissions

OK - thanks! as a rule, should I try and limit the number of concurrent submissions?

I know statistically that my current agent can score 10 on each seed so if I submit it enough times then at some point it should score that.

View post 0

Multiple dead submissions

@mohanty - I think I may have overloaded the system last night - I was trying to trigger multiple submissions so that I have them all queue up. Two show as Failed, Eight show as ImageBuildStarted after 15 hrs.

I’m not sure if there is anything I can do on my end

image.png2526×1318 416 KB

View post 0

(Win10) ModuleNotFoundError: No module named 'pwd'

I also tried aicrowd-repo2docker . and received the same error

View post 0