Episode in evaluation ends prematurely

My latest evaluation yielded this:

• Episode-5
  • State: Episode Complete
  • Floor Number: 2
  • Reward: 2.400
  • Steps: 649

I find this rich, considering that there is no way to die on the third floor without running out of time, and an agent starts out with 3000 frames of time even if they pick up no time orbs. Clearly, there is some bug in the environment or the evaluation.

Anyway, I just resubmitted and will hopefully not hit this bug again.

View post 0

Why I was blocked from uploading new models for 3 days

I had the same ModuleNotFoundError as you. I followed the solutions in

• [Solved] Build troubles with the new aicrowd-repo2docker?
• [Solved] No module named numpy (or similar)?

View post 2

Is the new v2.2 used for scoring?

Same issue here, stuck on generating video. Maybe that feature should simply be removed until it can be implemented reliably.

View post 0

UnityTimeOutException in evaluation

Spoke too soon. Just did another submission and got another timeout after 998s seconds (my timeout was set to 900). Must be non-deterministic.

View post 0

UnityTimeOutException in evaluation

@mohanty looks like your suggestion worked! Submission runs now.

View post 0

UnityTimeOutException in evaluation

I see this stack trace when I try to submit an agent that worked previously:

2019-05-24T17:42:52.825782788Z root
2019-05-24T17:43:05.17870298Z INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
2019-05-24T17:43:35.184856349Z Traceback (most recent call last):
2019-05-24T17:43:35.184901058Z   File "run.py", line 56, in <module>
2019-05-24T17:43:35.184908019Z     env = create_single_env(args.environment_filename, docker_training=args.docker_training)
2019-05-24T17:43:35.184929282Z   File "/home/aicrowd/util.py", line 16, in create_single_env
2019-05-24T17:43:35.184932961Z     env = ObstacleTowerEnv(path, **kwargs)
2019-05-24T17:43:35.184935919Z   File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 45, in __init__
2019-05-24T17:43:35.184939382Z     timeout_wait=timeout_wait)
2019-05-24T17:43:35.184942214Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/environment.py", line 69, in __init__
2019-05-24T17:43:35.184945802Z     aca_params = self.send_academy_parameters(rl_init_parameters_in)
2019-05-24T17:43:35.184948806Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/environment.py", line 491, in send_academy_parameters
2019-05-24T17:43:35.184952019Z     return self.communicator.initialize(inputs).rl_initialization_output
2019-05-24T17:43:35.184954878Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/rpc_communicator.py", line 80, in initialize
2019-05-24T17:43:35.184958142Z     "The Unity environment took too long to respond. Make sure that :\n"
2019-05-24T17:43:35.184963164Z mlagents_envs.exception.UnityTimeOutException: The Unity environment took too long to respond. Make sure that :
2019-05-24T17:43:35.184968225Z 	 The environment does not need user interaction to launch
2019-05-24T17:43:35.184972965Z 	 The Academy and the External Brain(s) are attached to objects in the Scene
2019-05-24T17:43:35.184977708Z 	 The environment and the Python interface have compatible versions.

I am definitely using v2.1 of the environment. Not sure what’s going on, but this happened several submissions in a row, and I see that nobody else has successfully submitted in a few days.

View post 0

Track submissions to the leaderboard

I created a script that tracks new submissions to the leaderboard and logs them to the console. Watch as people gradually all realize they can submit the Rainbow baseline. xD

import itertools
import time

from bs4 import BeautifulSoup
import requests

BASE_URL = 'https://www.aicrowd.com/challenges/unity-obstacle-tower-challenge/leaderboards'


def main():
    board = fetch_leaderboard()
    while True:
        time.sleep(60 * 5)
        new_board = fetch_leaderboard()
        print_diffs(board, new_board)
        board = new_board


def print_diffs(old, new):
    for k, v in new.items():
        if k not in old:
            print('new submission: %s -> %f' % (k, v))
        elif old[k] != v:
            print('new submission: %s -> %f (old: %f)' % (k, v, old[k]))
    for k, v in old.items():
        if k not in new:
            print('deleted submission: %s -> %f' % (k, v))


def fetch_leaderboard():
    result = {}
    for page in itertools.count():
        sub_result = fetch_leaderboard_page(page)
        if not len(sub_result):
            break
        result.update(sub_result)
    return result


def fetch_leaderboard_page(page):
    url = '%s?page=%d' % (BASE_URL, page)
    response = requests.get(url)
    data = response.text
    soup = BeautifulSoup(data, 'html.parser')
    table = soup.find('table', {'class': 'table-leaderboard'})
    result = {}
    for row in table.find('tbody').find_all('tr'):
        columns = row.find_all('td')
        result[columns[2].get_text().strip()] = float(columns[4].get_text().strip())
    return result


if __name__ == '__main__':
    main()

View post 2