Hit a connection error, exiting silently

Hello @tahir_javed_cs20d407

Can you give us a few submission IDs so that we can debug this further on our end?

View post 0

Hit a connection error, exiting silently

Hello @tahir_javed_cs20d407

This happens when the evaluation times out (2 hours is the current timeout) or the evaluation takes up too much of memory.

View post 0

Why is the evaluation failing at evaluation rollout phase?

Hello @utsav_dey_cs20s009

The issue should be fixed now. We are re-evaluating the effected submissions.

View post 0

Evaluation failing at last stage

Hello @tahir_javed_cs20d407

The issue should be fixed now. We are re-evaluating the effected submissions.

View post 1

Evaluation failing at last stage

Hello @tahir_javed_cs20d407

Thanks for pointing this out. We are looking into the issue and will soon fix it.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Hello @jon_grantham

is that without considering time spent in the agent?

Yes, it is without considering the time spent in the agent.

How much of that is network latency? I.e., if we have an agent capable of performing faster than that, does all of the agent compute time get hidden under the latency?

This includes the network latency, processing delay and everything that is needed to send a request and get the response. Any latency/compute time on the agent will be added to this.

For example, if you have something like

import aicrowd_gym
from tqdm import trange

env = aicrowd_gym.make("NetHackChallenge-v0")
env.reset()
for _ in trange(1000000):
    _, _, done, _ = env.step(1)
    if done:
        env.reset()

This should give you a throughput of 1500-2000 iterations per second during the evaluation.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Hello @jon_grantham

The evaluations run on AWS EC2 instances. The resources available are as follows

Note:
During the evaluations, you will receive a proxy NetHack env object instead of the actual environment. This proxy object talks to the actual NetHack env over the network and returns the values as needed. We do this to prevent participants from tampering with the env. This also adds an overhead. Based on our benchmarks, a single env should roughly give a throughput of 1500-2000 steps/second. Using something like a batched env increases the throughput.

Hope this helps and please feel free to reach out to us for any help.

View post 2

Iterating over Dict spaces in aicrowd-gym

Hello @anssi

Thanks for sharing this with us.

• Allow access to observation/action space variables that exist in normal Gym (e.g. spaces for Dict obs)

We will soon update the evaluator to allow access to a few more attributes. At the moment the following attributes/methods can be accessed.

On action space:

(
    "contains",
    "dtype",
    "sample",
    "shape",
    "n",
)

On observation space:

(
    "bounded_above",
    "bounded_below",
    "contains",
    "dtype",
    "high",
    "is_bounded",
    "low",
    "sample",
    "shape",
    "n",
)

• This is perhaps result of bad programming from my end, but make sure variables that do not exist in the environment return appropriate exceptions. For example, I did hasattr("something", env). something was not a variable in the environment, but with aicrowd-gym it was set to None. This caused hasattr to return True and subsequently things failing.

I believe this happens with gym.Env objects as well. For example, if I do

import gym
env = gym.make("CartPolve-v0")
setattr(env, "something", "some value")

it doesn’t return any error. If possible, can you share with a simple example the expected behaviour?

Note: During the evaluation, the env object you have access to is a proxy object and not the actual gym.Env instance. So setting attributes from your code will not set any attributes on the actual env object. If you have any specific use case for doing this, please feel free to reach out to us and we can try to accommodate your use case.

Edit: Sorry, misread the last part. We will try to get the read access to all the env attributes in our future releases.

View post 1

Problems with starter kit setup

Hello @dylan_miller

It looks like one of your apt repositories is giving the error. Kitware also made cmake installable using pip. You can run the following to install the latest version of cmake from kitware.

pip install -U cmake

Reference: https://pypi.org/project/cmake/

View post 1

Can I use GPU for prediction?

Hello @woosung_choi @Li_Xue_Han

Sorry, my bad. GPUs are not available for this competition. Apologies again for the confusion.

View post 4

Can I use GPU for prediction?

Hello @Li_Xue_Han

Edit: GPUs are not available for this competition

You can use GPUs for inference. Please refer How to enable GPU for your submission? for more information on using GPUs for your submission.

View post 0

"pod deleted during operation" during generating predictions

Hello @Piteryo

The evaluation ran out of memory. Can you check how much RAM is being used when you run this locally?

Here is a screenshot of the RAM usage for reference.

Screenshot 2021-06-20 at 6.28.09 PM1326×510 65.2 KB

View post 0

Failed at "Build Image" with no logs

Hello @kim_min_seok

Thanks for notifying us about this. This was an issue on our end and it is fixed now. We queued the effected submissions for re-evaluation.

View post 3

Colab Notebook (baseline submission)

Hello @charel_van_hoof

You probably want to post this in https://discourse.aicrowd.com/c/neurips-2021-nethack-challenge/

View post 0

Discussion on SiamMOT baseline

Hello @octo

Replied on the thread. Let us know if you need any further help on this

View post 1

Getting error in siam-mot baseline submission "pack exceeded maximum allowed size"

Hello @paulids

You need to use Git LFS to push the large files in your repository (your saved models, etc.,). You can find more information on how to use LFS here

View post 0

AssertionError: Incorrect prediction directory structure. View the submission for more details

Hello @SenticLab.UAIC

It looks like # Prediction phase heading is missing in your notebook. Please add this heading after your training code block has ended. We use these to figure out which part of the notebook needs to be executed during evaluation.

For more information please refer this discussion.

View post 1

Dataset: transport endpoint not connected

Hello @e_t

Can you please drop a mail to servicedesk@aridhia.com regarding this?

View post 0

Submission Error : 'DATASET_PATH' not recognised as an internal or external command

Hello @jigsaw

Can you share any output that you might have got after running the command? Do you see something like

[NbConvertApp] Executing notebook with kernel: python

when you run the command?

View post 0

Submission Error : 'DATASET_PATH' not recognised as an internal or external command

Hello @jigsaw

Can you try this?

View post 0