BUG in evaluation timing

Hello @dami

Thanks for reporting this issue. It is a minor glitch on our side that showed the wrong end time on the issue page. We released a fix for this and future submissions should show the correct end time. Please note that these time stamps are not used for timeouts.

View post 1

[Resolution] Bugs With Getting Started Of Round 2

@gaurav_singhal We updated the dataset file names and the extracted directory names. The zip files should extract debug, training, unlabelled and validation directories. Also updated the colab notebook with the new dataset paths.

Thanks for your help!

View post 1

[Resolution] Bugs With Getting Started Of Round 2

@gaurav_singhal The updated notebook already has some code changes in the magic box cell on the lines of your suggested changes. Sharing the link to the updated notebook just in case,

View post 1

[Resolution] Bugs With Getting Started Of Round 2

Hello @gaurav_singhal

Thanks for pointing these out.

We will update the file names to match the names shown in the listing.

Regarding the colab notebook part, are you using the notebook that was released for round 1? We have an updated notebook for round 2. The updated notebook should have all these fixes along with additional code needed to run the post_purchase_training_phase. Can you try using the new notebook in case if you haven’t yet?

View post 0

L2R Docker container

Hello @nandan_tumu

Unfortunately, aicrowd/learn-to-race:base is not available publicly and only works when you submit your code to AIcrowd.

The base image is based on ubuntu and has l2r repo along with the simulator. The image is not released publicly as it has the data need for VegasNorthRoad circuit (the private track to be used for next round).

You can specify any packages that need be installed using apt-get in the apt.txt file.

Example: apt.txt · master · Learn to Race / l2r-starter-kit · GitLab

The image has conda pre-installed, so you can also specify any conda commands that you want to run.

Example: Dockerfile · master · Learn to Race / l2r-starter-kit · GitLab

Finally, you can specify the pip packages in the requirements.txt file.

Example: requirements.txt · master · Learn to Race / l2r-starter-kit · GitLab

You can update the Dockerfile in your repo as you seem fit except for the base image part (the first line of the Dockerfile). In case you are facing any issues in setting up the runtime dependencies during the evaluation, please feel to post on the forums or reach out to us.

View post 0

DL Frameworks supported during evaluation

Hello @max333

Is there some default CUDA installed? Or we have to modify this file to get any CUDA?

We do not have CUDA installed by default. You can install the version that you need by uncommenting the lines in the Dockerfile as mentioned in the above reply. Please feel free to reach out to us or the organizers if you need any further help with setting this up.

I guess also the torch version in requirements.txt should be changed to a CUDA version.

Yes, you can update the requirements.txt based in your need.

Also is there some info on the server hardware (GPU, number of cores).

The evaluations run on AWS g4dn.xlarge nodes. They have 4 vCPUs, 16 GB RAM and 1x Nvidia Tesla T4 GPU.

View post 2

DL Frameworks supported during evaluation

Hello @rsakthivel

You can choose any framework. The easiest way to do this is to specify the framework/library in your requirements.txt file.

If you need to choose a CUDA version, you can uncomment one of these lines based on your requirement.

You can also edit the Dockerfile as you see fit. However, the base image needs to be aicrowd/learn-to-race:base.

View post 1

KeyError: ‘success_rate’

Hello all!

The way we register lap wise metrics had a bug that was causing the evaluator to skip registering a few metrics if the lap was completed by the end of the first episode. This issue is fixed and all the effected submissions were re-evaluated. Please let us know if you are still facing this issue or any of your submissions did not get re-evaluated.

View post 1

KeyError: ‘success_rate’

@denis9 @boliu0 Thanks for reporting this. Can you check if this fixes the issue?

If this commit doesn’t fix the issue, it would help us pinpoint the problem if you can share the traceback for when this exception is raised.

View post 1

How to run the simulator on Mac OS using Docker?

Hello @themaroonknight

The base docker image mentioned in the dockerfile is a protected image that contains a few evaluation tracks. Unfortunately, you can’t use it to build the image locally and only works on evaluation servers.

The simulator needs an Nvidia graphics card to run and the simulator binaries are built for Linux. So running it on Mac is not feasible at this point. However, as @siddha_ganju mentioned, you can start an EC2 instance on AWS (g4dn.xlarge instance) and train your agents.

View post 1

Hit a connection error, exiting silently

Hello @tahir_javed_cs20d407

Can you give us a few submission IDs so that we can debug this further on our end?

View post 0

Hit a connection error, exiting silently

Hello @tahir_javed_cs20d407

This happens when the evaluation times out (2 hours is the current timeout) or the evaluation takes up too much of memory.

View post 0

Why is the evaluation failing at evaluation rollout phase?

Hello @utsav_dey_cs20s009

The issue should be fixed now. We are re-evaluating the effected submissions.

View post 0

Evaluation failing at last stage

Hello @tahir_javed_cs20d407

The issue should be fixed now. We are re-evaluating the effected submissions.

View post 1

Evaluation failing at last stage

Hello @tahir_javed_cs20d407

Thanks for pointing this out. We are looking into the issue and will soon fix it.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Hello @jon_grantham

is that without considering time spent in the agent?

Yes, it is without considering the time spent in the agent.

How much of that is network latency? I.e., if we have an agent capable of performing faster than that, does all of the agent compute time get hidden under the latency?

This includes the network latency, processing delay and everything that is needed to send a request and get the response. Any latency/compute time on the agent will be added to this.

For example, if you have something like

import aicrowd_gym
from tqdm import trange

env = aicrowd_gym.make("NetHackChallenge-v0")
env.reset()
for _ in trange(1000000):
    _, _, done, _ = env.step(1)
    if done:
        env.reset()

This should give you a throughput of 1500-2000 iterations per second during the evaluation.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Hello @jon_grantham

The evaluations run on AWS EC2 instances. The resources available are as follows

Note:
During the evaluations, you will receive a proxy NetHack env object instead of the actual environment. This proxy object talks to the actual NetHack env over the network and returns the values as needed. We do this to prevent participants from tampering with the env. This also adds an overhead. Based on our benchmarks, a single env should roughly give a throughput of 1500-2000 steps/second. Using something like a batched env increases the throughput.

Hope this helps and please feel free to reach out to us for any help.

View post 2

Iterating over Dict spaces in aicrowd-gym

Hello @anssi

Thanks for sharing this with us.

• Allow access to observation/action space variables that exist in normal Gym (e.g. spaces for Dict obs)

We will soon update the evaluator to allow access to a few more attributes. At the moment the following attributes/methods can be accessed.

On action space:

(
    "contains",
    "dtype",
    "sample",
    "shape",
    "n",
)

On observation space:

(
    "bounded_above",
    "bounded_below",
    "contains",
    "dtype",
    "high",
    "is_bounded",
    "low",
    "sample",
    "shape",
    "n",
)

• This is perhaps result of bad programming from my end, but make sure variables that do not exist in the environment return appropriate exceptions. For example, I did hasattr("something", env). something was not a variable in the environment, but with aicrowd-gym it was set to None. This caused hasattr to return True and subsequently things failing.

I believe this happens with gym.Env objects as well. For example, if I do

import gym
env = gym.make("CartPolve-v0")
setattr(env, "something", "some value")

it doesn’t return any error. If possible, can you share with a simple example the expected behaviour?

Note: During the evaluation, the env object you have access to is a proxy object and not the actual gym.Env instance. So setting attributes from your code will not set any attributes on the actual env object. If you have any specific use case for doing this, please feel free to reach out to us and we can try to accommodate your use case.

Edit: Sorry, misread the last part. We will try to get the read access to all the env attributes in our future releases.

View post 1

Can I use GPU for prediction?

Hello @woosung_choi @Li_Xue_Han

Sorry, my bad. GPUs are not available for this competition. Apologies again for the confusion.

View post 4

Can I use GPU for prediction?

Hello @Li_Xue_Han

Edit: GPUs are not available for this competition

You can use GPUs for inference. Please refer How to enable GPU for your submission? for more information on using GPUs for your submission.

View post 0