Activity
Ratings Progression
Challenge Categories
Challenges Entered
A benchmark for image-based food recognition
Latest submissions
ASCII-rendered single-player dungeon crawl game
Latest submissions
See Allgraded | 163439 | ||
graded | 162868 | ||
graded | 160769 |
Participant | Rating |
---|
Participant | Rating |
---|
-
AutoAscend NeurIPS 2021 - The NetHack ChallengeView
NeurIPS 2021 - The NetHack Challenge
Are the specs for the machine that the evaluations are run on available anywhere?
About 3 years agoHello @jyotish,
If the throughput is in 1500-2000 range, it indicates that the maximum average number of steps is only {1500--2000} * 0.5 * 3600 / 128 = 21k -- 28k
per game.
Also a note, that the example you gave doesnβt really take into account the environment step delay, because env.step(1)
(1 is CompassDirection.E
) is a no-op after a few steps when the character hits the wall (the turn counter stops to tick after that).
28k steps per game is an extremely tight limit if one were to go for ascension (we are!). Assuming that we need 200k turns for ascension, which can be roughly equivalent to 400k steps, indicates that we have to early drop at least 93% of all games. We already have >40k steps in average, and have to do hacks to maximize the median score, like quitting after exceeding the median (in fact Panic
team does the same judging from the leaderboard).
Of course the assumption that agent takes no time to execute is not realistic. In our case environment takes ~15% of the entire execution time (measured locally so thereβs no communication delay).
The competition goal is to develop the best agent, but right now itβs more like an performance optimization problem instead. Citing the challenge motivation: βThe only restriction is on the compute and runtime during evaluation, though these will be set to very generous limits to support a wide range of possible implementationsβ. Currently the compute limit is far from being βvery generousβ and to do this, we believe the time limit should be increased at least 10 times.
So, we encourage you to increase the time limit as much as possible to give participants a better chance to beat the game. With the current limit ascension is extremely unlikely, but itβs quite possible if the limit is increased.
Evaluate rollouts timed out
About 3 years agoAdmins responded to me that the video recording timed out and they will release a fix for it soon.
Evaluate rollouts timed out
About 3 years agoI got a timeout on βEvaluate Rolloutsβ step. Iβm not sure what the step is exactly doing, but I guess it doesnβt involve running the agent. If thatβs the case, can you increase time limit for this step?
What does this step do and why does it take so long? Some of our episodes are quite long (>100k steps), maybe thatβs the reason?
π§ Requesting Feedback and Suggestions
About 3 years agoHey @dipam,
I just gave some feedback about tight time limits in this comment.