Intermittent UNAUTHENTICATED: Session expired (timeout: 120s) across all games (local + remote)

The sessions are ideally supposed to reauthenticate and continue normally. Can you share with us some script using which we can reproduce this error?

View post 0

Supemario, game screenshot works?

Hello @RickySong @ChoiSoojin,

On the evaluation server, we run the games using the same scripts as the ones in the starter kit. To keep the behavior consistent, we updated the starter kit to run in headless mode as well.

If you’d like to enable visualizations again, you can switch the environment creation line in super_mario_env.py (around line 217):

to:

self.env = gym_super_mario_bros.make(
    'SuperMarioBros-1-1-v1',
    render_mode='human',
    apply_api_compatibility=True
)

View post 0

Session cannot get data including mcp_urls

Hello @ns601023

It seems like you are using an older version of the starter kit. Can you please pull the new changes and try again?

git pull origin master --rebase

View post 1

Pokemon map is broken?

@ilya_gusev is there a particular submission you are referring to that I can check?

We didn’t change anything specific to pokemon and wouldn’t expect this change. Maybe the game hasn’t started yet? The game ROM we received from Krafton team starts at the menu screen and the “Map on Screen” wouldn’t be defined for that screen.

Is there something I can cross check on the server logs or can you give the exact steps to replicate this issue so that we can pass this to Krafton team?

View post 0

Starcraft submission

@mikhail1 @cheong_wei_xun

can you please take a look at this

View post 0

Model submission

While running remote mode, it seems like the Star Craft failed after entering the second episode, is it possible that there’s a problem with the code in the MCP server or game envs of AIcrowd?

The issue was due to requests getting queued and dequeued arbitrarily at MCP server. We moved to a gRPC based implementation to get around this issue.

Please pull the recent changes made to starter kit and let us know if you still run into any issues.

long will the evaluation process usually take?

It’s subjective. However, if you use the random agents (not really random, they simply repeat the same static action), this is what you should expect.

image1694×460 101 KB

Do score only appear after all games are completed?

Yes. Your submission must complete all games to be marked as graded. Your submission won’t appear on leaderboard otherwise.

will it directly show the score on the submission page once all games are completed or it will take some additional time to make the score shown on the submission page

Your submission would be marked as failed if it doesn’t complete all the games. In case your evaluation times out i.e. doesn’t finish all the games in 12 hours, the submission would eventually get marked as failed.

View post 0

Clarification on ORAK Scoring Standards and Remote Mode Episode Settings

When running the games in remote mode, how many episodes are executed for each game?

Three episodes each for all games.

And for games like 2048, is the final score taken as the average of three rounds

Final score for each game would be the average scores across episodes.

View post 0

Super_mario

Hey, we released a few patches to the starter kit that would remove fastmcp dependencies. Can you please pull the recent changes, give it a try and reach out to us if you are still running into problems?

View post 0

📢 Starter Kit Update

View post 0

📢 Starter Kit Update

Hello everyone! A quick heads up about an important stability update to the starter kit.

We’ve migrated the transport/communication backend (previously based on MCP/FastMCP) to gRPC to make the interaction between your agent and the game environment more robust and predictable .

What changed?

• The underlying transport layer is now gRPC-based .
• The agent-facing interface and APIs are unchanged . Your existing agent code should continue to work as is.
• The main goal of this change is better resilience under long games and reconnections .

Why this matters?

Some of you were seeing:

• Random stalls / timeouts during longer runs
• Reconnection issues
• Episodes hanging with no clear error

These issues were caused by how requests were queued and retried in the previous MCP-based setup. With gRPC, we now strictly enforce “one client, one action in flight” , and we get clearer error handling, which should eliminate these stalls and make reconnect behavior much more reliable.

What you need to do

1. Update your local starter kit
   • Pull the latest changes from the repo (e.g., git pull --rebase ).
2. Reinstall/refresh dependencies if needed
   • uv sync
3. Run your existing agents as usual
   • No changes should be required to your agent logic or environment interaction code.

If you still see issues

If you run into:

• Timeouts
• Stalls
• Reconnection problems

please share:

• Logs (client + server, if possible)
• Approximate episode length and map
• Steps to reproduce

This will help us quickly track down any remaining edge cases.

Thanks for your patience while we tracked this down.

View post 0

StarCraft Stuck After Episode 1 (‘Client is not connected’ Issue)

We use the Linux headless binary for evaluations, and the steps below should help you get everything set up clearly:

• Download the last available version (4.10) from GitHub - Blizzard/s2client-proto: StarCraft II Client - protocol definitions used to communicate with StarCraft II..
• Check the README for instructions on unzipping the archive. It’s password protected, and the password is provided in the README itself.
• Extract the game into your $HOME directory. After extraction, the expected path should be $HOME/StarCraftII.
• Depending on your burnysc2 version, the maps directory may need to be either $HOME/StarCraftII/Maps or $HOME/StarCraftII/maps. To avoid issues with missing map directories, create a symlink:

  ln -s $HOME/StarCraftII/Maps $HOME/StarCraftII/maps
  

• For additional details on configuring the maps, see: Question about SC2 map setting in starter kit

View post 1

Model submission

1. The final evaluations would be manually run by Krafton AI team and the prizes would be decided based on the outcome of the manual runs. Krafton AI team would verify the size of the model you submit during the final evaluations.
2. No, we do not require any sort of access to your LLM server. However, for the final evaluations, you would need to include precise instructions and code that is needed to start your LLM server and ensure that Krafton AI team is able to run everything end-to-end.

View post 0

Model submission

No, it doesn’t need to be hosted externally. We simply provide the endpoints that let you get game observations, and your agent only needs to return the actions to execute in the game. How you produce those actions whether through local models, external services, or any other setup is entirely up to you.

This means your machine must be able to access whichever models or services you rely on during inference.

mermaid-diagram-2025-12-10-1332523840×2764 410 KB

In Local Mode, everything (your runner, game launcher, agents, and the MCP game servers) runs directly on your machine. The runner starts the game servers, initializes your agents, and your agents communicate with the servers over localhost. There is no connection to the AIcrowd backend.

mermaid-diagram-2025-12-10-1332433840×2764 339 KB

In Remote Mode, your runner and agents (including the LLM calls you make etc.,) still run locally, but the MCP game servers run on AIcrowd’s remote infrastructure. Your runner creates a session via the Session API, receives the MCP server URLs, and your agents interact with the remote servers over HTTPS.

mermaid-diagram-2025-12-10-1332233840×2764 374 KB

• Local Mode hosts the entire stack on your machine.
• Remote Mode keeps your agents local while offloading the game servers and environments to AIcrowd.

Hope this makes it clear

View post 0

ModuleNotFoundError: No module named 'omegaconf'

uv looks for pyproject.toml and automatically manages the virtual environment at the repo level.

Can you verify that your environment is actually being used?

# check which Python uv is running
uv run python

# try importing a starter-kit–specific library, for example:
import sc2

If this import works, then uv is configured correctly.

You can install additional packages with:

uv add <package>

Although uv pip install -r pyproject.toml works, it’s generally better to use:

uv sync

This installs the exact dependency versions listed in uv.lock, matching the environment used during starter-kit testing.

View post 0

Model submission

You don’t need to submit your model directly for this challenge. When you make a submission, we automatically launch an instance of each game and provide your agent with a unique MCP address for that run. Each game reports its score back to us, and we update the leaderboard accordingly.

View post 0

Submissions stuck at "Compiling model for Neuron"

@artist @whoamananand it seems like the models are hitting memory limits. Can you share the the config params you used to compile the model on trn1.2xlarge so that we can investigate this further?

View post 0

Submitting models to Neuron: pick the right `--neuron.model-type` (and tune vLLM if you need to)

Hey @whoamananand

Your model is hitting memory limits and crashing the node on which the evaluation was running. Can you try submitting a smaller model?

We will figure out a way to relay the OOM errors properly on the submission details page.

View post 1

Submitting models to Neuron: pick the right `--neuron.model-type` (and tune vLLM if you need to)

View post 0

Submitting models to Neuron: pick the right `--neuron.model-type` (and tune vLLM if you need to)

When you run aicrowd submit-model, the platform spins up a vLLM server for your model. You can pass a handful of --vllm.* flags to control things like max context length, dtype, batching limits, LoRA settings, and a few inference-time parameters.

But there’s one flag you must get right for Neuron hardware:

--neuron.model-type <model-type>

Why this matters (Neuron compilation in one paragraph)

AWS Inferentia/Trainium (Neuron) doesn’t run your PyTorch model “as-is” the way a typical GPU setup might. The model needs to be compiled into a Neuron-compatible artifact before it can run on the accelerator.

Because the compilation path is model-architecture-specific, the submission system needs to know which backend/architecture you’re using. Hence --neuron.model-type.

If you don’t set --neuron.model-type, the submission will default to qwen3.

Supported model types (backends)

The NxD Inference model hub currently supports architectures including: Llama (text), Llama (multimodal), Llama4, Mixtral, DBRX, Qwen2.5, Qwen3, and FLUX.1 (beta).

(For the others, the exact string is typically the obvious lowercase name, aligned to the supported architecture list above. If you’re unsure, match it to your model family—e.g., Mixtral → mixtral, Qwen3 → qwen3—because picking the wrong type can lead to compile failures or incorrect behavior.)

Supported vLLM server flags

aicrowd submit-model currently supports these vLLM arguments:

--vllm.max-model-len
--vllm.dtype
--vllm.kv-cache-dtype
--vllm.quantization
--vllm.load-format
--vllm.rope-theta
--vllm.rope-scaling
--vllm.max-num-batched-tokens
--vllm.max-num-seqs
--vllm.enforce-eager true
--vllm.enable-lora true
--vllm.lora-dtype
--vllm.lora-extra-vocab-size
--vllm.enable-prefix-caching true
--vllm-env.allow-long-max-model-len

# inference time parameters
--vllm-inference.max-tokens

Example: complete submission command

At minimum, set your repo/tag and the Neuron model type:

aicrowd submit-model \
  --hf-repo <repo> \
  --hf-repo-tag <branch/tag> \
  --neuron.model-type llama

Or, for DBRX:

aicrowd submit-model \
  --hf-repo <repo> \
  --hf-repo-tag <branch/tag> \
  --neuron.model-type dbrx

If you need tighter control over serving behavior, add the relevant --vllm.* flags on the same command.

View post 0

Concerns About Challenge Readiness - Seeking Clarification

Hello @sankar_ram

1. We are already accepting submissions. In an unlikely case that there are any more changes to the evaluation setup, we will re-evaluate the submissions as needed. You can start submitting your models right away.
2. As of now, you can make 5 submissions per day.
3. “Global Chess Challenge” is the name of the competition. “AWS Trainium Challenge 2025” was a placeholder we used during our internal development.

View post 0