Pokemon map is broken?

@ilya_gusev is there a particular submission you are referring to that I can check?

We didn’t change anything specific to pokemon and wouldn’t expect this change. Maybe the game hasn’t started yet? The game ROM we received from Krafton team starts at the menu screen and the “Map on Screen” wouldn’t be defined for that screen.

Is there something I can cross check on the server logs or can you give the exact steps to replicate this issue so that we can pass this to Krafton team?

View post 0

Starcraft submission

@mikhail1 @cheong_wei_xun

can you please take a look at this

View post 0

Model submission

While running remote mode, it seems like the Star Craft failed after entering the second episode, is it possible that there’s a problem with the code in the MCP server or game envs of AIcrowd?

The issue was due to requests getting queued and dequeued arbitrarily at MCP server. We moved to a gRPC based implementation to get around this issue.

Please pull the recent changes made to starter kit and let us know if you still run into any issues.

long will the evaluation process usually take?

It’s subjective. However, if you use the random agents (not really random, they simply repeat the same static action), this is what you should expect.

image1694×460 101 KB

Do score only appear after all games are completed?

Yes. Your submission must complete all games to be marked as graded. Your submission won’t appear on leaderboard otherwise.

will it directly show the score on the submission page once all games are completed or it will take some additional time to make the score shown on the submission page

Your submission would be marked as failed if it doesn’t complete all the games. In case your evaluation times out i.e. doesn’t finish all the games in 12 hours, the submission would eventually get marked as failed.

View post 0

Clarification on ORAK Scoring Standards and Remote Mode Episode Settings

When running the games in remote mode, how many episodes are executed for each game?

Three episodes each for all games.

And for games like 2048, is the final score taken as the average of three rounds

Final score for each game would be the average scores across episodes.

View post 0

Super_mario

Hey, we released a few patches to the starter kit that would remove fastmcp dependencies. Can you please pull the recent changes, give it a try and reach out to us if you are still running into problems?

View post 0

📢 Starter Kit Update

View post 0

📢 Starter Kit Update

Hello everyone! A quick heads up about an important stability update to the starter kit.

We’ve migrated the transport/communication backend (previously based on MCP/FastMCP) to gRPC to make the interaction between your agent and the game environment more robust and predictable .

What changed?

• The underlying transport layer is now gRPC-based .
• The agent-facing interface and APIs are unchanged . Your existing agent code should continue to work as is.
• The main goal of this change is better resilience under long games and reconnections .

Why this matters?

Some of you were seeing:

• Random stalls / timeouts during longer runs
• Reconnection issues
• Episodes hanging with no clear error

These issues were caused by how requests were queued and retried in the previous MCP-based setup. With gRPC, we now strictly enforce “one client, one action in flight” , and we get clearer error handling, which should eliminate these stalls and make reconnect behavior much more reliable.

What you need to do

1. Update your local starter kit
   • Pull the latest changes from the repo (e.g., git pull --rebase ).
2. Reinstall/refresh dependencies if needed
   • uv sync
3. Run your existing agents as usual
   • No changes should be required to your agent logic or environment interaction code.

If you still see issues

If you run into:

• Timeouts
• Stalls
• Reconnection problems

please share:

• Logs (client + server, if possible)
• Approximate episode length and map
• Steps to reproduce

This will help us quickly track down any remaining edge cases.

Thanks for your patience while we tracked this down.

View post 0

StarCraft Stuck After Episode 1 (‘Client is not connected’ Issue)

We use the Linux headless binary for evaluations, and the steps below should help you get everything set up clearly:

• Download the last available version (4.10) from GitHub - Blizzard/s2client-proto: StarCraft II Client - protocol definitions used to communicate with StarCraft II..
• Check the README for instructions on unzipping the archive. It’s password protected, and the password is provided in the README itself.
• Extract the game into your $HOME directory. After extraction, the expected path should be $HOME/StarCraftII.
• Depending on your burnysc2 version, the maps directory may need to be either $HOME/StarCraftII/Maps or $HOME/StarCraftII/maps. To avoid issues with missing map directories, create a symlink:

  ln -s $HOME/StarCraftII/Maps $HOME/StarCraftII/maps
  

• For additional details on configuring the maps, see: Question about SC2 map setting in starter kit

View post 1

Model submission

1. The final evaluations would be manually run by Krafton AI team and the prizes would be decided based on the outcome of the manual runs. Krafton AI team would verify the size of the model you submit during the final evaluations.
2. No, we do not require any sort of access to your LLM server. However, for the final evaluations, you would need to include precise instructions and code that is needed to start your LLM server and ensure that Krafton AI team is able to run everything end-to-end.

View post 0

Model submission

No, it doesn’t need to be hosted externally. We simply provide the endpoints that let you get game observations, and your agent only needs to return the actions to execute in the game. How you produce those actions whether through local models, external services, or any other setup is entirely up to you.

This means your machine must be able to access whichever models or services you rely on during inference.

mermaid-diagram-2025-12-10-1332523840×2764 410 KB

In Local Mode, everything (your runner, game launcher, agents, and the MCP game servers) runs directly on your machine. The runner starts the game servers, initializes your agents, and your agents communicate with the servers over localhost. There is no connection to the AIcrowd backend.

mermaid-diagram-2025-12-10-1332433840×2764 339 KB

In Remote Mode, your runner and agents (including the LLM calls you make etc.,) still run locally, but the MCP game servers run on AIcrowd’s remote infrastructure. Your runner creates a session via the Session API, receives the MCP server URLs, and your agents interact with the remote servers over HTTPS.

mermaid-diagram-2025-12-10-1332233840×2764 374 KB

• Local Mode hosts the entire stack on your machine.
• Remote Mode keeps your agents local while offloading the game servers and environments to AIcrowd.

Hope this makes it clear

View post 0

ModuleNotFoundError: No module named 'omegaconf'

uv looks for pyproject.toml and automatically manages the virtual environment at the repo level.

Can you verify that your environment is actually being used?

# check which Python uv is running
uv run python

# try importing a starter-kit–specific library, for example:
import sc2

If this import works, then uv is configured correctly.

You can install additional packages with:

uv add <package>

Although uv pip install -r pyproject.toml works, it’s generally better to use:

uv sync

This installs the exact dependency versions listed in uv.lock, matching the environment used during starter-kit testing.

View post 0

Model submission

You don’t need to submit your model directly for this challenge. When you make a submission, we automatically launch an instance of each game and provide your agent with a unique MCP address for that run. Each game reports its score back to us, and we update the leaderboard accordingly.

View post 0

Question about SC2 map setting in starter kit

You can search for the map names directly and you should be able to find them.

Maps ending with * AIE and * LIE are patched versions. They allow the latest maps to work with the last StarCraft II binary released for Linux. You can find these * AIE maps here: Maps - AI Arena Wiki

The map Ancient Cistern LE can be found here: Large-Language-Models-play-StarCraftII/Maps at main · histmeisah/Large-Language-Models-play-StarCraftII · GitHub

If you are using the latest StarCraft II version (and not the Linux headless binary), you do not need to patch the maps. You should be able to run the games as-is.

Just make sure you’ve upgraded both burnysc2 and s2clientprotocol to their latest versions.

View post 0

StarCraft Stuck After Episode 1 (‘Client is not connected’ Issue)

We are investigating this issue and will get back as soon as we have an update

View post 0

Question about SC2 map setting in starter kit

Ancient Cistern LIE is a ported version of Ancient Cistern LE. Since our evaluation environment uses the Linux headless SC2 binary, and the latest available release is still v4.10, it cannot load the newer official maps. To address this, we ported Ancient Cistern LE ourselves using a community-provided patch: GitHub - aiarena/sc2patch.

View post 1

💬 Feedback & Suggestions

@imeintanis Using AIcrowd | Flextrack Challenge 2025 | Submissions (https://www.aicrowd.com/challenges/flextrack-challenge-2025/submissions?my_submissions=true_ should display the list of all submissions you made to the challenge.

You can get to this page by using the submission filters option.

View post 0

💬 Feedback & Suggestions

Hello @jack_vandyke

Thanks for bringing this up. A minor misconfigured on our end resulted in this. All the pending evaluations are graded now. You should get your evaluation results almost instantly without any wait time.

View post 0

Issues regarding CPDC 2025 rerun

Hello @yiyang_zheng

We were able to run your submission fine. It was indeed an issue with git LFS files.

View post 0

Request for investigation into the cause of the error

Hello @NineGates

We are re-evaluating 289818 (seems like some network error caused it to fail with the error “failed to load artifacts”) and 289826 (evaluation didn’t start).

We can’t re-evaluate the rest as they failed due to timeout errors. Please note that it’s not the average prediction time we look at it, there is hard constraint on per step timeout. One step that violates this can fail the submission.

• 289762 violated the overall timeout.
• 289784 failed due to a per step timeout

2025-06-18 08:18:43.290 INFO main:run_with_timeout:154 - Running batch_generate_response with timeout 80
2025-06-18 08:20:28.554 INFO main:run_with_timeout:164 - Executed batch_generate_response in 105.26403450965881 seconds

• 289820 failed due to a per step timeout

2025-06-18 07:06:50.363 INFO main:run_with_timeout:154 - Running batch_generate_response with timeout 80
2025-06-18 07:08:14.969 INFO main:run_with_timeout:164 - Executed batch_generate_response in 84.60583829879761 seconds

View post 1

Submission Status Change From "Generating" to "Prepare Generate" and Stuck

In case your submission emits logs that can further help, let us know. We will clean up any dataset related log lines and share it with you.

View post 0