Question on Scoring

I think we need clarification on this. I’ll pause work on this competition until then

View post 1

Question on Scoring

One issue I see with the tie-breaking criteria is that it is fairly simple to create rule-based scripts to achieve perfect scores in the games. There’s probably already code for this online- so what is preventing teams from relying primarily on rule-based scripts to perform most of the work while a minimal LLM is included just to ensure the team has the optimal tie-breaking solution?

Since there’s nothing in the rules disallowing this, it is likely that the winning solutions will work this way.

View post 0

Question on Scoring

The top team in the Track 1 leaderboard currently has a perfect score of 1.0 for all 4 games. I imagine there will soon be more teams with this score. I understand that the final evaluation will include hidden test cases/scenarios, but how can we determine if our agents are improving relative to other teams if we all have perfect scores of 1.0? Perhaps the current evaluation cases should be more difficult so we can better compare different teams progress?

View post 0

Question about SC2 map setting in starter kit

It would be useful to provide the Ancient Cistern LIE file (along with any other map files required for running evaluation in linux). I’m still unsuccessful in finding or generating this file after searching through the zips on Maps - AI Arena Wiki and following the manual patching process in Large-Language-Models-play-StarCraftII/Maps at main · histmeisah/Large-Language-Models-play-StarCraftII · GitHub

View post 1

AICrowd Gitlab Offline?

Appears to be back online now.

View post 0

AICrowd Gitlab Offline?

This link is currently not working. Maybe gitlab is offline?
https://gitlab.aicrowd.com/aicrowd/challenges/orak-game-agent-challenge-2025/orak-2025-starter-kit

View post 2