When is Deadline to Team up?

Hi Chris,

Sorry for the delayed response. My understanding is that the team was awaiting an official statement from Meta before publicly responding. However, in the absence of such confirmation, here’s the official recommendation from AIcrowd to Meta, which I feel fairly confident about:

1. Deadline to Team Up:

There appears to have been a discrepancy in the announced dates. Given this confusion, I confirm that we will enforce the later date. Therefore, May 28th should be the official deadline for team mergers.

2. Clarification on Submission Limits After Team Mergers:

The rules could benefit from clearer language regarding submission limits after teams merge. My understanding is that the intention was to continue to have a limit of 6 submissions per week during Phase 2 as well, consistent with the overall submission rules. After merging, teams will maintain this limit of 6 submissions per week.

However, during the week in which the team merger occurs, if the combined submissions of both teams exceed the weekly limit (6), the newly merged team will not be allowed any additional submissions for that week - until the submission quota resets in a week, like with any other team.

View post 1

Evaluation Method of Leaderboard

I agree.
And having being closely involved in the final due-diligence of many competitions on AIcrowd, I assure you, that we do reserve the right to disqualify submissions that are clearly trying to exploit a loophole, or are not aligned with the spirit of the competition.

We will check in with the Meta team, on how best to include this officially in the Rules of the competition, if its not already addressed.

View post 0

During Submission How Do We Download Web Search URL?

@Chris_Deotte : appreciate the suggestion. And acknowledge the current gap in being able to get the full page contents for the search queries.

We already have a safe approach deployed, where the search API responds on the eval servers with the full page contents directly in its response (so the time limit is also agnostic of the web requested related overheads).

It’s just not documented yet, and we hope to rollout some documentation about how to use it effectively over the weekend. We are aiming for an interface that works transparently as a drop in replacement for your local dev flow as well.

View post 1

What's the G6e instance size for evaluation?

@cobochan : The instance we use is a g6e-2xlarge instance. And the disk storage available is 300GB per evaluation container.
You are right to note that 4vCPUs are available, even the instance type has 8vCPUs. We make 4 vCPUs available to the evaluation container, and the other 4 vCPUs might be used up by the background services running on the same instance.

View post 0

How to Submit (Gated) Hugging Face Models Without Sharing Them with Other Teams 🔒

View post 0

Does "search_pipeline" source change during LB submission

@Chris_Deotte : Yes, the evaluators use a different index with the same search_pipeline interface. This means that the same query can produce different results locally versus during evaluation.

The index used by the evaluators is constructed with the assosiated test set in mind and is optimized to return results with high relevance wrt specific test set.

View post 1

How to use private model

Hi @Chris_Deotte ,

The docs for referencing private HF repos is added to the (soon to be released) starter kit here.
We will make an announcement soon once the evaluators are ready to support the new starter kit.

Update: We have slightly changed the interfacing with Huggingface, and how we expect teams to securely submit their models via Hugginface. The key change is that, we want people to have their models available as “Public Gated” repos on Huggingface instead of Private repos. We have added step by step instructions for the same here: Using Gated Hugging Face Models in Your Submission

View post 1

💬 Feedback & Suggestions

This is already addressed in the upcoming release of the starter kit. We will post an announcement as soon as it’s ready.

View post 2

📢 Dataset Release: CRAG-MM v0.1.1 🚀

Dear All,

We are pleased to announce that the v0.1.1 release of the CRAG-MM dataset is now publicly available on Hugging Face:

• Single-Turn: https://huggingface.co/datasets/crag-mm-2025/crag-mm-single-turn-public
• Multi-Turn: https://huggingface.co/datasets/crag-mm-2025/crag-mm-multi-turn-public

We encourage you to visit the Hugging Face Dataset Card to explore the dataset. Below is a minimal example to help you get started quickly:

# pip install datasets

from datasets import load_dataset

# For single-turn dataset
dataset = load_dataset("crag-mm-2025/crag-mm-single-turn-public", revision="v0.1.1")

# For multi-turn dataset
dataset = load_dataset("crag-mm-2025/crag-mm-multi-turn-public", revision="v0.1.1")

# View available splits
print(f"Available splits: {', '.join(dataset.keys())}")

# Access example data
example = dataset["validation"][0]
print(f"Session ID: {example['session_id']}")
print(f"Image: {example['image']}")
print(f"Image URL: {example['image_url']}")

"""
Note:
Either 'image' or 'image_url' will be provided, but not necessarily both.
When the actual image is unavailable, only the image URL will be provided.
The evaluation servers will always include the loaded 'image' field.
"""

# Display image
import matplotlib.pyplot as plt
plt.imshow(example['image'])


def prepare_feature_vocabularies(dataset_split):
    """Extract feature vocabularies for category encoding from dataset.
    
    These vocabularies allow conversion between integer indices and string labels.
    """
    return {
        "domain": dataset_split.features["turns"][0]["domain"],
        "query_category": dataset_split.features["turns"][0]["query_category"],
        "dynamism": dataset_split.features["turns"][0]["dynamism"],
        "image_quality": dataset_split.features["turns"][0]["image_quality"],
    }


def print_conversation(example: Dict[str, Any], feature_vocabularies: Dict[str, Any]) -> None:
    """Print a conversation in an indented format.
    
    Args:
        example: A single dataset example containing conversation turns
        feature_vocabularies: Mapping of features to their vocabularies for encoding/decoding from idx to str
    """
    # Print session ID
    print(f"Session ID: {example['session_id']}")
    
    # Print image info
    print(f"Image: {example['image']}")
    print(f"Image URL: {example['image_url']}")
    """
    Note: Either 'image' or 'image_url' will be provided in the dataset, but not necessarily both.
    When the actual image cannot be included, only the image_url will be available.
    The evaluation servers will nevertheless always include the loaded 'image' field.
    """
    image_quality_str = feature_vocabularies["image_quality"].int2str(example['image_quality'])
    print(f"Image Quality: {image_quality_str}")
    
    # Determine if single-turn or multi-turn
    is_single_turn = len(example['turns']) == 1
    print(f"Type: {'Single-turn' if is_single_turn else 'Multi-turn'} ({len(example['turns'])} turns)")
    
    # Create answer lookup dictionary if answers exist
    answer_lookup = {}
    if 'answers' in example and example['answers'] is not None:
        answer_lookup = {a["interaction_id"]: a["ans_full"] for a in example["answers"]}
    
    # Print each turn
    print("\nConversation:")
    for i, turn in enumerate(example['turns']):
        # For multi-turn, show turn number
        if not is_single_turn:
            print(f"\tTurn {i+1}:")
        
        # Convert metadata to string representations
        domain_str = feature_vocabularies["domain"].int2str(turn['domain'])
        category_str = feature_vocabularies["query_category"].int2str(turn['query_category'])
        dynamism_str = feature_vocabularies["dynamism"].int2str(turn['dynamism'])
        
        # Print metadata
        prefix = "\t\t" if not is_single_turn else "\t"
        print(f"{prefix}Domain: {domain_str} | Category: {category_str} | Dynamism: {dynamism_str}")
        
        # Print query and answer with fixed tab indentation
        print(f"{prefix}Q: {turn['query']}")
        
        ans = answer_lookup.get(turn['interaction_id'], "No answer available")
        print(f"{prefix}A: {ans}")
        
        if not is_single_turn and i < len(example['turns']) - 1:
            print()  # Add blank line between turns in multi-turn conversations
    
    print("\n" + "-" * 60 + "\n")  # Add separator between examples


split_to_use = "validation"
feature_vocabularies = prepare_feature_vocabularies(dataset[split_to_use])
print_conversation(example, feature_vocabularies)

Best,
The CRAG-MM Team

View post 0

💬 Welcome to the Meta CRAG - MM Challenge 2025!

View post 0

How to use Mock API in submission

Please refer to the example here:

You do not need to include the MockAPI in your submission, it will be automatically made available in your evaluation environment, and the correct URI will be populated in the environment variable referenced in the example above (CRAG_MOCK_API_URL)

View post 0

Function calling arguments in local_evaluation.py is mismatching with dummy_model's interface

@evelynintech : Can you please ensure that you are using git lfs pull to pull the example dataset files via git lfs ?
You can also verify the contents of the example data files in the example_data folder to ensure that they are locally available.

View post 1

Submit error

@yong_deng : Can you please try submitting again now ?
As the error says, we had a scheduled maintenance yesterday, so we were not accepting submissions when you made your submission.

View post 0

Track2 Validation failed:No module named 'aicrowd_gym'

@carpe : We have requeued the submission which failed with "No module named ‘aicrowd_gym’”, it was because of a transient error at our end. You are not expected to manually install aicrowd_gym.

View post 1

How to add SSH key to Gitlab?

@momi193 : Thanks for pointing it out, we have updated the links in the main message as well.

Best,
Mohanty

View post 0

Submission failure for Task 2

Hi Iris,

The issue reported seems specific to the submissions made by your account, as there are numerous successful submissions after the timestamp reported in your message.

We acknowledge that the error propagation still has some cluster specific issues that we are trying to resolve as we speak.
In the meantime, please find attached the error for your submission.

submission-248780-evaluation-4331-188153-bl8rn-1190196615: /home/aicrowd/.conda/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py:515: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   warnings.warn(
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Traceback (most recent call last):
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Evaluation started
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Init wrapper
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/run.py", line 3, in <module>
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     start_test_client()
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/client_launcher.py", line 20, in start_test_client
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     client.run_agent()
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 193, in run_agent
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     raw_response, status, message = self.process_request(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 99, in process_request
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     "data": self.route_agent_request(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 129, in route_agent_request
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self.execute(target_attribute, *args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 142, in execute
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return method(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/aicrowd_wrapper.py", line 30, in classify_link
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self.agent.classify_link(test_data_batch)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/agents/deberta_v3_large_nli_comFact_agent.py", line 56, in classify_link
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     head_logits = self.model(**head_inputs).logits
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1300, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     outputs = self.deberta(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     encoder_outputs = self.encoder(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     output_states = layer_module(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     attention_output = self.attention(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     self_output = self.self(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 721, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     rel_att = self.disentangled_attention_bias(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 819, in disentangled_attention_bias
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     p2c_att = torch.gather(
submission-248780-evaluation-4331-188153-bl8rn-1190196615: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.29 GiB. GPU 0 has a total capacty of 14.75 GiB of which 1.02 GiB is free. Process 38611 has 13.73 GiB memory in use. Of the allocated memory 11.06 GiB is allocated by PyTorch, and 2.54 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Best,
Mohanty

View post 0

🚨 Important Update: Challenge Deadline Extended & Update to Rules

Hi kevin,

There was indeed an issue with compute node assignment earlier today, but it has been fixed now. We can see that the image building for your latest submission has already started.

View post 0

Resources section

@soham17 : The starter kit for the challenge is available here: AIcrowd / Challenges / Generative Interior Design Challenge 2024 / Generative Interior Design 2024 starter kit · GitLab

It is also linked at the top of the Challenge Overview Page here: AIcrowd | Generative Interior Design Challenge 2024 | Challenges

Best of Luck,
Mohanty

View post 0

No Submission Slots Remaining

@crlandsc : The submission quotas have not changed. The submission quotas are checked against the number of submissions made by your (or any of your team members) in the last 24 hour window.
So for example, if you make 3 submissions at 23:55 on Day 1, then you can make only 2 more submissions until 23:55 on Day 2 (assuming the submission quota is 5 submissions / day).
The daily submission quotas are not automatically reset at midnight each day.

In the last few days, these are the only submissions we see from your account:

• April 28th 04:13 UTC
• April 28th 04:33 UTC
• April 28th 04:45 UTC
• April 28th 15:34 UTC
• April 29th 19:34 UTC

All of the said submissions failed to evaluate for unrelated reasons (for more details please refer to the relevant debug logs in the issues associated with the failed submissions), and not because of exceeding submission quotas.

Additionally, we are updating the quotas for the competition, to allow up to 5 failed submissions which would not count towards your daily submission quota of 5 submissions.

View post 1

Submissions are being stuck on "Preparing the cluster for you" for a long time

@kimberley_jensen : There is a spike in the number of submissions because of the approaching deadline, and all the submissions are queued on the evaluation servers, and are all being eventually evaluated. Some submissions are timing out because of being in the queue for too long, and we are manually re-queuing them when that happens.

We will keep a close eye on the submission queues, and intervene as required.
Thank you for your patience.

View post 1