💬 Feedback & Suggestions

@CPMP 5 submissions per team should be enforced by the UI on a per day basis (00:00 to 00:00 UTC fixed window). We have two settings that the organizers can choose from, 24 hour rolling window and 24 hour fixed window. For this challenge its set to fixed window.

Is your team able to make more than 5 submissions in the fixed window?

View post 0

💬 Feedback & Suggestions

On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.

Apologies, I wasn’t aware of this.

I’ll mention that we generally do not run competitions without private test sets, which is where this is a much bigger issue. This was an oversight on our part.

I checked the leaderboards and see that there is one team that has managed to make a lot of submissions doing this. After the team merge deadline we’ll check for any more unfair merges, and let the Amazon team decide if they want to provide everyone else with additional submissions to keep things fair.

View post 0

💬 Feedback & Suggestions

Agreed, but this is the system we have, it applies to everyone. If I understand correctly it is also the system on some other platforms.

I understand it’s an outsized advantage for this challenge as it’s not a code based challenge and doesn’t have a private test set.

We’re open to alternative ideas for future challenges, please share your feedback here or in dm.

View post 0

💬 Feedback & Suggestions

Hi @BenediktSchifferer

Yes, every team is limited to 5 submissions per day. After merging teams, submissions limits will apply to the whole team.

View post 0

💬 Feedback & Suggestions

@ECNU_Wei_Zhang Please ask all you team members to dm me and confirm that they want to leave the team.

View post 0

💬 Feedback & Suggestions

@ECNU_Wei_Zhang Okay your team has been removed

View post 2

Clarification and More Information for Phase 2

@gaozhanfire Thanks for letting me know. Leaderboards should be displayed correctly now.

View post 0

Clarification and More Information for Phase 2

@tereka Yes, you can use the Phase 1 test set as you like.

View post 0

Clarification and More Information for Phase 2

@tereka The competition will be ranked based on the results of Phase 2. The deadline may be extended, that is upto Amazon, they haven’t notified us yet.

View post 0

Clarification and More Information for Phase 2

@BenediktSchifferer The only difference between the Phase 1 and Phase 2 is the test sets. All new submissions are scored on Phase 2 test sets only.

View post 0

Clarification and More Information for Phase 2

The Phase 2 datasets are available on the Resources page.

They have been available since 9 hours ago when the Phase 2 submissions opened.

View post 0

Submission Times

@quickpepper947 We do not check for identical commits when counting submissions. I missed that the date you transitioned from debug false to true was not the date you got 6 submissions. In that case I’ll need to further dig into why you got 6 submissions.

I suspect that it may be related to the tags having identical timestamps as you mentioned, I guess further investigation is required.

View post 0

Submission Times

@XavierJ , yes, all teams had the possibility to use 6 submissions due to this issue. However, to my best of my knowledge no one has used it intentionally. I apologize again that your team ended up having fewer submissions due to this.

I also checked @quickpepper947’s extra submission and 2 out of the 6 made on 8th May are actually identical commits.

View post 0

Submission Times

@XavierJ , thanks for the rebuttal post. Indeed some deeper investigation was required on my part, and now I believe I’ve found the root cause of the issue.

Along with the previously mentioned setting difference in submission counting window I mentioned in the previous post. There is another issue that silently affected your team, due to an unfortunate mistake on our end.

The issue lies in the key debug: true in aicrowd.json that yoir team seems to have in all your submissions. This key has long been deprecated, and we do not include it in our starter kits. Due to legacy reasons, it is still present in our evaluations codebase, but will only allow 1 submission per day if the key is set to true. The submission just runs normally on the same dataset and the same settings.

However, it seems the mention of this key has not been removed from our autogenerated documentation for the Create Submission page. I presume this is where you got the key and added it to aicrowd.json? Normally, people do not add this key, and this issue went undetected.

In your case, since your team has 4 members, this led to the misunderstanding that every team member gets 1 submission each. Instead, they were failing because the debug key was set.

This is also the case with submissions from @quickpepper947, he was not using the debug key throughout the competition. But added it as true for one of the submissions, and then set it to false in the next submission. These two settings created seperate counters for the submissions, 1 for the debug set to true and 5 for the debug set to false. This led to 6 submissions going scored in one day.

We sincerely apologize for the confusion caused due to the debug key not being properly deprecated and removed from all communication channels. We’ll fix this issue as soon as possible.

View post 0

Submission Times

Dear @JusperLee ,

Thank you for bringing your concerns to our attention. We have thoroughly investigated the issue and have found that there was a misunderstanding, for which we apologize.

Regarding the submission quota, we would like to clarify the following points. Prior to the extension of the deadline, the submission limit was initially set to 10 submissions per day. However, due to a high submission load that caused many submissions to become stuck, we made the decision to extend the deadline by one week. During this extension period, the submission limit was 5 per day.

Regardless of the size of the team, every team is granted 5 submissions per day (or 10 when the limit was 10). This allocation is not based on the number of team members. Therefore, if only one member of your team submits, they can utilize all 5 submissions for that day. If multiple people submit, it counts towards the entire team’s quota.

It is important to note that our platform does not have a mechanism in place to allow teams or participants to have a different number of submissions. The submission limit remains consistent for all teams throughout the challenge.

Lastly, there was a wrong communication regarding the submission window. Our platform supports two options: a Rolling 24-hour window or a Fixed window between UTC 00:00:00 and UTC 23:59:59. While the intention was to utilize the Rolling 24-hour window, an unintended bug caused the Fixed window to be used for the CDX challenge. We acknowledge that this was not accurately reflected in the forum post. Upon reviewing all the submissions made in the challenge, we can confirm that no team has exceeded the limit of 5 submissions in a day within the fixed window of quota reset (UTC 00:00:00 to UTC 23:59:59). Your screenshots will also confirm the same.

Once again, we apologize for any confusion and inconvenience caused. We value your feedback and will take it into consideration as we strive to enhance the clarity and accuracy of our communications. Thank you for your understanding.

View post 2

MDX Final Leaderboard A Error

@kin_wai_cheuk You’re right, it had wrongly used the Round 2 dataset’s SDR instead of the full SDR. This is fixed now.

View post 1

Something wrong with Final Leaderboard B in CDX23 track

It was a mistake on my part, used the same selection key. Its fixed now.

View post 2

Are these wrong leaderboard submissions? Will they be removed?

@subatomicseer , I inspected the inference code and indeed the submissions you mentioned were baseline models, not allowed for leaderboard B and C. Thank you for mentioning these.

View post 1

Are these wrong leaderboard submissions? Will they be removed?

Thanks for pointing this out, I’ll follow up with the participants.

Of course in general, we cannot prevent wrongly marked submissions from showing up on the leaderboard if the users don’t tell us to remove them.

However, for the data constrained leaderboards of both CDX (LB A) and MDX (LB A and B), we will collect the training code from the winners and reproduce the results, hence any wrongly marked submissions will certainly be removed in the final leaderboards.

View post 2

Aicrowd_gym ModuleNotFoundError

Hi @crlandsc

Apologies for the confusion. We are making some changes to our evaluation setup, hence the difference from the documentation provided. However, I understand that since the challenge is ending using environment.yml might be important, hence we’ve added back the support for it.

I’ve made some minor changes to your environment.yml and made a submission. I believe now there is some error for the code to run, please check the logs.

Hope this helps.

View post 0