Loading
AIcrowd
Challenges
Organize a challenge FAQ Contact Jobs
Log In Sign Up

MeltingPot Challenge 2023

Challenge Rules

MELTINGPOT CHALLENGE 2023 OFFICIAL RULES

PLEASE READ THESE OFFICIAL RULES CAREFULLY. ENTRY INTO THIS COMPETITION CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL RULES. IF YOU DO NOT AGREE TO ANY PART OF THESE OFFICIAL RULES, PLEASE DO NOT ENTER THIS CHALLENGE.

NO PURCHASE IS NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY KIND WILL NOT INCREASE YOUR CHANCES OF WINNING VOID WHERE PROHIBITED.

1. Competition Description

Melting Pot Contest at NeurIPS 2023 is organised by researchers from the Cooperative AI Foundation, MIT, and Google DeepMind. This new competition challenges researchers to push the boundaries of multi-agent reinforcement learning (MARL) for mixed-motive cooperation. The contest leverages the cutting-edge Melting Pot environment suite to rigorously evaluate how well agents can adapt their cooperative skills to interact with novel partners in unforeseen situations. Success requires demonstrating true cooperative intelligence. This event offers participants a unique opportunity to tackle open challenges in cooperative AI and to advance the frontier of deployable real-world AI systems.

To help you "Get Started" we have a starter kit available. We hope they are helpful; if you find any bugs, typos, or improvements, please send us a pull request. This Challenge will be run in accordance with these Official Rules ("Rules").

2. Sponsors

The Competition is organized by AICrowd SA, EPFL Innovation Park, Bâtiment C, c/o Fondation EPFL Innovation Park, 1015 Lausanne, Switzerland and Google Deepmind, 5 New Street Square, London, EC4A 3TW . These third parties will be known as “Sponsors”. Notwithstanding the foregoing, the terms in these Official Rules are between AI Crowd, the Cooperative AI Foundation and you (the participant).

3. Organizers Admins

“Organizers Admins” are any companies or organizations authorized by Organizers to aid them with the administration, sponsorship or execution of this Challenge including but not limited to AICrowd SA. The organizers for this challenge include Cooperative AI Foundation, Courtenay House, Pynes Hill, Exeter, EX2 5AZ . The Cooperative AI Foundation is registered with the Charity Commission for England and Wales under charity number 1201294, and incorporated as a company limited by guarantee established in England for charitable purposes only under company number 13485176.

4. Eligibility

You may enter this challenge individually or as a team of two or more members ("Team"). You are eligible to enter this Challenge if you (and each member of your Team) meet all of the following requirements as of the time and date of entry:

  • You are an individual;
  • You are 16 years of age or older but in no event less than the age of majority in your place of residence; and
  • You have internet access, an email account, and access to a personal computer.

The organizers will not be able to transfer the travel grant and Macbook Pro to accounts of participants who reside in any of the following countries or regions. (Please note that residents of these countries or regions are still allowed to participate in the Challenge, but no prizes will be available.)

  • Russia
  • The Crimea region of Ukraine
  • Cuba
  • Iran
  • North Korea
  • Sudan
  • Syria
  • Quebec, Canada
  • Brazil
  • Italy

Please Note: it is entirely your responsibility to review and understand your employer's and country's policies about your eligibility to participate in this Challenge. If you participate in violation of your employer's or country's policies, you and your entry may be disqualified from the Challenge. The organizers disclaim any and all liability or responsibility with respect to disputes arising between an employer and such employer's employee or between a country and its resident in relation to the Challenge.

5. Competition Start and End Dates

  • Start date: August 25
  • End date: 15th November, 2023

6. Prizes

The prizes for the challenge are as follows:

  • 🥇 First place : $5,000
  • 🥈 Second place : $3,000
  • 🥉 Third place : $2,000

The challenge offer $10,000 in prizes for top performing teams, in adiditon to the $20,000 in travel and compute credits to broaden participation from underrepresented groups and lower resource barriers. The travel and compute credit distribution details and eligibility will be shared during the course of the challenge.

To be eligible for the prizes, participants will have to release the training and inference code to their solutions under a permissive e.g. BSD, MIT, Apache, (not exclusive) or copyleft e.g. MPL, EPL, LGPL, AGPL, GPL (not exclusive) license from the OSI Approved Licenses https://opensource.org/licenses/. The submitted training and inference code is expected to be reproducible and should produce the same score on the leaderboard.

The winning participants will have to provide a valid and unexpired ID card or passport which AIcrowd will use only for the purpose of verifying the individual's identity for internal record keeping.

Any and all prize(s) is(are) non transferable. All taxes, fees and expenses associated with participation in the Challenge or receipt and use of a prize are the sole responsibility of the Prize Winner(s). No substitution of prize or transfer/assignment of prize to others or request for the cash equivalent by winners is permitted. Acceptance of prize constitutes permission for AIcrowd to use winner’s name and entry for purposes of advertising and trade without further compensation unless prohibited by law.

The participation of the Organizers and Organizer Admins, its employees and its affiliates, parents, agents, representatives, advertising and promotional agencies and members of the immediate family (parents, children or siblings) of these employees or any person with whom they are domiciled, are permitted to participate in the challenge but will not be eligible to win the prize.

In accordance with our terms and conditions, Cooperative AI operates under limited liability with respect to the disbursement of prize money.

7. Data Use Agreement (if applicable)

By participating in this challenge, you ("the Participant") agree to the following terms and conditions regarding data use:

  1. Transparency: The Participant agrees to maintain complete transparency when incorporating additional data from external sources. This includes clear documentation of methods used and adherence to ethical standards.

  2. Dataset Declaration: In instances where the Participant collects their own dataset for training purposes, it must be made available to other participants and organizers, under a license that permits use in this competition. A link to the dataset should be shared bybefore the end of the challenge by 3012th October, 2023.

  3. Use Justification: When declaring the use of a dataset, the Participant must provide an explanation for its use in relation to this challenge. The Participant must explicitly state that the usage does not violate any terms of use for the dataset.

Violation of these terms and conditions may result in disqualification from the challenge. By accepting these terms and conditions, the Participant confirms understanding and agreement to the above stipulations.

8. Is the Entry an Eligible Entry?

To be eligible to be considered for a prize, as solely determined by the Organizers: The Entry MUST:

The Entry MUST:

  • be compatible with the official submission format;
  • be self-contained and function without a dependency on any external services and network access
  • be in English;
  • be the Team’s own original work;
  • not have been submitted previously in any promotion of any kind;
  • not contain material or content that: is inappropriate, indecent, obscene, offensive, sexually explicit, pornographic, hateful, tortious, defamatory, or slanderous or libellous; or promotes bigotry, racism, hatred or harm against any group or individual or promotes discrimination based on race, gender, ethnicity, religion, nationality, disability, sexual orientation, or age; or promotes alcohol, illegal drugs, or tobacco; or violates or infringes another’s rights, including but not limited to rights of privacy, publicity, or their intellectual property rights; or is inconsistent with the message, brand, or image of Organizers, is unlawful; or is in violation of or contrary to the laws or regulations of any jurisdiction in which the Entry is created; and

The Team members MUST:

  • designate one person as the team leader who will be solely responsible for receiving communications from and communicating with Sponsor;
  • ensure the Team has obtained any and all consents, approvals, or licences required for submission of the Entry;
  • obtain any consents necessary from all members of the Team with respect to the sharing of such member’s personal information as outlined herein;
  • obtain the agreement of all members of the Team to these Rules;
  • not generate the Entry by any means which violate these Rules, - the Organizers Terms of Service or the Organizers Privacy Policy;
  • not engage in false, fraudulent, or deceptive acts at any phase during participation in the Challenge; and
  • not tamper or abuse any aspect of this Challenge.

9. Challenge Specific Rules

Agent Design

Participants are granted the freedom to implement, train, and design agents in their preferred manner, with the stipulation that agents are prohibited from attempting to establish connections with external resources during evaluation. Any external resources required must be included locally.

Contest Structure

The contest is divided into two phases: a development phase focused on iterative enhancement and a final test phase for official scoring. During Phase I, participants can submit an 2 submissions per day per team/participant number of entries, and the leaderboard rankings will initiate on August 10. Submissions will undergo assessment on multiple seeded episodes, and the results will be furnished within 24 hours. The test phase enables up to 3 submissions, with the highest score considered for final rankings. Only error reports and ultimate scores will be furnished. The test phase leaderboard will remain concealed from participants until the NeurIPS conference.

Time Limits

Submissions will be subjected to a liberal time constraint to prevent abuse of computation resources, such as 24 hours or 100 million steps per episode. Each team/participant is allowed to make two submission per day.

Code Requirements

While it is encouraged, releasing source code is not mandatory for leaderboard ranking. Nevertheless, the organizers retain the authority to withhold prize money from entries that do not disclose their code. For test phase submissions, the source code must be shared privately with the organizers solely for validation purposes.

Multiple Entries

Each participant should submit just one entry per team. The organizers may scrutinize code for resemblances and may disqualify teams attempting multiple submissions. Each team/participant is allowed to make two submission per day.

Systems Description

Test phase submissions necessitate the inclusion of a free-form textual description of the system. This description should elucidate the method, guided by provided prompts. Although comprehensive details are encouraged, they are not mandatory.

Adjudication

Rankings will be determined through a unified track. Solutions with minimal resource demands may be spotlighted.

Presentations

The top three teams are required to furnish concise video summaries of their systems before the conference. These summaries will be showcased on-site.

By participating, you consent to confer non-exclusive rights to the organizers over submitted code and trajectories exclusively for the purposes of contest administration, validation, and analysis. We retain the prerogative to disqualify teams infringing these regulations. If you have further inquiries about participating, please reach out to us. Best of luck!

10. Disqualification

If you, any Team member, or the Entry is found to be ineligible for any reason, including but not limited to conflicts within Teams and noncompliance with these Rules, Organizers and Organizers Affiliates reserve the right to disqualify the Entry and/or you and/or your Team members from this Challenge and any other contest or promotional activity sponsored or administered in any way by the Organizers.

A participant is not allowed to create more than one account to participate in the challenge. Violating this will result in disqualification from the challenge.

Participants should not attempt to get around the limited number of submissions during the test phase by entering several teams into the competition. Participants should only be associated with one Entry. If two teams have overlap of participants, or if the organizers deem two entries to be effectively similar modulo small changes, they reserve the right to disqualify both teams.

If a participating team does not receive the Prizes for any of the above-mentioned reasons, the prizes will be offered to the next eligible team on the final leaderboard.

11. How may the Entry potentially be used?

The Entry may be used in a few different ways. Organizers do not claim to own your Team’s Entry, however, by submitting the Entry you and each member of your Team:

  • to be used in a report submitted to Organizers after the competition has ended, and possibly released as an open-source dataset; Admins to use the rights and licenses granted hereunder;
  • acknowledges and agrees that the Team will not be compensated and may not be credited (at Organizer’s sole discretion) for the use of the Entry as described in these Rules;
  • acknowledges that the Organizers may have developed or commissioned materials similar to the Entry and waive any claims resulting from any similarities to the Entry;
  • understand and acknowledge that, subject to the provision of Prizes, Organizers are not obligated to use the Entry in any way, even if the Entry is selected as a winning Entry.
  • agrees that each member will execute any necessary paperwork for Organizers and Organizers Admins to use the rights and licenses granted hereunder;

The solution developed in this challenge can be utilised for generating neural network models suitable for production or real-time implementation. Personal data you submit in relation to this Challenge will be used by Organisers.

12. How will Winners be Selected and Notified?

A list of all winners of this Challenge will be posted on AIcrowd Site and may be announced at Organizers' discretion via Organizers' Twitter, Facebook, Blog, or Website, or at an Organizer or Organizer, Admins sponsored or hosted event.

13. Your Odds of Winning

ODDS OF WINNING A PRIZE ARE SUBJECT TO THE TOTAL NUMBER OF ELIGIBLE ENTRIES RECEIVED AND HOW YOUR ENTRY SCORES IN ACCORDANCE WITH THE JUDGING CRITERIA.

14. When will prizes be awarded?

The prizes will be awarded within a commercially reasonable time frame to the designated Team Leader unless otherwise agreed to by Team Leader, remaining Team members and Organizer. All members of a Team may be required to complete and sign additional documentation, such as non-disclosures, representations and warranties, liability and publicity releases (unless prohibited by applicable law), and tax documents, or other similar documentation in the manner and within the timeframe specified by Organizer in order for the potentially winning team to claim the prize. Organizers will in no way be involved in any dispute with respect to receipt of a prize by any other members of a Team, including, without limitation, division of the prize value among Team members.

Winners are responsible for any tax liability that may result from receipt of any prize. Only prizes claimed in accordance with these Rules will be awarded.

15. Your Personal Data and Privacy

Organizers may use cookies and/or collect IP addresses for the purpose of implementing or exercising its rights or obligations under the Rules, for information purposes, identifying your location, including without limitation for the purpose of redirecting you to the appropriate geographic website, if applicable, or for any other lawful purpose in accordance with the AIcrowd Privacy Policy.

Organizers may use the personal data you provide via your participation in this Challenge:

  • to contact you in relation to the Challenge;
  • to confirm the details of your Entry;
  • to administer and execute this Challenge, including sharing it with Organizer Admins;
  • at Organizers’ discretion, to credit you and/or your Team for the Entry, identify you and/or your Team as a Winner, or other similar notice; and
  • as otherwise noted in these Rules or as necessary for Organizers to meet their obligations under these Rules or applicable law.

Organizers only require name and email address to be submitted for you to participate in this Challenge for its uses as outlined in the Participation Terms. Please read the AIcrowd Terms and Conditions, Participation Terms carefully to understand how your data may be used by AIcrowd SA.

16. Additional Terms and Conditions

  • The organizer reserves the right to amend the Rules and timeline without giving prior notification or any reasons thereof.
  • In case of conflicts, the decision of the Organizers will be final and binding.
  • Violation of the rules or other unfair activity may result in disqualification.
  • By participating in the Challenge, each Participant agrees that: (1) any and all disputes, claims, and causes of action arising out of or in connection with the Challenge, or any prize awarded, shall be resolved individually without resort to any form of class action; and under no circumstances will a Participant be permitted to obtain any award for, and each Participant hereby waives all rights to claim, punitive, incidental or consequential damages and any and all rights to have damages multiplied or otherwise increased and any other damages.
  • If there is a conflict between these Official rules and the AIcrowd Terms of use, the Aicrowd Terms of use will govern.