MEDIQA 2021 - Radiology Report Summarization (RRS)
ACL-BioNLP Shared Task
MEDIQA 2021 tackles three summarization tasks in the medical domain:
- Consumer Health Question Summarization (QS@AIcrowd),
- Multi-Answer Summarization (MAS@AIcrowd),
- Radiology Report Summarization (RRS@AIcrowd).
In this shared task, we will also explore the use of different evaluation metrics for summarization.
MEDIQA 2021 will be organized at the NAACL-BioNLP 2021 workshop.
The automatic summarization of radiology reports has several clinical applications such as accelerating the radiology workflow and improving the efficiency of clinical communications.
This task aims to promote the development of clinical summarization models that are able to generate radiology impression statements by summarizing textual findings written by radiologists.
Training Data: A subset from the MIMIC-CXR Dataset [13,14] could be used for training. Instructions and scripts to download this training set are described here: https://github.com/abachaa/MEDIQA2021/tree/main/Task3.
Participants can use available external resources. But, please note that the rest of the MIMIC-CXR reports as well as the Indiana dataset should not be used for training.
Validation set: A subset from the MIMIC-CXR and Indiana datasets, available here: https://github.com/abachaa/MEDIQA2021/tree/main/Task3.
Test Set: A subset from the MIMIC-CXR and Indiana datasets, and a new test set from the Stanford University School of Medicine. The test set will be available for the registered participants under the Resouces Section.
The registration & data usage agreement form is available under the Resources section of the AIcrowd projects. The form covers the three tasks. You can download it from any of the three MEDIQA projects: QS@AIcrowd, MAS@AIcrowd & RRS@AIcrowd.
To register, you need to complete, sign, and upload the form. When approved, you will be able to download the official test sets and to submit your runs on the AIcrowd submission systems.
January 29, 2021: Release of the validation sets.
February 26, 2021: Release of the test sets. Run submission opens on AIcrowd.
March 5, 2021: Run submission deadline. Participants' ROUGE scores will be available on AIcrowd.
March 10, 2021: Release of the official results.
March 17, 2021: Papers due date (Submission website and instructions).
April 15, 2021: Notification of acceptance.
April 26, 2021: Camera-ready papers due (hard deadline).
🖊 Evaluation Metrics
ROUGE will be used as the main metric to rank the participating teams, but we will also use several evaluation metrics more adapted to each task.
🚀 Submission format
- Format: ID [tab] Summary
-- Task 1 & Task 2 => question_id [tab] summary
-- Task 3 => study_id [tab] summary
- The summary must fit in one line (no line breaks).
1) Each team is allowed to submit a maximum of 10 runs.
2) Please choose a username that represents your team, and update your profile with the following information: First name, Last nam, Affiliation, Address, City, Country.
3) For each run submission, it is mandatory to fill in the submission description field of the submission form with a short description of the methods, tools and resources used for that run.
4) The final results will not be considered official until a working notes paper with the full description of the methods is submitted.
📱 Contact us
- We strongly encourage you to use our mailing list for communications between the participants and the organizers: https://groups.google.com/d/forum/bionlp-mediqa
- In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at: firstname.lastname@example.org