Is there a limit on the total number of submissions when merging teams?

Hi @good-good-study, yes, it is 5 submissions/task/day.

We have just resetted the submission limit due to issue with code submissions earlier which should solve the wrong counting. Please let us know incase you still face any issues.

View post 1

Predicition submission

Hi @wxl121228, yes the prediction submissions are closed right now in favour of code submissions round.

More info about the code submission round:

View post 0

🚀 Code Submission Round Launched 🚀

Hi @amiruddin_nagri, the default leaderboard is “Code Submissions” which starter recently.

You can select “Prediction Submissions” to view the previous submissions and their relevant scores.

View post 0

🚀 Code Submission Round Launched 🚀

Hi @wufanyou,

The absolute file paths are available as parameters of predict function.
(to get rid of any confusion participants may face with directory structures)

We have data, models, etc folders available as examples for the getting started section and a better local development experience. You are free to upload your models and relevant files in folder/structure of your choice. The only constraint is that the entrypoint to your code evaluations will be run.py.

The idea to provide product catalogue by default sounds good.
Please let us check internally and we will update on the request soon.

View post 1

🚀 Code Submission Round Launched 🚀

Hi @wufanyou,

Thanks for your questions.

The submission process is as follows

image1028×168 5.14 KB

The public test set phase is available for debugging purposes.
You can run your code end to end on the public split and the logs for this phase will be available, etc. The time taken in image building and public split stages, etc would not be counted towards your 30 minutes timeout (and have their own timeouts to avoid misuse).

The private split run however have 30 minutes purely reserved for your end to end run on the private dataset.

Yes, the default docker image is nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04, however you are not restricted to use that and can choose to use requirements.txt, environment.yml to even custom images & Dockerfile.
Please note, in case you are using your custom image, you need to make sure that they have relevant Nvidia drivers installed to make use of the GPU.

A full detail about the runtime is available in starter kit’s description as well as on Discourse here.

Please check more details in the starter kit’s README and let us know if you still have any follow up questions.

View post 1

🚀 Code Submission Round Launched 🚀

View post 0

Are there any way to modify the team name？

Hi @wufanyou,

Unfortunately you can’t modify the team name directly right now.

Please drop a message with your previous & new team name.
We will get it changed shortly!

View post 0

Is there a limit on the total number of submissions when merging teams?

Hi @heya5,

This challenge has daily submissions limit of 5 submissions/day/task/team.

While I understand the reason behind submission * total_day condition for team merging on other platforms.
We do not have this check currently for this challenge.

Please note, that, to tackle above issue, we generally have team freeze deadline after which new team members or teams can’t be created. It is on July 1, 2022 for this challenge.

Best,
Shivam

View post 0

🚀 Dataset Update: `v0.3` of the dataset has been released 🚀

Hi @trenusch, please verify example_id column in your submission.
It looks like you are using indices [0....n] instead of the the example_id which are required for a valid submission, due to which your submissions are failing.

You can check sample submission for the correct example_ids as well.

image998×430 69.1 KB

View post 0

🚀 Dataset Update: `v0.3` of the dataset has been released 🚀

Hi @trenusch,

The new test dataset files are already available on the resources page along with this announcement.

image894×626 23.2 KB

The new submissions since the release have the scores calculated on the new dataset (along with older submissions re-evaluated automatically).

We will start accepting the code submissions from 21st of June, 2022, until then you can continue submitting your predictions on the v0.2 test sets. However, the evaluators will only consider the subset of the test set which are present in the v0.3 of the test sets.

Please let us know in case you are still facing some issue in accessing or submitting solution on new Task 3 dataset.

View post 0

🚀 Dataset Update: `v0.3` of the dataset has been released 🚀

Yes, you can use it for training.

View post 0

Any chance to use the dataset for my thesis?

Hi @rivima

Please check this response

View post 0

Which is the deadline?

Hi, there is no deadline.
It’s an always running benchmark.

View post 1

No way to load SequenceTagger

Hi @Bonum, here is an working example [should work on Colab] on how to use SequenceTagger:
https://www.aicrowd.com/showcase/sequencetagger-example

Yup, the files are missing in GitHub repository, while available in HF, so I have just merged them both to make it work.

All the best !

View post 0

No way to load SequenceTagger

Hi @Bonum, the notebook has following example:

from transformers import AutoTokenizer, AutoModel
tokeniser = AutoTokenizer.from_pretrained("pranaydeeps/Ancient-Greek-BERT")
model = AutoModel.from_pretrained("pranaydeeps/Ancient-Greek-BERT")  

These models are hosted on HF due to which will work directly.

While you are trying to use SuperPeitho-FLAIR-v2/final-model.pt which isn’t, as the author told in the README.

We provide a pre-trained POS Tagging model in the directory SuperPeitho-FLAIR-v2

That means, you need to clone the main repository and use the model file directly.

$> git clone git@github.com:pranaydeeps/Ancient-Greek-BERT.git
$> cd Ancient-Greek-BERT

[...]
## run your python code along with above load command

All the best !

View post 0

💡 Solutions are Public

View post 0

Local Run Produces Different AP

Hi all,

Thanks for opening this discussion, and it is indeed an important one.
I am happy that we all can brainstorm and improve the pipeline for all the participants together.

The scores calculated right now we’re using the complete annotation files as @Mykola_Lavreniuk suspected, while the predictions made by your codebases get added for the calculation purposes on the go. The intention has always been to provide real-time feedback to you to show the approximate scores as soon as possible.

Issue #1: Score gradually increases

The flaw
We run COCOEval using complete annotations BUT with the subset of images (predictions generated ones), due to which the score starts with a much lower value and grows to your final scores.

Future update
We will be making an update in the pipeline based on this thread to create a subset of annotations file dynamically, calculate the scores with that for live reporting.

Issue #2: Validation scores don’t make sense

The flaw
We were using only 100 images in the validation phase because it worked as an integration test. We missed providing a proper scores table for the validation phase that you can use to compare local v/s online performance – which is our bad. The scores get calculated with annotation files for the complete validation subset, but only 100 images are present in the validation phase (as in Issue #1), causing much lower scores.

Future update
We will continue use a tiny subset to limit the compute used in validation phase and believe the score on 100 images should be reliable enough to compare local v/s online scoring. The list of these ~100 images will be shared in the resources section in future so you can reproduce the scores locally. (meanwhile the current scores were quite low, example 0.09AP while full evaluation for a lot more, it was happening due to Issue #1)

View post 1

End of Round 2⏱️

Round 2 of Food Recognition Benchmark has ended !!

This round saw a lot more activity and a lot more rank swaps on the leaderboard.
Thanks to you all for making it a success.

While a few of submissions are being re-evaluated at the moment, the public leaderboard looks like this right now:

image1920×1307 61.5 KB

The number of participants who crossed the official baseline has increased from 12 to 15.
At the same time, participants having score >= 0.200 AP is increased from 6 to 11 as well!

We can’t wait to complete the due diligence before making the winners’ announcement.
Stay tuned for it!

Why Food Recognition Benchmark matters?

It’s a good time to re-share about exciting updates on how the community collaboration has worked so far and what it’s enabling! [Blog Post]

The Food Recognition Benchmark is not a competition, but an ongoing collaboration with a community of inspiring individuals like yourself, to deliver the future of diet tracking in research cohorts around the world.

What can you do meanwhile?

Start some interesting discussions with other challenge participants’ here on the Forums or on Discord Chat.
Share what worked and what didn’t during the competition. And maybe some interesting approaches you may want to try out in future?
Share any feedbacks you have for future Rounds either in this thread of here.
Guess the winners in this thread

View post 3

Gitlab seems to be down

Hi, it will be back up within 5 minutes.

View post 1

Submission deadline

It ends on 06 May 23:59 UTC.

View post 0