11 Follower
0 Following
Chris Deotte










Ratings Progression


Challenge Categories


Challenges Entered

Improve RAG with Real-World Benchmarks

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 270741
graded 270740
graded 270655

Latest submissions

See All
graded 235811
graded 235350
graded 235349

Latest submissions

See All
graded 235349
graded 235348
graded 235347

Latest submissions

See All
graded 235350
graded 235166
graded 235125
Participant Rating
mincheolyoon 0
happystat 0
unna97 0
linchia 0
Karrich 0
gaozhanfire 0
pengbo_wang 0
pp_ 0
pengyue_jia3 0
GenpengXu 0
xiaopeng_li 0
Participant Rating
  • NVIDIA-Merlin Amazon KDD Cup '23: Multilingual Recommendation Challenge
  • Team_NVIDIA Amazon KDD Cup 2024: Multi-Task Online Shopping Challenge for LLMs

Amazon KDD Cup 2024: Multi-Task Online Shopping Ch

Note for our final evaluation

13 days ago

@yilun_jin , Sometimes the exact same code will succeed one time and fail another time. For example we submitted the exact same code [here] and [here]. The first succeeded and the second failed. During re-run what happens if code fails that has previously succeeded? Will the admins run it a second time?

Also, can you tell us why the second link above failed?

When we select our final 2 submissions for each track, should we just select our best scoring submission twice in case it fails the first time it is re-run?

All Submissions Are Failing

20 days ago

Our team’s last 6 submissions failed. And when I look at the list of submissions from the other teams in the past 4 hours, all other teams failed too. Is there a problem with AIcrowd server?

Here are the links of our team’s last two failures [here] and [here]

Can an admin please investigate? Thank you.

Push gitlab and cannot find issue

27 days ago

The same thing has just happened to me. I have have created 5 new tags. They all appear in my GitLab but none appear in my issues.

They are tags submission-200, submission-202, submission-203, submission-204, submission-205. Some code are duplicates of each other because I tried submitting the same thing twice without success.

All Submissions "Waiting In Queue" for 12 Hours

About 1 month ago

FYI, all submissions (from all teams) have been “waiting in queue” for the past 12 hours. Perhaps an admin can investigate. Thanks.

Submission stuck on "evaluation initiated"

About 1 month ago

The following two submissions [here] and [here] are stuck with label “evaluation initiated” even though they have failed.

Can an admin switch the GitLab label to failed? Because as is, they are using 2 submission quotas. Thanks.

Submission Failed - Please Tell Us When Submission Works Again

About 1 month ago

Yes, this is not fixed. I just submitted and got
Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists..
The GitLab issue is [here]

For the past 2 days, no team has been able to submit to track 5.

Please fix this issue and let us know when it is fixed. Thank you

Submissions fail

About 1 month ago

I am also seeing weird submission behavior today. I posted a discussion describing the errors I have been seeing today [here]

Submission Failed - Please Tell Us When Submission Works Again

About 1 month ago

Hi, for the past 4 hours, I have been receiving " Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists.." when submitting to track 5. An example GitLab issue (for admins to review) is [here].

I have tried 3 times and received 3 “failed” submissions. I do not want to try anymore because I do not want to use up my failed submission quota. Can an admin tell us when submissions are working for track 5 again? Thanks.

Track 2 LB Doesn't Show Retrieval Score

About 1 month ago

Hi, Can admins @yilun_jin fix the track 2 leaderboard webpage to show each teams’ retrieval score? Thank you.

Phase 2 launching!

2 months ago

I notice that AIcrowd website says “Round 2: 21 days left” which implies that phase 2 ends on June 15th. It this the correct end of phase 2?

Another Frozen Evaluation

3 months ago

Thank you for fixing our previous frozen evaluation.

We have another evaluation here. The GitLab issue page shows that it failed but the AIcrowd website is still showing that the submission is being evaluated.

As such, we cannot submit again to this track because the AIcrowd website thinks that a submission is in progress. Can an admin @yilun_jin update the AIcrowd website to acknowledge that our submission failed thus allowing us to make a new submission?

Thank you.

Our Evaluation is Frozen

3 months ago

Hi. Our submission to track 1 [here] has frozen. It shows 97% completed and it appears to be within all time limits. There has been no update for the past 3 hours.

Can an admin @yilun_jin please unfreeze our submission and post the results? Thank you

Are some errors caused by AIcrowd server and not submission code?

3 months ago

Hi. Can an admin please solve the following problem and provide a score for our last submission?

We just submitted our other code that failed yesterday and this time the AIcrowd GitLab issue says the error is caused by AIcrowd server. The server could not load the sentence-transformers/all-MiniLM-L6-v2 to evaluate the score. Below is copy and paste from reason for failure.

Evaluate Scores: OSError: We couldn’t connect to ‘https://huggingface.co’ to load this file, couldn’t find it in the cached files and it looks like sentence-transformers/all-MiniLM-L6-v2 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at ‘https://huggingface.co/docs/transformers/installation#offline-mode’.

Our code successfully predicted all questions within all time limits and then AIcrowd server failed to load sentence transformer to compute our LB score

Are some errors caused by AIcrowd server and not submission code?

3 months ago

I just submitted the exact same code to the exact same track that failed earlier today. This time it succeeded. Does this mean that we need to repeatedly submit our failed code to AIcrowd server?

What is causing the inconsistent behavior of AIcrowd server? Do fails that occur because of AIcrowd server count toward are weekly track 1-4 limit of 20 and weekly track 5 limit of 3?

Are some errors caused by AIcrowd server and not submission code?

3 months ago

Today it took 10+ hours to receive the result of two submissions that we made. After submitting, we were watching the progress at AIcrowd GitLab issues. The submissions were successfully predicting questions. Then the updates froze and the submissions were labeled “evaluation_initated” for the next 8 hours. After 8 hours we saw that the submissions were labeled “failed”.

The debug logs do not show any errors. This is the same code that worked before (on a different track) and/or made more progress before (on same track).

Is it possible that the errors and/or inconsistent behavior is caused by AIcrowd servers? Should we just submit the same code again?

Has any team submitted the exact same code twice. And one time the code failed and one time the code was successful?

The maximum number of players per team.

3 months ago

There are currently two teams with 7 members. Do you mean to say “the maximum is 7”?

We Cannot View Submissions

3 months ago

Thank you admins for adding the “Submission” button to the individual track 1, 2, 3, 4, 5 webpages.

Can you also add the “Submission” button to the main challenge page? Currently if we go to our profile page and click “See All” referring to see all submission in this competition it does not work. This is because it attempts to use URL:
which is currently disabled by admins.

What Does 2 Submission Per Week Mean?

3 months ago

What does 2 submissions per week mean? Is this based on a calendar week, like we can only make 2 submissions Monday thru Sunday? Or is this based on a sliding 7 day window, like we can only make maximum 2 submissions per 7 day sliding window?

We Cannot View Submissions

3 months ago

At the top of the track 1 page there are 8 buttons named “Overview, Leaderboard, Notebooks, Discussion, Insights, Resources, Submissions, Rules”.

If we click the button “Submission” we see all submissions for track 1.

There is no way to see submissions for tracks 2, 3, 4, 5. Because neither the main page nor track 2 nor track 3 nor track 4 nor track 5 has a “Submission” button.

Can an administrator, add “Submission” button to each track? Thank you!

Why Doesn't Leaderboard Show Ranking Score?

3 months ago

I noticed that the leaderboard for each track shows the score for the tasks: Multiple Choice, NER, Retrieval, and Generation. But no leaderboards show the results for Ranking?

Is this on purpose or a bug?

Earned a BA in mathematics then worked as a graphic artist, photographer, carpenter, and teacher. Earned a PhD in computational science and mathematics with a thesis on optimizing parallel processing. Now work as a data scientist and researcher.