US
Activity
Ratings Progression
Challenge Categories
Challenges Entered
Completed
Meta
Improve RAG with Real-World Benchmarks
Latest submissions
Amazon Search
Revolutionise E-Commerce with LLM!
Latest submissions
See All| graded | 270741 | ||
| graded | 270740 | ||
| graded | 270655 |
AIcrowd
| Participant | Rating |
|---|---|
mincheolyoon
|
0 |
happystat
|
0 |
unna97
|
0 |
linchia
|
0 |
Karrich
|
0 |
gaozhanfire
|
0 |
pengbo_wang
|
0 |
pp_
|
0 |
pengyue_jia3
|
0 |
GenpengXu
|
0 |
xiaopeng_li
|
0 |
| Participant | Rating |
|---|
-
NVIDIA-Merlin Amazon KDD Cup '23: Multilingual Recommendation ChallengeView
-
Team_NVIDIA Amazon KDD Cup 2024: Multi-Task Online Shopping Challenge for LLMsView
Amazon KDD Cup 2024: Multi-Task Online Shopping Ch
All Submissions Are Failing
23 days agoOur team’s last 6 submissions failed. And when I look at the list of submissions from the other teams in the past 4 hours, all other teams failed too. Is there a problem with AIcrowd server?
Here are the links of our team’s last two failures [here] and [here]
Can an admin please investigate? Thank you.
Push gitlab and cannot find issue
About 1 month agoThe same thing has just happened to me. I have have created 5 new tags. They all appear in my GitLab but none appear in my issues.
They are tags submission-200, submission-202, submission-203, submission-204, submission-205. Some code are duplicates of each other because I tried submitting the same thing twice without success.
All Submissions "Waiting In Queue" for 12 Hours
About 1 month agoFYI, all submissions (from all teams) have been “waiting in queue” for the past 12 hours. Perhaps an admin can investigate. Thanks.
Submission stuck on "evaluation initiated"
About 1 month agoThe following two submissions [here] and [here] are stuck with label “evaluation initiated” even though they have failed.
Can an admin switch the GitLab label to failed? Because as is, they are using 2 submission quotas. Thanks.
Submission Failed - Please Tell Us When Submission Works Again
About 1 month agoYes, this is not fixed. I just submitted and got
Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists..
The GitLab issue is [here]
For the past 2 days, no team has been able to submit to track 5.
Please fix this issue and let us know when it is fixed. Thank you
Submissions fail
About 1 month agoI am also seeing weird submission behavior today. I posted a discussion describing the errors I have been seeing today [here]
Submission Failed - Please Tell Us When Submission Works Again
About 1 month agoHi, for the past 4 hours, I have been receiving " Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists.." when submitting to track 5. An example GitLab issue (for admins to review) is [here].
I have tried 3 times and received 3 “failed” submissions. I do not want to try anymore because I do not want to use up my failed submission quota. Can an admin tell us when submissions are working for track 5 again? Thanks.
Track 2 LB Doesn't Show Retrieval Score
About 1 month agoHi, Can admins @yilun_jin fix the track 2 leaderboard webpage to show each teams’ retrieval score? Thank you.
Phase 2 launching!
2 months agoI notice that AIcrowd website says “Round 2: 21 days left” which implies that phase 2 ends on June 15th. It this the correct end of phase 2?
Another Frozen Evaluation
3 months agoThank you for fixing our previous frozen evaluation.
We have another evaluation here. The GitLab issue page shows that it failed but the AIcrowd website is still showing that the submission is being evaluated.
As such, we cannot submit again to this track because the AIcrowd website thinks that a submission is in progress. Can an admin @yilun_jin update the AIcrowd website to acknowledge that our submission failed thus allowing us to make a new submission?
Thank you.
Our Evaluation is Frozen
3 months agoHi. Our submission to track 1 [here] has frozen. It shows 97% completed and it appears to be within all time limits. There has been no update for the past 3 hours.
Can an admin @yilun_jin please unfreeze our submission and post the results? Thank you
Are some errors caused by AIcrowd server and not submission code?
3 months agoHi. Can an admin please solve the following problem and provide a score for our last submission?
We just submitted our other code that failed yesterday and this time the AIcrowd GitLab issue says the error is caused by AIcrowd server. The server could not load the sentence-transformers/all-MiniLM-L6-v2 to evaluate the score. Below is copy and paste from reason for failure.
Evaluate Scores: OSError: We couldn’t connect to ‘https://huggingface.co’ to load this file, couldn’t find it in the cached files and it looks like sentence-transformers/all-MiniLM-L6-v2 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at ‘https://huggingface.co/docs/transformers/installation#offline-mode’.
Our code successfully predicted all questions within all time limits and then AIcrowd server failed to load sentence transformer to compute our LB score
Are some errors caused by AIcrowd server and not submission code?
3 months agoI just submitted the exact same code to the exact same track that failed earlier today. This time it succeeded. Does this mean that we need to repeatedly submit our failed code to AIcrowd server?
What is causing the inconsistent behavior of AIcrowd server? Do fails that occur because of AIcrowd server count toward are weekly track 1-4 limit of 20 and weekly track 5 limit of 3?
Are some errors caused by AIcrowd server and not submission code?
3 months agoToday it took 10+ hours to receive the result of two submissions that we made. After submitting, we were watching the progress at AIcrowd GitLab issues. The submissions were successfully predicting questions. Then the updates froze and the submissions were labeled “evaluation_initated” for the next 8 hours. After 8 hours we saw that the submissions were labeled “failed”.
The debug logs do not show any errors. This is the same code that worked before (on a different track) and/or made more progress before (on same track).
Is it possible that the errors and/or inconsistent behavior is caused by AIcrowd servers? Should we just submit the same code again?
Has any team submitted the exact same code twice. And one time the code failed and one time the code was successful?
The maximum number of players per team.
3 months agoThere are currently two teams with 7 members. Do you mean to say “the maximum is 7”?
We Cannot View Submissions
3 months agoThank you admins for adding the “Submission” button to the individual track 1, 2, 3, 4, 5 webpages.
Can you also add the “Submission” button to the main challenge page? Currently if we go to our profile page and click “See All” referring to see all submission in this competition it does not work. This is because it attempts to use URL:
https://www.aicrowd.com/challenges/amazon-kdd-cup-2024-multi-task-online-shopping-challenge-for-llms/submissions
which is currently disabled by admins.
What Does 2 Submission Per Week Mean?
4 months agoWhat does 2 submissions per week mean? Is this based on a calendar week, like we can only make 2 submissions Monday thru Sunday? Or is this based on a sliding 7 day window, like we can only make maximum 2 submissions per 7 day sliding window?
We Cannot View Submissions
4 months agoAt the top of the track 1 page there are 8 buttons named “Overview, Leaderboard, Notebooks, Discussion, Insights, Resources, Submissions, Rules”.
If we click the button “Submission” we see all submissions for track 1.
There is no way to see submissions for tracks 2, 3, 4, 5. Because neither the main page nor track 2 nor track 3 nor track 4 nor track 5 has a “Submission” button.
Can an administrator, add “Submission” button to each track? Thank you!
Why Doesn't Leaderboard Show Ranking Score?
4 months agoI noticed that the leaderboard for each track shows the score for the tasks: Multiple Choice, NER, Retrieval, and Generation. But no leaderboards show the results for Ranking?
Is this on purpose or a bug?
Note for our final evaluation
15 days ago@yilun_jin , Sometimes the exact same code will succeed one time and fail another time. For example we submitted the exact same code [here] and [here]. The first succeeded and the second failed. During re-run what happens if code fails that has previously succeeded? Will the admins run it a second time?
Also, can you tell us why the second link above failed?
When we select our final 2 submissions for each track, should we just select our best scoring submission twice in case it fails the first time it is re-run?