🏆 Meta CRAG Challenge 2024: Winners Announcement

Hello Participants,

We are excited to announce the winners of the Meta CRAG Challenge 2024! We deeply appreciate the efforts and contributions of every participant in helping advance Retrieval-Augmented Generation (RAG) systems. Over the last four months, the challenge saw over 2,000 participants from across the globe, with more than 5,500 submissions. Below are the winners for each task. You can find details of the final evaluation process here.

🧩 Task 1: Retrieval Summarisation

• 🥇 1st Place: Team db3
• 🥈 2nd Place: Team md_dh
• 🥉 3rd Place: Team ElectricSheep

Category Winners:

• 🌸 Simple with Condition: Team dummy_model
• 🌸 Set: Team dummy_model
• 🌸 Comparison: Team dRAGonRAnGers
• 🌸 Aggregation: Team dummy_model
• 🌸 Multi-hop: Team bumblebee7
• 🌸 Post-processing: Team dRAGonRAnGers
• 🌸 False Premise: Team ETSLab

🌐 Task 2: Knowledge Graph + Web Retrieval

• 🥇 1st Place: Team db3
• 🥈 2nd Place: Team APEX
• 🥉 3rd Place: Team md_dh

Category Winners:

• 🌸 Simple with Condition: Team ElectricSheep
• 🌸 Set: Team ElectricSheep
• 🌸 Comparison: Team dRAGonRAnGers
• 🌸 Aggregation: Team ElectricSheep
• 🌸 Multi-hop: Team ElectricSheep
• 🌸 Post-processing: Team ElectricSheep
• 🌸 False Premise: Team Future

🤖 Task 3: End-to-End Retrieval-Augmented Generation

• 🥇 1st Place: Team db3
• 🥈 2nd Place: Team APEX
• 🥉 3rd Place: Team vslyu-team

Category Winners:

• 🌸 Simple with Condition: Team StarTeam
• 🌸 Set: Team md_dh
• 🌸 Comparison: Team dRAGonRAnGers
• 🌸 Aggregation: Team md_dh
• 🌸 Multi-hop: Team ETSLab
• 🌸 Post-processing: Team md_dh
• 🌸 False Premise: Team Riviera4

🔦 Winner Spotlight Series

Let's see some standout solutions from 2024. These approaches offer valuable insights for participants aiming to build robust multi-modal RAG systems.

🌟 Team db3 – Overall Winner (1st in all three tasks)

• Team Members: Jiazun Chen and Yikuan Xia
• Affiliation: Peking University (PhD candidates under Prof. Gao Jun)
• Expertise: Community search in large graphs, table fusion, and cross-domain graph alignment.

Solution Highlights:

• Task 1: Used HTML parsing (BeautifulSoup), LangChain chunking, bge-base-en-v1.5 retriever, custom reranker, and fallback prompts (“I don’t know”) for uncertain outputs.
• Tasks 2 & 3: Combined web and KG data using a Parent-Child Chunk Retriever, orchestrated LLM-controlled API calls, and added tight reranking constraints.
• Hallucination Control: Fine-tuned generation on grounded evidence, used calculators for numerical reasoning, and constrained LLM output behaviour.

🔗 Read Full Solution

🌟 Team dRAGonRAnGers – 2nd in Task 1, 3rd in Tasks 2 & 3

• Team Members: Students from POSTECH's Data Systems Lab
• Motivation: Build cost-efficient, real-world-ready RAG pipelines.

Solution Highlights:

• Retrieval Gating: Skipped retrieval when LLM confidence was high (likely via fine-tuned LLaMA variants).
• Answer Verification: Implemented a self-consistency pass to validate generated responses.
• Optimisation Focus: Balanced latency, retrieval cost, and hallucination reduction with a two-stage architecture.

🔗 Read Full Solution

🌟 md_dh – 3rd in Task 1, Category Winner in Tasks 2 & 3

• Team Member: Mitchell DeHaven
• Affiliation: ML Engineer at Darkhive; previously at USC’s ISI
• Background: NLP, speech systems, and logic-based reasoning.

Solution Highlights:

• Pipeline (MARAGS): Used BeautifulSoup-based chunking, cross-encoder reranking, and modular LoRA adapters per task.
• Reliability Features: Filtered for “hittability”, added fallback prompts, and verified API responses through execution (e.g., eval()).
• Reasoning Tools: Used Chain-of-Thought prompting for complex logic and table operations.

🔗 Read Full Solution