🏆 Behind the Winning Strategy of Team db3 [Meta CRAG 2024]

View post 0

🏆 Winner Spotlight Series md_d [Meta KDD Cup 2024]

View post 0

🏆 Learning from Team dRAGonRAnGers’ Strategy [Meta CRAG 2024]

View post 0

🏆 Winner Spotlight Series md_d [Meta KDD Cup 2024]

In this spotlight, we explore the modular, logic-aware system from Mitchell DeHaven, username md_dh, secured 3rd place in Task 1.

Mitchell’s Pipeline – MARAGS System Overview

A custom-built Multi-Adapter Retrieval-Augmented Generation System (MARAGS) included:

• Document chunking via BeautifulSoup (<2000 characters per segment)
• Cross-encoder reranking of segments by query relevance
• Modular LoRA adapters fine-tuned per subtask
• CoT prompting for complex reasoning
• Evaluation through code execution of API responses

Hallucination Control and Answer Reliability

• Contexts were pre-filtered for “hittability”—whether they included the actual answer
• If uncertain, the model was explicitly prompted to output: “I don’t know”
• API call responses were tested via eval() for execution correctness

What Carries Over to 2025?

Mitchell’s strategy remains highly relevant for this year’s MM-RAG focus:

• Modular tuning scales to multi-modal pipelines via adapter switching
• Hittability filtering helps reduce noise across web, image, and KG fusion
• Evaluation via execution mirrors this year’s emphasis on verifiability and trust
• Chain-of-Thought prompting supports visual reasoning and multi-hop QA

Mitchell DeHaven is a machine learning engineer at Darkhive, with prior experience in NLP and speech systems at USC’s ISI.

Stay tuned for more insights from past CRAG standouts—and good luck with your submissions!

Read other winning strategies here: db3 and dRAGonRAnGers

View post 0

🏆 Learning from Team dRAGonRAnGers’ Strategy [Meta CRAG 2024]

Winner Spotlight Series: Day 2 – dRAGonRAnGers
2nd Place in Task 1 | 3rd Place in Task 2 & 3

As we look ahead to the Meta CRAG-MM Challenge 2025, it’s worth revisiting the inventive strategies that emerged last year. In this edition of the Winner Spotlight series, we highlight Team dRAGonRAnGers from POSTECH, who earned podium finishes across all three tasks with their pragmatic and efficiency-driven approach to RAG. You can also read the complete technical report over here.

Their work is a lesson in thoughtful engineering—optimising for real-world constraints without compromising answer quality.

Challenge Recap: A Demanding Test of RAG

The 2024 CRAG Challenge pushed participants to develop Retrieval-Augmented Generation systems that could reason over web documents and structured graphs with minimal hallucinations. Success depended not just on accurate retrieval, but also on balancing cost, latency, and model robustness.

Core Insight: Trust the Model—But Verify

At the heart of the dRAGonRAnGers’ approach was an elegant refinement of the RAG pipeline aimed at:

• Avoiding unnecessary retrievals when the LLM already had a high-confidence response
• Preventing hallucinations by validating outputs through self-reflection

Their strategy revolved around a two-stage enhancement process:

Step 1: Retrieval Bypass via LLM Confidence

Rather than treating retrieval as mandatory, the team built a mechanism to assess the confidence of the LLM’s internal knowledge (likely using fine-tuned LLaMA variants). When confidence crossed a defined threshold, the system skipped retrieval entirely, saving compute and latency.

This adaptive gating proved particularly effective for factoid or frequently seen questions—an increasingly relevant optimisation for production-grade QA systems.

Step 2: Post-Generation Answer Verification

Even when retrieval was bypassed or ambiguous data was returned, the team added a verification layer: a second pass through the LLM to judge the trustworthiness of the output.

This form of self-consistency checking acted as a safeguard, filtering out hallucinations and improving answer reliability.

Outcome: Efficient, Accurate, Scalable

The combination of selective retrieval and post-hoc verification resulted in:

• Lowered system load without sacrificing accuracy
• Fewer hallucinations, particularly in borderline or low-signal queries
• Improved responsiveness for multi-turn and interactive scenarios

In a challenge that increasingly reflects real-world constraints, their system offered a compelling balance between precision and pragmatism.

Meet the Minds Behind dRAGonRAnGers

The team hails from the Data Systems Lab at POSTECH, blending deep academic research with a drive to tackle applied AI problems.

Their participation was driven by a shared goal: explore the real-world trade-offs of building reliable, cost-efficient RAG systems.

Lessons for 2025: Efficiency Is a Competitive Advantage

While the CRAG-MM Challenge 2025 introduces multi-modal and multi-turn elements, the principles behind dRAGonRAnGers’ design carry forward:

• Retrieval Gating: In image-heavy queries, selectively triggering retrieval (e.g., only when OCR or visual tags lack clarity) could save valuable inference time.
• Answer Verification: With more complex inputs (e.g., image-KG hybrids), validating generated answers before surfacing them remains crucial.
• Resource-Aware Design: Their cost-conscious pipeline offers a strong blueprint for systems facing real-time or on-device constraints.

Stay tuned for more Winner Spotlights—and best of luck as you shape your own strategy for this year’s challenge.

Read other winning strategies here: db3 and md_dh

View post 0

🏆 Behind the Winning Strategy of Team db3 [Meta CRAG 2024]

As we gear up for the new round of Meta CRAG MM Challenge 2025, let’s revisit the standout approaches from last year’s competition. In this Winner Spotlight, we dive into the strategy behind Team db3, who took the top spot across all three tasks in the Meta KDD Cup 2024 – CRAG Challenge. You can also read the complete technical report over here.

This deep dive is designed to inform and inspire participants aiming to push boundaries in retrieval-augmented generation (RAG) this year.

Challenge Overview: What Was CRAG 2024 All About?

The 2024 CRAG challenge focused on building RAG systems capable of sourcing relevant knowledge from web documents and mock knowledge graphs to answer complex queries. It tested not just retrieval and generation quality but also robustness and hallucination control.

Team db3, comprising Jiazun Chen and Yikuan Xia from Peking University, achieved:

• 1st in Task 1 – Retrieval Summarisation (28.4%)
• 1st in Task 2 – Knowledge Graph + Web Retrieval (42.7%)
• 1st in Task 3 – End-to-End RAG (47.8%)

Task 1: Retrieval Summarisation

Team db3 engineered a layered retrieval-generation pipeline:

• Parse HTML with BeautifulSoup
• Chunk text using LangChain into retrievable segments
• Retrieve with the bge-base-en-v1.5 model
• Rerank results using a custom relevance model
• Add dynamic fallback: prompt the model to say “I don’t know” when uncertain

Tasks 2 & 3: Knowledge Graph + Web Integration

Their architecture evolved with more complex inputs and integrations:

• Combine structured data (mock KGs) and unstructured web pages
• Implement a Parent-Child Chunk Retriever for fine-grained retrieval
• Use a tuned LLM to orchestrate API calls via a controlled, regularised set
• Perform heavy reranking to ensure only the most relevant data reached the generator

Hallucination Mitigation

To keep outputs grounded and reliable, the team:

• Fine-tuned the model to rely only on retrieved evidence
• Added constraints to reduce overconfident generations
• Used Python-based calculators for numerical reasoning tasks

Meet the Team

Jiazun Chen and Yikuan Xia are third-year PhD candidates at Peking University, advised by Professor Gao Jun.

Their research focuses on:

• Community search in massive graph datasets
• Graph alignment for cross-domain analysis
• Table data fusion across heterogeneous sources

What Carries Over from 2024 to 2025?

While the Meta CRAG-MM Challenge 2025 takes a leap into multi-modal and multi-turn territory, several principles from db3’s approach remain highly applicable:

• Structured + Unstructured Retrieval
  db3’s integration of knowledge graphs and web data directly informs Task 2 of CRAG-MM, which fuses image-KG with web search.

• Hallucination Mitigation
  Their use of grounded generation and standardised fallback (“I don’t know”) is vital in MM-RAG, where conciseness and truthfulness are tightly evaluated.

• Reranking and Retrieval Granularity
  Techniques like Parent-Child Chunk Retrieval can be adapted to visual-context-aware retrieval in 2025.

• LLM-as-Controller
  db3’s LLM-mediated API selection prefigures the multi-turn query orchestration required in this year’s task 3.

In short: while the modality has evolved, the core disciplines—retrieval quality, grounding, and structured reasoning—remain front and centre. Studying the 2024 winning strategy is still a powerful head start for 2025.

Stay tuned for the next Winner Spotlight—and good luck with your submissions.

Read other winning strategies here: dRAGonRAnGers and md_dh

View post 1

📢 📕 Updated Starter-Kit Release - v0.1.1

View post 0

💬 Feedback & Suggestions

Hi @bunnyveil
This issue is now fixed! You should be able to invite new team members now.

View post 0

The cragmm_search file is missing

Hi,

Please check out this post for dataset details

View post 0

💬 Welcome to the Meta CRAG - MM Challenge 2025!

View post 0

CPDC Winner Spotlight: 💡 Strategies to Improve Solutions For Task 2

Hope you’re enjoying Round 1 of the CPDC 2025 Challenge! As you prepare for the upcoming round, we’re excited to share a spotlight on the winning strategies from CPDC 2023. These highlights offer practical insights and implementation tips to help strengthen your approach.

The solutions for Task 1 of CPDC 2023 (a dialogue generation task) are most closely related to Task 2 of CPDC 2025, which also focuses on persona-consistent dialogue generation.

Task 1 Winner
First Place: Kaihua Ni
Key Insight: Combining LLM Fine-tuning with Advanced Prompt Engineering
Username: @ni_kai_hua

Background: AI graduate from University of Leeds with experience at Augmentum and CareerBuilder. Specialises in AI, deep learning, and language dynamics.

Winning Strategy:

Two-Pronged Approach:

• Fine-tuned an LLM to emulate specific individuals
• Engineered precise, persona-aligned prompts to guide output generation

Key Methods:

Fine-Tuning with Transfer Learning:

• Used curated datasets (dialogues, writings) aligned with target personas
• Adapted models to reflect individual styles and semantics

Advanced Prompt Engineering:

• Defined clear conversational goals
• Subtly incorporated persona traits
• Maintained coherence across multiple dialogue turns

Dialogue Coherence:

• Applied attention window tuning and context control

Custom Evaluation Loop:

• Built bespoke evaluation metrics aligned with CPDC scoring
• Iterative refinement based on metrics

Ethical Safeguards:

• Embedded privacy protections
• Prevented harmful/inappropriate content
• Ensured ethical persona emulation

Insight: Demonstrated how LLMs can generate nuanced, human-like dialogue without compromising integrity

Implementation Tips
Want to apply Kaihua’s approach to your solution? Here are some practical steps:

For the fine-tuning component:

• Start with a smaller, more efficient LLM as your base model
• Create a curated dataset that specifically represents your target personas
• Focus on preserving stylistic elements in your training data, not just semantic content

For the prompt engineering component:

• Structure your prompts with clear sections for conversation goal, persona traits, and dialogue history
• Experiment with different attention window sizes to find optimal context retention
• Implement a simple evaluation loop to measure improvements against CPDC’s scoring criteria

Task 1 Runner-Up
Second Place: Zhiyu Wang
Key Insight: Principles-Driven Prompt Engineering for Persona Alignment
Username: @wangzhiyu918
Team: Zhiyu Wang, Puhong Duan, Zhuojun Xie, Wang Liu, Bin Sun, Xudong Kang, Shutao Li

Background: PhD candidate at Hunan University focusing on vision-language understanding, LLMs, and multi-modal LLMs.

Winning Strategy:

Core Focus: Prompt engineering inspired by recent LLM advancements (ChatGPT, LLaMA)

Key Methods:

• Studied The Art of ChatGPT Prompting guide
• Based strategy on three principles:
  • Clarity: Specific language for accurate comprehension
  • Conciseness: Avoided unnecessary verbosity
  • Relevance: Ensured alignment with dialogue context and persona
• Refined prompts using GPT-4
• Deployed carefully designed prompt (available in their repository)

Insight: The methodical and prompt-focused design contributed to generating highly coherent, persona-aligned responses

Implementation Tips
Want to apply Zhiyu’s approach to your solution? Here are some practical steps:

Study effective prompting techniques:

• Review prompting guidelines and best practices from established sources
• Analyze the structure of successful prompts for persona-based dialogue

Apply the three core principles:

• Clarity: Replace vague instructions with specific directives
• Conciseness: Remove redundant or tangential information from prompts
• Relevance: Ensure every element of your prompt directly contributes to persona alignment

Iterative refinement:

• Use GPT-4 or similar models to test prompt variations
• Create a systematic testing framework to compare prompt performance

Task 1 Third Place
Third Place: Kaidi Yan
Key Insight: Strategic Minimalism in Prompt Design
Username: @kevin_yan
Team: Kaidi Yan, Jiayu Liu

Background: Software engineer at a large technology company, primarily working on server-side C++ development, with recent focus on LLMs.

Winning Strategy:

Core Focus: Targeted prompt engineering, carefully adapted to new scoring rules and aimed at simulating natural dialogue flow

Key Methods:

• Defined clear objective at the start of the prompt
• Designed special prompts for initial utterances to simulate realistic conversation openers
• Merged all prior utterances into a single user prompt instead of user/system pairs
• Post-processed model responses for completeness and fluency
• Deliberately kept prompt length short to avoid overfitting

Insight: While brevity may have limited peak performance, his approach prioritised adaptability and relevance — a strategic trade-off for generalisation

Implementation Tips
Want to apply Kaidi’s approach to your solution? Here are some practical steps:

Simplify your prompt structure:

• Start with a clear, concise objective statement
• Remove unnecessary complexity and instructions
• Focus on the essential elements needed for persona alignment

Improve conversation handling:

• Create specialised handling for conversation starters
• Experiment with merging dialogue history into unified context
• Implement lightweight post-processing for response quality

Balance brevity with performance:

• Test incrementally shorter prompts while monitoring performance
• Identify which prompt elements contribute most to score improvement
• Find the optimal balance between prompt length and effectiveness

View post 0

🎮 Welcome to Commonsense Persona-Grounded Dialogue Challenge 2025

View post 0

🎮 Welcome to Commonsense Persona-Grounded Dialogue Challenge 2025

View post 0

💬 Feedback & Suggestions

Hello,

The organisers at Sony are currently conducting the final round of human evaluation. As per the challenge rules, the top entries on the final leaderboard will be assessed through human evaluation, and the winning teams will be selected based on the results of this subjective assessment.

We are awaiting the outcome from the organisers and will share an update as soon as we receive it.

Thank you for your patience.

View post 1

About the Integrating Contextual Dialogue and Task Execution category

View post 0

About the Task-Oriented Dialogue (Task 1) category

View post 0

About the Context-Aware Dialogue (Task 2) category

View post 0

About the Test Challenge 4/4-186134 category

View post 0

About the Test Challenge 4/4-186134 category

View post 0

About the Test Challenge 4/4-186134 category

View post 0