RAG and Agent Memory: Chunking, Pitfalls, Hallucinations

Q: True or false: 'To make an LLM master a batch of new factual knowledge, fine-tuning is usually more reliable than RAG.'

False。The evidence is the opposite — for injecting new facts, RAG consistently beats unsupervised fine-tuning; fine-tuning is better for changing style / format, not loading facts.

Why RAG: the model's knowledge is 'baked in'

Knowledge in the model's weights (parametric memory) has two flaws: it goes stale and it can't be edited (you'd have to retrain). RAG = parametric memory + non-parametric memory (a retrievable external vector index), feeding retrieved documents into generation. The upside: swap the index to update knowledge without retraining; it makes generation more specific and more factual, lowering hallucination — note that's lower, not eliminate (retrieve the wrong thing and it still makes things up).

Retrieval comes in two flavors: sparse retrieval (BM25 / TF-IDF, keyword matching) and dense retrieval (learned vector embeddings, semantic matching that handles 'different wording, same meaning'). And a common myth to bust: want the model to remember new knowledge, so you fine-tune? The evidence is the opposite — unsupervised fine-tuning struggles to teach an LLM new facts; for injecting knowledge, RAG consistently beats fine-tuning.

⚠️Myth: 'new knowledge = fine-tuning.' For injecting facts, prefer RAG, not fine-tuning; fine-tuning is better for changing style / format.

RAG's engineering reality: chunking, lost-in-the-middle, and its seven pitfalls

RAG isn't done the moment you bolt on a vector DB. Chunking decides retrieval granularity: chunks too big dilute meaning and may exceed the embedding model's context and get truncated; too small loses context. There's no magic size — test against your queries. And a counterintuitive phenomenon: Lost-in-the-Middle — LLMs use long context in a U-shape: relevant info does best at the start or end, and drops noticeably in the middle (can drop over 30%). So how retrieval results are ordered and placed really matters.

RAG failures can be catalogued — the seven failure points: missing content / not retrieved / not in context / not extracted / wrong format / wrong granularity / incomplete. Don't eyeball evaluation either: RAGAS uses 'reference-free' metrics (no human labels needed) — Faithfulness (how much of the answer is supported by the retrieved context) + Answer Relevancy. On memory: an agent's memory has two layers — short-term (in-thread conversation history, tied to the session) + long-term (persisted across sessions, usually fetched via vector retrieval).

💡Callback: you met Lost-in-the-Middle back in 'Context Engineering' — RAG pushes it from 'how to write the prompt' to 'how to order the retrieved results.'

自测 · 学完检查一下

想真正动手做题、记进度、攒连胜？到互动课里练。

Compared with using the model's built-in knowledge directly, RAG's core advantage is?

答案：Knowledge is retrievable and updatable without retraining, and answers are more factual, lowering (not eliminating) hallucination

RAG = parametric + non-parametric memory; swap the index to update, and it lowers hallucination but doesn't eliminate it (wrong retrieval still makes things up).

What does 'Lost-in-the-Middle' describe?

答案：LLMs use long context in a U-shape — info placed in the middle is most easily ignored

A U-shaped curve — start and end do well, the middle drops noticeably — so the ordering/position of retrieved results matters.

True or false: 'To make an LLM master a batch of new factual knowledge, fine-tuning is usually more reliable than RAG.'

答案：False

The evidence is the opposite — for injecting new facts, RAG consistently beats unsupervised fine-tuning; fine-tuning is better for changing style / format, not loading facts.