PROJECT 02 · IntermediatePROEarly draft

RAG Knowledge Assistant

A company has 300 support documents, and customers ask questions answered inside them. Your mission: an assistant that answers from the source — accurately, with citations, without inventing.

Intermediate15–20h< $5Python · pgvector

Start building

What you'll build

A full RAG system: document chunking, semantic search, reranked retrieval, cited answers, and hallucination guardrails — with an eval set proving it works.

Why it matters

RAG is the market's most-demanded AI skill. This system is the hardest-hitting portfolio piece for the job market.

Skills you'll gain

Chunking strategy · L4Embeddings & semantic searchRetrieval-quality measurementHallucination control & citationsRAG evaluationAdvanced retrieval & reranking

Prerequisites

PRJ-1 evaluated + gates M4–M5 + the provided corpus.

Requirements

RQ-1: answer faithfulness ≥ 80% on the eval set.
RQ-2: retrieval hit rate ≥ 70%.
RQ-3: 100% refusal for questions with no answer in the source.
RQ-4: every claim in an answer carries a citation resolving to its source chunk.
RQ-5: full-index cost + 100 eval questions tracked and under the declared cap.

Architecture

Product-level: documents → chunking → embeddings → vector index → retrieval + reranking → cited, guarded generation. The explanation teaches that answer quality is capped by retrieval quality — which is why retrieval is measured on its own before generation is touched.

Build — 6 stages

1Corpus to chunksE2 E3

2Meaning becomes vectorsE1 E3

3Measure retrieval firstE5 E3

4Answer from the source — onlyE1

5The verdict, in numbersE5 E4

6Harden it, meter itE5 E1

THE BAR — VISIBLE FROM THE START

QualityAn eval set of ≥50 ground-truthed questions: faithfulness ≥80% (claim-by-claim) and hit rate ≥70%.

Robustness100% refusal on unanswerable questions, ≤10% false refusal, and the code-question + ugly-input suites pass.

CostFull indexing + the entire eval run under $5, with a measured table.

Failures you'll fix yourself

Confidently wrong, source in handThe system answers with full confidence and a plausible-looking citation — but the answer is wrong, even though the correct information really is in the corpus.

Retrieval returns garbage for codesOrdinary questions work, but any question with a product code or error number ('what does ERR-4032 mean?') returns totally unrelated chunks, and hit rate on those questions is near zero.

Over-refusal after the crackdownAfter you tightened the faithfulness instructions to hit 80%, the system now refuses questions whose answers really are in the corpus — 'not found in the documents' for questions it used to answer correctly.

The docs moved on, the index sleptAn update to 20 documents arrives (in the asset pack). The system gives answers that used to be correct — but are now stale: old prices, changed procedures — with valid citations to chunks that really are in the index.