RAG· Lesson 4 of 4
What Do You Build RAG With? — LangChain, LlamaIndex, or No Framework?
In one line · **The real question isn't "which framework is best" — it's: how much control and how many dependencies are you trading for ready-made orchestration comfort? And in most BuildAI cases: start with no framework — here's why…**
- A framework adds no intelligence — the model, the vector store, and the embeddings are identical in every case. What changes: who writes the code that wires them together.
- Three real options: no framework (full control, more boilerplate), LlamaIndex (strongest when the problem is data/indexing), LangChain (the broadest orchestration ecosystem, the heaviest abstraction).
- BuildAI's position: for learning and for small production systems, start with no framework — so you understand what's happening. Reach for a framework when wiring complexity genuinely becomes the problem.
- 💸 The framework doesn't change your bill: cost lives in model calls, and those are the same in all three cases.
- 🔥 This landscape moves fast — the names and APIs on this page are reviewed every 30 days.
This is literally the first question anyone building RAG asks: "Do I start with LangChain or what?" And it's answered wrong constantly — by hype: "every tutorial uses it, so it must be right."
That mistake gets paid for later in two ways. First, at debugging time: retrieval returned garbage, and you have no idea what the final prompt sent to the model looked like, because five layers of abstraction built it on your behalf. Second, at upgrade time: a new release broke the API, and you're facing a refactor of code you never understood in the first place — it just worked.
The opposite mistake is real too: teams hand-writing thousands of lines of ingestion boilerplate for 15 sources when a mature library does it better. This is an engineering decision with criteria — not team allegiance.
Recall the RAG pipeline you built: load, split, embed, store, search, build the prompt, generate. Seven steps, all ordinary code.
A framework hands you those steps pre-written and pre-wired: loaders for every file type, ready-made splitters, connectors for every vector store, and chains that tie it all together in two lines.
What you gain: a fast start and ready-made spare parts. What you pay: a layer between you and what's happening — when it works perfectly, and when it breaks too.
A real example — the same pipeline, two faces
Without a framework, the retrieval and generation you wrote in the project is about 30 lines — you can see the final prompt character by character, and top-k is a number you typed yourself.
With a framework, the same thing looks like this:
> 🔥 Volatile fragment *(API names change between releases — the idea is what lasts, not the lines)*: > > ``python > # LlamaIndex — the "it's magic" version (API surface as of Aug 2026) > index = VectorStoreIndex.from_documents(documents) > answer = index.as_query_engine().query("What is the return policy?") >
Two lines instead of thirty. Beautiful? Very. But ask yourself: what chunk size did the splitting use? How many results did the search return? What exactly is in the prompt that went to the model? The answer: default decisions were made on your behalf, and you never saw them. Every one of them has a setting you can change — if you knew it existed. The difference between a beginner and a professional with frameworks is precisely knowing those defaults.
| # | ❌ Wrong | ✅ Right |
|---|---|---|
| 1 | Choosing by hype: "every tutorial uses it" | Tutorials optimize for the quick demo — you optimize for understanding and maintenance. The decision table above |
| 2 | A framework on top of a pipeline never learned: your first-ever RAG in two magic lines | Build the pipeline once by hand — then use whatever you like, knowing exactly what the framework saves you |
| 3 | Unreviewed defaults: chunk size, top-k, and the prompt template all left as defaults | Day one with any framework: print the final prompt that goes to the model, and review every default touching retrieval |
| 4 | Unpinned versions: pip install with no pinning in a 🔥 market | Pin versions + scheduled, tested upgrades — not accidental upgrades on the day something falls over |
| 5 | Random framework mixing: some LangChain, some LlamaIndex, nobody knows who owns what | If you split, split on clear boundaries (one for ingestion only, say) — by documented decision, not copy-paste accretion |
Cost — the reassuring truth
Good news: this is not a money decision. Your bill lives in model calls — input and output tokens — and those calls are identical whether sent by hand-written code or a ready-made chain. The ~$570/month from the "why does RAG exist" scenario stays ~$570 in all three cases.
Latency? The extra layers add milliseconds — next to seconds of LLM generation, that margin is invisible. The one exception: if a ready-made chain performs extra model calls you didn't account for (automatic query rewriting, intermediate summarization…) — then the bill genuinely rises, not because of the framework, but because of calls you didn't know were happening. Which brings us back to the same rule: know what happens inside.
| Your situation | Decision | Why |
|---|---|---|
| Learning RAG for the first time | No framework | The goal is understanding the pipeline itself — a framework will hide it from you exactly when you need it most |
| Small/medium production system, one or two data sources | No framework / a thin layer | The pipeline is simple; control and debuggability are worth more than saving limited boilerplate |
| Heavy ingestion: 10+ sources in different formats, varied indexes | LlamaIndex | Rewriting dozens of loaders and parsers by hand is waste — this is exactly the problem it was built for |
| Complex orchestration: many components, multiple models, branching wiring patterns | LangChain | The integrations and chains ecosystem saves real wiring work — here the abstraction earns its price |
| The flow has become an agent with state and branching | Look at LangGraph | Out of this page's scope — covered fully in "What do you build agents with?" |
| Production with a minimum-dependencies constraint (compliance, small team, long lifespan) | No framework | Every dependency is a maintenance and upgrade commitment — in a 🔥 market like this, that commitment is expensive |
- Don't go frameworkless if the task isn't educational, the ingestion is a many-headed beast, and the deadline is close — at that point you're reinventing a mature library at your product's expense.
- Don't bring in a framework because "we might need its capabilities later" — YAGNI applies to the AI stack like any stack. Bring it when the need is in front of you, not in your imagination.
- And don't mix the two the wrong way: a framework for ingestion + direct code for retrieval and generation = a legitimate, smart split. The mistake is a full framework on top of a pipeline you never understood in the first place.
Alternatives — the three real options, honestly
### (a) No framework — direct API calls + a vector store client
| ✅ Strength | ❌ Weakness |
|---|---|
| Full control: you see every prompt and every decision with your own eyes | More boilerplate: loaders, retries, and splitting are on you |
| Direct debugging: the stack trace is your own code | New feature = new implementation (no ready "spare parts") |
| Near-zero dependencies: the model SDK + the vector store client, done | As sources and components multiply, your code starts becoming a small framework — written once and undocumented |
| No upgrade churn: no release ever breaks you | — |
This is the BuildAI default — for learning and for small production systems: "so you understand what's happening." The RAG Knowledge Assistant project is deliberately built this way, end to end.
### (b) LlamaIndex — when the problem is the data
| ✅ Strength | ❌ Weakness |
|---|---|
| Its strongest suit: ingestion, indexing, and query engines — the data-centric problems | Outside RAG (agents, general orchestration) it's not its home turf |
| Ready loaders for dozens of sources and formats (PDFs, Notion, databases…) | Abstraction hides retrieval decisions — you must learn its defaults |
| Varied index structures ready when your needs exceed a plain vector index | Real upgrade churn — the APIs keep moving 🔥 |
### (c) LangChain — when the problem is the wiring
| ✅ Strength | ❌ Weakness |
|---|---|
| The broadest integration ecosystem in the field: any model, any vector store, any tool — there's probably a connector | The heaviest abstraction of the three — the longest distance between you and the final prompt |
| Ready chains for common patterns + a community and examples without end | Dependency weight and API velocity — the highest churn on the list 🔥 |
| A paved road for scaling up: LangGraph when you need complex agent flows (a pointer only — details live in the Agents decision guide) | Many of the field's famous debugging horror stories start with "a ready-made chain I didn't know what it did inside" |
### Anyone else?
> 🔥 Volatile fragment: yes — Haystack (strong for production search pipelines) and others, and the list grows monthly. We don't cover them in detail: the criteria on this page are the same — apply them to any new name that appears.
Trade-offs
| Criterion | No framework | LlamaIndex | LangChain |
|---|---|---|---|
| Speed to start | ⚠️ Slower — you write everything | ✅ Fast for RAG specifically | ✅ Fast, with examples for everything |
| Transparency & debuggability | ✅ Complete — the prompt is right there | ⚠️ Medium — defaults to learn | ❌ Hardest — the longest layer chain |
| Ready-made ingestion power | ❌ All on you | ✅ The strongest | ⚠️ Decent |
| Integration breadth | ❌ You write them | ⚠️ Good | ✅ The widest |
| Lock-in | ✅ Zero | ⚠️ Medium | ⚠️ Medium-to-high — it spreads through your codebase if you let it |
| Upgrade churn 🔥 | ✅ Zero | ❌ Real | ❌ Historically the highest |
| Community & docs | ⚠️ The raw API docs (excellent anyway) | ✅ Strong | ✅ The biggest |
⚠️ Mind the churn row: "the framework breaks APIs between releases" isn't slander — it's the natural price of a field that moves monthly, with frameworks chasing it. Pin versions, and budget framework upgrades as a recurring maintenance item, not a surprise.
Interview questions
Q1: "What did you build your RAG with? And why?" This question isn't about the name — it's about having criteria. A strong answer: "Direct calls + a vector store client, because the sources were limited and prompt control and debugging mattered more than saving boilerplate" — or the reverse with a clear criterion ("15 data sources, so I chose LlamaIndex for ingestion"). The weak answer: "I used X because it's the most popular."
Q2: What are the problems with heavy abstraction in LLM applications specifically? In LLM applications, quality lives entirely in the details abstraction hides: the final prompt, chunk sizes, result counts, intermediate calls. When quality degrades — and it will — diagnosis requires seeing exactly those things. Add churn on top: a field changing monthly means frameworks breaking APIs constantly.
Q3: Your team wants to rewrite a working LangChain RAG in direct code "for control." Your view? A judgment question: rewrites carry cost and risk, and abstract "control" isn't sufficient justification. The mature answer asks first: what's the actual problem? If recurring debugging keeps getting stuck on framework layers, or upgrades keep breaking things — there's a real case, done gradually (most painful component first). If the system works and is stable — leave it, and spend the effort improving retrieval itself.
- Inside BuildAI: Provider abstraction — writing switchable code → Advanced retrievers → "What do you build agents with?"
- The tools' own docs: 🔥 read the LlamaIndex and LangChain documentation after building the pipeline by hand once — you'll read it then as an engineer evaluating a tool, not as someone lost looking for a life raft.