A beta tester broke my chatbot on his third message.

He was poking at chat.kubecraftlabs.io, my self-hosted RAG bot for KubeCraft material. He asked a question, got a good answer, asked a follow-up — and the follow-up got an answer that ignored everything they’d just discussed. He posted the report in the testing channel:

Each Prompt act as separate prompt there is NO LINKAGE to next prompt no link from last prompt done (follow up questions not works) each prompts works isolated call.

He was right. I went to look at the code, and the bug was right there in front of me — three lines that explained the entire missing feature.

def _extract_question(messages):
    for msg in reversed(messages):
        if msg.role == "user":
            return msg.content.strip()

The function reverses the messages array, returns the first user message it finds, and discards everything else. The dialogue history never reaches the retriever. It never reaches the model. The conversation is structurally invisible.

That’s why every RAG demo “works” until turn three.

The Real Problem Behind The Bug

This isn’t a coding mistake. It’s a missing concept.

When you ask a search question, you say it in full: “How do I install KubeCraft on a Kubernetes cluster?” When you ask a follow-up, you don’t repeat yourself. You say “and on ARM?” or “how about for CKAD?” or “what about the GPU driver?” The full meaning lives in the conversation, not in the words you just typed.

A stateless RAG retriever sees only the last sentence. It embeds “and on ARM?” and gets back chunks about the most popular meaning of “ARM” in the corpus, which has nothing to do with your install question. The model then composes an answer that looks confident and is completely wrong.

I had a clear bug, an obvious file to edit, and a strong urge to start typing.

I didn’t.

I Did The Research First

I have a rule for production work I haven’t done before: don’t write the fix until you know what the field already figured out. So I burned an hour of compute on three parallel research agents — one searching X for the discourse over the last 30-60 days, one searching the production-architecture web, one chewing through vendor docs.

Five patterns came back. Every conversational RAG system in production uses some combination of these:

1. Standalone query rewrite. Before retrieval, a small LLM rewrites the bare follow-up into a self-contained search query. “And on ARM?” becomes “How do I install KubeCraft on ARM-based Kubernetes nodes?” This is the single most-cited fix in the entire corpus — LangGraph dedicates a graph node to it, LlamaIndex publishes the canonical primitives, LangChain ships history-aware retrieval chains. The consensus is unanimous.

2. Verbatim turn buffer. Keep the most recent four to eight user-and-assistant turn pairs fully in the generation prompt. This isn’t an LLM call — it’s a slicing rule on the message list. The model sees the actual back-and-forth, not a paraphrase of it.

3. Rolling summarization. When older turns fall out of the verbatim window, a small LLM condenses them into a running summary that takes their place. Triggered on either turn count or token threshold.

4. Persistent cross-session memory. A separate vector index for facts and episodes that span sessions. Mem0 is the most-cited dedicated memory layer; Cloudflare just shipped Agent Memory in their Agents SDK in April; Letta is the emerging name. Different problem from per-conversation memory — defer it.

5. Token budget management. Five rules in priority order: protect the system prompt and the latest retrieved context, evict oldest turns first, compress when budget is exceeded, cap verbatim turns at eight, prefer selective injection over full dumps.

This is the part most production blog posts don’t write down. So now you know.

The Three Things I Did Not Do

The reason this kind of post earns its keep is what it rules out, not what it ships. Three approaches I considered and rejected, with reasons.

Concatenating the last N user messages. Zero industry endorsement. The pattern doesn’t appear as a recommended approach in any vendor doc or production-architecture writeup. People propose it because it’s cheap and feels like memory; it isn’t. The query you embed ends up being a Frankenstein of two unrelated questions and the retriever does worse than with the bare last message.

Embedding the full conversation as the retrieval query. Actively warned against. Topic drift across turns pollutes the query vector — the embedding averages across mixed topics and retrieves none of them well. This is the kind of mistake you only make once.

Adopting LangGraph for orchestration. LangGraph is the right destination eventually, when you want grade-and-reflect loops and CRAG-style self-correction. It is the wrong starting point for a two-call refactor of a working FastAPI route. Introducing a graph framework alongside a small fix is over-scoped. Revisit when there’s evidence you actually need it.

One quick aside that surprised me. OpenAI is sunsetting the Assistants API on August 26, 2026. The Assistants API was their auto-managed conversation primitive — Threads, magic state, no work for you. They’re moving customers to the Responses API plus a Conversations resource, where the application owns the conversation state explicitly. Even OpenAI is saying “hold the dialogue yourself” now. That’s the direction.

The Constraint That Reshaped The Design

Here’s where my situation diverged from the textbook.

The industry consensus is precise: use a small fast model for the rewriter, reserve the large model for generation. Haiku-class, GPT-4o-mini-class, or self-hosted Llama-3.1-8B are the named choices. The cost win is real — there’s a much-cited claim of an eighty-seven percent cost reduction with no accuracy loss when you tier your models properly.

I have one GPU. One model on the GPU at a time.

I can’t run a small fast rewriter alongside Gemma. There’s no slot for it. Deploying a second model means either buying a second GPU or context-switching weights on every turn — the first isn’t in scope, the second is unacceptable latency.

So the rewriter is the same Gemma as the generator. Different system prompt. Different max_tokens cap. Same weights. Same GPU. Sequential, not parallel.

This is the part most production blog posts gloss over. What do you do when you don’t have the resources for the textbook architecture?

You adapt. You take the parts of the consensus that are about information flow (rewrite for retrieval, history for generation, four-to-eight verbatim turns, history before context) and you keep them. You drop the part that’s about deployment topology (separate small model). And you compensate with mitigations.

What The Implementation Actually Looks Like

A new module called rewriter.py. One async function. The system prompt is short and unambiguous:

You are a search-query rewriter. Given a conversation history
and the user's latest message, produce a single self-contained
search query that captures the user's intent. Resolve all
pronouns and references using the conversation context. Output
ONLY the rewritten query — no explanation, no preamble.

Temperature zero. max_tokens capped at 256 — generous, but enough to clip a runaway essay if Gemma decides to elaborate.

The split that took me a minute to internalize: the rewrite is only for retrieval. The generator still gets the user’s actual original question. We don’t put words in their mouth. The rewriter’s job is to make the vector search work; the generator’s job is to answer what the user actually said. Many implementations conflate these and end up with answers to the rewritten query instead of the user’s question.

There’s also a turn-1 skip. If there’s no prior history, the rewriter does nothing — it returns the question unchanged without making an LLM call. First-message latency is identical to the old code. The new path only kicks in when it’s needed.

Fail-Open As A Feature

The piece I’m most happy with is the failure mode.

Any error in the rewriter — timeout, LLM unreachable, JSON parse, unexpected exception — the system catches it, logs it, increments a counter, and falls back to the bare original question. The pipeline degrades to current production behavior. It never blocks.

except LLMError as exc:
    metrics.REWRITER_REQUESTS.labels(status="error").inc()
    logger.warning("rewriter: fail-open, error=%s", exc)
    return question

This is the design principle behind shipping new pipeline stages safely on day one: every new step you add must be able to silently disappear and leave the rest of the system working as it did before. Fail-open is not a defensive afterthought. It’s how you make the new thing additive instead of disruptive.

If the rewriter is broken at three a.m., the system answers the bare question. Nobody pages me. The Prometheus counter rag_rewriter_requests_total{status="error"} ticks up, and I look at it the next morning.

Defense-In-Depth Against Prompt Injection

One more thing the implementer caught in code review.

The single-message version of _extract_question already strips role-boundary tokens — <|user|>, <|assistant|>, that family of strings — to prevent a hostile message from breaking out of its role and impersonating the system prompt. When I added history, I forgot to apply the same sanitization to the prior turns.

So now _extract_history runs every history message through the same role-token stripper before either the rewriter or the generator sees it. A multi-turn injection attempt has to defeat the same defense as a single-turn one.

This is what defense-in-depth looks like in a real codebase: not a separate module, not a shiny mitigation framework — just a single function applied at every boundary where untrusted text enters trusted state.

What Goes In The Generation Prompt

Final order, after everything is assembled:

[system prompt]
[verbatim recent turns, up to 8]
[<<<CONTEXT>>> retrieved excerpts <<<END CONTEXT>>>]
[Question: <original user question>]

History before retrieved context. Current question last. This is the consensus order from the production research — not what the original issue body proposed. The model treats the most recent item in the prompt as the operative instruction; placing the question last and the retrieved evidence immediately above it keeps generation focused on answering this turn with this evidence, while still seeing the conversational arc.

The implementation took seven test-driven tasks. Failing tests first, then code. Seventy-seven tests pass at the end. Pull request 151 merged, version 0.11.0 deployed to staging, then to production. Same day as the research.

Two Things To Take Home

If you’ve read this far, here are the two ideas worth carrying.

Conversation memory is not one thing. It’s a stack of three different mechanisms with different triggers and different storage effects:

  • The rewriter — every turn with prior history, transient, no storage.
  • The verbatim buffer — a slicing rule, no LLM call.
  • The summarizer — overflow only, persisted as running state.

Mixing them up is the most common confusion in this space. They are not alternatives. They compose. When you see “we added memory” in a vendor blog post, ask which of the three they mean.

Fail-open is a feature. Designing the new code so it can always degrade to the old behavior is what makes shipping new pipeline stages on day one safe. Catch every error. Log it. Count it. Return the input unchanged. The Prometheus counter tells you whether the new path is healthy; the user never sees the difference.

The rewriter could break tonight and the bot would keep working. That’s the whole point.

Keep building, Mischa