How RAG chatbots actually answer without making things up

A retrieval-augmented generation (RAG) chatbot is any assistant that reads the visitor’s question, searches a body of your own content for relevant passages, and gives those passages to a language model as context before the model writes its reply. The point is not fancier phrasing. It’s a chain of custody. If the answer came from a document you own, you can prove it.

That’s the promise. The rest of this piece is the operational reality behind it.

What retrieval actually does

The chatbot doesn’t “read” your documents in the way a person reads a report. It pre-processes them once:

1. Extract — pull the text out of the file, whatever the format.
2. Clean — strip headers, footers, navigation, boilerplate.
3. Chunk — split into passages of roughly 300–500 tokens, with overlap between neighbours so a sentence that straddles a boundary can still be found.
4. Embed — send each chunk to an embedding model that returns a numeric vector. Vectors that mean similar things end up close in high-dimensional space.
5. Store — write vectors and text into a vector database (in Anserra’s case, PostgreSQL with pgvector).

At question time, the same embedding function turns the visitor’s query into a vector, and the database returns the k passages closest to it. Those passages are placed in the model’s context window along with the visitor’s question and a system prompt.

That last part is what stops the model from generating from its own opinions: “Answer only using the CONTEXT below. If the answer is not there, say so.”

Why “just vector search” isn’t enough

Semantic embeddings are excellent at paraphrase and synonym. They fail on named entities and exact strings. “Do you have part number 42–B-9182 in stock?” is essentially a database key; embeddings smear it into “some question about product 42”.

The fix is hybrid retrieval: run the same query through full-text search too, and combine the ranks. Anserra uses Reciprocal Rank Fusion — a passage that both halves agree on outranks one that merely tops a single list. This is why exact SKUs and paraphrased questions both land.

The three things that can still go wrong

Even with hybrid search:

• The right document was never indexed. The chatbot can only cite what you gave it. Most “the chatbot got it wrong” tickets we investigate are actually “the correct policy PDF isn’t in the knowledge base.”
• The chunk was too small to carry the whole answer. If a paragraph starts “This applies to enterprise plans only” three sentences before “Refunds within 30 days”, chunking may separate them. Overlap and passage-level chunking mitigate this; passage citation makes it visible.
• The model ignored the context and answered from its parametric knowledge. A well-written system prompt reduces this. A relevance threshold (drop matches below score X) removes it — if nothing is relevant enough, the bot declines rather than improvises.

What to check before you buy a RAG chatbot

Ask a vendor these five questions. If any answer is vague, keep looking.

1. Does every answer show its sources? Not “some of them.” Every one. If a claim has no source, it wasn’t retrieved — and the model made it up.
2. What’s the minimum relevance threshold? A vendor that can’t explain what happens when nothing relevant is found is a vendor whose bot hallucinates in that scenario.
3. Do you support hybrid retrieval? Semantic only sounds smarter and fails on the specific queries that convert.
4. Where are embeddings stored? If the answer involves a hosted third-party vector database you can’t inspect, migrating away is going to be painful.
5. How is content deleted? If you remove a source, are its embeddings gone the same second? Ask them to explain the sequence.

What honest limits look like

Anserra tells you when the chatbot didn’t know. Nothing is worse than a customer support bot that confidently invents a refund policy. The product’s own analytics page surfaces the questions that produced a thumbs-down or no citation — that’s the docs you should write next, not evidence that the chatbot is broken.

A RAG chatbot that never says “I don’t know” isn’t a good chatbot. It’s a liability with a nicer accent.

The short version

Retrieval-augmented generation works when: your content is complete, retrieval is hybrid, the model is forced to cite, and low-confidence answers are refused. The rest is theatre.

If you want to try it against your own documents, start free — no credit card, no drop-off, the widget is live in about five minutes.