Case study · AI Infrastructure & RAG
A support agent grounded in your own knowledge
When a support bot answers from a model's general training, it guesses at the wrong moments and cannot show where an answer came from. Stallwart builds a production-grade AI support agent: answers drawn from your documentation and policies with citations, controls against guessing, and a clean handoff to a human when retrieval does not support a confident answer.
- 01Knowledge ingestion
- 02Retrieval
- 03Grounded answer
- 04Guardrails
- 05Human escalation
- 06Evaluation & analytics
- Client
- a SaaS and technology company
- Problem
- Generic support bots answer from training data, guess when unsure, and cannot cite a source.
- Approach
- A RAG agent grounded in company knowledge, with citations, escalation, permissions, and evaluation.
The problem
A support bot that answers from a model's general training is confidently wrong at exactly the wrong moments. It does not know this product's policies, it cannot point to where an answer came from, and when it is unsure it guesses instead of escalating. That erodes trust faster than having no bot at all.
The failure is rarely the model. It is the absence of grounding, controls, and a path to a human. What a support agent needs to be safe in production is grounding in the company's own knowledge, citations a reader can check, and escalation when it is not sure.
The reality behind the problem
Pointing a chatbot at the documentation is not enough. A model retrieves nothing by default, it answers from its general training, and a more capable model guesses more fluently, not more honestly. The gap is not the model's intelligence, it is whether each answer is tied to a passage in the company's own knowledge that a reader can open and check.
The harder problem is the unsure case. A support question the documentation does not cover is where a bot does the most damage: it produces a confident, wrong answer about a refund, an outage, or a security setting, and a customer acts on it. A disclaimer or a confidence score bolted on afterward does not hold, because the model has no reliable sense of what it does not know. Honest deflection has to be built into how the agent decides to answer at all.
What was assumed
A good model makes a good support agent.
What Stallwart asked
What does the agent do when it is not sure, and can it show where an answer came from?
The desired outcome
The target is not a cleverer chatbot. It is an agent that is safe to put in front of customers, because every answer is traceable and every unsure case is handled honestly.
- Every answer traceable to a passage in the company's own knowledge, with a citation a reader can open.
- Honest deflection: when retrieval does not support a confident answer, the agent escalates instead of guessing.
- A clean handoff that carries the conversation and context to a human, so the customer is not sent back to the start.
- Answers that respect access permissions, so a user never sees knowledge they are not entitled to.
- Knowledge that stays current as documentation and policies change, without retraining the model.
- Visibility into answer quality and into the questions the knowledge base does not yet cover.
The system we designed
This system ingests the company's own knowledge, such as documentation, product details, and policies, and indexes it for retrieval. At answer time it uses retrieval-augmented generation (RAG): it retrieves the relevant passages, answers from them, and cites the sources, so a reader can verify the answer rather than trust it blind.
Guardrails keep it honest. When retrieval does not support a confident answer, the agent escalates to a human rather than inventing one. Access permissions control what any given user can see, feedback loops and evaluation track answer quality over time, and analytics show what customers actually ask and where the knowledge base is thin.
How it works
- 01
Ground every answer in retrieved sources
The agent answers from passages retrieved out of the company's own knowledge, not from the model's general memory, and cites them. Grounding plus citations is what makes an answer checkable instead of plausible.
- 02
Make 'I am not sure' a first-class outcome
When retrieval does not support a confident answer, escalation is the correct behavior, not a failure. A clean handoff carries the conversation and context to a human so the customer is not sent in a loop.
- 03
Measure quality, do not assume it
Evaluation on real questions, feedback capture, and analytics turn the agent from a launch into a system that improves, and they show where the knowledge base needs to be written, not guessed.
The architecture
Retrieve, ground, cite, and escalate.
- 01
Knowledge ingestion
Docs, product, policies indexed
- 02
Retrieval
Relevant passages for the question
- 03
Grounded answer
Generated from sources, with citations
- 04
Guardrails
Low support, no confident answer
- 05
Human escalation
Handoff with context when unsure
- 06
Evaluation & analytics
Quality tracked, gaps surfaced
The agent answers from retrieved knowledge or hands off to a human.
Architecture
What sits underneath.
Underneath the workflow are four layers, each doing one job, so grounding and escalation are decisions the system makes on purpose rather than side effects of a prompt.
Knowledge ingestion and indexing
Documentation, product detail, and policies are parsed, chunked, and indexed for retrieval, and each passage keeps a link back to its source so an answer can cite exactly where it came from.
Retrieval and grounding
At answer time the agent retrieves the passages relevant to the question and composes the answer only from them, so the response is tied to the company's own knowledge rather than the model's general memory.
Decision and escalation layer
Before answering, the agent judges whether the retrieved passages actually support a confident answer. When they do not, it escalates to a human with the conversation and context attached rather than filling the gap.
Permissions and observability
Access controls decide which knowledge a given user can retrieve, and every answer, citation, and escalation is logged so quality and coverage can be measured rather than assumed.
The hard parts
Where the engineering judgment was.
RAG over fine-tuning for company knowledge
Product knowledge and policies change. Retrieval keeps answers current as the source updates, where fine-tuning bakes in a snapshot that goes stale and cannot cite itself.
The trade-off
Retrieval quality becomes the thing to engineer and evaluate, which is why ingestion and evaluation are part of the build, not an afterthought.
Permissions and escalation before scale
A support agent touches real customer data and real answers. Access controls and a human path are what make it safe to put in front of users at all.
The trade-off
More to build than a thin wrapper over a model, which is the difference between a demo and production.
Reliability & guardrails
How it avoids the wrong call.
The guardrails are about one thing: the agent never gives a confident, wrong support answer, because that costs more than no answer at all.
Grounded or not at all
The agent answers only from retrieved passages and cites them. If retrieval returns nothing that supports the question, it does not compose an answer from general training, it says so and hands off.
Deflection over a confident guess
A wrong, confident support answer is more costly than no answer. The agent is tuned to escalate when support is weak, because sending a customer away with a plausible but wrong instruction is the failure that erodes trust fastest.
Citations a reader can check
Every answer carries links to the passages it came from, so a customer or an agent can verify it rather than trust it blind, and a wrong source is traceable instead of hidden.
Permission-aware retrieval
Retrieval is scoped to what the user is allowed to see, so the agent cannot ground an answer in a document that user has no access to.
Productionization
What turns it from a demo into software.
What turns the build from a demo that answers happy-path questions into software a support team can run.
Knowledge kept in sync
Ingestion re-runs as documentation and policies change, so the index tracks the live source instead of drifting into a stale snapshot that answers yesterday's policy.
Evaluation on real questions
The agent is tested against real support questions, including the ones it should refuse, so a regression in grounding or in deflection is caught before customers meet it.
Feedback and analytics
Thumbs, escalations, and unanswered questions are captured and surfaced, showing where the knowledge base is thin and which topics drive the most contacts.
Observability and escalation routing
Every answer, citation, and handoff is logged and routed to the right human queue, so a failure is visible and a customer in an unsure case reaches someone who can help.
What changed
The change is in what happens when the agent is unsure. Instead of a confident, unsourced guess, a customer gets an answer tied to the company's own documentation with a citation they can open, or a clean handoff to a human who already has the context. The agent stops being a liability at the exact moments that matter most.
Support teams get a system rather than a launch. They can see which questions the agent answers well, which it escalates, and where the knowledge base needs to be written rather than guessed, and they can trust that an answer in front of a customer is grounded in a source rather than invented.
- Answers are grounded in the company's own knowledge and cite a source a reader can open.
- When retrieval is weak, the agent escalates to a human instead of guessing.
- Handoffs carry the full conversation and context, so customers are not sent back to the start.
- Analytics show where the knowledge base is thin, so it is written from real questions.
Ownership & handover
What you receive, and keep.
At handover, the agent, its knowledge pipeline, and the controls that keep it honest are yours.
- The RAG agent and its retrieval, grounding, and citation logic.
- The knowledge ingestion and indexing pipeline, and the process that keeps it in sync with your documentation.
- The escalation routing and the human-handoff integration.
- The permission model over what each user can retrieve.
- The evaluation suite, feedback capture, and analytics dashboards, with runbooks for extending them.
What we learned
The insight that generalizes: a support agent's value is decided by what it does when it does not know. A confident wrong answer about a refund or a security setting costs more than silence, so honest deflection has to be a designed behavior, not a fallback bolted on at the end. Ground every answer in a checkable source, make escalation a first-class outcome rather than a failure, and the agent becomes something a support team can trust in front of customers.
This work connects to
- AI Infrastructure & RAG
- Retrieval-augmented generation
- AI customer support
- Hallucination control and evaluation
- Human-in-the-loop systems
Frequently asked
What is a RAG customer support agent?
It is a support agent that uses retrieval-augmented generation: instead of answering from a model's general training, it retrieves relevant passages from the company's own documentation and policies, answers from them, and cites the sources so the answer can be verified.
How do you stop an AI support agent from hallucinating?
By grounding answers in retrieved sources, citing them, and setting guardrails so that when retrieval does not support a confident answer the agent escalates to a human rather than inventing one. Evaluation on real questions tracks whether that is working.
What makes an AI support agent production-grade rather than a demo?
Grounding and citations, controlled escalation to humans, access permissions over what each user can see, evaluation and feedback loops, analytics, and observability. A demo answers happy-path questions; a production agent handles the unsure cases safely.
How does the agent decide when to escalate to a human instead of answering?
Before it answers, the agent checks whether the passages it retrieved from your knowledge actually support a confident answer. When they do not, it escalates rather than guessing, and the handoff carries the full conversation and context to a human. A wrong, confident support answer is more costly than an honest handoff, so escalation is treated as correct behavior, not a failure.
Want a support agent that cites its sources and knows when to escalate?
Point us at your documentation and policies. We will scope a grounded agent built for production, not a demo.
Last updated: October 13, 2026