Pilot with guardrails, from first prompt to production

'Pilot with guardrails' is a phrase that has eaten its own meaning. Three years into production LLM deployments, most AI pilot plans still use the words as a safety gesture, which does nothing. A guardrail you cannot point to in a diff is not a guardrail. It is a sentence the team wrote down to make the deck feel measured.

This post is the deep version of [what 'pilot with guardrails' actually means](/blog/pilot-with-guardrails-explained). The companion is the overview. This one is the mechanism: the six classes of code that run before and after every model call, the specific failure each one prevents, the pattern that implements it in 2026, and the pre-committed criteria that decide whether a working pilot widens, holds, or dies. If you are an engineer, treat the sections below as a build backlog. If you are the operator who signed the budget, treat them as a gate.

The 95 percent figure from the MIT NANDA 'State of AI in Business' report, covered by Fortune in August 2025, is the clearest signal that scoping a pilot is only the first move. Pilots that scoped the model well and the system around it poorly still produced no measurable P&L impact, because the system around the model is where the real failure modes live. Guardrails are the shortest answer to that gap. They are the code that stops a working pilot from leaking data, burning budget, or producing an output the business cannot stand behind.

The industry's open security framework for LLMs is the OWASP Top 10 for LLM Applications, 2025 edition. It names the specific attack surfaces that guardrails are the counter-move for. If a vendor cannot tell you which OWASP item each of their guardrails addresses, they are describing a feeling, not a defense. Six classes cover the field. Any pilot missing one of them is one of the six failure modes waiting to happen.

The two guardrail classes that run on every request sit on either side of the model: input guardrails before the model is called, output guardrails before the response is used. Both are code. Both have well-worn patterns. Both are where most pilots skip the hardest work.

Input guardrails do five jobs on every incoming request, usually in this order. Schema validation (does the payload conform to the shape this endpoint accepts). Length and encoding bounds (reject anything over the token or byte ceiling, reject non-UTF-8). Language and scope detection (if the pilot handles English support tickets, non-English or out-of-scope inputs route to a rejection path). Prompt-injection screening (a classifier or rule-based detector for strings that attempt to override the system prompt, exfiltrate it, or jailbreak the model). And PII handling (detect, redact, or route to a path with the right data residency). In 2026 the implementations teams actually ship are NVIDIA's NeMo Guardrails input rails configured in Colang, Guardrails AI's RAIL specifications, Pydantic schema validation combined with a prompt-injection classifier (Lakera Guard, Protect AI Rebuff, or an in-house fine-tuned classifier), and Anthropic's constitutional classifier approach described in their January 2025 research. The specific library matters less than the pattern. A working input layer is code that fails closed on anything it was not designed to see.

Output guardrails do three jobs, also in order. Shape enforcement (parse the response as the typed object the endpoint promised, reject or regenerate on parse failure). Content classification (a policy check against the categories the business flagged, run as a cheap classifier call separate from the main model). And grounding verification (for RAG and retrieval workflows, verify every claim in the output is supported by a retrieved document, or flag the output as low-confidence). Shape enforcement is where structured-output tooling pays off. Every major LLM provider in 2026 supports constrained generation: OpenAI's response_format with strict JSON Schema, Anthropic's tool-use for typed outputs, open-source model servers like vLLM with JSON-Schema-constrained decoding. The pattern a mature team ships: define the output as a Pydantic (Python) or Zod (TypeScript) schema, use the provider's constrained-generation feature, validate on receipt, and treat any parse failure as a retry with a lower temperature or a hard error if the retry fails. Free-text outputs still exist, and they pass through a classifier gate before any user sees them.

In plain English for the non-engineer reading along: inputs get checked before the model runs. Outputs get checked before anyone acts on them. Both checks are code the team can point to. If the team says the model will handle it, you are looking at a pilot without guardrails.

OWASP added Excessive Agency as its own top-ten category in the 2025 edition because the first wave of agent systems failed in a predictable way. The system was given a tool it should not have had, used it on an input it should not have accepted, and did a thing nobody signed off on. The fix is not better prompting. The fix is scoping the tool surface in code, outside the model.

Three patterns hold up in production. First, allow-lists at the agent level. A tool the agent cannot see is a tool the agent cannot call. If the pilot drafts email, it has a draft_email tool and does not have a send_email tool. If the pilot classifies tickets, it has a set_ticket_category tool and does not have an escalate_to_manager tool. An allow-list is the simplest form of blast-radius control, and the one that most reliably survives a prompt-injection payload.

Second, approval tiers. Tools are classified by what they can change: read-only, write-with-approval, and write-automatic. The agent's approval policy is a function of the tier and the request's confidence score. A write-with-approval tool call pauses the workflow and routes to a human. A write-automatic tool call with low model confidence is rejected or escalated. The tier is not a model prompt. It is a decision table enforced by the orchestration code around the model.

Third, parameter bounds enforced outside the LLM. A send_payment tool does not accept any dollar value. It accepts the specific range the business approved for the specific tier of request. The LLM proposes the parameter. The code enforces the bound and rejects anything outside it. If the model proposes a parameter that falls back on a default, the default is the safe one.

The failure mode this prevents is specific and expensive. Through 2024 and 2025 several reported production incidents involved customer support agents given tools that could issue refunds or close accounts, which were called by prompt-injection payloads pasted into customer messages. The agents behaved exactly as instructed by the injected prompt. The fix in each case was the same. Scope the tool, bound the parameter, require a human on actions over a threshold. The behavior of the model never had to change, because the behavior of the model was never the control point.

A monthly budget is a report. A cost ceiling is code. The difference matters because the retry loop that overspends does not wait for the monthly report. Three layers of enforcement, each operating at a different scale:

Alongside the circuit breaker, cost per request is a logged metric with its own alert band. A silent model switch (the provider updates the default model behind the API), a retrieval regression (the retrieved context grows unexpectedly), or a prompt bloat (an engineer added a thousand tokens of instructions to the system prompt) all show up first in cost per request, usually weeks before they show up as output quality problems. A pilot that watches cost per request catches these in a day. A pilot that watches total monthly spend catches them at the invoice.

The right pattern: log tokens-in, tokens-out, model name, prompt version, and the resulting dollar amount for every request. Alert on cost per request moving two standard deviations outside its rolling band for more than a short window. Treat a sustained drift as the same severity as a quality drift, because it almost always is. For the specific metrics and merge-gate pattern that turn evaluation into a shipping discipline, see [how to evaluate AI systems](/blog/how-to-evaluate-ai-systems).

In plain English for the operator: budgets are for reports; ceilings are for code. The ceiling keeps a bug or a bot from producing a five-figure overnight bill the finance team learns about at month-end, and the per-request metric catches a silent degradation before anyone else notices.

Human-in-the-loop (HITL) is the guardrail class that gets implemented loosest, because teams treat it as a safety net rather than a routing rule. A safety net catches whatever falls on it. A routing rule routes specific classes of request to specific humans with specific context in a specific time window. The second one is what actually gets used in production.

The HITL rule has four parts. The trigger condition: which request classes or confidence levels route to a human, written as a decision table, not a prompt instruction. The context package: what the human receives (the input, the model's proposed action, the confidence score, the retrieval context the model used, and the suggested disposition) so the decision can be made in under a minute. The time window: how long the human has before the request escalates further or defaults to a safe no-op. And the recorded outcome: every human decision is logged into the eval set, so the trigger threshold can be re-tuned over time and the human review is the source of training signal, not a bottleneck that disappears into a shared mailbox.

Three failure modes HITL guardrails prevent, each with a different root cause. The request the system should have escalated but did not, because no trigger was defined. The escalation nobody acts on, because the context package was so thin the human could not act in under a minute. And the escalation path that was defined on launch day and never updated, because no feedback loop measured how often the human overrode the system. The last one is why the recorded outcome matters: an intervention rate that is not declining over time means the trigger threshold is wrong or the model's grounding is drifting, and either way the pilot is not ready to widen.

For a technically-literate operator reading this: HITL is not 'a human can intervene'. It is a documented routing contract between the system and a named role, with a specific time budget and a feedback loop. If the team cannot show you the trigger conditions and the context package format, they have not built HITL. They have built a feeling.

A pilot that implements the six guardrail classes above is a pilot that can run for a sprint without doing damage on a bad day. That is not the same as a system ready for production. Two decisions sit between the two, and they use different criteria.

The widen-or-kill decision closes the pilot. It runs on three pre-committed metrics. Accuracy against a random-sample human review (what percentage of outputs are correct, scored weekly on a sample large enough to be meaningful). Intervention rate (how often humans override, correct, or reject the system, declining over time means the system is earning trust). And cost per unit of work (what one output costs, compared against the human cost of the same work). The criteria are written down before the pilot starts. Widen means all three metrics are inside their agreed band for three consecutive weeks. Hold means one metric is at threshold but another is drifting; keep the scope, investigate, set a two-week deadline. Kill means two consecutive weeks below threshold on accuracy, or a critical guardrail firing on an input the system was never supposed to accept. The decision runs on a cadence. It does not run when someone feels ready.

The production-approval decision is a different conversation and runs on a different checklist. A pilot that passes widen criteria is evidence that production is worth building. The seven items that separate the two are specific and uncontroversial. Continuous automated evaluation (the eval set grown from real traffic, run as a merge gate on every prompt or model change). Observability dashboards (latency, error rate, cost per request, which prompt and model are live, which guardrails fired on which requests). Isolated rollback (model, prompt, retrieval index, and application code versioned independently and revertible in minutes). Named ownership (the engineer who is paged on failure, the owner of the eval set, the owner of the prompt library). Audit trail (every decision logged with inputs, outputs, prompt and model versions, and which guardrails fired). Stress and load testing (the system has been run at target volume before launch). And a written runbook (the top five known failure modes and the response for each). The twelve-item version of this list is the ground that [the AI pilot to production checklist](/blog/ai-pilot-to-production-checklist-2026) walks through, and the structural reasons the gap exists at all are covered in [why AI pilots fail to reach production](/blog/why-ai-pilots-dont-reach-production).

The pattern that separates a team shipping production AI from a team producing demos is the sequence: scope the pilot with the six guardrail classes written in code, measure against three pre-committed metrics on a cadence, make the widen-or-kill call when the cadence says, and only then open the production budget conversation. 'Pilot with guardrails' is that whole sequence, not a sentence in a slide deck.

If you want a second set of eyes on the specific pilot you are running, Stallwart does this work: production AI engineering for US B2B teams that have a working demo and need the guardrails and the widen-or-kill contract that turn it into a system someone can run without you. The deliverable from a pilot engagement is a decision you can defend, with the code and the pre-committed criteria the decision depends on.