From AI Prototype to Production: What Changes

A prototype proves that a model can produce a good answer on inputs you chose. Production proves that a system keeps producing acceptable answers on inputs you did not choose, at a cost you can afford, without falling over when something breaks. Those are different problems, and the second one is mostly engineering, not modeling.

This trips up almost everyone the first time. The demo works on the tenth try in a notebook, the screen recording looks convincing, and it feels like the hard part is done. In practice the notebook was maybe ten percent of the work. The other ninety percent is the part nobody films: handling the input you did not anticipate, deciding what happens when the model is wrong, measuring whether it is getting better or worse, and keeping the whole thing observable once real users are hitting it.

The good news is that the missing ninety percent is not mysterious. It is a known set of concerns, and you can learn to see them. This article walks through each one and ends with a checklist you can hold in your head.

In a notebook you feed the model clean, well-formed examples, often the same handful you have been staring at for a week. Real users send truncated text, the wrong language, empty strings, pasted PDFs full of layout junk, adversarial prompts, and inputs ten times longer than anything you tested. The distribution of real traffic is wider and weirder than your test set, always.

This is the single biggest reason demos that looked finished fall apart in week one. Your prototype was implicitly overfit to the inputs you happened to try. The fix is not cleverness, it is defensive boundaries: validate and normalize input before it reaches the model, define what a malformed request looks like and reject it early with a clear error, and cap sizes so a single huge input cannot blow your latency or your token budget.

A useful habit is to keep a growing file of real inputs that broke something, and to treat each one as a permanent test case. The gap between test inputs and real inputs never fully closes, but it shrinks every time you feed a real failure back in.

A notebook cell that throws an exception is fine, you just re-run it. A production request that throws leaves a user staring at a spinner or a stack trace. Model calls fail in ways ordinary code does not: the API times out, returns a rate-limit error, produces malformed JSON when you asked for JSON, hallucinates a field, or simply returns something confidently wrong.

So you have to design for the wrong answer, not just the missing answer. That means timeouts and retries with backoff on the network layer, schema validation on structured output so a bad shape is caught rather than passed downstream, and a defined fallback for when the model cannot produce something usable. The fallback might be a cached response, a simpler rule-based path, or an honest message that the system could not complete the request. What it cannot be is undefined behavior.

The mental shift is this: in a prototype the happy path is the whole story, and in production the happy path is one branch among many. Most of your production code will be about the branches where things go wrong.

In a demo, you are the eval. You look at the output, decide it is good, and move on. That does not scale and it does not survive a change. The moment you tweak a prompt, swap a model, or adjust retrieval, you need to answer a hard question: did that make things better or worse, across the range of inputs I care about, not just the one I am looking at right now?

An eval is a repeatable way to score outputs against expectations. At minimum it is a set of representative inputs paired with either known-good answers or a scoring rule. Scoring can be exact match for structured tasks, a metric like accuracy or F1 for classification, a similarity or overlap measure for retrieval, or an LLM-as-judge for open-ended text where you prompt a separate model to rate the answer against a rubric. None of these is perfect, and LLM-judges have their own biases, but any repeatable eval beats eyeballing.

The discipline that matters most: build the eval set before you start tuning, and never optimize against numbers you cannot measure. A change that improves your favorite example while quietly regressing ten others is a change you will ship if you have no eval, and catch if you do.

A notebook has no service-level agreement. It can take thirty seconds and cost whatever it costs, because it runs once for an audience of one. A production endpoint has a latency budget and a per-request cost that multiplies by every user, and both are easy to ignore until the bill or the complaints arrive.

Latency compounds. If your system makes three sequential model calls plus a retrieval step, the user waits for the sum. Techniques that help include running independent calls in parallel, streaming tokens so the user sees progress instead of a blank screen, caching results for repeated inputs, and using a smaller or cheaper model for the easy majority of requests while reserving the expensive model for the hard ones. Cost follows the same logic: token usage per request times request volume is your real number, and prompt size, context stuffed into retrieval, and retries all quietly inflate it.

Make these visible early. Log tokens and latency per request from day one, because you cannot manage a number you never measured.

Once real traffic arrives, you are flying by instruments. Monitoring means logging inputs and outputs (with attention to privacy), tracking latency and error rates, and watching quality drift over time, because a model that was good in March can degrade as user behavior or upstream data shifts. Without monitoring you learn about failures from angry users instead of dashboards.

Security for AI systems adds concerns ordinary apps do not have. Prompt injection lets a malicious input hijack your instructions, so never trust model output as if it were code or a command, and never let a model call a dangerous tool without checks. Keep secrets out of prompts, do not log sensitive data carelessly, and treat anything the model generates as untrusted until validated. Deployment then follows normal engineering: version your prompts and models the way you version code, roll out changes gradually, and keep the ability to roll back fast.

All of this feeds a loop. Real inputs reveal failures, failures become eval cases, evals gate your changes, monitoring surfaces drift, and each pass makes the system a little sturdier. A prototype is a snapshot. A production system is a process you maintain.