How to choose an LLM for production

Choosing an LLM for production is not picking the highest-scoring model on a public leaderboard. It is finding the cheapest, fastest model that clears the quality bar for your specific task, under your privacy and latency constraints, without locking you to one vendor. The task defines the bar, so the decision starts with the task and not the model.

This matters because the models that top general benchmarks are usually the most expensive and the slowest, and most production tasks are narrower than the benchmark. Classifying a support ticket, extracting fields from an invoice, or drafting a reply from a retrieved context does not need the same capability as open-ended reasoning across a long document. Matching model capability to task difficulty is where most of the cost and latency savings live.

So the useful frame is not "which model is best" but "which model is sufficient here, and what does it cost me on the four axes that actually bind: quality, cost, latency, and risk." The rest of this article is those axes and how to measure them on your own data.

Frontier models, the large hosted models from the major labs, buy you capability and convenience. They handle ambiguous instructions, multi-step reasoning, and long context well, and they need less prompt engineering to reach a working result. You pay for that per token, and you accept that the weights and the roadmap belong to someone else.

Small and open-weight models flip the trade. A well-chosen small model, fine-tuned or prompted for a narrow task, can match a frontier model on that task at a fraction of the cost and latency, and you can run it where you want. The cost moves from per-token spend to the engineering and infrastructure needed to host, tune, and maintain it. That cost is real and recurring, which is why small models pay off on high-volume, well-defined tasks and rarely on low-volume, open-ended ones.

In practice most production systems use both. A capable model handles the hard or rare path, a smaller model handles the high-volume common path, and a router decides which one a given request needs. The decision is per task inside a system, not one model for the whole product.

Hosted APIs remove operational burden. You get scaling, availability, and model updates without running GPUs, and you start in an afternoon. The costs are per-token pricing at scale, less control over latency tails and model versions, and the requirement that your data leave your environment to reach the provider.

Self-hosting, whether an open-weight model on your own GPUs or a dedicated deployment in your cloud, gives you control over data residency, latency, versioning, and unit economics at high volume. The price is that you now own capacity planning, GPU supply, autoscaling, and the on-call rotation for an inference service. This is an engineering commitment, not a checkbox, and it only earns out above a volume threshold or when a compliance requirement forces it.

The honest default for most teams is to start hosted, instrument everything, and self-host only the specific workloads where the numbers or the compliance rules justify it. Deciding to self-host the whole stack before you have production traffic usually buys control you are not yet using at a cost you cannot yet amortize.

Beyond the frontier-versus-small and hosted-versus-self-hosted framings, a production choice comes down to a short list of measurable properties. Score every candidate on each, on your task, not in the abstract.

The only reliable way to compare candidates is an evaluation set built from your own data. Assemble a few dozen to a few hundred representative inputs with known-good expected outputs, including the awkward edge cases that break demos, then run every candidate model through the same set and score them the same way. This turns model selection from an argument into a measurement.

Score what the task cares about. For extraction or classification, exact-match or field-level accuracy is enough. For generation, use a rubric, and where a human rubric does not scale, an LLM-as-judge scored against a reference can approximate it, as long as you spot-check the judge. Record cost and latency in the same run so quality is never compared without its price attached.

Keep the eval set in version control and rerun it whenever you change a prompt, a model version, or a provider. A model that scored well six months ago may have been quietly updated, and a cheaper model released last week may now clear your bar. The eval is not a one-time gate, it is the instrument you keep pointed at the decision.

The model you pick today will not be the best or cheapest option next quarter, so the goal is a system where swapping models is a configuration change, not a rewrite. Put a thin abstraction layer between your application and the provider so model choice, prompt templates, and routing live in one place behind a stable internal interface. Your application should ask for a capability, not call a named vendor directly.

Guard against the couplings that quietly create lock-in: provider-specific prompt formats baked through the codebase, reliance on one vendor's exact structured-output behavior, and evals that only run against the incumbent. Normalize requests and responses at the boundary, and keep at least one alternative model wired into the eval harness so a switch is always one measured step away.

This is the layer where model selection stops being a one-time decision and becomes an operational capability. It is also the part teams skip under deadline pressure, which is exactly how a proof of concept turns into a system that can only ever run on the vendor it was prototyped against. Building that abstraction, the evals behind it, and the routing on top is the kind of production engineering Stallwart does around a customer's own workflow and constraints, rather than handing over a model recommendation and leaving.