Most products that add an LLM feature do not break because the model is bad. They break because a slow, probabilistic dependency was wired straight into a request path that used to be fast and deterministic. A support page that loaded in 300 ms now waits eight seconds for a summary, a parser chokes on a stray sentence the model added, and nobody can say which prompt version produced last Tuesday’s odd output.
The fix is mostly ordinary engineering discipline applied to an unusual component. Below is the approach we use when adding LLM features to products that already have users, data and uptime expectations.
Start with one narrow feature behind a flag
Pick a single feature with a clear input, a clear output and a tolerable failure mode. Good first candidates: drafting a reply that a human edits before sending, tagging incoming tickets, summarizing a long record into three bullet points, or extracting fields from an uploaded document into a form the user reviews. Poor first candidates: anything that moves money, changes permissions or writes directly to a system of record without review.
Put the feature behind a feature flag from day one. The flag should be controllable per account or per user, not just globally, so you can turn it on for internal staff, then a handful of friendly customers, then a percentage of traffic. A flag also gives you a kill switch that does not require a deploy, which you will want the first time a provider has an incident.
Isolate the model behind an internal interface
Application code should never call a provider SDK directly. Create an internal service or module with a narrow contract, something like summarizeTicket(ticketId) returns Summary, and keep everything model-specific inside it: the prompt template, the provider client, the model name, parameters, parsing and validation.
This boundary pays for itself quickly:
- You can swap providers or models without touching product code.
- You can stub the service in tests so your existing test suite stays fast and deterministic.
- You get one place to enforce timeouts, logging, redaction and cost tracking.
- Callers deal with typed results and typed errors, not raw text.
Whether this is a separate network service or a module inside your monolith matters less than the discipline of the contract. For most teams a module is enough at first.
Plan for slow, failing and unavailable models
Treat the model like any other external dependency with variable latency. Response times for a single call can range from under a second to tens of seconds depending on output length, model size and provider load, so the request path needs explicit rules.
Timeouts and retries
Set a hard timeout that fits the user experience, not the model’s worst case. Retry only on errors that are likely transient, such as rate limits, connection resets and 5xx responses, with exponential backoff and jitter, and cap the total retry budget. Do not retry validation failures blindly; a second identical call often fails the same way.
Fallbacks
Decide in advance what the user sees when the model does not answer. Options include a smaller or alternative model, a cached previous result, a rule-based default, or simply hiding the feature and showing the page without it. The worst fallback is an error that blocks the core workflow the user came to do.
Move long work off the request path
If a task regularly takes more than a couple of seconds, run it in a background job and update the UI when it finishes. Generating a summary when a ticket is created, rather than when it is opened, removes most of the perceived latency.
Validate outputs and test prompts like code
Structured outputs
Free text is fine for something a human reads. For anything your code consumes, ask for structured output, ideally using the provider’s JSON schema or tool-calling features, and validate the response against a schema before it touches your database.
Validation should check more than shape. If the model extracts an invoice total, confirm it parses as a number and matches the sum of line items within a tolerance. If it picks a category, confirm the category exists. If it returns an ID, confirm the ID belongs to the current account. When validation fails, log the failure, try one repair attempt if it is cheap, and otherwise fall back. Never let unvalidated model output write to a system of record.
An evaluation set before prompt tuning
Prompt changes are code changes with unpredictable blast radius. A tweak that fixes one edge case can quietly degrade twenty others. The defense is an evaluation set: a collection of real or realistic inputs with expected outputs or grading criteria.
Start small. Thirty to a hundred well-chosen cases covering common inputs, known edge cases and past failures is enough to catch most regressions. For each case, define how to score it: exact match for extracted fields, schema validity for structure, and either rubric-based human review or a carefully checked model grader for open-ended text.
Run the set in CI whenever the prompt, model, parameters or parsing logic change, and track the score over time. When a user reports a bad output, add that input to the set. Over a few months this becomes the most valuable asset in the feature, because it lets you change models with evidence rather than hope.
Hold to cost, latency and logging rules
Budgets
Give every LLM feature an explicit budget: a target p95 latency and a cost ceiling per request and per month. Token usage is easy to underestimate, because retrieved context, conversation history and system prompts all count on every call.
Practical controls that keep budgets intact:
- Record input and output tokens per call, tagged by feature and account.
- Cap the context you send; trim history and retrieve only the most relevant chunks.
- Use smaller models for classification and extraction, and reserve larger models for tasks that measurably need them.
- Cache results for identical inputs, and use provider prompt caching where it applies.
- Set per-account rate limits so one heavy user cannot consume the month’s budget.
Logging without storing sensitive data
You need logs to debug and evaluate, but full prompts and responses often contain customer data you should not keep indefinitely. Log metadata by default: request ID, feature, prompt version, model, token counts, latency, validation result and fallback path. Store full payloads only when needed, with redaction of obvious personal data, restricted access and a short retention period.
A useful pattern is sampled payload logging for internal and opted-in accounts only, feeding directly into the evaluation set after review. Check, too, what your provider retains on its side and whether that fits your agreements with customers.
Roll out gradually and version everything
Treat the prompt template, model identifier and generation parameters as one versioned unit. Store prompts in the repository, not in a dashboard someone edits by hand, and stamp every output with the version that produced it. When a provider deprecates a model or ships a new snapshot, you change one version string, run the evaluation set, and roll out like any other release.
Rollout itself follows the flag: internal users, then a small cohort, then widening percentages. At each step, watch a short list of signals:
- Error, timeout and fallback rates.
- Validation failure rate.
- p95 latency and cost per request.
- User signals such as edits, rejections, thumbs-down or the feature being switched off.
Set thresholds in advance so the decision to pause is mechanical rather than a debate.
Where to go from here
If you are adding a first LLM feature to an existing product, a sensible next step is a short written plan: the one feature you will ship, the interface it sits behind, its fallback, its evaluation set of fifty cases, and its latency and cost budget. That one page will surface most of the hard questions before any code is written.
This is the kind of work we do in our AI native custom development engagements, where the goal is an AI feature that behaves like the rest of a well-run codebase. If you want a second opinion on your plan before you build, we are happy to review it with you.