Skip to main content
OpsRoadmaps

Checklists / llm-feature-production-readiness

LLM Feature Production Readiness Checklist

Before a feature built on someone else's model reaches users: version pinning, an evaluation set, cost and latency limits, guardrails, a tested fallback, and a clear account of what leaves your boundary.

9 steps · 8 required · last reviewed (next review 2026-12-20) · updated 2026-09-20

Related focus areas:LLMOps

Sign in to run this checklist

Run this before an LLM-backed feature is user-facing. The distinguishing risk is that your most important dependency can change without you deploying anything, and that the same input can produce different output on two consecutive calls.

The three items teams most often skip — version pinning, a cost ceiling and a defined provider-failure behaviour — account for most early production incidents in this space.

  1. The model and version are pinned and recorded with each response

    required

    Without this, a provider-side change is indistinguishable from a bug in your own code.

  2. An evaluation set of real cases runs before release with a threshold that can block it

    required

    Fifty representative examples that actually run beat a sophisticated harness that does not exist.

  3. Cost per request is measured and a spend alert exists

    required

    In this domain the surprise usually arrives as an invoice rather than an outage.

  4. Latency is measured at p95 including retries and fallbacks

    required

    Average latency hides exactly the tail that makes a feature feel broken.

  5. Input and output pass guardrails proportionate to the risk of the output

    required

    The appropriate check for a drafting aid and for something touching a customer record are not the same check.

  6. Behaviour when the provider is slow, rate-limited or down is defined and tested

    required

    Undefined fallback means the failure mode is whatever the HTTP client does by default.

  7. What is sent to the provider is documented and defensible to a privacy reviewer

    required

    The data leaving your boundary is a privacy decision, and it is easier to answer before it is asked.

  8. A named owner can disable the feature without a deploy

    required

    A quality problem you can only fix by shipping is a quality problem that persists for hours.

  9. Provider deprecation has been drilled and the switching time is known

    optional

    Vendors schedule deprecations on their calendar, not yours.

Run it against your real gaps

A checklist is most useful when you know which controls you are actually missing. A free OpsRoadmaps assessment scores your controls with explainable findings and builds a roadmap from the results.

Assess my stack

Checklists document self-reported practice. Items were last reviewed 2026-09-20 — adapt them to your own environment.