AI Application

ARCHITECTURE The application calls a model provider through a single abstraction rather than scattering SDK calls through the codebase — that boundary is what makes provider change, fallback and cost attribution possible later. Prompts are versioned artifacts fetched from a registry (see /promptops), not string literals. Every request and response is traced with the model version, token counts, latency and cost attached. Guardrails sit on both sides: input checks before the call, output checks before the result reaches a user or a downstream system. A fallback path is defined for provider failure, and it degrades to something honest rather than failing opaquely. ASSUMPTIONS A user-facing feature where a wrong or slow answer has real consequence, and a team that can maintain an evaluation set. It assumes you do not control the model — if you train and serve your own, the operational centre of gravity moves to /mlops instead. It also assumes someone has answered the privacy question about what leaves your boundary. That is a prerequisite, not a later step. SECURITY Treat model output as untrusted input. If it reaches a shell, a query, a browser or another system, it needs the same validation any user input would get — prompt injection is the realistic attack and the mitigation is not "a better prompt", it is not granting the output authority. Scope what is sent to the provider, and record it. Provider API keys in a secret manager with rotation. OBSERVABILITY Per-request latency at p95 including retries, cost per feature, evaluation pass rate on every release, and the model version in every trace. Track refusal and fallback rates — a quietly rising fallback rate is the signal that the provider changed something. Capture enough of the interaction to diagnose a complaint, with retention and access controls appropriate to what it contains. AVAILABILITY Providers rate-limit, slow down and deprecate on their schedule. Define the degraded experience before you need it: a cached answer, a simpler non-model path, or an honest "unavailable" beats a spinner. Timeouts must be explicit and short enough that a slow provider does not consume your own capacity. Rehearse a provider outage and a model deprecation (see /resilience-engineering). COST CONSIDERATIONS Cost scales with tokens, and tokens scale with context. The common surprises are retrieval stuffing far more context than needed and retries multiplying spend during an incident. Measure cost per thousand requests per feature, alert before the invoice, and cap spend per request. Smaller models for the subtasks that do not need the large one is usually the largest available saving (see /finops). IAC GUIDANCE Provider configuration, rate limits, fallback routing and spend alerts in code. Prompts and evaluation sets in version control with review — an unreviewed prompt change is an unreviewed behaviour change. Keep model version pins in configuration so a change is a visible diff. PRODUCTION READINESS Before users: model and version pinned and logged, an evaluation set that can block a release, cost and latency measured with alerting, guardrails proportionate to the output's authority, provider-failure behaviour defined and tested, a named owner who can disable the feature without a deploy, and a privacy review of what is sent. The LLM feature readiness checklist at /checklists is the gate; /llmops is the practice.