Event-Driven Architecture

ARCHITECTURE Producers publish events to a durable broker; consumers process independently with their own retry and failure handling. A schema registry governs the event contract. Dead-letter destinations capture what cannot be processed, and something watches them — an unwatched dead-letter queue is a silent data-loss mechanism. The decisions that matter are made early and are expensive to change: ordering guarantees, delivery semantics, and whether consumers must be idempotent. In practice, assume at-least-once delivery and make every consumer idempotent. Exactly-once is a property of a narrow set of configurations and usually not the one you are running. Events describe what happened, in the producer's language. An event shaped around what one consumer wants is a remote procedure call with extra latency. ASSUMPTIONS Teams that can own a consumer end to end, tolerance for eventual consistency in the user experience, and enough operational maturity to debug a flow that has no single stack trace. It assumes someone owns the broker. If your services are synchronous by nature and the team is small, this adds debugging cost without buying decoupling you need. SECURITY Authentication and authorization per topic, not per cluster. Encryption in transit, and at rest if events carry personal data — which they often do, and which makes retention a privacy obligation rather than a storage setting (see /privacyops). Be deliberate about what goes into an event body: a durable log is the worst place to discover you published something you must now delete, because deleting from a compacted log is a project. OBSERVABILITY Consumer lag per consumer group is the primary health signal — it tells you whether the system is keeping up before users notice. Trace context must propagate through the broker, or every asynchronous hop becomes a debugging dead end. Track dead-letter volume, redelivery counts and end-to-end latency from produce to final effect, which is the number the business actually experiences. AVAILABILITY Broker replication across zones with a replication factor and minimum in-sync replica setting you have reasoned about — the defaults usually favour availability over durability, and you should choose rather than inherit that. Consumers should degrade by falling behind, not by failing: a consumer that crashes on a bad message and restarts into the same message is an infinite loop, which is what dead-lettering exists to prevent. Replaying is your recovery tool. Make it deliberate: decide retention with replay in mind, and rehearse it. COST CONSIDERATIONS Storage and inter-zone transfer dominate. Retention is the main lever, and it is a trade against your ability to replay. Over-partitioning costs money and rarely buys throughput you need; under-partitioning caps consumer parallelism. Managed brokers remove operational cost and add a per-throughput bill that grows quietly (see /finops and /noops for the honest accounting). IAC GUIDANCE Topics, partitions, retention and access policy belong in code, not in a console — topic configuration created by hand is the classic drift source here. Schemas belong in version control with compatibility checking in the pipeline, so a breaking change fails a build rather than a consumer. PRODUCTION READINESS Before production: consumers idempotent, dead-letter destinations defined and monitored, consumer lag alerting with an owner, trace context propagating across the broker, schema compatibility enforced in CI, retention and replay decided, and a rehearsed replay. See /dataops for the pipeline discipline and /observability-engineering for the tracing this depends on.