Data Platform

ARCHITECTURE Sources land raw in object storage or a warehouse staging layer, unchanged. Transformation runs as version-controlled code producing modelled tables, and a semantic layer defines metrics once for downstream consumption (see /analyticsops). Orchestration coordinates dependencies and retries. Lineage is captured across the whole path. Keep the raw layer immutable. The ability to reprocess from the original bytes is what turns a transformation bug from a data-loss incident into a backfill. Schema contracts at the producer boundary are the difference between a platform and a permanent firefight. Without them every upstream deploy is an unannounced change to your inputs. ASSUMPTIONS More than one team consumes the data, someone owns the pipelines operationally, and the business will act on what the platform produces. It assumes a warehouse-centric approach is acceptable — if sub-second serving is required, that is a different design and should not be bolted onto this one. SECURITY Access governed at the model layer rather than by restricting dashboards, because the query interface is not the only door. Column-level controls for sensitive fields and masking in non-production environments. Personal data needs a retention schedule that is applied, not documented — a warehouse is where retention obligations quietly accumulate (see /privacyops). Service credentials for pipelines scoped per source, so one compromised connector is not a full-estate credential. OBSERVABILITY Freshness per dataset measured against a stated expectation — the defining failure of a data platform is silence, not noise. Row-count and distribution checks on load, test pass rates tracked over time, and pipeline failures that page an owner with enough context to act. Lineage makes impact analysis possible before a change rather than after, which is the difference between a review and an incident. AVAILABILITY Data platforms usually need recoverability more than uptime: a few hours late is survivable, silently wrong is not. Design for reprocessing — idempotent loads, deterministic transformations, and a documented backfill procedure that does not double-count. Orchestrator failure should delay, not corrupt. Know your recovery position for the warehouse itself and test a restore. COST CONSIDERATIONS Warehouse compute is usually the largest line and it is driven by query patterns, not data volume. Uncontrolled ad-hoc querying and full-refresh transformations that could be incremental are the two common causes. Storage tiering for raw data is cheap and worth doing early. Attribute spend per team or the incentive to improve sits with nobody (see /finops). IAC GUIDANCE Warehouse objects, roles and grants in code — permissions granted by hand are the hardest drift to detect and the most consequential. Transformation code, tests and orchestration definitions in version control with review. Keep environment differences to configuration rather than separate code paths. PRODUCTION READINESS Before teams depend on it: every dataset has an owner and a freshness expectation, automated freshness checks exist, quality tests can block publication rather than annotate it, schema contracts agreed with producers, backfill documented and performed once, lineage recorded, and access governed at the model layer. The pipeline readiness checklist at /checklists covers it, and /dataops is the practice.