Data Contracts: Enforcing Schema Governance at Scale

The Schema Registry Isn't Enough

Confluent Schema Registry validates compatibility. But it doesn't tell you: who owns this field? What does "user_id" mean? Is "status" deprecated? Who do I ping when the pipeline breaks?

Data Contracts = Schema + Semantics + SLA

We define contracts as YAML files owned by producer teams:

# contracts/events/user_created.yaml
version: 1.2.0
owner: identity-team
slack: #identity-platform
sla:
  freshness: 5m
  availability: 99.9%
fields:
  user_id:
    type: string
    format: uuid
    description: "Globally unique user identifier"
    pii: false
  email:
    type: string
    format: email
    description: "Primary email for login and notifications"
    pii: true
    encryption: required
  created_at:
    type: timestamp
    description: "Account creation timestamp (UTC)"
    pii: false

Automated Enforcement

• CI gate: Producer PRs run contract validation (schema compatibility, required fields, PII tags)
• Breaking change detection: Compare new version vs latest in registry. Fail on field removal, type change, required→optional
• Consumer notification: Auto-generate PRs in consumer repos with migration guide when deprecation announced
• Runtime validation: Sidecar validates every event against contract. Dead letter queue for violations.

Results

• Schema-related incidents: 47/month → 3/month
• Time to onboard new consumer: 2 weeks → 2 hours
• Producer teams now treat data as a product, not a byproduct