Skip to content
Data Platform Infrastructure

Operating Model

Data Platform uses a same-account model for each active environment. The EKS cluster, service IAM roles, Glue catalog, Lake Formation settings, and lakehouse S3 bucket for an environment live in the same <env>-dataops account.

flowchart LR
  Engineer[Engineer SSO profile] --> Terraform[Terraform root]
  CI[GitHub Actions OIDC] --> Provisioner[env-dataops-data-platform-provisioner]
  Terraform --> Provisioner
  Provisioner --> AWS[AWS resources]
  Pod[Workload pod] --> IRSA[Service IRSA role]
  IRSA --> LF[Lake Formation and Glue]
  LF --> S3[Iceberg S3 data]
  Polaris[Polaris] --> Glue[Glue catalog]
  Kyuubi[Kyuubi Spark] --> Polaris
  Kyuubi --> Glue

Principles

  • Deployments use environment-owned provisioner roles.
  • Workloads use service-specific IRSA roles.
  • Lake Formation grants are made to the consuming principal, not to the human or CI identity that deployed the resource.
  • Prod deployments go through protected CI/CD. Local prod write deployment is intentionally not the normal path.
  • Cross-environment reads use Glue, Lake Formation, S3, Kyuubi, and Polaris configuration. They do not require services in one environment to connect to another environment’s Polaris RDS.

Checked Against

  • docs/data-platform-auth-and-data-flow.html.
  • implementations/2026-05-21-dl-419-prod-dataops-runtime-progress.md.
  • implementations/2026-05-22-dl-419-dev-catalog-interactive-write-exception-progress.md.
  • implementations/2026-05-26-dl-388-terraform-cutover-progress.md.