An EKS platform that scales data jobs from the queue.

The challenge

The platform runs data-science batch jobs, such as IP-change processing, clustering, grouping, and email sanitising, that arrive in bursts. Fixed node groups were either idle or short of capacity, and environments needed to be reproducible across accounts and regions.

The constraints

• Two AWS accounts, development and production, that must never be confused.
• No long-lived cloud credentials in CI.
• Secrets must stay encrypted in Git.

Decisions & tradeoffs

• Terragrunt with the account, environment, and region hierarchy, and an allowed-account guard in every generated provider.
• Two stacks per environment, infrastructure and Kubernetes commons, linked by a dependency.
• Karpenter with consolidation and a mix of spot and on-demand capacity, keeping Cluster Autoscaler only as a disabled fallback.
• KEDA ScaledJobs that start workers from SQS queue depth instead of schedules.
• Argo CD with Image Updater: new semver tags in ECR are picked up and written back to Git.

The implementation

GitHub Actions deploys through AWS OIDC roles, so CI holds no keys. Secrets use SOPS with KMS through helm-secrets and External Secrets with Secrets Manager. Argo CD signs in through GitHub SSO, logs go to CloudWatch through Fluent Bit, and production nodes run an endpoint security sensor.

Outcomes

Batch capacity now follows the queue, and the same code grew the platform from one development region to four, plus two production clusters.

Handover & ongoing ownership

The Terragrunt and Argo CD repositories with their conventions, maintained by the client's engineers.