Production Kubernetes Platform — Case Study

A platform that scales sideways and survives an audit

Two properties are cheap to design in and brutally expensive to add later: horizontal headroom, and an audit trail. Both had to be there from the first diagram, because our platforms get resold to enterprises whose security teams inspect them before signing.

This is the platform our SaaS products run on. It autoscales on demand rather than on peak provisioning, it spans availability zones, and every control an assessor asks about was switched on before they asked.

Traffic path and blast-radius boundaries. The data subnet has no route to the internet; the security plane is enabled at account level, not per workload.

Why each piece is what it is

A platform diagram tells you what was built. What it does not tell you is what was rejected, which is where the actual engineering sits.

1Karpenter over managed node groups. Node groups mean choosing instance types in advance and provisioning for peak, so capacity sits idle overnight. Karpenter provisions against actual pending pods and consolidates when demand falls — the cost curve follows load instead of the worst hour of the week.

2Multi-AZ from the start, not multi-region. Multi-region doubles operational surface and cost for a recovery objective our platforms did not need. Two availability zones covers the failure mode that actually occurs; the DR plan covers the one that does not.

3Terraform with remote state, no console changes. 500+ resources under code. The rule that matters is not that it is automated — it is that nobody fixes anything by hand, because a console fix is invisible to the next apply.

4Bastion behind MFA rather than a VPN. Fewer moving parts, and the access path is one an assessor can inspect in a single screenshot.

Node groups require choosing capacity in advance, so you provision for the worst hour and pay for it all night. Karpenter provisions against pending pods and consolidates when demand falls.

The security plane is not a layer, it is the account

GuardDuty, Security Hub, Inspector, Config, KMS and ACM are enabled across the account rather than bolted onto the cluster. A control scoped to one workload has to be re-argued for every new workload; a control at account level is simply true, and stays true for whatever gets deployed next.

Observability from inside and outside

Prometheus, Grafana and Loki inside the cluster for metrics and logs — and Site24x7 checking from outside the perimeter, because an internal monitor cannot tell you the platform is unreachable. If DNS, the certificate or the load balancer is the problem, everything inside still reports healthy while users see nothing.

That external view is also what makes an uptime figure defensible. A number a customer's assessor can independently reason about is worth more than one derived from your own instrumentation.

Delivery runs on Terraform with S3 state, GitHub and ECR, with staging built to mirror production. Environments that diverge are the reason defects reach users; keeping them the same is unglamorous and pays for itself every release.

Cost controls belong in the first diagram

Rightsizing, reserved planning and spot automation came after the platform was running, as a response to a bill rather than a design input. The savings were real, but the exercise was archaeology — working out what each workload actually needed, months after the decisions were made.

Designing with cost as a constraint from the start would have been considerably cheaper than optimising later. It is now the first thing I ask about a new environment, and the reason the cloud agent I built posts a daily cost report rather than a monthly one.