Skip to content
All articles

AWS · VPC networking

98% of the VPC is free, and pod 697 won't start

On EKS with the VPC CNI, every pod takes a real address from its node's subnet. A /16 VPC cut into /24 subnets runs out at 696 pods while 98% of its addresses sit unused.

· 4 min read

A VPC, 10.0.0.0/16, with 65,536 addresses. Its subnets take 1,152, and the other 64,384 (98.2%) belong to no subnet, so no pod can use them. Three zones, ap-south-1a, 1b and 1c, each have a private /24 subnet with 251 usable addresses, filled by eight m5.large nodes of 30 addresses and 29 pods each. 8 nodes × 29 pods × 3 zones = 696 pods. Pod 697 fails with FailedCreatePodSandBox: only 11 addresses are left in each subnet, too few for another node. The same VPC with private /20 subnets has 4,091 usable addresses in each, 136 nodes per zone and 11,832 pods: 17 times the capacity, for one digit typed on the first afternoon.

Subnet sizes are usually decided in an account's first hour, and the cost of that decision arrives much later. One version of it looks like this: new pods fail with FailedCreatePodSandBox, and the team debugs it as a Kubernetes problem for hours before anyone counts addresses, because nothing in the error mentions a subnet.

Where the addresses went

A VPC is a private network with a CIDR block you choose. Subnets are carved out of it, one per availability zone per tier, and each one is permanently sized. AWS reserves five addresses in every subnet (the network address, the VPC router, DNS, one for future use, and broadcast), so a /24 gives you 251 usable addresses, not 256.

On EKS with the VPC CNI, every pod takes a real, routable VPC address from the subnet its node sits in, not a virtual one from an overlay. An m5.large holding 29 pods consumes 30 addresses. The course's lab runs the numbers for a cluster that has to run 3,000 pods across three zones:

VPC /16                    65,536 addresses
public /26 x 3 AZs         192 allocated, 59 usable each
private /24 x 3 AZs        768 allocated, 251 usable each
data /26 x 3 AZs           192 allocated, 59 usable each
unallocated in the VPC     64,384 (98.2% of it)
m5.large + VPC CNI         30 addresses per node
-> 8 nodes per AZ x 29 pods x 3 AZs = 696 pods   OUT OF ADDRESSES

Sixty-five thousand addresses, 98 per cent of them free, and the cluster cannot start its 697th pod. A pod can only take an address from its own subnet. The free space in the VPC is not available to it, and no zone can lend to another.

One digit on the first afternoon

The same /16 with /20 private subnets has 4,091 usable addresses in each, room for 136 nodes per zone and 11,832 pods in all: 17 times the capacity.

The expensive case is the VPC that started too small. A /20 VPC, chosen because "we only have a few services" and cut into /22 private and /26 public and data subnets, leaves 640 unallocated addresses, too few for a fourth subnet of any useful size. A VPC's primary CIDR cannot be changed, so every remedy adds something: a secondary CIDR block (EKS custom networking typically puts pods in a secondary CIDR from the 100.64.0.0/10 range, for exactly this reason) or IPv6, each with routing, peering and operational costs of its own.

Engineering Insight

Take a /16, cut /20 private subnets, and leave half the VPC unallocated. Unused private address space costs nothing, and every subnet you do not carve today is one you can add later without a migration. It is the cheapest reversibility you will buy in AWS.

Get one diagram a week

A short article built around one engineering diagram, from the same library as these courses.

One diagram-led article a week on AI and systems engineering. We email you once to confirm, and every newsletter has an unsubscribe link. Privacy policy