Two things came out of this work. The first is a NAT migration that made outbound traffic for one workload about 40% cheaper and roughly 6× faster, saving around €4,300 a year going forward. The second we tripped over by accident while staring at the billing data: an unrelated firewall line that had quietly cost us about €15,700 in total since March 2024, most of it in the year before we caught it. I’ll tell both stories, starting with the one that took us eighteen months to finish.
The workload that broke Cloud NAT
Almost everything we run reaches the internet through Cloud NAT, Google’s fully-managed egress gateway. For the vast majority of our services it just works and we never think about it. That’s the whole point of a managed service, and for most of the fleet I’d pick it again tomorrow.
Then there’s gathertime, our cinema-showtimes scraper and part of the Khronos project. It’s the exception that never stopped being the exception. It fires bursts of short-lived HTTP requests at hundreds of external sites from Cloud Run containers that scale out on demand: something like 4,000 to 5,000 simultaneous connections at peak across ~50 containers. It’s a tiny fraction of our fleet, and on its own it hits outbound harder than everything else combined.
This is europe-west1’s whole Cloud NAT profile, every workload in the region,
not just gathertime. But gathertime is most of what you’re looking at: nothing
else we run comes close to that connection count or that bursty a shape.
Under those bursts, connections started failing intermittently. From inside the app it looked like this: thousands of outbound fetches dying with “no network route to the specified host”.
The same failure from the app’s side: thousands of errored scrape requests
during the incidents.
What made it maddening is that we were getting OUT_OF_RESOURCES while using
under 1% of the gateway’s port capacity. Failing at 1% utilisation isn’t a
capacity problem. It’s an allocation problem, and those are a lot harder to see.
Two causes: our code, and how Cloud NAT hands out ports
Working the problem turned up two things, and I want to be upfront that one of them was ours.
The scraper had lost its HTTP connection pooling somewhere along the way. It opened a fresh connection, and burned a new source port, for every single request instead of reusing them. Boxoffice Team fixed it: a shared, pooled HTTP client, and we dropped a legacy proxy layer in the same pass. It helped. It also didn’t stop the failures, which is the part that sent us looking deeper.
The second cause was structural, and Google’s engineering team confirmed it for us. Cloud NAT doesn’t give a workload all its ports up front. It hands them out in blocks, and only reaches for the next block once the current one is 80% full, an allocation that takes several seconds. During those seconds, a burst that outruns the last 20% of the block has nowhere to go, and connections drop. A scraper firing hundreds of new connections at a fresh site sails straight past that headroom every time.
Dropped packets during scrape bursts. The drop to near-zero follows a
port-tuning change, a band-aid that moved the pain rather than removing it.
Cloud NAT’s port allocation is tuned for steady traffic, not bursts, and no amount of retry logic in the app changes the shape of the mechanism underneath.
The dead end we spent eighteen months in
There’s exactly one knob: hand out bigger blocks so bursts don’t run out. But bigger blocks come from a fixed pool of public IPs, and we bring our own, a deliberately small set. So the trade was brutal. Blocks too small and bursty jobs drop connections. Blocks too big and a single intensive job reserves so many ports that other jobs can’t get an address at all and fail to start.
There was no setting that kept both the scraper and everything else happy. Every attempt to tune one side burned through our public-IP budget on the other. We chased that trade-off for about eighteen months, and I think we knew it was zero-sum long before we admitted it.
Why we finally moved: the 10× wall
We could have limped along at today’s volume. What changed the maths: while scoping a new system that would land in the same Khronos project as gathertime, it became clear Cloud NAT wouldn’t hold up once that load landed too. gathertime’s own traffic is projected to grow roughly 10× this year on top of that. Extrapolating the ports-versus-IPs trade to that combined volume made the answer obvious: the managed model was not going to carry us there.
I want to be clear this isn’t “Cloud NAT is bad”. It’s an excellent fit for most of what we run. This was a capacity-planning call. Our single worst workload had outgrown the abstraction, and it was about to get ten times worse.
The fix: let the kernel do the NAT
A Linux box has done NAT for decades (it’s what we ran on-premises too, before the move to cloud), and its model is the opposite of Cloud NAT’s: no blocks, no pre-allocation. The kernel assigns a source port from the full range of roughly 64,000 at connection time, and the whole range is available to every connection on that IP. Bursts get handled at kernel speed, because there’s no “allocate the next block” pause to fall into. And since each machine offers the full port range on a single IP, the ports-versus-addresses trade that cornered us simply disappears.
So we built our own NAT: a pool of small Debian VMs running the kernel’s native NAT, behind a load balancer, with health checks and their own fixed public IPs.
Your workload (opts in with a network tag)
│
▼
Load balancer ──► ┌── NAT VM ──► internet
(health-checked) ├── NAT VM ──► internet
└── NAT VM ──► internet
(each with a fixed public IP)For the curious: traffic is steered by a VPC route tied to a per-region network
tag, the load balancer spreads it across VMs in 3 zones, and each VM does
nftables masquerading with the kernel’s connection tracker sized for ~260k
concurrent connections. The VMs are disposable. We rebuild them from the latest
patched image on every change, with no downtime, so there’s nothing to maintain
by hand.
There’s a real cost to this that I don’t want to gloss over: we now own the reliability of something Google’s SRE org used to own for us. We took that on deliberately, and we cover it with health-checked redundancy across 3 zones, automated image rebuilds, and monitoring on the gateway itself. But it’s a genuine tradeoff, and anyone inheriting this should know that going in.
What it actually cost
These are billed actuals from the billing export, not volume-normalised numbers. Monthly traffic varies, so read them as a representative before/after rather than a lab result.
Before, europe-west1’s Cloud NAT bill averaged about €886/month, driven by the per-GB data-processing fee. After migrating that region, for a stable month:
| €/month | |
|---|---|
| Cloud NAT (traffic not yet migrated) | 264 |
| VM NAT (europe-west1) | 263 |
| Total | 527 |
So roughly €886 down to €527 a month, about 40% lower, or around €4,300/year. And that’s before the bigger saving we tripped over on the way out.
VM NAT isn’t free either, and I’d rather you hear the catch from me than find it in a bill later. The VMs themselves are cheap; the bigger line is inter-zone data transfer, the return traffic crossing zone boundaries between the load balancer and the VMs. What we really did was trade Cloud NAT’s expensive per-GB processing fee for cheap compute plus a much cheaper per-GB transfer fee. Cheaper overall, not zero.
What changed for the application
Start with the same dashboard we opened with:
Same region, same graph, a few weeks later after migration. Once gathertime’s bursts stopped
hitting Cloud NAT, the connection count and data-processing rate for the whole
region just flattened. Whatever’s left is everything else we run, which was
never the problem.
This is the part that matters most to you, though: here’s gathertime’s own latency before and after:
gathertime request latency. Before: a jittery ~30 seconds, spiking past 40.
After moving off Cloud NAT (around 10:00): a stable ~5 seconds, roughly 6×
lower, with the variance gone too. The retries and stalled connections just
disappear.
The OUT_OF_RESOURCES-at-1% failures, the burst-time drops, the
sleep-and-retry hacks people had bolted into their code: all gone, because the
conditions that caused them don’t exist on the new gateway.
No more errors, no more retries, and a 6× drop in latency. The app is faster, simpler, and cheaper to run.
The firewall line that cost us €15,700
While we were deep in the network and billing data, we found an unrelated trap that ended up costing more than the entire NAT migration saved. GCP’s firewall has two tiers: a free basic tier, and a paid advanced tier that bills a per-gigabyte fee on all inspected traffic, plus a policy-coverage charge. You get silently upgraded to the paid tier the moment any rule uses a hostname instead of an IP address.
One rule, allowing Windows license activation by hostname
(kms.windows.googlecloud.com), was keeping our entire shared firewall on the
paid tier. We don’t have an exact date for when it was added: the
policy-coverage charge shows up in the billing export as far back as March 2024,
while the per-gigabyte data-processing charge only starts appearing in June 2025. Whatever the full history, by the time we caught it in April 2026 the two
charges together were running at about €870/month, org-wide, across every
project on the shared network. Summed across every month it billed, this rule
cost us about €15,700 in total.
The fix was one line, swapping the hostname for its IP:
- dest_fqdns = ["kms.windows.googlecloud.com"]
+ dest_ip_ranges = ["35.190.247.13/32"] # kms.windows.googlecloud.com| Policy coverage | Data processing | Total | |
|---|---|---|---|
| 2024-03 (earliest billed month) | €53 | — | €53 |
| 2025-06 (data processing starts) | €423 | €356 | €779 |
| 2026-04 (last full month) | €352 | €517 | €869 |
| 2026-05 (fix landed mid-month) | €64 | €62 | €126 |
| June 2026 onward | €0 | €0 | €0 |
| Total, Mar 2024 – May 2026 | €11,431 | €4,297 | €15,728 |
About €15,700 total, from one line of Terraform. If you write firewall rules, the takeaway is simple: a hostname in a rule isn’t free. A single FQDN reference, in one rule, changes what the entire shared firewall costs for everyone on it, and it can keep costing you for a long time before anyone notices.
What we need you to do
If you take one thing from this post, take this:
⚠️ Retry your long-lived connections. When we roll a NAT VM for routine patching, it drains for up to 120 seconds and then resets any connection still open. Short requests are unaffected. But if you hold long-lived connections (WebSockets, gRPC streams, persistent keep-alive pools, database connections over the NAT), your app must reconnect and retry with backoff. It was good practice already; now it’s load-bearing. Not sure whether this affects you? Ask us.
Beyond that, most of you don’t need to do anything. If your outbound traffic is modest, you’re on Cloud NAT today, because it’s still the right default. We’re migrating everyone gradually: right now it’s one region and one workload on VM NAT, and we’ll move the rest region by region. europe-west1’s remaining traffic and europe-west9 alone still carry about €314/month in Cloud NAT cost between them, roughly €3,760/year, which gives you a sense of how much is still left to shift. Each migration shifts more traffic off Cloud NAT’s per-GB fee and onto the faster, burst-tolerant path, so expect lower NAT cost and better outbound performance over time, with no action needed beyond the retry point above.
What I’d take away from all this
A problem you keep re-tuning is usually architectural, not a tuning problem. Eighteen months of port tuning was the signal, and we read it late. Fixing the connection-pooling bug first mattered too, not because it solved anything, but because it told us honestly that the real cause was somewhere deeper. And watch the whole bill, not just the line you expect: the 40% NAT saving is real, but a hostname in a firewall rule and inter-zone transfer are exactly the kind of hidden lines that move the number when you’re not looking.
Cloud NAT is still great for the common case. Knowing precisely where it stops fitting, which for us was high-burst, IP-constrained, high-scale egress, turned out to be the actual job.
