When we set out to make Aurora faster, we didn’t expect to find quite this much room. The team had been chipping at the latency budget for months — better caching, smaller prompts, the usual moves. But the numbers stayed stubborn.
This post is a walkthrough of what changed, what didn’t, and the three improvements that, taken together, dropped P50 latency from 64ms to 34ms — a 47% reduction over a single quarter.
1. We rewrote the inference layer
The single biggest win came from replacing our request-routing layer. The old system bounced every request through a queue + worker pool that made sense at 10K requests/day. At 10M, the queue was the bottleneck.
The new architecture uses direct connection to per-region model replicas with circuit breakers and a streaming response handler. The win wasn’t the technology — it was admitting we’d outgrown the original design.
2. Prompt caching, but per-tenant
Our customers were sending the same system prompts thousands of times an hour. Caching at the model layer felt obvious in retrospect, but the privacy implications took us a few weeks to get right.
The cache wins were 4× larger when we keyed by tenant + prompt rather than just prompt. Customers don’t share workloads, but their prompts often look similar — and we’d been treating each request as fully independent.
3. Edge regions, finally
The third improvement was deploying to twelve regions instead of three. Latency for our European and APAC customers dropped from “fine” to “imperceptible” — and the cost added up to less than 8% on top of our infra bill.
What’s next
We’re looking at the next 30% — and it’s going to be harder. The cheap wins are gone. But customers tell us “fast enough” is now table stakes; what they want is predictable. So that’s where we’re pointing the team for Q2.
/ JOIN THE CONVERSATION
Leave a comment.