Our streaming inference layer started as a proof of concept and ended up handling production traffic for two years. It was time to rewrite.
What was wrong
The old system passed every token through three queues, four async hops, and two serialization boundaries. It worked but added 12-15ms of overhead per request — measurable when our SLA is 35ms.
What we built
The new layer is a single-process, lock-free response pipeline with direct stream-to-socket transport. We deleted ~6,000 lines of code and the hot path got simpler.
The biggest lesson: the old design wasn’t wrong; it was right for a different scale. Recognizing that earlier would have saved us six months of patches.
Numbers
P50 latency dropped 22%. P99 dropped 41%. CPU footprint shrunk by ~30%. The codebase is smaller, the bug rate is lower, and the on-call rotation is happier.
/ JOIN THE CONVERSATION
Leave a comment.