How we cut model latency by 47% in one quarter
A walkthrough of the infra changes our team shipped between October and December — with the actual numbers, the dead ends, and the wins that surprised us.
Category: Engineering
A better way to do streaming inference
We rewrote our inference layer over a long weekend. Here’s the architecture and why it matters.