Model inference benchmarks are everywhere. Groq does 800 tokens per second. Fireworks AI does 300. Together AI competes on price-per-million-tokens.
None of this matters if your agent takes 9 seconds to respond.
Where the time actually goes
We profiled a production agent handling customer support queries. Average end-to-end latency: 8.2 seconds. Here's the breakdown:
- Model inference: 1.4 seconds
- Tool call execution (API lookups): 2.1 seconds
- Context assembly (fetching history, building prompt): 1.8 seconds
- Orchestration overhead (framework, serialization, callbacks): 1.6 seconds
- Network round trips between services: 1.3 seconds
The model was fast. Everything around the model was slow.
The fixes nobody talks about
Parallelize tool calls. If your agent needs data from three APIs, don't call them sequentially. Most orchestrators do this by default, but most custom implementations don't.
Pre-fetch context. Don't wait for the model to ask for user history. Load it when the request arrives. By the time the model needs it, it's already in memory.
Co-locate inference and tools. If your model runs on us-east-1 and your tools run on eu-west-1, every tool call adds 80ms of network latency. Multiply by 8 tool calls and you've added 640ms of pure geography.
Stream the first token. Users perceive latency from the first visible response, not the last. Start streaming the model's output immediately. Even if the full response takes 5 seconds, seeing text appear after 400ms changes the experience completely.
Switch to a faster model? Maybe. Fix your orchestration? Definitely.
If agent latency is killing your user experience, we optimize the parts benchmarks ignore.