Reduce high Retell end-to-end latency — pick a faster LLM, shorten your prompt, enable fast tier, and diagnose slow LLM, ASR, and TTS components per call.
If your agent is slow to respond, work through this guide top to bottom. Across agents, the two most common causes of high latency are LLM model choice and prompt length, so those are listed first — but the dominant factor varies from one agent to the next, and these aren’t always the culprit. Before concluding what’s slow for a specific agent, confirm it against that agent’s per-call latency breakdown rather than assuming.
These three changes resolve most latency issues. They tend to have the highest impact, but confirm which one actually applies to your agent using the per-call breakdown before acting — not every agent is bottlenecked by the same thing.
1
Choose a faster LLM model
Model choice is often one of the largest factors in LLM response time, though how much it matters depends on the agent — verify against the call’s LLM latency before assuming the model is the bottleneck.
Smaller models respond faster. If your use case can tolerate slightly less reasoning, switching to a smaller model noticeably reduces latency.
Larger, reasoning-oriented models are slower to first token — they trade latency for capability.
Latency also varies between providers for models of a similar size, so if you’ve already trimmed model size and prompt and still see high LLM latency, trying a comparable model from a different provider can help.
Switching models changes your agent’s behavior, not just its speed. Re-test your prompt and flows after switching — treat it as a deliberate change, not a one-click toggle.
2
Shorten your prompt
Every token in your prompt adds to the time-to-first-token, and that cost is paid on every turn of the conversation.
Keep system prompts tight and focused on the current task.
Move rarely-needed detail into the knowledge base or into tool descriptions rather than the main prompt.
Agent Handbook presets also add tokens to every interaction (the estimated count is shown on hover). Turn off any preset you don’t need.
Prompts beyond roughly 8k tokens become noticeably slower. If you are well above that, trimming the prompt is one of the easiest wins available.
3
Enable fast tier
Fast tier routes your LLM calls through dedicated, high-priority infrastructure, reducing both average response time and call-to-call variance.Fast tier costs more than the standard model rate, so weigh it against your use case — but it is one of the most reliable ways to tighten an inconsistent response time.
After you stop speaking, Retell waits a short silence window — called endpointing — to confirm you’ve actually finished before it responds. A longer endpointing setting waits for more context and produces more accurate transcripts, but it adds directly to perceived latency.
If you’re using the accuracy-optimized transcription mode, note that it intentionally waits longer (about 200ms more) than the speed-optimized mode. Switch to the speed setting if responsiveness matters more than transcript precision.
Turn off Boosted Keywords if enabled, as it can add transcription latency.
Features marked with a turtle icon 🐢 add estimated latency. Aim to keep estimated latency under 1.5s, and disable any turtle-marked feature you don’t need.
Turtle-marked settings show a latency warning on hover.
2
Check your response wait time
The Response Wait time setting in speech settings adds a deliberate pause before every response, up to 5.5 seconds at its most patient. Any wait it adds appears as its own row in the estimated latency breakdown. If the agent feels slow but the per-call components look normal, lower this setting.
3
Consider geographic distance
International calls add latency from the physical distance between regions. If you’re calling across countries or continents, use a phone number in the same region as your users.
