Skip to main content
If your agent is slow to respond, work through this guide top to bottom. Across agents, the two most common causes of high latency are LLM model choice and prompt length, so those are listed first — but the dominant factor varies from one agent to the next, and these aren’t always the culprit. Before concluding what’s slow for a specific agent, confirm it against that agent’s per-call latency breakdown rather than assuming.

​
Start with the biggest levers

These three changes resolve most latency issues. They tend to have the highest impact, but confirm which one actually applies to your agent using the per-call breakdown before acting — not every agent is bottlenecked by the same thing.
1

Choose a faster LLM model

Model choice is often one of the largest factors in LLM response time, though how much it matters depends on the agent — verify against the call’s LLM latency before assuming the model is the bottleneck.
  • Smaller models respond faster. If your use case can tolerate slightly less reasoning, switching to a smaller model noticeably reduces latency.
  • Larger, reasoning-oriented models are slower to first token — they trade latency for capability.
  • Latency also varies between providers for models of a similar size, so if you’ve already trimmed model size and prompt and still see high LLM latency, trying a comparable model from a different provider can help.
Switching models changes your agent’s behavior, not just its speed. Re-test your prompt and flows after switching — treat it as a deliberate change, not a one-click toggle.
2

Shorten your prompt

Every token in your prompt adds to the time-to-first-token, and that cost is paid on every turn of the conversation.
  • Keep system prompts tight and focused on the current task.
  • Move rarely-needed detail into the knowledge base or into tool descriptions rather than the main prompt.
  • Agent Handbook presets also add tokens to every interaction (the estimated count is shown on hover). Turn off any preset you don’t need.
  • Prompts beyond roughly 8k tokens become noticeably slower. If you are well above that, trimming the prompt is one of the easiest wins available.
3

Enable fast tier

Fast tier routes your LLM calls through dedicated, high-priority infrastructure, reducing both average response time and call-to-call variance.Fast tier costs more than the standard model rate, so weigh it against your use case — but it is one of the most reliable ways to tighten an inconsistent response time.

​
Diagnose by component

If the levers above don’t resolve it, use the per-call latency breakdown to find which component dominates, then target that component.

​
High LLM latency

If LLM latency dominates your end-to-end time and is suddenly worse than usual, your provider may be under transient heavy load.
  1. Check the Status Page for ongoing incidents.
  2. If there is an active incident, wait for it to resolve.
  3. Otherwise, revisit model choice and prompt length above — a consistently high LLM latency usually points to one of those.
For a voice agent whose primary model is usually suitable but sometimes slow, configure fast fallback. It starts a second model concurrently and prefers the primary for the configured period. The backup can incur additional charges even when its response is not used, so test both response quality and cost. Fast fallback is separate from a model’s speed tier and from Response Wait time. Its timeout is a primary preference period, not an end-to-end latency guarantee.

​
High ASR (transcription) latency

After you stop speaking, Retell waits a short silence window — called endpointing — to confirm you’ve actually finished before it responds. A longer endpointing setting waits for more context and produces more accurate transcripts, but it adds directly to perceived latency.
  1. Check the active transcription mode, including any node override. Try Optimize for speed if responsiveness matters more than transcript precision. With Custom Settings, review the provider’s endpointing. The latency difference between presets varies by language and provider; it is not a fixed 200ms.
  2. Turn off Boosted Keywords if enabled, as it can add transcription latency.

​
High TTS latency

The voice provider you choose affects how quickly the first audio byte is produced.
  1. Some voice settings — such as enabling emotion — can increase TTS latency in certain cases. Disable them if you don’t need them.
  2. Latency also varies by provider, so if TTS dominates your latency, try a different voice provider.

​
Choppy or unstable audio

This is usually network jitter or call-server load rather than model latency.
  1. Run a ping test from your client to api.retellai.com. A round-trip time consistently above 300ms can cause latency spikes.
  2. To confirm jitter or packet loss on a specific call, analyze its PCAP file in Wireshark.
  3. If your ping is low and the problem persists, contact support with an example call ID.

​
Other factors

1

Turn off unnecessary turtle-icon features

Features marked with a turtle icon 🐢 add estimated latency. Aim to keep estimated latency under 1.5s, and disable any turtle-marked feature you don’t need.
The Response Wait time setting in Speech settings with a turtle icon beside its label. A tooltip anchored to the icon reads 'When tuned down, the latency will increase', and the slider sits at the far right showing 0.4s.

Turtle-marked settings show a latency warning on hover.

2

Check your response wait time

The Response Wait time setting in speech settings adds a deliberate pause before every response, up to 5.5 seconds at its most patient. Any wait it adds appears as its own row in the estimated latency breakdown. If the agent feels slow but the per-call components look normal, lower this setting.
3

Consider geographic distance

International calls add latency from the physical distance between regions. If you’re calling across countries or continents, use a phone number in the same region as your users.

​
Contact support

If the steps above don’t resolve your latency issues:
  1. Locate your call ID.
  2. Message support.
  3. Include your call ID, the steps you’ve already tried, and your current latency measurements.