Retell AI connects with your server, and get responses / actions from your custom LLM.
We recommend following the Integrate Your LLM Guide, where
it would show step by step instructions. This doc is to show the underlying protocols
of what our server would send to yours, and what we expect to receive.
This socket shall connect directly to your server, where you would get live transcript and
other relevant inputs from us, and provide responses back to us using your custom LLM.
In short, this WebSocket controls what the agents says, and controls actions like ending the call.Retell AI will initiate this WebSocket when starting the call,
and your server should get prepared to handle it.
Retell and your server would conform to the following protocol sending the WebSocket messages to communicate.All message event type is “text”, where the data attribute of message event is
a JSON object stringified.
The connection would start by your server sends an optional config event to Retell,
and a response event
that serves as the begin message for agent to speak.
Set content to empty string if you want the agent to wait for user to start the conversation.Retell would send back a call details event if that’s enabled in config.
Retell and your server would periodically send ping pong events
to keep the connection alive if configured in config.As the call goes, Retell would send over live transcript and other updates in
update only events, and determine when is appropriate to
ask for responses / reminders in
(response / reminder required events)[/api-references/llm-websocket#response-reminder-required-events]. Your server will be
sending back response events accordingly.
Not all your responses would get spoken out, cause the user might
continue to speak even when Retell thought it would be agent’s turn.If at certain point, you want agent to jump in the conversation and speak something immediately, you can send an
agent interrupt event.
When you set call_details to true in the config event,
Retell will send call details events to your server right away so that you can save the time of retriving call detail from
Get Call API.
Event type. This event is simply a update containing the latest transcript or turntaking information, no response required.Available options: update_only
Complete live transcript collected in the call so far. Presented in the form of a list of
utterances.See transcript_object field of Get Call API Response for the detailed schema
for the object in the list.
Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to
true in the config event.It precisely captures when (at what utterance, which word) the tool was invoked and what was the result
if the tool calls were sent timely in this LLM websocket. See
tool call invocation event
and tool call result event for more information of how to send
tool call invocations and results.See transcript_with_tool_calls field of Get Call API Response for the detailed schema
for the object in the list.
Indicates change of speaker (turn taking). This field will be present when speaker changes to user (user turn), or right before
agent is about to speak (agent turn). This field can be helpful determining when to call functions in the call.Available options: agent_turn, user_turn
Retell would continuously assess if it’s a good time for agent to speak, and would
ask for content for response / reminder when appropriate, but not all
responses Retell ask for would get spoken out (as user might continue to speak).
This unique auto incrementing id is used to track the response Retell needs, and used to identify the
responses streamed from your server, as you can
send multiple events to stream back responses, and we need an id to group them.When a new response is needed, a new event with response id will
be sent, and all previous responses will be discarded.
Complete live transcript collected in the call so far. Presented in the form of a list of
utterances.See transcript_object field of Get Call API Response for the detailed schema
for the object in the list.
Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to
true in the config event.It precisely captures when (at what utterance, which word) the tool was invoked and what was the result
if the tool calls were sent timely in this LLM websocket. See
tool call invocation event
and tool call result event for more information of how to send
tool call invocations and results.See transcript_with_tool_calls field of Get Call API Response for the detailed schema
for the object in the list.
If set to true, Retell will send ping pong events to your server, and would expect ping pong events back
from your server every 2s. Once there’s 5s without ping pong event, Retell would close the current connection
and restart a new connection to your server for up to 2 times.
If set to true, Retell will populate an additional field in the update only events,
response / reminder required events.
This additional field will contain
transcript of the call weaved with tool call invocation and results.You need to send tool call invocations and results to us in websocket so that we can construct it. See
tool call invocation event
and tool call result event for more information of how to send
tool call invocations and results.
You can send agent update events at any time of the call to update some of the agent configurations.
We might add more configuration to this event in the future.This can be useful when you want to modify the agent behavior during the call, like when
you want to use reminders as a way to let agent continue speaking, or you wish to make agent
less responsive when user is looking up information.
Controls how responsive is the agent. Value ranging from [0,1].
Lower value means less responsive agent (wait more, respond slower),
while higher value means faster exchanges (respond when it can).
Controls how sensitive the agent is to user interruptions. Value
ranging from [0,1]. Lower value means it will take longer / more
words for user to interrupt agent, while higher value means it’s
easier for user to interrupt agent.
If set (in milliseconds), will trigger a reminder to the agent to
speak if the user has been silent for the specified duration after
some agent speech. Must be a positive number.
If set, controls how many times agent would remind user when user is
unresponsive. Must be a non negative integer. Set to 0 to disable agent from
reminding.
When you set auto_reconnect to true in the config event,
you need to send ping_pong events to signal that your server
is still alive. Retell would expect a ping_pong event back every 2s, and would close the connection if there’s no ping_pong
event for 5s.
Your server needs to respond to response / reminder required events
so that agent can speak in time.
You can stream the response or send it in one go, although streaming is
recommended for lower latency. When a newer event that require response / reminder is received, you can
stop responding to the previous response / reminder required events, as it would not get used. You still need to respond to
previous response / reminder required event if newer update_only events are received.
If set to true, Retell would end the call after content associated with this id is fully spoken. If agent was interrupted
during speaking, the end call signal would get discarded.
If set, Retell would transfer the call to the number specified here after content associated with this id is fully spoken.
Only applicable to Retell numbers or imported numbers. If your voice agent is using custom telephony via the dial to
SIP endpoint route, you need to write your own call transfer logic.
If set, Retell would press the input digit or digits to send DTMF tones after content associated with this id is fully spoken
(although you probably do not want the voice agent to speak anything when pressing digits).
Only applicable if you are already using latest Call API V2.
If at certain point, you want to jump in the conversation and speak something immediately, you can send this event.
It will stop the agent speech if the agent is speaking, or it will interrupt the user if the user is speaking.
Used to group the interrupt events. This is a unique id maintained in your server. If interrupt events with same
ids are received, the content would get spoken in order of the interrupt events received. If interrupt events with
different ids are received, the previous interrupt events are discarded.
If set to true, agent would not get interrupted by user for content in this event.
This is recommended to set to true here because without this setting,
if user is talking, and agent interrupts here, the two party would
speak at the same time, and agent would get interrupted quickly.
If set, we will transfer the call to the number specified here after content is fully spoken.
Only applicable to numbers purchased through Retell.
For call transfer with your own Twilio account, you can trigger it from your server directly.
If set, Retell would press the input digit or digits to send DTMF tones after content associated with this id is fully spoken
(although you probably do not want the voice agent to speak anything when pressing digits).
Only applicable if you are already using latest Call API V2.
If you send the tool call invocation and results in the websocket,
Retell would populate the transcript_with_tool_calls field in the Get Call API Response
after call ends. We would weave the transcript and precisely capture when (at what utterance, which word)
the tool was invoked and what was the result.If you also set transcript_with_tool_calls to true in the config event,
Retell would populate the transcript_with_tool_calls
field in the update only event with the tool call invocations and results.
This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session
and wish to have Retell manages it for you.
If you send the tool call invocation and results in the websocket,
Retell would populate the transcript_with_tool_calls field in the Get Call API Response
after call ends. We would weave the transcript and precisely capture when (at what utterance, which word)
the tool was invoked and what was the result.If you also set transcript_with_tool_calls to true in the config event,
Retell would populate the transcript_with_tool_calls
field in the update only event with the tool call invocations and results.
This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session
and wish to have Retell manages it for you.
Sometimes you may wish to pass some data from the server where the LLM is running to the frontend during a web call,
for animation purposes or other stuff. It can be challenging to make sure the frontend of the call is connecting from
can connect to the server where the LLM is running. In this case, you can send metadata events to the Retell and Retell will
forward it to the frontend.See frontend metadata event for the forwarded event.