LLM WebSocket
Retell AI connects with your server, and get responses / actions from your custom LLM.
Overview
This socket shall connect directly to your server, where you would get live transcript and other relevant inputs from us, and provide responses back to us using your custom LLM. In short, this WebSocket controls what the agents says, and controls actions like ending the call.
Retell AI will initiate this WebSocket when starting the call, and your server should get prepared to handle it.
Endpoint
{your-server-websocket-endpoint}/{call_id}Path Parameters
Unique call id to identify the call.
Protocol
Retell and your server would conform to the following protocol sending the WebSocket messages to communicate.
All message event type is “text”, where the data attribute of message event is
a JSON object stringified.
Event Flow
The connection would start by your server sends an optional config event to Retell, and a response event that serves as the begin message for agent to speak. Set content to empty string if you want the agent to wait for user to start the conversation.
Retell would send back a call details event if that’s enabled in config. Retell and your server would periodically send ping pong events to keep the connection alive if configured in config.
As the call goes, Retell would send over live transcript and other updates in update only events, and determine when is appropriate to ask for responses / reminders in (response / reminder required events)[/api-references/llm-websocket#response-reminder-required-events]. Your server will be sending back response events accordingly. Not all your responses would get spoken out, cause the user might continue to speak even when Retell thought it would be agent’s turn.
If at certain point, you want agent to jump in the conversation and speak something immediately, you can send an agent interrupt event.
Retell -> Your Server Event Spec
There will be a couple events that Retell can send to your server in this websocket,
to differentiate between them, check the interaction_type field:
ping_pong: (optional) to check for disconnection and keep the connection aliveupdate_only: (required) to send real time updates about the call like live transcriptresponse_required: (required) ask for response content from your serverreminder_required: (required) ask for reminder content from your server
Ping Pong Event
When you set auto_reconnect to true in the config event,
Retell will send ping_pong events to your server
every 2s to keep the connection alive.
Differentiate what this event is.
Available options: ping_pong
Timestamp (milliseconds since epoch) of when Retell send this event. You can use this to calculate the time taken for the round trip.
Call Details Event
When you set call_details to true in the config event,
Retell will send call details events to your server right away so that you can save the time of retriving call detail from
Get Call API.
Differentiate what this event is.
Available options: call_details
Contains the response from Register Call API.
Update Only Event
Retell would send event when transcript updates — either user speaks, or agent speaks. Retell also sends this event when turntaking happens.
Event type. This event is simply a update containing the latest transcript or turntaking information, no response required.
Available options: update_only
Complete live transcript collected in the call so far. Presented in the form of a list of utterances.
See transcript_object field of Get Call API Response for the detailed schema
for the object in the list.
Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to
true in the config event.
It precisely captures when (at what utterance, which word) the tool was invoked and what was the result if the tool calls were sent timely in this LLM websocket. See tool call invocation event and tool call result event for more information of how to send tool call invocations and results.
See transcript_with_tool_calls field of Get Call API Response for the detailed schema
for the object in the list.
Indicates change of speaker (turn taking). This field will be present when speaker changes to user (user turn), or right before agent is about to speak (agent turn). This field can be helpful determining when to call functions in the call.
Available options: agent_turn, user_turn
Response / Reminder Required Events
Retell would continuously assess if it’s a good time for agent to speak, and would ask for content for response / reminder when appropriate, but not all responses Retell ask for would get spoken out (as user might continue to speak).
Determines what do we need from your server.
response_required: Require a response from your server for the current live transcript.reminder_required: User has not spoken for a while, a reminder is needed from your server.
Available options: response_required, reminder_required
This unique auto incrementing id is used to track the response Retell needs, and used to identify the responses streamed from your server, as you can send multiple events to stream back responses, and we need an id to group them.
When a new response is needed, a new event with response id will be sent, and all previous responses will be discarded.
Complete live transcript collected in the call so far. Presented in the form of a list of utterances.
See transcript_object field of Get Call API Response for the detailed schema
for the object in the list.
Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to
true in the config event.
It precisely captures when (at what utterance, which word) the tool was invoked and what was the result if the tool calls were sent timely in this LLM websocket. See tool call invocation event and tool call result event for more information of how to send tool call invocations and results.
See transcript_with_tool_calls field of Get Call API Response for the detailed schema
for the object in the list.
Your Server -> Retell Event Spec
There will be a couple events that your server can send to Retell in this websocket,
to differentiate between them, set the response_type field:
config: (optional) the initial config for configuring reconnection, whether to send call details etcping_pong: (optional) to check for disconnection and keep the connection aliveresponse: (required) to send back responses to the user when requestedagent_interrupt: (optional) to jump in conversation and speak content in it immediately, interrupts both agent and user.tool_call_invocation: (optional) to bookkeep and weave tool call invocations and results in the transcript.tool_call_result: (optional) to bookkeep and weave tool call invocations and results in the transcript.metadata: (optional) to pass some data from the server where the LLM is running to the frontend during a web call.
Config Event
You can send a config at connection open to configure reconnection, whether to send call details etc.
Differentiate what this event is.
Available options: config
Configuration object to control whether to auto reconnect, and whether Retell sends a call detail over.
Update Agent Event
You can send agent update events at any time of the call to update some of the agent configurations. We might add more configuration to this event in the future.
This can be useful when you want to modify the agent behavior during the call, like when you want to use reminders as a way to let agent continue speaking, or you wish to make agent less responsive when user is looking up information.
Differentiate what this event is.
Available options: update_agent
Set what agent configuration you want to update. All fields are optional.
Ping Pong Event
When you set auto_reconnect to true in the config event,
you need to send ping_pong events to signal that your server
is still alive. Retell would expect a ping_pong event back every 2s, and would close the connection if there’s no ping_pong
event for 5s.
Differentiate what this event is.
Available options: ping_pong
Timestamp (milliseconds since epoch) of when your server send this event. Retell would use this to calculate the time taken for the round trip.
Response Event
Your server needs to respond to response / reminder required events so that agent can speak in time. You can stream the response or send it in one go, although streaming is recommended for lower latency. When a newer event that require response / reminder is received, you can stop responding to the previous response / reminder required events, as it would not get used. You still need to respond to previous response / reminder required event if newer update_only events are received.
Differentiate what this event is.
Available options: response
Indicates which requested response this is answering.
Partial or full content response content.
Whether the content is complete. When streaming responses back, only the last event of the response should have this field set to true.
If set to true, agent would not get interrupted by user for content in this event. Useful for conveying important information.
If set to true, Retell would end the call after content associated with this id is fully spoken. If agent was interrupted during speaking, the end call signal would get discarded.
If set, Retell would transfer the call to the number specified here after content associated with this id is fully spoken. Only applicable to Retell numbers or imported numbers. If your voice agent is using custom telephony via the dial to SIP endpoint route, you need to write your own call transfer logic.
If set, Retell would press the input digit or digits to send DTMF tones after content associated with this id is fully spoken (although you probably do not want the voice agent to speak anything when pressing digits). Only applicable if you are already using latest Call API V2.
Agent Interrupt Event
If at certain point, you want to jump in the conversation and speak something immediately, you can send this event. It will stop the agent speech if the agent is speaking, or it will interrupt the user if the user is speaking.
Differentiate what this event is.
Available options: agent_interrupt
Used to group the interrupt events. This is a unique id maintained in your server. If interrupt events with same ids are received, the content would get spoken in order of the interrupt events received. If interrupt events with different ids are received, the previous interrupt events are discarded.
Partial or full content response content.
Whether the content is complete. When streaming responses back, only the last event of the response should have this field set to true.
If set to true, agent would not get interrupted by user for content in this event. This is recommended to set to true here because without this setting, if user is talking, and agent interrupts here, the two party would speak at the same time, and agent would get interrupted quickly.
If set to true, and if this response is used for agent to speak, we would end the call after content is fully spoken.
If set, we will transfer the call to the number specified here after content is fully spoken. Only applicable to numbers purchased through Retell. For call transfer with your own Twilio account, you can trigger it from your server directly.
If set, Retell would press the input digit or digits to send DTMF tones after content associated with this id is fully spoken (although you probably do not want the voice agent to speak anything when pressing digits). Only applicable if you are already using latest Call API V2.
Tool Call Invocation Event
If you send the tool call invocation and results in the websocket,
Retell would populate the transcript_with_tool_calls field in the Get Call API Response
after call ends. We would weave the transcript and precisely capture when (at what utterance, which word)
the tool was invoked and what was the result.
If you also set transcript_with_tool_calls to true in the config event,
Retell would populate the transcript_with_tool_calls
field in the update only event with the tool call invocations and results.
This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session
and wish to have Retell manages it for you.
Differentiate what this event is.
Available options: tool_call_invocation
Tool call id, globally unique.
Name of the function in this tool call.
Arguments for this tool call, it’s a stringified JSON object.
Tool Call Result Event
If you send the tool call invocation and results in the websocket,
Retell would populate the transcript_with_tool_calls field in the Get Call API Response
after call ends. We would weave the transcript and precisely capture when (at what utterance, which word)
the tool was invoked and what was the result.
If you also set transcript_with_tool_calls to true in the config event,
Retell would populate the transcript_with_tool_calls
field in the update only event with the tool call invocations and results.
This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session
and wish to have Retell manages it for you.
Differentiate what this event is.
Available options: tool_call_result
Tool call id, globally unique.
Result of the tool call, can be a string, a stringified json, etc.
Metadata Event
Sometimes you may wish to pass some data from the server where the LLM is running to the frontend during a web call, for animation purposes or other stuff. It can be challenging to make sure the frontend of the call is connecting from can connect to the server where the LLM is running. In this case, you can send metadata events to the Retell and Retell will forward it to the frontend.
See frontend metadata event for the forwarded event.
Differentiate what this event is.
Available options: metadata
You can put anything here that can be json serialized.
Sample Events
Retell -> Your Server Sample Events
Your Server -> Retell Sample Events
Was this page helpful?