We recommend following the Integrate Your LLM Guide, where it would show step by step instructions. This doc is to show the underlying protocols of what our server would send to yours, and what we expect to receive.

​
Overview

This socket shall connect directly to your server, where you would get live transcript and other relevant inputs from us, and provide responses back to us using your custom LLM. In short, this WebSocket controls what the agents says, and controls actions like ending the call.

Retell AI will initiate this WebSocket when starting the call, and your server should get prepared to handle it.

​
Endpoint

WebSocket Endpoint: {your-server-websocket-endpoint}/{call_id}

​
Path Parameters

call_id
string
required

Unique call id to identify the call.

​
Protocol

Retell and your server would conform to the following protocol sending the WebSocket messages to communicate.

All message event type is “text”, where the data attribute of message event is a JSON object stringified.

​
Event Flow

The connection would start by your server sends an optional config event to Retell, and a response event that serves as the begin message for agent to speak. Set content to empty string if you want the agent to wait for user to start the conversation.

Retell would send back a call details event if that’s enabled in config. Retell and your server would periodically send ping pong events to keep the connection alive if configured in config.

As the call goes, Retell would send over live transcript and other updates in update only events, and determine when is appropriate to ask for responses / reminders in (response / reminder required events)[/api-references/llm-websocket#response-reminder-required-events]. Your server will be sending back response events accordingly. Not all your responses would get spoken out, cause the user might continue to speak even when Retell thought it would be agent’s turn.

If at certain point, you want agent to jump in the conversation and speak something immediately, you can send an agent interrupt event.

​
Retell -> Your Server Event Spec

There will be a couple events that Retell can send to your server in this websocket, to differentiate between them, check the interaction_type field:

  • ping_pong: (optional) to check for disconnection and keep the connection alive
  • update_only: (required) to send real time updates about the call like live transcript
  • response_required: (required) ask for response content from your server
  • reminder_required: (required) ask for reminder content from your server

​
Ping Pong Event

When you set auto_reconnect to true in the config event, Retell will send ping_pong events to your server every 2s to keep the connection alive.

interaction_type
enum<string>
required

Differentiate what this event is.

Available options: ping_pong

timestamp
integer
required

Timestamp (milliseconds since epoch) of when Retell send this event. You can use this to calculate the time taken for the round trip.

​
Call Details Event

When you set call_details to true in the config event, Retell will send call details events to your server right away so that you can save the time of retriving call detail from Get Call API.

interaction_type
enum<string>
required

Differentiate what this event is.

Available options: call_details

call
object
required

Contains the response from Register Call API.

​
Update Only Event

Retell would send event when transcript updates — either user speaks, or agent speaks. Retell also sends this event when turntaking happens.

interaction_type
enum<string>
required

Event type. This event is simply a update containing the latest transcript or turntaking information, no response required.

Available options: update_only

transcript
object[]
required

Complete live transcript collected in the call so far. Presented in the form of a list of utterances.

See transcript_object field of Get Call API Response for the detailed schema for the object in the list.

transcript_with_tool_calls
object[]

Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to true in the config event.

It precisely captures when (at what utterance, which word) the tool was invoked and what was the result if the tool calls were sent timely in this LLM websocket. See tool call invocation event and tool call result event for more information of how to send tool call invocations and results.

See transcript_with_tool_calls field of Get Call API Response for the detailed schema for the object in the list.

turntaking
enum<string>

Indicates change of speaker (turn taking). This field will be present when speaker changes to user (user turn), or right before agent is about to speak (agent turn). This field can be helpful determining when to call functions in the call.

Available options: agent_turn, user_turn

​
Response / Reminder Required Events

Retell would continuously assess if it’s a good time for agent to speak, and would ask for content for response / reminder when appropriate, but not all responses Retell ask for would get spoken out (as user might continue to speak).

interaction_type
enum<string>
required

Determines what do we need from your server.

  • response_required: Require a response from your server for the current live transcript.
  • reminder_required: User has not spoken for a while, a reminder is needed from your server.

Available options: response_required, reminder_required

response_id
integer
required

This unique auto incrementing id is used to track the response Retell needs, and used to identify the responses streamed from your server, as you can send multiple events to stream back responses, and we need an id to group them.

When a new response is needed, a new event with response id will be sent, and all previous responses will be discarded.

transcript
object[]
required

Complete live transcript collected in the call so far. Presented in the form of a list of utterances.

See transcript_object field of Get Call API Response for the detailed schema for the object in the list.

transcript_with_tool_calls
object[]

Transcript of the call weaved with tool call invocation and results. Populated when transcript_with_tool_calls field is set to true in the config event.

It precisely captures when (at what utterance, which word) the tool was invoked and what was the result if the tool calls were sent timely in this LLM websocket. See tool call invocation event and tool call result event for more information of how to send tool call invocations and results.

See transcript_with_tool_calls field of Get Call API Response for the detailed schema for the object in the list.

​
Your Server -> Retell Event Spec

There will be a couple events that your server can send to Retell in this websocket, to differentiate between them, set the response_type field:

  • config: (optional) the initial config for configuring reconnection, whether to send call details etc
  • ping_pong: (optional) to check for disconnection and keep the connection alive
  • response: (required) to send back responses to the user when requested
  • agent_interrupt: (optional) to jump in conversation and speak content in it immediately, interrupts both agent and user.
  • tool_call_invocation: (optional) to bookkeep and weave tool call invocations and results in the transcript.
  • tool_call_result: (optional) to bookkeep and weave tool call invocations and results in the transcript.
  • metadata: (optional) to pass some data from the server where the LLM is running to the frontend during a web call.

​
Config Event

You can send a config at connection open to configure reconnection, whether to send call details etc.

response_type
enum<string>
required

Differentiate what this event is.

Available options: config

config
object
required

Configuration object to control whether to auto reconnect, and whether Retell sends a call detail over.

​
Update Agent Event

You can send agent update events at any time of the call to update some of the agent configurations. We might add more configuration to this event in the future.

This can be useful when you want to modify the agent behavior during the call, like when you want to use reminders as a way to let agent continue speaking, or you wish to make agent less responsive when user is looking up information.

response_type
enum<string>
required

Differentiate what this event is.

Available options: update_agent

agent_config
object
required

Set what agent configuration you want to update. All fields are optional.

​
Ping Pong Event

When you set auto_reconnect to true in the config event, you need to send ping_pong events to signal that your server is still alive. Retell would expect a ping_pong event back every 2s, and would close the connection if there’s no ping_pong event for 5s.

response_type
enum<string>
required

Differentiate what this event is.

Available options: ping_pong

timestamp
integer
required

Timestamp (milliseconds since epoch) of when your server send this event. Retell would use this to calculate the time taken for the round trip.

​
Response Event

Your server needs to respond to response / reminder required events so that agent can speak in time. You can stream the response or send it in one go, although streaming is recommended for lower latency. When a newer event that require response / reminder is received, you can stop responding to the previous response / reminder required events, as it would not get used. You still need to respond to previous response / reminder required event if newer update_only events are received.

response_type
enum<string>
required

Differentiate what this event is.

Available options: response

response_id
integer
required

Indicates which requested response this is answering.

content
string
required

Partial or full content response content.

content_complete
boolean
required

Whether the content is complete. When streaming responses back, only the last event of the response should have this field set to true.

no_interruption_allowed
boolean

If set to true, agent would not get interrupted by user for content in this event. Useful for conveying important information.

end_call
boolean

If set to true, Retell would end the call after content associated with this id is fully spoken. If agent was interrupted during speaking, the end call signal would get discarded.

transfer_number
string

If set, Retell would transfer the call to the number specified here after content associated with this id is fully spoken. Only applicable to numbers purchased through Retell. For call transfer with your own Twilio account, you need to trigger it from your server directly.

​
Agent Interrupt Event

If at certain point, you want to jump in the conversation and speak something immediately, you can send this event. It will stop the agent speech if the agent is speaking, or it will interrupt the user if the user is speaking.

response_type
enum<string>
required

Differentiate what this event is.

Available options: agent_interrupt

interrupt_id
integer
required

Used to group the interrupt events. This is a unique id maintained in your server. If interrupt events with same ids are received, the content would get spoken in order of the interrupt events received. If interrupt events with different ids are received, the previous interrupt events are discarded.

content
string
required

Partial or full content response content.

content_complete
boolean
required

Whether the content is complete. When streaming responses back, only the last event of the response should have this field set to true.

no_interruption_allowed
boolean

If set to true, agent would not get interrupted by user for content in this event. This is recommended to set to true here because without this setting, if user is talking, and agent interrupts here, the two party would speak at the same time, and agent would get interrupted quickly.

end_call
boolean

If set to true, and if this response is used for agent to speak, we would end the call after content is fully spoken.

transfer_number
string

If set, we will transfer the call to the number specified here after content is fully spoken. Only applicable to numbers purchased through Retell. For call transfer with your own Twilio account, you can trigger it from your server directly.

​
Tool Call Invocation Event

If you send the tool call invocation and results in the websocket, Retell would populate the transcript_with_tool_calls field in the Get Call API Response after call ends. We would weave the transcript and precisely capture when (at what utterance, which word) the tool was invoked and what was the result.

If you also set transcript_with_tool_calls to true in the config event, Retell would populate the transcript_with_tool_calls field in the update only event with the tool call invocations and results. This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session and wish to have Retell manages it for you.

response_type
enum<string>
required

Differentiate what this event is.

Available options: tool_call_invocation

tool_call_id
string
required

Tool call id, globally unique.

name
string
required

Name of the function in this tool call.

arguments
boolean
required

Arguments for this tool call, it’s a stringified JSON object.

​
Tool Call Result Event

If you send the tool call invocation and results in the websocket, Retell would populate the transcript_with_tool_calls field in the Get Call API Response after call ends. We would weave the transcript and precisely capture when (at what utterance, which word) the tool was invoked and what was the result.

If you also set transcript_with_tool_calls to true in the config event, Retell would populate the transcript_with_tool_calls field in the update only event with the tool call invocations and results. This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session and wish to have Retell manages it for you.

response_type
enum<string>
required

Differentiate what this event is.

Available options: tool_call_result

tool_call_id
string
required

Tool call id, globally unique.

content
string
required

Result of the tool call, can be a string, a stringified json, etc.

​
Metadata Event

Sometimes you may wish to pass some data from the server where the LLM is running to the frontend during a web call, for animation purposes or other stuff. It can be challenging to make sure the frontend of the call is connecting from can connect to the server where the LLM is running. In this case, you can send metadata events to the Retell and Retell will forward it to the frontend.

See frontend metadata event for the forwarded event.

response_type
enum<string>
required

Differentiate what this event is.

Available options: metadata

metadata
object
required

You can put anything here that can be json serialized.

​
Sample Events

​
Retell -> Your Server Sample Events

{
  "interaction_type": "ping_pong",
  "timestamp": 1703302407333
}

​
Your Server -> Retell Sample Events

{
  "response_type": "config",
  "config": {
    "auto_reconnect": true,
    "call_details": true
  }
}

Was this page helpful?

List VoicesCreate Phone Call (Deprecated)