LLM WebSocket

​

Overview

This socket shall connect directly to your server, where you would get live transcript and other relevant inputs from us, and provide responses back to us using your custom LLM. In short, this WebSocket controls what the agents says, and controls actions like ending the call.

Retell AI will initiate this WebSocket when starting the call, and your server should get prepared to handle it.

​

Endpoint

​

Path Parameters

​

Protocol

Retell and your server would conform to the following protocol sending the WebSocket messages to communicate.

All message event type is “text”, where the data attribute of message event is a JSON object stringified.

​

Event Flow

The connection would start by your server sends an optional config event to Retell, and a response event that serves as the begin message for agent to speak. Set content to empty string if you want the agent to wait for user to start the conversation.

Retell would send back a call details event if that’s enabled in config. Retell and your server would periodically send ping pong events to keep the connection alive if configured in config.

As the call goes, Retell would send over live transcript and other updates in update only events, and determine when is appropriate to ask for responses / reminders in (response / reminder required events)[/api-references/llm-websocket#response-reminder-required-events]. Your server will be sending back response events accordingly. Not all your responses would get spoken out, cause the user might continue to speak even when Retell thought it would be agent’s turn.

If at certain point, you want agent to jump in the conversation and speak something immediately, you can send an agent interrupt event.

​

Retell -> Your Server Event Spec

There will be a couple events that Retell can send to your server in this websocket, to differentiate between them, check the interaction_type field:

• ping_pong: (optional) to check for disconnection and keep the connection alive
• update_only: (required) to send real time updates about the call like live transcript
• response_required: (required) ask for response content from your server
• reminder_required: (required) ask for reminder content from your server

​

Ping Pong Event

When you set auto_reconnect to true in the config event, Retell will send ping_pong events to your server every 2s to keep the connection alive.

​

Call Details Event

When you set call_details to true in the config event, Retell will send call details events to your server right away so that you can save the time of retriving call detail from Get Call API.

​

Update Only Event

Retell would send event when transcript updates — either user speaks, or agent speaks. Retell also sends this event when turntaking happens.

​

Response / Reminder Required Events

Retell would continuously assess if it’s a good time for agent to speak, and would ask for content for response / reminder when appropriate, but not all responses Retell ask for would get spoken out (as user might continue to speak).

​

Your Server -> Retell Event Spec

There will be a couple events that your server can send to Retell in this websocket, to differentiate between them, set the response_type field:

• config: (optional) the initial config for configuring reconnection, whether to send call details etc
• ping_pong: (optional) to check for disconnection and keep the connection alive
• response: (required) to send back responses to the user when requested
• agent_interrupt: (optional) to jump in conversation and speak content in it immediately, interrupts both agent and user.
• tool_call_invocation: (optional) to bookkeep and weave tool call invocations and results in the transcript.
• tool_call_result: (optional) to bookkeep and weave tool call invocations and results in the transcript.
• metadata: (optional) to pass some data from the server where the LLM is running to the frontend during a web call.

​

Config Event

You can send a config at connection open to configure reconnection, whether to send call details etc.

​

Update Agent Event

You can send agent update events at any time of the call to update some of the agent configurations. We might add more configuration to this event in the future.

This can be useful when you want to modify the agent behavior during the call, like when you want to use reminders as a way to let agent continue speaking, or you wish to make agent less responsive when user is looking up information.

​

Ping Pong Event

When you set auto_reconnect to true in the config event, you need to send ping_pong events to signal that your server is still alive. Retell would expect a ping_pong event back every 2s, and would close the connection if there’s no ping_pong event for 5s.

​

Response Event

Your server needs to respond to response / reminder required events so that agent can speak in time. You can stream the response or send it in one go, although streaming is recommended for lower latency. When a newer event that require response / reminder is received, you can stop responding to the previous response / reminder required events, as it would not get used. You still need to respond to previous response / reminder required event if newer update_only events are received.

​

Agent Interrupt Event

If at certain point, you want to jump in the conversation and speak something immediately, you can send this event. It will stop the agent speech if the agent is speaking, or it will interrupt the user if the user is speaking.

​

Tool Call Invocation Event

If you send the tool call invocation and results in the websocket, Retell would populate the transcript_with_tool_calls field in the Get Call API Response after call ends. We would weave the transcript and precisely capture when (at what utterance, which word) the tool was invoked and what was the result.

If you also set transcript_with_tool_calls to true in the config event, Retell would populate the transcript_with_tool_calls field in the update only event with the tool call invocations and results. This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session and wish to have Retell manages it for you.

​

Tool Call Result Event

If you send the tool call invocation and results in the websocket, Retell would populate the transcript_with_tool_calls field in the Get Call API Response after call ends. We would weave the transcript and precisely capture when (at what utterance, which word) the tool was invoked and what was the result.

If you also set transcript_with_tool_calls to true in the config event, Retell would populate the transcript_with_tool_calls field in the update only event with the tool call invocations and results. This is helpful when you do not wish to maintain a copy of transcript and function calls locally during the call session and wish to have Retell manages it for you.

​

Metadata Event

Sometimes you may wish to pass some data from the server where the LLM is running to the frontend during a web call, for animation purposes or other stuff. It can be challenging to make sure the frontend of the call is connecting from can connect to the server where the LLM is running. In this case, you can send metadata events to the Retell and Retell will forward it to the frontend.

See frontend metadata event for the forwarded event.

​

Sample Events

​

Retell -> Your Server Sample Events

{
  "interaction_type": "ping_pong",
  "timestamp": 1703302407333
}

​

Your Server -> Retell Sample Events

{
  "response_type": "config",
  "config": {
    "auto_reconnect": true,
    "call_details": true
  }
}