> ## Documentation Index
> Fetch the complete documentation index at: https://docs.retellai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio WebSocket (Deprecated)

> Deprecated low-level audio WebSocket protocol that connects Retell to your audio source — kept for reference; use Custom Telephony or Web Call SDKs.

<Note>We recommend following the [Custom Telephony phone call setup guide](/deploy/custom-telephony) and
[web call setup guide](/deploy/web-call). In these guides, the SDK will do all the work and
you can ignore this doc. This doc is to show the underlying protocols
for those few folks that want to work with audio bytes, websocket, and native js directly. </Note>

## Overview

This WebSocket shall connect directly to user audio source, whether it's from web,
or from phone through Twilio websocket. You should first
[Register Call](/api-references/register-call), then generate this endpoint, and
pass it to your frontend, or to Twilio.

Different audio sources have different protocols, and have different requirements on
websocket messages, leading to different ways to read and send audio bytes.
There are multiple protocols that our system would support, and only
one of them is defined by us.

The Web frontend / programmable communication tools like Twilio will initiate this WebSocket,
and Retell AI will handle it. Upon receiving this socket, the call is considered ongoing,
and Retell AI will initiate [LLM WebSocket](/api-references/llm-websocket)
to your server to get agent text responses.

## Endpoint

<Note>Websocket Endpoint: `wss://api.retellai.com/audio-websocket/{call_id}`</Note>

### Path Parameters

<ParamField query="call_id" type="string" required>
  Call id to identify call spec and to authenticate.
  Generated from [Register Call](/api-references/register-call).
</ParamField>

### Query Parameters

<ParamField query="enable_update" type="boolean" default="false">
  Whether to send live update about the call (currently contains transcript) in the websocket. Only supported in `Web` Protocol.
  Commonly used to render live transcript in web frontend.
</ParamField>

<ParamField query="enable_audio_alignment" type="boolean" default="false">
  With this field set to true, the audio will not be sent as raw audio bytes, but encoded to base64 and
  sent in a json where it contains the text corresponding to the audio. This is useful for alignment and
  animation purposes. Currently the frontend SDK does not support this option.
</ParamField>

## Web Frontend Code Walkthrough

We provide an example React frontend code. Detailed guide and github code [here](/deploy/web-call).

## Supported Protocol: Web

This protocol is defined by us, and is only relevant to you when you directly handle
user audio bytes (typically in web frontend).

User raw audio bytes with no encoding shall be sent in chunks in real-time (streaming
buffer sizes should be between 20 milliseconds and 250 milliseconds of audio,
smaller chunks have lower latency), while the server would send back raw audio bytes
in chunks for frontend to buffer for a smooth playback. The audio encoding and sample
rate is contained in the call detail. The message type is "binary".

When turn changes from agent back to user (for example, user interrupts the agent), server would
send a "clear" event to signal frontend to clear its buffer, so that agent stops speaking
ASAP when it should stop.

The connection would close on errors and a reason would get displayed in `close`
event when closed on error.

### Frontend -> Retell Message Event Spec

WebSocket sends mic input to Retell server for speech recognition

content: raw audio bytes

### Retell -> Frontend Message Event Spec

There are a couple of events that can be sent from Retell server to your frontend, depending
on your configuration. They are as follows:

#### Raw Audio Response Event

This event contains Agent audio response in raw audio bytes. The audio is sent whenever the generation is complete,
so the audio can be faster than real-time.

* This event is populated when query parameter `enable_audio_alignment` is false.
* content: raw audio bytes

#### Clear Event

Retell sends audio as they synthesize, and the frontend buffers it to play in real-time. However, there are cases where
the audio needs to be cleared, for example when the user interrupts the agent. This event signals to clear all audio in buffer.

* content: "clear"

#### Update Event

Contains Real-time information about the call like transcript, turntaking information, etc.

* This event is populated when query parameter `enable_update` is true.
* content: json that contains following property

<ResponseField name="event_type" type="update" required>
  The text content corresponding to the audio.
</ResponseField>

<ParamField query="transcript" type="object[]" required>
  Complete live transcript collected in the call so far. Presented in the form of a list of
  utterances.

  <Expandable title="properties">
    <ResponseField name="role" type="enum<string>" required>
      Indicates whether this utterance is from agent or user.
      Available options: `agent`, `user`
    </ResponseField>

    <ResponseField name="content" type="string" required>
      Text content of the utterance.
    </ResponseField>

    <ResponseField name="words" type="object[]" required>
      List of words in the utterance. Useful for animation or calculating speaking rate.

      <Expandable title="properties">
        <ResponseField name="word" type="string" required>
          The word itself.
        </ResponseField>

        <ResponseField name="start" type="number" required>
          Start time of the word in seconds relative to the beginning of call.
        </ResponseField>

        <ResponseField name="end" type="number" required>
          End time of the word in seconds relative to the beginning of call.
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ParamField>

<ParamField query="turntaking" type="enum<string>">
  Indicates change of speaker (turn taking). This field will be present when speaker changes to user (user turn), or right before
  agent is about to speak (agent turn). This field can be helpful in determining when to call functions in the call.
  Available options: `agent_turn`, `user_turn`
</ParamField>

#### Text Audio Alignment Event

Sometimes you might want to show text in frontend, and usually that's achievable by subscribing to the Update Event,
but sometimes you might want to align the text with the audio precisely (for example, for avatar use cases).
This event sends audio along with the text alignment information. Note that the frontend SDK does not yet have support for this event.

* This event is populated when query parameter `enable_audio_alignment` is true.
* The Raw Audio Event will not be sent when this event is sent.
* content: json that contains following property

<ResponseField name="event_type" type="audio_alignment" required>
  The text content corresponding to the audio.
</ResponseField>

<ResponseField name="text" type="string" required>
  The text content corresponding to the audio.
</ResponseField>

<ResponseField name="audio" type="string" required>
  Base64 encoded raw audio bytes.
</ResponseField>

#### Metadata Event

When your custom LLM passes a [metadata event](/api-references/llm-websocket#metadata-event) to Retell,
it will be forwarded to your frontend here.

<ParamField query="event_type" type="enum<string>" required>
  Differentiate what this event is.

  Available options: `metadata`
</ParamField>

<ParamField query="metadata" type="object" required>
  You can put anything here that can be json serialized.
</ParamField>

### Sample Events

Frontend -> Retell

<CodeGroup>
  ```json Raw Audio theme={null}
  // audio bytes
  [0x24, 0xFF, 0xAE, 0x11, ... , 0x1A, 0x3F]
  ```
</CodeGroup>

Retell -> Frontend

<CodeGroup>
  ```json Raw Audio theme={null}
  // audio bytes
  [0x24, 0xFF, 0xAE, 0x11, ... , 0x1A, 0x3F]
  ```

  ```json Clear Buffer theme={null}
  "clear"
  ```

  ```json Update theme={null}
  {
    "event_type": "update",
    "transcript": [
      {
        "role": "user",
        "content": "Hey.",
        "words": [
          {
            "word": "Hey.",
            "start": 3.175,
            "end": 3.615
          }
        ]
      },
      {
        "role": "agent",
        "content": "Hey how are you?",
        "words": [
          {
            "word": "Hey.",
            "start": 4.375,
            "end": 4.615
          },
          {
            "word": "How",
            "start": 4.615,
            "end": 4.855
          },
          {
            "word": "are",
            "start": 4.855,
            "end": 5.030156
          },
          {
            "word": "you?",
            "start": 5.030156,
            "end": 5.2053127
          }
        ]
      }
    ]
  }
  ```

  ```json Audio Alignment theme={null}
  {
    "event_type": "audio_alignment",
    "text": "Hey how are you?",
    // audio bytes encoded in base64
    "audio": "UklGRiQAAABXQVZFZm10IBAAAAABAAEARKwAABCxAgAEABAAZGF0YUksAAAA..."
  }
  ```

  ```json Metadata theme={null}
  {
    "event_type": "metadata",
    "metadata": {
      "avatar_emotion": "Angry",
      "user_id": "1234"
    }
  }
  ```
</CodeGroup>

## Supported Protocol: Twilio WebSocket

You don't need to worry about handling the socket and format, as we will connect directly to
Twilio server. When using Twilio for phone call, you don't need to touch any audio bytes, just to
respond the correct websocket url to Twilio is enough.

See this [websocket protocol](https://www.twilio.com/docs/voice/twiml/stream#message-media)
for details. Basically audio is encoded in base64 and stored in media event under `media.payload`.

## Android and iOS support

We don't have Android or iOS SDK for now, but you can follow web implementation to implement for Android and iOS

1. Implement an audio websocket to connect to your server. It will be responsible for sending and receiving
   audio bytes between client and Retell server.
   ([Code](https://github.com/RetellAI/retell-client-js-sdk/blob/main/src/audioWsClient.ts))

2. Call `register-call` on your server to get the call id

3. Pass microphone stream to the websocket. ([Code](https://github.com/RetellAI/retell-client-js-sdk/blob/main/src/index.ts))

4. Handle "clear" event from the server, it means the user has interrupted the agent and you need to clear the local audio data.

## FAQ

<AccordionGroup>
  <Accordion title="When do I have to write audio capture and playback code?">
    You typically don't need to write audio capture code when making a phone call. However, to place
    call from your web frontend, you would need to capture mic and follow the protocol `web` to
    send, receive, and play audio bytes.
  </Accordion>

  <Accordion title="Do I have to deal with audio bytes myself?">
    Only when you are capturing and playing the audio in your web frontend,
    you might need certain processing over the bytes. Otherwise, it's either just forwarding the bytes
    or programmable communication tools like Twilio handles that together with us.
  </Accordion>

  <Accordion title="Can I still mutate the audio bytes when making a phone call?">
    Yes, you can do this by directly connecting to programmable communication tools like Twilio, modify the
    bytes, send it to us using a protocol. Doing this would introduce some extra network latency.
  </Accordion>
</AccordionGroup>