This guide is outdated (relying on deprecated audio websocket). Will update soon.

Check out the code at Node Demo / Python Demo for a practical example. All code in this guide is from this repository.

In this guide, we will show how to integrate Retell AI with phone call using Twilio in your server. This way, you can use your own numbers and have full control over telecommunications functionalities.

You can use your own Twilio with both Retell LLM or Custom LLM.

​
How Data Flow During Phone Call

1

Get Phone Number / Bind Phone Number With Agent

This will set up the voice webhook of the number, and the number is good to accept inbound calls.

2

Setup Voice Webhook

This webhook will be called whenever a phone rings for inbound and outbound calls. You will also register call with Retell at this stage.

3

(Optional) Make Outbound Call

Supply the callee number to make an outbound call with your configured number in step 1.

4

Audio Websocket

Retell’s server with handle audio bytes with Twilio.

5

LLM Websocket

Retell’s server communicate with your server to get responses for agents.

​
Setup Twilio Account And Your Server

If you haven’t, sign up account with Twilio to get your account id and auto token. Please note that Twilio free account can only create one number, so if you need more than one number in your development, you’d have to upgrade to a Twilio paid account.

Same as the previous guide for setting up your LLM WebSocket server, you need your server exposed to public network so Twilio can call.

this.twilio = twilio(
  process.env.TWILIO_ACCOUNT_ID,
  process.env.TWILIO_AUTH_TOKEN,
);
this.retellClient = new Retell({
  apiKey: process.env.RETELL_API_KEY,
});

​
Buy A Phone Number From Twilio

Get a new number under the area code, associate an agent with it.

Now every time inbound / outbound call happens for this number, it would call ${your-ip-address}/twilio-voice-webhook/agent-id which you will set up later.

You can grab the agent id from you Retell LLM agent or Custom LLM agent

The Retell agent id in the path parameter would let your voice webhook be able to know which agent to use.

CreatePhoneNumber = async (areaCode: number, agentId: string) => {
  try {
    const localNumber = await this.twilio
      .availablePhoneNumbers("US")
      .local.list({ areaCode: areaCode, limit: 1 });
    if (!localNumber || localNumber[0] == null)
      throw "No phone numbers of this area code.";

    const phoneNumberObject = await this.twilio.incomingPhoneNumbers.create({
      phoneNumber: localNumber[0].phoneNumber,
      voiceUrl: `${process.env.NGROK_IP_ADDRESS}/twilio-voice-webhook/${agentId}`,
    });
    console.log("Getting phone number:", phoneNumberObject);
    return phoneNumberObject;
  } catch (err) {
    console.error("Create phone number API: ", err);
  }
};

​
Bind Your Existing Twilio Numbers With Retell Agent

Change the voice url of your existing Twilio number. Can also use to update the number you created from last step.

Now every time inbound / outbound call happens for this number, it would call ${your-ip-address}/twilio-voice-webhook/agent-id which you will set up later.

The Retell agent id in the path parameter would let your voice webhook be able to know which agent to use.

RegisterPhoneAgent = async (number: string, agentId: string) => {
  try {
    const phoneNumberObjects = await this.twilio.incomingPhoneNumbers.list();
    let numberSid;
    for (const phoneNumberObject of phoneNumberObjects) {
      if (phoneNumberObject.phoneNumber === number) {
        numberSid = phoneNumberObject.sid;
      }
    }
    if (numberSid == null) {
      return console.error(
        "Unable to locate this number in your Twilio account, is the number you used in BCP 47 format?"
      );
    }

    await this.twilio.incomingPhoneNumbers(numberSid).update({
      voiceUrl: `${process.env.NGROK_IP_ADDRESS}/twilio-voice-webhook/${agentId}`,
    });
  } catch (error: any) {
    console.error("failer to retrieve caller information: ", error);
  }
};

​
Setup Twilio Voice Webhook (Inbound Setup Complete)

Twilio will call this endpoint when there’s an incoming/out-going call for your Twilio number to get an address to stream the call to. Here you will pass Retell’s Audio Websocket server address to it.

Retell Audio Websocket Endpoint: wss://api.retellai.com/audio-websocket/{call_id}

The following code does 2 things:

  1. Register Call: sends the detail of the call (agent, encoding, etc) to Retell server and gets a call_id from our server. This call_id is the path parameter of the audio websocket endpoint. By registering the call, and pass that id to Twilio, which in turn use the id to connect to Retell server, we would know how to handle this call correctly.
  2. Start Twilio Stream and asks it to connect to Retell: here you would pass the audio websocket URL to Twilio, and Twilio would connect to Retell server by calling that, and we will take care of all the audio bytes exchanges from there.
ListenTwilioVoiceWebhook = (app: expressWs.Application) => {
  app.post("/twilio-voice-webhook/:agent_id",
    async (req: Request, res: Response) => {
      const agentId = req.params.agent_id;
      const answeredBy = req.body.AnsweredBy;
      try {
        // Respond with TwiML to hang up the call if its machine
        if (answeredBy && answeredBy === "machine_start") {
          this.EndCall(req.body.CallSid);
          return;
        }

        const callResponse = await this.retellClient.registerCall({
          agentId: agentId,
          audioWebsocketProtocol: AudioWebsocketProtocol.Twilio,
          audioEncoding: AudioEncoding.Mulaw,
          sampleRate: 8000,
        });
        if (callResponse.callDetail) {
          // Start phone call websocket
          const response = new VoiceResponse();
          const start = response.connect();
          const stream = start.stream({
            url: `wss://api.retellai.com/audio-websocket/${callResponse.callDetail.callId}`,
          });
          res.set("Content-Type", "text/xml");
          res.send(response.toString());
        }
      } catch (err) {
        console.error("Error in twilio voice webhook:", err);
        res.status(500).send();
      }
    }
  );
};

Now your webhook is listening at ${your-ip-address}/twilio-voice-webhook and expects a path parameter for agent Id.

​
Make An Outbound Call with Retell Agent

Specify the fromNumber as a twilio number you just set up. toNumber as callee number.

CreatePhoneCall = async (fromNumber: string, toNumber: string, agentId: string) => {
  try {
    await this.twilio.calls.create({
      machineDetection: "Enable", // detects if the other party is IVR
      machineDetectionTimeout: 8,
      asyncAmd: "true", // call webhook when determined whether it is machine
      asyncAmdStatusCallback: `${process.env.NGROK_IP_ADDRESS}/twilio-voice-webhook/${agentId}`, // Webhook url for machine detection
      url: `${process.env.NGROK_IP_ADDRESS}/twilio-voice-webhook/${agentId}`, // Webhook url for registering call
      to: toNumber,
      from: fromNumber,
    });
    console.log(`Call from: ${fromNumber} to: ${toNumber}`);
  } catch (error: any) {
    console.error("failer to retrieve caller information: ", error);
  }
};

Was this page helpful?

Retell NumbersWeb Call