Overview

LLM is slow, and when you need LLM to conduct tool calling, it will take 2 LLM calls which adds more latency.

LLM responses can be inconsistent, it can hallucinate and goes out of the character especially when prompts are long.

Tool calling (also known as function calling) is the ability for LLM to decide when to call an external tool for extracting knowledge, calling APIs, etc. This is crucial for enabling voice agents to perform tasks in the real world. However, when there’re many tools for LLM, it can easily choose the wrong tool or choose the wrong time to call a tool.

There’re traits about real-time conversation that makes it unique and requires a lot of tuning. For example, when would you execute a tool call, how do you handle interruptions (how about interruptions during a tool call), how do you handle long pauses, etc.

We have benchmarked many LLM models in terms of latency, function calling accuracy, etc. We provide multiple LLM options, potentially fine-tune the model for voice conversation, package in a developer friendly API, and handle things like rate limit, fallback. This way you can get a sweet spot in terms of accuracy, latency, and cost without hassle.

All LLM calls are done in streaming mode, which means the latency that matters the most now is not time to completion, but time to first sentence. Even tool calls are done in streaming mode.

We have abstracted away the hurdle to handle function calling: alignment of function call with transcript, timing of tool calling, speaking during and / or after tool calling, handling interruptions for tool calls etc. We have a set of APIs that allow you to choose from a list of predefined tools and or to define your own tools. Read more at Tool Calling.

The API let users define states, where you can break down your call into multiple states, so that each stage has shorter prompt and less tool choice. This can improve overall prompt following abilities and reduce hallucination. Read more at LLM States.

We understand that for each call, the prompt might change slightly based on customer, and it’s sort of that prompt is a template and you might want to pass some parameters during each call. We let you define these dynamic parameters in prompts and tool call descriptions, and we will replace them with actual values during the call. Read more at Dynamic Variables.

We provide a set of templates and examples that you can use to quickly get started with Retell LLM Framework.