Voice AI & Voice Agents: An Illustrated Primer

https://voiceaiandvoiceagents.com
Admin
A comprehensive guide to the state-of-the-art in conversational voice AI, covering core...
Tokens:3,592
Snippets:19
Trust Score:7
Update:1 year ago
Show doc for...
Tokens:
Raw
### Example of LLM Composite Function Chaining

Source: https://voiceaiandvoiceagents.com/index

This example demonstrates how an LLM can perform composite function calling by chaining multiple functions ('list_files()' and 'load_resource()') to fulfill a user request. The LLM independently determines the sequence of calls based on the user's instruction and available tools.

```APIDOC
User: Claude, load the most recent picture I have of the Eiffel Tower.
function call request: <list_files()>
function call response: <['eiffel_tower_1735838843.jpg', 'empire_state_building_1736374013.jpg', 'eiffel_tower_1737814100.jpg', 'eiffel_tower_1737609270.jpg',
'burj_khalifa_1737348929.jpg']
function call request: <load_resource('eiffel_tower_1737814100.jpg')>
function call response: <{ 'success': 'Image loaded successfully', 'image': … }>
LLM: I have loaded an image of the Eiffel Tower. The image shows the Eiffel
Tower on a cloudy day.
```

--------------------------------

### LLM Prompt for Pronunciation Steering

Source: https://voiceaiandvoiceagents.com/index

Example language for an LLM prompt designed to steer the pronunciation of specific words by instructing the LLM to output 'sounds-like' spellings. This technique is useful when the voice service lacks direct phonetic steering capabilities for proper nouns or unusual words.

```Prompt
              Replace "NVIDIA" with "in vidia" and replace

              "GPU" with "gee pee you" in your responses.

```

--------------------------------

### Gemini System Instruction for Audio Transcription

Source: https://voiceaiandvoiceagents.com/index

This system instruction (prompt) guides the Gemini model to function specifically as an audio transcriber. It instructs the model to provide an exact text transcription of the input audio, using conversation history only for context, and adhering to strict output rules including a special string for unclear audio.

```Prompt
You are an audio transcriber. You are receiving audio from a user. Your job is to transcribe the input audio to text exactly as it was said by the user.

You will receive the full conversation history before the audio input, to help with context. Use the full history only to help improve the accuracy of your transcription.

Rules:
- Respond with an exact transcription of the audio input.
- Do not include any text other than the transcription.
- Do not explain or add to your response.
- Transcribe the audio input simply and precisely.
- If the audio is not clear, emit the special string "".
- No response other than exact transcription, or "", is allowed.
```

--------------------------------

### Pipecat Default VAD Configuration Parameters

Source: https://voiceaiandvoiceagents.com/index

This snippet illustrates the default configuration parameters used by the Pipecat framework for Voice Activity Detection (VAD). These parameters define the thresholds for detecting the start and end of user speech, the confidence required for speech classification, and the minimum volume level considered as speech. Adjusting these values is crucial for optimizing turn detection behavior for specific voice AI applications.

```Python
# Pipecat's names and default values
# for the four configurable VAD
# parameters
VAD_STOP_SECS = 0.8
VAD_START_SECS = 0.2
VAD_CONFIDENCE = 0.7
VAD_MIN_VOLUME = 0.6
```

--------------------------------

### Maintaining Conversation Context with Stateless LLMs

Source: https://voiceaiandvoiceagents.com/index

This conceptual example illustrates how a stateless Large Language Model (LLM) requires the entire conversation history to be resent with each new turn to maintain context. It shows the cumulative nature of the input required for the LLM to generate coherent responses across multiple turns.

```Text
Turn 1:
  User: What's the capital of France?
  LLM: The capital of France is Paris.

Turn 2:
  User: What's the capital of France?
  LLM: The capital of France is Paris.
  User: Is the Eiffel Tower there?
  LLM: Yes, the Eiffel Tower is in Paris.

Turn 3:
  User: What's the capital of France?
  LLM: The capital of France is Paris.
  User: Is the Eiffel Tower there?
  LLM: Yes, the Eiffel Tower is in Paris.
  User: How tall is it?
  LLM: The Eiffel Tower is about 330 meters tall.
```

--------------------------------

### Cartesia API Word-Level Timestamps Example

Source: https://voiceaiandvoiceagents.com/index

A JSON response snippet from the Cartesia API illustrating word-level timestamp metadata. This data is crucial for maintaining accurate conversation context in voice AI applications, enabling the reconstruction of heard text back to the original input.

```JSON
{
  "type": "timestamps",
  "context_id": "test-01",
  "status_code": 206,
  "done": false,
  "word_timestamps": {
    "words": ["What's", "the", "capital", "of", "France?"],
    "start": [0.02, 0.3, 0.48, 0.6, 0.8],
    "end": [0.3, 0.36, 0.6, 0.8, 1]
  }
}
```

--------------------------------

### LLM Prompt for Handling Speech-to-Text Transcription Errors

Source: https://voiceaiandvoiceagents.com/index

This prompt guides a large language model (LLM) to act as a helpful, concise, and reliable voice assistant. It instructs the LLM to silently correct transcription errors, prioritize intended meaning, provide short and direct answers, and respond in plain, unformatted text suitable for text-to-speech conversion. It also includes instructions for handling time-dependent questions and misunderstandings.

```Prompt
You are a helpful, concise, and reliable voice assistant. Your primary goal is to understand the user's spoken requests, even if the speech-to-text transcription contains errors. Your responses will be converted to speech using a text-to-speech system. Therefore, your output must be plain, unformatted text.

When you receive a transcribed user request:
1. Silently correct for likely transcription errors. Focus on the intended meaning, not the literal text. If a word sounds like another word in the given context, infer and correct. For example, if the transcription says "buy milk two tomorrow" interpret this as "buy milk tomorrow".
2. Provide short, direct answers unless the user explicitly asks for a more detailed response. For example, if the user says "what time is it?" you should respond with "It is 2:38 AM". If the user asks "Tell me a joke", you should provide a short joke.
3. Always prioritize clarity and accuracy. Respond in plain text, without any formatting, bullet points, or extra conversational filler.
4. If you are asked a question that is time dependent, use the current date, which is February 3, 2025, to provide the most up to date information.
5. If you do not understand the user request, respond with "I'm sorry, I didn't understand that."

Your output will be directly converted to speech, so your response should be natural-sounding and appropriate for a spoken conversation.
```

--------------------------------

### Configuring a Pipecat Pipeline with Krisp for Speaker Isolation

Source: https://voiceaiandvoiceagents.com/index

This Python code snippet demonstrates the structure of a `Pipecat` pipeline, integrating various stages for voice AI processing. It specifically highlights the inclusion of `krisp_filter` for primary speaker isolation, alongside other components like transport, VAD, speech-to-text (STT), large language model (LLM), and text-to-speech (TTS). This setup is designed to improve transcription accuracy by suppressing background speech.

```Python
pipeline = Pipeline(
    [
      transport.input(),
      krisp_filter,
      vad_turn_detector,
      stt,
      context_aggregator.user(),
      llm,
      tts,
      transport.output(),
      context_aggregator.assistant(),
    ]
  )
```

--------------------------------

### Simple RAG: Merging User Info into LLM System Instructions

Source: https://voiceaiandvoiceagents.com/index

This snippet demonstrates a basic Retrieval Augmented Generation (RAG) approach for a voice AI agent. It fetches user information at the beginning of a session and dynamically incorporates it into the LLM's system prompt, enhancing the agent's context with personalized data.

```Python
user_info = fetch_user_info(user_id)

system_prompt_base = "You are a voice AI assistant..."

system_prompt = (
  system_prompt_base
  + f"""
The name of the patient is {user_info["name"]}.
The patient is {user_info["age"]} years old.
The patient has the following medical history: {user_info["summarized_history"]}.
"""
)
```

--------------------------------

### Pipecat Pipeline for Context-aware Turn Detection

Source: https://voiceaiandvoiceagents.com/index

This Pipecat pipeline demonstrates context-aware turn detection and parallel greedy conversation inference using Gemini 2.0 Flash native audio input. The output is gated until the turn detection inference identifies a phrase endpoint, allowing for natural conversation flow and improved responsiveness.

```Python
    [
      transport.input(),
      vad,
      audio_accumulater,
      ParallelPipeline(
      [
        FunctionFilter(filter=block_user_stopped_speaking),
      ],
      [
        ParallelPipeline(
        [
          classifier_llm,
          completeness_check,
        ],
        [
          tx_llm,
          user_aggregator_buffer,
        ],
        )
      ],
      [
        conversation_audio_context_assembler,
        conversation_llm,
        bot_output_gate,
      ],
      ),
      tts,
      transport.output(),
      context_aggregator.assistant(),
    ],
```

--------------------------------

### Pipecat Pipeline for Parallel Gemini Inference

Source: https://voiceaiandvoiceagents.com/index

This Python code defines a Pipecat pipeline that orchestrates two parallel inference processes: one for generating conversation responses and another for transcribing user speech. It integrates audio collection, context aggregation, and text-to-speech, demonstrating how to manage distinct LLM modes within a single flow.

```Python
  pipeline = Pipeline(
    [
        transport.input(),
        audio_collector,
        context_aggregator.user(),
        ParallelPipeline(
            [ # transcribe
                input_transcription_context_filter,
                input_transcription_llm,
                transcription_frames_emitter,
            ],
            [ # conversation inference
                conversation_llm,
            ],
        ),
        tts,
        transport.output(),
        context_aggregator.assistant(),
        context_text_audio_fixup,
    ]
  )
```

--------------------------------

### LLM Function Call: Initial In-Progress Context

Source: https://voiceaiandvoiceagents.com/index

Demonstrates the initial state of an LLM conversation where a function call is initiated. It shows the user's query, the LLM's immediate response, and the function call request message paired with an 'IN_PROGRESS' placeholder response in the context. This pattern ensures the LLM understands the function call is ongoing.

```LLM Interaction
User:  Please look up the price of 1000 widgets.
LLM: Please wait while I look up the price for 1000 widgets.
function call request: { name: "price_lookup", args: { item: "widget", quantity: 1000 } }
function call response: { status: IN_PROGRESS }
```

--------------------------------

### LLM Function Call: Multiple Concurrent In-Progress Calls

Source: https://voiceaiandvoiceagents.com/index

Shows how multiple function calls can be initiated and remain in an 'IN_PROGRESS' state concurrently within the LLM context. By using placeholder responses, the conversation can continue without confusing the LLM, even while multiple background operations are pending completion.

```LLM Interaction
User:  Please look up the price of 1000 widgets.
LLM: Please wait while I look up the price for 1000 widgets.
function call request: { name: "price_lookup", args: { item: "widget", quantity: 1000 } }
function call response: { status: IN_PROGRESS }

User: Please lookup the price of 1000 pre-assembled modules.
LLM: Please wait while I also look up the price for 1000 pre-assembled modules.
function call request: { name: "price_lookup", args: { item: "pre_assembled_module", quantity: 1000 } }
function call response: { status: IN_PROGRESS }
```

--------------------------------

### RAG During Session: LLM Function Call for External System Lookup

Source: https://voiceaiandvoiceagents.com/index

This asynchronous function illustrates RAG during an active session, where the LLM calls an external system to retrieve information (e.g., order status). It includes a pre-emptive speech output to manage user expectations regarding latency, and then returns the lookup results to the LLM.

```Python
async def query_order_system(function_name, tool_call_id, args, llm, context, result_callback):
  "First push a speech frame. This is handy when the LLM response might take a while."
  await llm.push_frame(TTSSpeakFrame("Please hold on while I look that order up for you."))

  query_result = order_system.get(args["query"])
  await result_callback({
    "info": json.dumps({
     "lookup_success": True,
     "order_status": query_result["order_status"],
     "delivery_date": query_result["delivery_date"],
    })
 })

llm.register_function("query_order_system", query_order_system)
```

--------------------------------

### LLM Function Call: Completed Context

Source: https://voiceaiandvoiceagents.com/index

Illustrates the context after a function call has successfully completed. This snippet shows the user's original query, the function call request, and the final function call response containing the actual result. This completed pair updates the LLM's context with the outcome of the function call.

```LLM Interaction
User:  Please look up the price of 1000 widgets.
function call request: { name: "price_lookup", args: { item: "widget", quantity: 1000 } }
function call response: { result: { price: 12.35 } }
```

--------------------------------

### Corrected Asynchronous Function Signature for LLM

Source: https://voiceaiandvoiceagents.com/index

This snippet provides a workaround for asynchronous LLM function calls. Instead of returning an iterator, the function initiates a background task and immediately returns a status, with a callback mechanism to update context later. This adheres to the requirement that request/response messages appear together.

```Pseudo-code
create_interest_task_and_return_success_immediately
    (interest: string, context_queue_callback: Callable[Message]) ->
      Literal["in_progress", "canceled", "success", "failure"]
```

--------------------------------

### LLM Adjusting Behavior Based on User Feedback

Source: https://voiceaiandvoiceagents.com/index

This snippet illustrates how an LLM (Agent) can learn from user corrections within a multi-turn conversation (in-context learning). Initially misinterpreting "MNI" as "Miami Dolphins", the agent corrects its understanding after user clarification and biases future interpretations towards the stock ticker.

```Text
User: How has MNI performed recently?
Agent: The Miami Dolphins won their game yesterday 21
to 3 and now lead the AFC East with two games remain-
 ing in the regular season.
User: No, I meant the stock MNI.
Agent: Ah, my apologies! You're asking about the
stock performance of MNI, which is the ticker symbol
 for McClatchy Company …
 From this point on, the model will bias towards
interpreting phonemes or transcribed text as "MNI"
 rather than "Miami".
```

--------------------------------

### Incorrect Asynchronous Function Signature for LLM

Source: https://voiceaiandvoiceagents.com/index

This snippet illustrates an incorrect function signature for handling asynchronous operations with LLMs. Direct asynchronous return types like 'Iterator[Message]' are not currently supported, as LLM function call request/response messages must appear together in the context.

```Pseudo-code
register_interest_generator(interest: string) -> Iterator[Message]
```

--------------------------------

### LLM Function Call: Handling Interruptible Cancellation

Source: https://voiceaiandvoiceagents.com/index

Demonstrates the context when an interruptible function call is canceled due to new user input while it was in progress. The function call response shows a 'CANCELLED' status, indicating the operation was aborted, while another function call might still be ongoing.

```LLM Interaction
User:  "Please look up the price of 1000 widgets."
LLM: "Please wait while I look up the price for 1000 widgets."
function call request: { name: "price_lookup", args: { item: "widget", quantity: 1000 } }
function call response: { status: CANCELLED }

User: Please lookup the price of 1000 pre-assembled modules.
LLM: Please wait while I look up the price for 1000 pre-assembled modules.
function call request: { name: "price_lookup", args: { item: "pre_assembled_module", quantity: 1000 } }
function call response: { status: IN_PROGRESS }
```
Tokens:
Raw