Speech service

https://learn.microsoft.com/en-us/azure/ai-services/speech-service
Admin
The Speech service allows users to recognize speech, synthesize speech, get real-time translations,...
Tokens:1,314,460
Snippets:7,293
Trust Score:10
Update:19 hours ago
Show doc for...
Tokens:
Raw
### Print usage instructions for VoiceLive setup

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-quickstart

Prints usage instructions for setting up environment variables and command line options for the VoiceLive quickstart. Includes examples for using application.properties, command line arguments, Azure CLI authentication, and custom model/voice.

```java
    /**
        * Prints usage instructions for setting up environment variables.
        */
    private static void printUsage() {
        System.err.println("\n═══════════════════════════════════════════════════════════════");
        System.err.println("Usage: mvn exec:java [options]");
        System.err.println("═══════════════════════════════════════════════════════════════");
        System.err.println("\nConfiguration (in priority order):");
        System.err.println("  1. Command line arguments (--endpoint, --api-key, etc.)");
        System.err.println("  2. Environment variables (AZURE_VOICELIVE_ENDPOINT, etc.)");
        System.err.println("  3. application.properties file");
        System.err.println("\nCommand Line Options:");
        System.err.println("  --endpoint <url>         VoiceLive endpoint URL");
        System.err.println("  --api-key <key>          API key for authentication");
        System.err.println("  --model <model>          Model to use (default: gpt-realtime)");
        System.err.println("  --voice <voice>          Voice name (e.g., en-US-Ava:DragonHDLatestNeural)");
        System.err.println("  --instructions <text>    Custom system instructions");
        System.err.println("  --use-token-credential   Use Azure CLI authentication");
        System.err.println("\nExamples:");
        System.err.println("  # Using application.properties:");
        System.err.println("  mvn exec:java");
        System.err.println("\n  # Using command line arguments:");
        System.err.println("  mvn exec:java -Dexec.args=\"--endpoint https://... --api-key <key>\"");
        System.err.println("\n  # Using Azure CLI authentication:");
        System.err.println("  mvn exec:java -Dexec.args=\"--use-token-credential\"");
        System.err.println("\n  # With custom model and voice:");
        System.err.println("  mvn exec:java -Dexec.args=\"--model gpt-4.1 --voice en-US-JennyNeural\"");
        System.err.println("═══════════════════════════════════════════════════════════════\n");
    }
```

--------------------------------

### Build and run the Go quickstart

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-recognize-speech

Builds the Go program and runs it to start speech recognition.

```bash
go build
go run quickstart

```

--------------------------------

### StartAsync method

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-voice-live-mcp-server

Starts the voice assistant by connecting to the VoiceLive API, starting audio playback and capture, and logging readiness. Prints example phrases to the console.

```csharp
public async Task StartAsync(CancellationToken cancellationToken = default)
{
    try
    {
        _logger.LogInformation("Connecting to VoiceLive API with model {Model}", _model);

        _session = await _client.StartSessionAsync(_model, cancellationToken).ConfigureAwait(false);
        _audioProcessor = new AudioProcessor(_session, _loggerFactory.CreateLogger<AudioProcessor>());

        await SetupSessionAsync(cancellationToken).ConfigureAwait(false);

        await _audioProcessor.StartPlaybackAsync().ConfigureAwait(false);
        await _audioProcessor.StartCaptureAsync().ConfigureAwait(false);

        _logger.LogInformation("Voice assistant with MCP ready!");
        Console.WriteLine();
        Console.WriteLine(new string('=', 70));
        Console.WriteLine("🎤 VOICE ASSISTANT WITH MCP READY");
        Console.WriteLine("Try saying:");
        Console.WriteLine("  • 'What is the GitHub repo fastapi about?'");
        Console.WriteLine("  • 'Search the Azure documentation for Voice Live API.'");
        Console.WriteLine("You may need to approve some MCP tool calls in the console.");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"❌ Error: {ex.Message}");
        return 1;
    }

    return 0;
}
```

--------------------------------

### Get help on advanced setup for datastore configuration

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/spx-data-store-configuration

Run this command to get more details about datastore files, including default configuration files like @spx.default, @default.config, and @*.default.config.

```bash
spx help advanced setup
```

--------------------------------

### Return to the quickstart folder with cd ..

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-agents-quickstart

Navigates back to the parent directory.

```bash
cd ..
```

--------------------------------

### Example output from successful helm install

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-container-howto-on-premises

Example output from a successful helm install execution, showing deployed resources such as pods, services, pod disruption budgets, deployments, and horizontal pod autoscalers.

```text
NAME:   onprem-speech
LAST DEPLOYED: Tue Jul  2 12:51:42 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
speech-to-text-7664f5f465-87w2d  0/1    Pending            0         0s
speech-to-text-7664f5f465-klbr8  0/1    ContainerCreating  0         0s
neural-text-to-speech-56f8fb685b-4jtzh  0/1    ContainerCreating  0         0s
neural-text-to-speech-56f8fb685b-frwxf  0/1    Pending            0         0s

==> v1/Service
NAME            TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)       AGE
speech-to-text  LoadBalancer  10.0.252.106  <pending>    80:31811/TCP  1s
neural-text-to-speech  LoadBalancer  10.0.125.187  <pending>    80:31247/TCP  0s

==> v1beta1/PodDisruptionBudget
NAME                                MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
speech-to-text-poddisruptionbudget  N/A            20%              0                    1s
neural-text-to-speech-poddisruptionbudget  N/A            20%              0                    1s

==> v1beta2/Deployment
NAME            READY  UP-TO-DATE  AVAILABLE  AGE
speech-to-text  0/2    2           0          0s
neural-text-to-speech  0/2    2           0          0s

==> v2beta2/HorizontalPodAutoscaler
NAME                       REFERENCE                  TARGETS        MINPODS  MAXPODS  REPLICAS  AGE
speech-to-text-autoscaler  Deployment/speech-to-text  <unknown>/50%  2        10       0         0s
neural-text-to-speech-autoscaler  Deployment/neural-text-to-speech  <unknown>/50%  2        10       0         0s


NOTES:
cognitive-services-speech-onpremise has been installed!
Release is named onprem-speech

```

--------------------------------

### Check Python installation on macOS

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/quickstarts/setup-platform

Open a terminal and run this command to verify Python is installed. If installed properly, you get a response like 'Python 3.8.10'.

```bash
python --version
```

--------------------------------

### Create voice-live-agents-quickstart.py

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-agents-quickstart-classic

This is the main script for the quickstart. It sets up logging, loads environment variables, and defines an AudioProcessor class for real-time audio capture and playback. It uses Microsoft Entra ID authentication via AzureCliCredential or DefaultAzureCredential.

```python
# -------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# -------------------------------------------------------------------------
from __future__ import annotations
import os
import sys
import argparse
import asyncio
import base64
from datetime import datetime
import logging
import queue
import signal
from typing import Union, Optional, TYPE_CHECKING, cast

from azure.core.credentials import AzureKeyCredential
from azure.core.credentials_async import AsyncTokenCredential
from azure.identity.aio import AzureCliCredential, DefaultAzureCredential

from azure.ai.voicelive.aio import connect
from azure.ai.voicelive.models import (
    AudioEchoCancellation,
    AudioNoiseReduction,
    AzureStandardVoice,
    InputAudioFormat,
    Modality,
    OutputAudioFormat,
    RequestSession,
    ServerEventType,
    ServerVad
)
from dotenv import load_dotenv
import pyaudio

if TYPE_CHECKING:
    # Only needed for type checking; avoids runtime import issues
    from azure.ai.voicelive.aio import VoiceLiveConnection

## Change to the directory where this script is located
os.chdir(os.path.dirname(os.path.abspath(__file__)))

# Environment variable loading
load_dotenv('./.env', override=True)

# Set up logging
## Add folder for logging
if not os.path.exists('logs'):
    os.makedirs('logs')

## Add timestamp for logfiles
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")

## Create conversation log filename
logfilename = f"{timestamp}_conversation.log"

## Set up logging
logging.basicConfig(
    filename=f'logs/{timestamp}_voicelive.log',
    filemode="w",
    format='%(asctime)s:%(name)s:%(levelname)s:%(message)s',
    level=logging.INFO
)
logger = logging.getLogger(__name__)

class AudioProcessor:
    """
    Handles real-time audio capture and playback for the voice assistant.

    Threading Architecture:
    - Main thread: Event loop and UI
    - Capture thread: PyAudio input stream reading
    - Send thread: Async audio data transmission to VoiceLive
    - Playback thread: PyAudio output stream writing
    """

    loop: asyncio.AbstractEventLoop

    class AudioPlaybackPacket:
        """Represents a packet that can be sent to the audio playback queue."""
        def __init__(self, seq_num: int, data: Optional[bytes]):
            self.seq_num = seq_num
            self.data = data

    def __init__(self, connection):
        self.connection = connection
        self.audio = pyaudio.PyAudio()

        # Audio configuration - PCM16, 24kHz, mono as specified
        self.format = pyaudio.paInt16
        self.channels = 1
        self.rate = 24000
        self.chunk_size = 1200 # 50ms

        # Capture and playback state
        self.input_stream = None

        self.playback_queue: queue.Queue[AudioProcessor.AudioPlaybackPacket] = queue.Queue()
        self.playback_base = 0
        self.next_seq_num = 0
        self.output_stream: Optional[pyaudio.Stream] = None

        logger.info("AudioProcessor initialized with 24kHz PCM16 mono audio")

    def start_capture(self):
        """Start capturing audio from microphone."""
        def _capture_callback(
            in_data,      # data
            _frame_count,  # number of frames
            _time_info,    # dictionary
            _status_flags):
            """Audio capture thread - runs in background."""
            audio_base64 = base64.b64encode(in_data).decode("utf-8")
            asyncio.run_coroutine_threadsafe(
                self.connection.input_audio_buffer.append(audio=audio_base64), self.loop
            )
            return (None, pyaudio.paContinue)

        if self.input_stream:
            return

        # Store the current event loop for use in threads
        self.loop = asyncio.get_event_loop()

        try:
            self.input_stream = self.audio.open(
                format=self.format,
                channels=self.channels,
                rate=self.rate,
                input=True,
                frames_per_buffer=self.chunk_size,
                stream_callback=_capture_callback,
            )
```

--------------------------------

### Install required Python packages

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-transcribe-2-streaming-realtime

Install the required packages for the Python transcription example: websockets, sounddevice, and azure-identity.

```bash
pip install websockets sounddevice azure-identity
```

--------------------------------

### Create and navigate to the translation-quickstart folder

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-speech-translation

Creates a new folder named 'translation-quickstart' and changes into it. This is the first step in setting up the environment for the quickstart.

```bash
mkdir translation-quickstart && cd translation-quickstart
```

--------------------------------

### Install Speech SDK for JavaScript

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/captioning-quickstart

Run this command to install the Speech SDK for JavaScript. This is required before running the captioning example.

```bash
npm install microsoft-cognitiveservices-speech-sdk
```

--------------------------------

### Create project folder

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live-agents-quickstart-classic

Creates a new folder named 'voice-live-quickstart' and changes into it.

```bash
mkdir voice-live-quickstart && cd voice-live-quickstart
```

--------------------------------

### Start continuous recognition

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-recognize-speech

Starts continuous recognition with startContinuousRecognitionAsync. A commented example shows how to stop recognition later with stopContinuousRecognitionAsync.

```javascript
speechRecognizer.startContinuousRecognitionAsync();

// Make the following call at some point to stop recognition:
// speechRecognizer.stopContinuousRecognitionAsync();
```

--------------------------------

### Speech translation output example

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-speech-translation

Example output from the speech translation quickstart when speaking into the microphone. Shows the recognized text and its translation into Italian.

```text
Speak into your microphone.
RECOGNIZED: Text=I'm excited to try speech translation.
Translated into 'it': Sono entusiasta di provare la traduzione vocale.

```

--------------------------------

### Create project folder

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-text-to-speech

Creates a new folder named synthesis-quickstart and navigates into it.

```bash
mkdir synthesis-quickstart && cd synthesis-quickstart
```

--------------------------------

### Get contextual help for Speech CLI output examples

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/spx-output-options

Run these commands to see examples of output options for recognize, synthesize, translate, and intent operations.

```bash
spx help recognize output examples
spx help synthesize output examples
spx help translate output examples
spx help intent output examples
```

--------------------------------

### mcp-quickstart.js - VoiceLive MCP quickstart sample

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-voice-live-mcp-server

Full code for the mcp-quickstart.js file. It imports VoiceLiveClient, AzureKeyCredential, and DefaultAzureCredential, sets up conversation logging, and defines printUsage and parseArguments functions. The sample supports options such as --api-key, --endpoint, --model, --voice, --instructions, --audio-input-device, --list-audio-devices, --use-token-credential, --no-audio, and -h/--help.

```javascript
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import "dotenv/config";
import { VoiceLiveClient } from "@azure/ai-voicelive";
import { AzureKeyCredential } from "@azure/core-auth";
import { DefaultAzureCredential } from "@azure/identity";
import { spawn } from "node:child_process";
import { existsSync, mkdirSync, appendFileSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";

const __dirname = dirname(fileURLToPath(import.meta.url));

const logsDir = join(__dirname, "logs");
if (!existsSync(logsDir)) mkdirSync(logsDir, { recursive: true });

const timestamp = new Date()
  .toISOString()
  .replace(/[:.]/g, "-")
  .replace("T", "_")
  .slice(0, 19);
const conversationLogFile = join(logsDir, `conversation_${timestamp}.log`);

function writeConversationLog(message) {
  appendFileSync(conversationLogFile, message + "\n", "utf-8");
}

function printUsage() {
  console.log("Usage: node mcp-quickstart.js [options]");
  console.log("");
  console.log("Options:");
  console.log("  --api-key <key>             VoiceLive API key");
  console.log("  --endpoint <url>            VoiceLive endpoint URL");
  console.log("  --model <name>              Model to use (default: gpt-realtime)");
  console.log(
    "  --voice <name>              Voice (default: en-US-Ava:DragonHDLatestNeural)",
  );
  console.log("  --instructions <text>       System instructions for the assistant");
  console.log("  --audio-input-device <name> Explicit SoX input device name (Windows)");
  console.log("  --list-audio-devices        List available audio input devices and exit");
  console.log("  --use-token-credential      Use Azure credential instead of API key");
  console.log("  --no-audio                  Connect and configure session without mic/speaker");
  console.log("  -h, --help                  Show this help text");
}

function parseArguments(argv) {
  const parsed = {
    apiKey: process.env.AZURE_VOICELIVE_API_KEY,
    endpoint: process.env.AZURE_VOICELIVE_ENDPOINT,
    model: process.env.AZURE_VOICELIVE_MODEL ?? "gpt-realtime",
    voice:
      process.env.AZURE_VOICELIVE_VOICE ?? "en-US-Ava:DragonHDLatestNeural",
    instructions:
      process.env.AZURE_VOICELIVE_INSTRUCTIONS ??
      "You are a helpful AI assistant with access to MCP tools. Always respond in English. When a user asks a question, use the appropriate tool once to find information, then summarize the results conversationally. IMPORTANT: Never call the same tool more than once per user question. After receiving a tool result, always respond to the user with what you found — do not search again. Some tools require user approval before they can be used. When you receive a system message asking you to request permission, you MUST clearly ask the user for their explicit approval before proceeding. Always wait for the user to say yes or no. Never skip the approval question or assume permission is granted. If a tool result arrives after the conversation has moved to a different topic, briefly introduce it as a late result before sharing the findings.",
  };
}
```

--------------------------------

### MCPQuickstart.cs - VoiceLive MCP server integration

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-voice-live-mcp-server

Complete C# program for the MCP quickstart. It reads configuration from appsettings.json or environment variables, supports API key or token credential authentication, checks the audio system, and starts an MCPVoiceAssistant. Requires the Azure.AI.VoiceLive, Azure.Identity, Microsoft.Extensions.Configuration, Microsoft.Extensions.Logging, and NAudio.Wave packages.

```csharp
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Channels;
using System.Threading.Tasks;
using Azure.AI.VoiceLive;
using Azure.Identity;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using NAudio.Wave;

namespace Azure.AI.VoiceLive.Samples
{
    /// <summary>
    /// MCP Quickstart - demonstrates MCP server integration with VoiceLive SDK.
    /// Shows how to define MCP servers, handle MCP tool calls, and implement
    /// an approval flow for tool calls that require user consent.
    /// </summary>
    public class Program
    {
        public static async Task<int> Main(string[] args)
        {
            // Setup configuration
            var configuration = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", optional: true)
                .AddEnvironmentVariables()
                .Build();

            var apiKey = configuration["VoiceLive:ApiKey"] ?? Environment.GetEnvironmentVariable("AZURE_VOICELIVE_API_KEY");
            var endpoint = configuration["VoiceLive:Endpoint"] ?? Environment.GetEnvironmentVariable("AZURE_VOICELIVE_ENDPOINT") ?? "https://your-resource-name.services.ai.azure.com/";
            var model = configuration["VoiceLive:Model"] ?? Environment.GetEnvironmentVariable("AZURE_VOICELIVE_MODEL") ?? "gpt-realtime";
            var voice = configuration["VoiceLive:Voice"] ?? Environment.GetEnvironmentVariable("AZURE_VOICELIVE_VOICE") ?? "en-US-Ava:DragonHDLatestNeural";
            var instructions = configuration["VoiceLive:Instructions"] ?? "You are a helpful AI assistant with access to MCP tools. Use the tools to help answer user questions. Respond naturally and conversationally. Some tools require user approval before they can be used. When you receive a system message asking you to request permission, you MUST clearly ask the user for their explicit approval before proceeding. Always wait for the user to say yes or no. Never skip the approval question or assume permission is granted. If a tool result arrives after the conversation has moved to a different topic, briefly introduce it as a late result before sharing the findings.";
            var useTokenCredential = args.Length > 0 && args[0] == "--use-token-credential";

            // Setup logging
            using var loggerFactory = LoggerFactory.Create(builder =>
            {
                builder.AddConsole();
                builder.SetMinimumLevel(LogLevel.Information);
            });

            var logger = loggerFactory.CreateLogger<Program>();

            // Validate credentials
            if (string.IsNullOrEmpty(apiKey) && !useTokenCredential)
            {
                Console.WriteLine("❌ Error: No authentication provided");
                Console.WriteLine("Set AZURE_VOICELIVE_API_KEY or use --use-token-credential.");
                return 1;
            }

            // Check audio system
            if (!CheckAudioSystem(logger))
                return 1;

            try
            {
                VoiceLiveClient client;
                if (useTokenCredential)
                {
                    client = new VoiceLiveClient(new Uri(endpoint), new DefaultAzureCredential(), new VoiceLiveClientOptions());
                    logger.LogInformation("Using Azure token credential");
                }
                else
                {
                    client = new VoiceLiveClient(new Uri(endpoint), new AzureKeyCredential(apiKey!), new VoiceLiveClientOptions());
                    logger.LogInformation("Using API key credential");
                }

                using var assistant = new MCPVoiceAssistant(client, model, voice, instructions, loggerFactory);
                using var cts = new CancellationTokenSource();

                Console.CancelKeyPress += (sender, e) =>
                {
                    e.Cancel = true;
                    cts.Cancel();
                };

                await assistant.StartAsync(cts.Token).ConfigureAwait(false);
            }
            catch (OperationCanceledException)
            {
                Console.WriteLine("\n👋 Voice assistant with MCP shut down. Goodbye!");
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Fatal error");
            }
        }
    }
}
```

--------------------------------

### Create project folder

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/llm-speech

Creates a new folder named 'llm-speech-quickstart' and navigates into it.

```bash
mkdir llm-speech-quickstart && cd llm-speech-quickstart
```

--------------------------------

### Example output from kubectl get all

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-container-howto-on-premises

Expected output from kubectl get all after the deployment is complete, showing pods, services, deployments, replicasets, and horizontal pod autoscalers in a Running state.

```text
NAME                                  READY     STATUS    RESTARTS   AGE
pod/speech-to-text-7664f5f465-87w2d   1/1       Running   0          34m
pod/speech-to-text-7664f5f465-klbr8   1/1       Running   0          34m
pod/neural-text-to-speech-56f8fb685b-4jtzh   1/1       Running   0          34m
pod/neural-text-to-speech-56f8fb685b-frwxf   1/1       Running   0          34m

NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
service/kubernetes       ClusterIP      10.0.0.1       <none>           443/TCP        3h
service/speech-to-text   LoadBalancer   10.0.252.106   52.162.123.151   80:31811/TCP   34m
service/neural-text-to-speech   LoadBalancer   10.0.125.187   65.52.233.162    80:31247/TCP   34m

NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/speech-to-text   2         2         2            2           34m
deployment.apps/neural-text-to-speech   2         2         2            2           34m

NAME                                        DESIRED   CURRENT   READY     AGE
replicaset.apps/speech-to-text-7664f5f465   2         2         2         34m
replicaset.apps/neural-text-to-speech-56f8fb685b   2         2         2         34m

NAME                                                            REFERENCE                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/speech-to-text-autoscaler   Deployment/speech-to-text   1%/50%    2         10        2          34m
horizontalpodautoscaler.autoscaling/neural-text-to-speech-autoscaler   Deployment/neural-text-to-speech   0%/50%    2         10        2          34m

```

--------------------------------

### Create MCPQuickstart.java

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-voice-live-mcp-server

Java class demonstrating MCP server integration with the VoiceLive SDK. Requires environment variables AZURE_VOICELIVE_ENDPOINT and AZURE_VOICELIVE_API_KEY (unless using --use-token-credential). Run with: mvn compile exec:java -Dexec.mainClass="MCPQuickstart" -q

```java
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import com.azure.ai.voicelive.VoiceLiveAsyncClient;
import com.azure.ai.voicelive.VoiceLiveClientBuilder;
import com.azure.ai.voicelive.VoiceLiveServiceVersion;
import com.azure.ai.voicelive.VoiceLiveSessionAsyncClient;
import com.azure.ai.voicelive.models.AudioEchoCancellation;
import com.azure.ai.voicelive.models.AudioInputTranscriptionOptions;
import com.azure.ai.voicelive.models.AudioInputTranscriptionOptionsModel;
import com.azure.ai.voicelive.models.AudioNoiseReduction;
import com.azure.ai.voicelive.models.AudioNoiseReductionType;
import com.azure.ai.voicelive.models.AzureStandardVoice;
import com.azure.ai.voicelive.models.ClientEventSessionUpdate;
import com.azure.ai.voicelive.models.InputAudioFormat;
import com.azure.ai.voicelive.models.InteractionModality;
import com.azure.ai.voicelive.models.MCPServer;
import com.azure.ai.voicelive.models.OutputAudioFormat;
import com.azure.ai.voicelive.models.ServerEventType;
import com.azure.ai.voicelive.models.ServerVadTurnDetection;
import com.azure.ai.voicelive.models.SessionUpdate;
import com.azure.ai.voicelive.models.SessionUpdateError;
import com.azure.ai.voicelive.models.SessionUpdateResponseAudioDelta;
import com.azure.ai.voicelive.models.VoiceLiveSessionOptions;
import com.azure.ai.voicelive.models.VoiceLiveToolDefinition;
import com.azure.core.credential.KeyCredential;
import com.azure.core.credential.TokenCredential;
import com.azure.core.util.BinaryData;
import com.azure.identity.AzureCliCredentialBuilder;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import javax.sound.sampled.TargetDataLine;

import java.io.FileInputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Queue;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import java.util.regex.Pattern;

/**
 * MCP Quickstart - demonstrates MCP server integration with the VoiceLive SDK.
 * Shows how to define MCP servers, handle MCP tool calls, and implement
 * an approval flow for tool calls that require user consent.
 *
 * <p><strong>Environment Variables Required:</strong></p>
 * <ul>
 *   <li>AZURE_VOICELIVE_ENDPOINT - The VoiceLive service endpoint URL</li>
 *   <li>AZURE_VOICELIVE_API_KEY - The API key (required if not using --use-token-credential)</li>
 * </ul>
 *
 * <p><strong>How to Run:</strong></p>
 * <pre>{@code
 * mvn compile exec:java -Dexec.mainClass="MCPQuickstart" -q
 * }</pre>
 */
public final class MCPQuickstart {

    private static final String DEFAULT_MODEL = "gpt-realtime";
    private static final String DEFAULT_VOICE = "en-US-Ava:DragonHDLatestNeural";
    private static final String DEFAULT_INSTRUCTIONS =
        "You are a helpful AI assistant with access to MCP tools. "
        + "Use the tools to help answer user questions. "
        + "Respond naturally and conversationally. "
        + "Some tools require user approval before they can be used. When you receive a "
        + "system message asking you to request permission, you MUST clearly ask the user "
        + "for their explicit approval before proceeding. Always wait for the user to say "
        + "yes or no. Never skip the approval question or assume permission is granted. "
        + "If a tool result arrives after the conversation has moved to a different topic, "
```

--------------------------------

### Example response for transcription status

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription-get

Example JSON response body from the Transcriptions - Get operation. It shows the transcription metadata and the status property, which is 'Succeeded' when the transcription is complete and results are available.

```json
{
  "self": "https://YourResourceName.cognitiveservices.azure.com/speechtotext/transcriptions/5cff1d03-118f-4c4c-b3ba-e1f1cd88c14d?api-version=2024-11-15",
  "displayName": "My Transcription",
  "locale": "en-US",
  "createdDateTime": "2025-05-24T13:36:57Z",
  "lastActionDateTime": "2025-05-24T13:37:13Z",
  "links": {
    "files": "https://YourResourceName.cognitiveservices.azure.com/speechtotext/transcriptions/5cff1d03-118f-4c4c-b3ba-e1f1cd88c14d/files?api-version=2024-11-15"
  },
  "properties": {
    "wordLevelTimestampsEnabled": true,
    "displayFormWordLevelTimestampsEnabled": false,
    "channels": [
      0,
      1
    ],
    "punctuationMode": "DictatedAndAutomatic",
    "profanityFilterMode": "Masked",
    "timeToLiveHours": 48,
    "languageIdentification": {
      "candidateLocales": [
        "en-US",
        "de-DE",
        "es-ES"
      ],
      "mode": "Continuous"
    },
    "durationMilliseconds": 3000
  },
  "status": "Succeeded"
}
```

--------------------------------

### Create project folder

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/fast-transcription-create

Creates a new folder named 'transcription-quickstart' and navigates into it.

```bash
mkdir transcription-quickstart && cd transcription-quickstart
```

--------------------------------

### Create and navigate to .NET console project

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/llm-speech

Use the .NET CLI to create a new console application named 'llm-speech-quickstart' and change into its directory. This is the first step in setting up the project.

```bash
dotnet new console -n llm-speech-quickstart
cd llm-speech-quickstart
```

--------------------------------

### Batch synthesis status response example

Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-synthesis

Example response body from the GET request. The outputs.result field contains a URL to download a ZIP file with the audio, summary, and debug details.

```json
{
  "id": "YourSynthesisId",
  "status": "Succeeded",
  "createdDateTime": "2024-03-12T07:23:18.0097387Z",
  "lastActionDateTime": "2024-03-12T07:23:18.7979669",
  "inputKind": "SSML",
  "customVoices": {},
  "properties": {
    "timeToLiveInHours": 168,
    "outputFormat": "riff-24khz-16bit-mono-pcm",
    "concatenateResult": false,
    "decompressOutputFiles": false,
    "wordBoundaryEnabled": false,
    "sentenceBoundaryEnabled": false,
    "sizeInBytes": 120000,
    "succeededAudioCount": 1,
    "failedAudioCount": 0,
    "durationInMilliseconds": 2500,
    "billingDetails": {
      "neuralCharacters": 29
    }
  },
  "outputs": {
    "result": "https://stttssvcuse.blob.core.windows.net/batchsynthesis-output/29f2105f997c4bfea176d39d05ff201e/YourSynthesisId/results.zip?SAS_Token"
  }
}
```
Tokens:
Raw