### create_app Function Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

Example of how to use the create_app function to start the aiohttp web application.

```python
from aiohttp import web
from app import create_app
import asyncio

async def main():
    app = await create_app()
    runner = web.AppRunner(app)
    await runner.setup()
    site = web.TCPSite(runner, 'localhost', 8765)
    await site.start()
    print("Server started on http://localhost:8765")
    await asyncio.Event().wait()

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Complete RTMT Setup Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/rtmt.md

This Python script demonstrates a full setup of the RTMT, including defining a custom tool, configuring Azure OpenAI credentials and deployment, and attaching the RTMT to an aiohttp web application.

```python
from aiohttp import web
from azure.core.credentials import AzureKeyCredential
from rtmt import RTMiddleTier, Tool, ToolResult, ToolResultDirection
import os

async def my_tool(args: dict) -> ToolResult:
    query = args.get("query", "")
    result = f"Query results for: {query}"
    return ToolResult(result, ToolResultDirection.TO_SERVER)

async def create_app():
    app = web.Application()
    
    credentials = AzureKeyCredential(os.environ["AZURE_OPENAI_API_KEY"])
    rtmt = RTMiddleTier(
        endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        deployment=os.environ["AZURE_OPENAI_DEPLOYMENT"],
        credentials=credentials,
        voice_choice="alloy"
    )
    
    rtmt.system_message = "You are a helpful assistant. Use the my_tool function to search for information."
    rtmt.temperature = 0.7
    rtmt.max_tokens = 500
    
    rtmt.tools["my_tool"] = Tool(
        target=my_tool,
        schema={
            "type": "function",
            "name": "my_tool",
            "description": "Search for information",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"}
                },
                "required": ["query"]
            }
        }
    )
    
    rtmt.attach_to_app(app, "/realtime")
    return app

if __name__ == "__main__":
    app = web.run_app(create_app(), host="localhost", port=8765)
```

--------------------------------

### Configuration Logging Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/configuration.md

Example output of configuration choices logged at application startup.

```text
Using AzureKeyCredential for Azure OpenAI
Using AzureKeyCredential for Azure Search
Realtime voice choice set to alloy
```

--------------------------------

### Recorder start Method Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Starts recording from a media stream. Sets up the Web Audio API graph with AudioWorklet processing.

```typescript
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
await recorder.start(stream);
```

--------------------------------

### Bash Command to Run Setup Script

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

Shows the command to execute the Python setup script.

```bash
python setup_intvect.py

```

--------------------------------

### setup_index Function Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

Creates (or reuses) an Azure AI Search index, data source connection, skillset, and indexer with integrated vectorization using Azure OpenAI embeddings.

```python
from azure.identity import AzureDeveloperCliCredential
from setup_intvect import setup_index

credential = AzureDeveloperCliCredential(tenant_id="your-tenant-id", process_timeout=60)

setup_index(
    azure_credential=credential,
    index_name="documents",
    azure_search_endpoint="https://myservice.search.windows.net",
    azure_storage_connection_string="DefaultEndpointsProtocol=https;AccountName=...",
    azure_storage_container="documents",
    azure_openai_embedding_endpoint="https://myservice.openai.azure.com",
    azure_openai_embedding_deployment="embedding-deployment",
    azure_openai_embedding_model="text-embedding-3-large",
    azure_openai_embeddings_dimensions=3072
)
```

--------------------------------

### Ragtools.py

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/docs/manual_setup.md

Example of updating field names in ragtools.py to match text/vector fields if an existing index was not created using the 'Import and vectorize data' option.

```python
from azure.search.documents.indexes.models import SearchField, SearchIndex, SimpleField, VectorSearch, VectorSearchAlgorithmConfiguration, HnswAlgorithmConfiguration, VectorizableFieldName, SearchableField, SemanticConfiguration, SemanticSearch, SemanticRanker, SemanticHybridSearch

def create_search_index(index_name: str, vector_search_algorithm: str = "hnsw", vector_search_metric: str = "cosine") -> SearchIndex:
    fields = [
        SimpleField(name="id", type="SearchFieldType.STRING", key=True),
        SearchableField(name="content", type="SearchFieldType.STRING", searchable=True, filterable=False, sortable=False, facetable=False),
        SearchableField(name="metadata", type="SearchFieldType.STRING", searchable=True, filterable=False, sortable=False, facetable=False),
        # Add your text/vector fields here
        # Example for text field:
        # SearchableField(name="your_text_field_name", type="SearchFieldType.STRING", searchable=True, filterable=True, sortable=True, facetable=True),
        # Example for vector field:
        # VectorSearchField(name="your_vector_field_name", type="SearchFieldType.COLLECTION(SearchFieldType.SINGLE)", searchable=True, vector_search_dimensions=1536, vector_search_algorithm_configurations=VectorSearchAlgorithmConfiguration(kind="hnsw", name=vector_search_algorithm, parameters=HnswAlgorithmConfiguration(metric=vector_search_metric)))
    ]

    vector_search = VectorSearch(
        algorithm_configurations=[
            VectorSearchAlgorithmConfiguration(
                kind="hnsw",
                name=vector_search_algorithm,
                parameters=HnswAlgorithmConfiguration(metric=vector_search_metric)
            )
        ],
        vector_fields=[],
        # Add your vector fields here
        # Example:
        # vector_fields=[
        #     VectorSearchField(name="your_vector_field_name", kind="vector", vector_search_dimensions=1536, vector_search_algorithm_configurations=VectorSearchAlgorithmConfiguration(kind="hnsw", name=vector_search_algorithm, parameters=HnswAlgorithmConfiguration(metric=vector_search_metric)))
        # ]
    )

    semantic_config = SemanticConfiguration(
        name="default",
        prioritized_elements=[
            SemanticRanker(name="default"),
            SemanticHybridSearch(name="default")
        ]
    )

    return SearchIndex(
        name=index_name,
        fields=fields,
        vector_search=vector_search,
        semantic_search=SemanticSearch(configurations=[semantic_config])
    )

```

--------------------------------

### Example Usage of upload_documents Function

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

Demonstrates how to use the upload_documents function with AzureDeveloperCliCredential.

```python
from azure.identity import AzureDeveloperCliCredential
from setup_intvect import upload_documents

credential = AzureDeveloperCliCredential(tenant_id="your-tenant-id", process_timeout=60)

upload_documents(
    azure_credential=credential,
    indexer_name="documents",
    azure_search_endpoint="https://myservice.search.windows.net",
    azure_storage_endpoint="https://myaccount.blob.core.windows.net",
    azure_storage_container="documents"
)

```

--------------------------------

### Example Client Integration

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Demonstrates how to integrate the useRealTime hook in a React component to start a voice chat session.

```typescript
import useRealTime from "@/hooks/useRealtime";

function App() {
    const { startSession, addUserAudio, inputAudioBufferClear } = useRealTime({
        // Connect to /realtime endpoint (relative to current origin)
        // useDirectAoaiApi: false (default) 
        
        onWebSocketOpen: () => {
            console.log("Connected to /realtime");
        },
        
        onReceivedResponseAudioDelta: (message) => {
            // Receive audio from model
            playAudio(message.delta);
        },
        
        onReceivedExtensionMiddleTierToolResponse: (message) => {
            // Receive tool results (citations)
            console.log(message.tool_name, JSON.parse(message.tool_result));
        }
    });

    return (
        <button onClick={() => startSession()}>
            Start Voice Chat
        </button>
    );
}
```

--------------------------------

### Direct Usage Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

An example demonstrating how to use the Recorder and Player components to record audio for 3 seconds and then play it back.

```typescript
import { Recorder } from "@/components/audio/recorder";
import { Player } from "@/components/audio/player";

async function recordAndPlay() {
    let recordedData: number[] = [];

    // Setup recorder
    const recorder = new Recorder((buffer) => {
        recordedData.push(...Array.from(buffer));
    });

    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    await recorder.start(stream);

    // Record for 3 seconds
    await new Promise(resolve => setTimeout(resolve, 3000));

    await recorder.stop();

    // Setup player
    const player = new Player();
    await player.init(24000);

    // Play the recorded audio
    const pcmData = new Int16Array(recordedData);
    player.play(pcmData);
}
```

--------------------------------

### Static File Serving Request Examples

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Examples of HTTP GET requests for serving static frontend assets like JavaScript, CSS, or images.

```http
GET /path/to/asset HTTP/1.1
Host: localhost:8765
```

```http
GET /app.js HTTP/1.1
GET /styles.css HTTP/1.1
GET /logo.svg HTTP/1.1
```

--------------------------------

### WebSocket /realtime Upgrade Request Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Example HTTP GET request to initiate a WebSocket connection to the /realtime endpoint.

```http
GET /realtime HTTP/1.1
Host: localhost:8765
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: [base64 key]
Sec-WebSocket-Version: 13
```

--------------------------------

### GET / Response Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Example HTTP response for the root endpoint, indicating a successful serving of the frontend HTML.

```http
HTTP/1.1 200 OK
Content-Type: text/html

<html>
<!-- Frontend application -->
</html>
```

--------------------------------

### Example .env File

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/configuration.md

An example .env file showing environment variables for Azure OpenAI, Azure AI Search, Azure Storage, and Azure Identity.

```dotenv
# Azure OpenAI
AZURE_OPENAI_ENDPOINT=https://myservice.openai.azure.com
AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview
AZURE_OPENAI_API_KEY=sk-...
AZURE_OPENAI_REALTIME_VOICE_CHOICE=alloy

# Azure AI Search
AZURE_SEARCH_ENDPOINT=https://myservice.search.windows.net
AZURE_SEARCH_INDEX=documents
AZURE_SEARCH_API_KEY=...
AZURE_SEARCH_SEMANTIC_CONFIGURATION=default
AZURE_SEARCH_IDENTIFIER_FIELD=chunk_id
AZURE_SEARCH_CONTENT_FIELD=chunk
AZURE_SEARCH_EMBEDDING_FIELD=text_vector
AZURE_SEARCH_TITLE_FIELD=title
AZURE_SEARCH_USE_VECTOR_QUERY=true

# Azure Storage (for setup_intvect.py)
AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=...
AZURE_STORAGE_CONTAINER=documents
AZURE_STORAGE_ENDPOINT=https://myaccount.blob.core.windows.net

# Azure OpenAI Embeddings (for setup_intvect.py)
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=embedding-deployment
AZURE_OPENAI_EMBEDDING_MODEL=text-embedding-3-large

# Azure Identity
AZURE_TENANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
```

--------------------------------

### Install frontend dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Commands to navigate to the frontend directory and install npm dependencies.

```bash
cd app/frontend
npm install
cd ../..
```

--------------------------------

### GET / Request Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Example HTTP GET request to the root endpoint, which serves the frontend application.

```http
GET / HTTP/1.1
Host: localhost:8765
```

--------------------------------

### Install and test updated dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to install dependencies after upgrading an existing one.

```bash
pip install -r app/backend/requirements.txt
# Test the application thoroughly
```

--------------------------------

### Complete Setup Workflow

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

This Python script demonstrates the complete workflow for setting up an Azure AI Search index with integrated vectorization. It loads environment variables, sets up the index with Azure OpenAI embeddings, and uploads documents for indexing.

```python
import os
from azure.identity import AzureDeveloperCliCredential
from setup_intvect import load_azd_env, setup_index, upload_documents

# Load azd environment
load_azd_env()

# Get configuration
AZURE_SEARCH_INDEX = os.environ["AZURE_SEARCH_INDEX"]
AZURE_OPENAI_EMBEDDING_ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"]
AZURE_OPENAI_EMBEDDING_DEPLOYMENT = os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"]
AZURE_OPENAI_EMBEDDING_MODEL = os.environ["AZURE_OPENAI_EMBEDDING_MODEL"]
EMBEDDINGS_DIMENSIONS = 3072
AZURE_SEARCH_ENDPOINT = os.environ["AZURE_SEARCH_ENDPOINT"]
AZURE_STORAGE_ENDPOINT = os.environ["AZURE_STORAGE_ENDPOINT"]
AZURE_STORAGE_CONNECTION_STRING = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
AZURE_STORAGE_CONTAINER = os.environ["AZURE_STORAGE_CONTAINER"]

# Create credentials
credential = AzureDeveloperCliCredential(
    tenant_id=os.environ["AZURE_TENANT_ID"],
    process_timeout=60
)

# Setup index with integrated vectorization
setup_index(
    azure_credential=credential,
    index_name=AZURE_SEARCH_INDEX,
    azure_search_endpoint=AZURE_SEARCH_ENDPOINT,
    azure_storage_connection_string=AZURE_STORAGE_CONNECTION_STRING,
    azure_storage_container=AZURE_STORAGE_CONTAINER,
    azure_openai_embedding_endpoint=AZURE_OPENAI_EMBEDDING_ENDPOINT,
    azure_openai_embedding_deployment=AZURE_OPENAI_EMBEDDING_DEPLOYMENT,
    azure_openai_embedding_model=AZURE_OPENAI_EMBEDDING_MODEL,
    azure_openai_embeddings_dimensions=EMBEDDINGS_DIMENSIONS
)

# Upload documents and start indexing
upload_documents(
    azure_credential=credential,
    indexer_name=AZURE_SEARCH_INDEX,
    azure_search_endpoint=AZURE_SEARCH_ENDPOINT,
    azure_storage_endpoint=AZURE_STORAGE_ENDPOINT,
    azure_storage_container=AZURE_STORAGE_CONTAINER
)
```

--------------------------------

### Install test dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Commands to install necessary Python packages for testing.

```bash
pip install pytest pytest-asyncio pytest-cov
```

--------------------------------

### Player init Method Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Initializes the audio player with a specific sample rate.

```typescript
import { Player } from "@/components/audio/player";

const player = new Player();
await player.init(24000);
```

--------------------------------

### Install updated dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to install dependencies after updating requirements.txt.

```bash
source .venv/bin/activate  # Ensure virtualenv is active
pip install -r app/backend/requirements.txt
```

--------------------------------

### load_azd_env Function Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

Loads environment variables from the Azure Developer CLI's (azd) active environment configuration.

```python
from setup_intvect import load_azd_env

load_azd_env()  # Now os.environ contains all azd variables
print(os.environ["AZURE_SEARCH_ENDPOINT"])
```

--------------------------------

### Start the application locally (Linux/Mac)

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/README.md

Starts the local development server on Linux or macOS using a shell script.

```bash
./scripts/start.sh
```

--------------------------------

### Install Python dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to install Python dependencies from requirements.txt.

```bash
pip install -r app/backend/requirements.txt
```

--------------------------------

### Development Server Startup

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

Code snippet that starts an aiohttp web server on localhost:8765.

```python
if __name__ == "__main__":
    host = "localhost"
    port = 8765
    web.run_app(create_app(), host=host, port=port)
```

--------------------------------

### Integration Example: Complete Voice Chat

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/frontend-hooks.md

Shows a comprehensive example of integrating `useRealtime`, `useAudioRecorder`, and `useAudioPlayer` hooks to build a real-time voice chat application.

```typescript
import { useState } from "react";
import useRealTime from "@/hooks/useRealtime";
import useAudioRecorder from "@/hooks/useAudioRecorder";
import useAudioPlayer from "@/hooks/useAudioPlayer";
import { GroundingFile, ToolResult } from "@/types";

function VoiceChat() {
    const [isRecording, setIsRecording] = useState(false);
    const [sources, setSources] = useState<GroundingFile[]>([]);

    // Realtime API hook
    const { startSession, addUserAudio, inputAudioBufferClear } = useRealTime({
        enableInputAudioTranscription: true,
        
        onWebSocketOpen: () => {
            console.log("Connected");
        },
        
        onReceivedResponseAudioDelta: (message) => {
            playAudio(message.delta);
        },
        
        onReceivedExtensionMiddleTierToolResponse: (message) => {
            const result: ToolResult = JSON.parse(message.tool_result);
            const files: GroundingFile[] = result.sources.map(x => ({
                id: x.chunk_id,
                name: x.title,
                content: x.chunk
            }));
            setSources(prev => [...prev, ...files]);
        }
    });

    // Audio recorder hook
    const { start: startRecording, stop: stopRecording } = useAudioRecorder({
        onAudioRecorded: addUserAudio
    });

    // Audio player hook
    const { reset: resetPlayer, play: playAudio, stop: stopPlayer } = useAudioPlayer();

    const handleToggleListening = async () => {
        if (!isRecording) {
            startSession();
            await startRecording();
            resetPlayer();
            setIsRecording(true);
        } else {
            await stopRecording();
            stopPlayer();
            inputAudioBufferClear();
            setIsRecording(false);
        }
    };

    return (
        <div>
            <button onClick={handleToggleListening}>
                {isRecording ? "Stop" : "Start"}
            </button>
            <div>
                {sources.map(source => (
                    <div key={source.id}>
                        <h4>{source.name}</h4>
                        <p>{source.content}</p>
                    </div>
                ))}
            </div>
        </div>
    );
}

export default VoiceChat;
```

--------------------------------

### Development server .env file

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/docs/existing_services.md

Example .env file contents for testing the solution locally with the azure-search-openai-demo index.

```bash
AZURE_TENANT_ID=<YOUR-TENANT-ID>
AZURE_OPENAI_ENDPOINT=https://<YOUR_OPENAI_ENDPOINT>.openai.azure.com
AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview
AZURE_OPENAI_REALTIME_VOICE_CHOICE=<choose one: echo, alloy, shimmer>
AZURE_SEARCH_ENDPOINT=https://<YOUR_SEARCH_SERVICE>.search.windows.net
AZURE_SEARCH_INDEX=gptkbindex
AZURE_SEARCH_SEMANTIC_CONFIGURATION=default
AZURE_SEARCH_IDENTIFIER_FIELD=id
AZURE_SEARCH_CONTENT_FIELD=content
AZURE_SEARCH_TITLE_FIELD=sourcepage
AZURE_SEARCH_EMBEDDING_FIELD=embedding
```

--------------------------------

### Static File Serving Response Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Example HTTP response for serving static files, including content type and length.

```http
HTTP/1.1 200 OK
Content-Type: [content-type based on file extension]
Content-Length: [file size]

[file content]
```

--------------------------------

### Integration Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/ragtools.md

Python code demonstrating how to set up voice RAG using RTMiddleTier and attach RAG tools.

```python
import os
from aiohttp import web
from azure.core.credentials import AzureKeyCredential
from azure.identity import DefaultAzureCredential
from rtmt import RTMiddleTier
from ragtools import attach_rag_tools
from app import create_app

async def setup_voice_rag():
    # Credentials
    llm_key = os.environ.get("AZURE_OPENAI_API_KEY")
    search_key = os.environ.get("AZURE_SEARCH_API_KEY")
    
    llm_credential = AzureKeyCredential(llm_key) if llm_key else DefaultAzureCredential()
    search_credential = AzureKeyCredential(search_key) if search_key else DefaultAzureCredential()
    
    # Create RTMiddleTier
    rtmt = RTMiddleTier(
        credentials=llm_credential,
        endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        deployment=os.environ["AZURE_OPENAI_REALTIME_DEPLOYMENT"],
        voice_choice=os.environ.get("AZURE_OPENAI_REALTIME_VOICE_CHOICE", "alloy")
    )
    
    # Set system message
    rtmt.system_message = """
        You are a helpful assistant. Always answer using the search tool first.
        Always cite sources using the report_grounding tool.
        Keep answers short for audio delivery.
    """.strip()
    
    # Attach RAG tools
    attach_rag_tools(
        rtmt=rtmt,
        credentials=search_credential,
        search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"],
        search_index=os.environ["AZURE_SEARCH_INDEX"],
        semantic_configuration=os.environ.get("AZURE_SEARCH_SEMANTIC_CONFIGURATION"),
        identifier_field=os.environ.get("AZURE_SEARCH_IDENTIFIER_FIELD", "chunk_id"),
        content_field=os.environ.get("AZURE_SEARCH_CONTENT_FIELD", "chunk"),
        embedding_field=os.environ.get("AZURE_SEARCH_EMBEDDING_FIELD", "text_vector"),
        title_field=os.environ.get("AZURE_SEARCH_TITLE_FIELD", "title"),
        use_vector_query=(os.getenv("AZURE_SEARCH_USE_VECTOR_QUERY", "true") == "true")
    )
    
    return rtmt
```

--------------------------------

### Start the application locally (Windows)

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/README.md

Starts the local development server on Windows using PowerShell.

```powershell
pwsh .\scripts\start.ps1
```

--------------------------------

### WebSocket /realtime Upgrade Response Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Example HTTP response indicating a successful WebSocket connection upgrade.

```http
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: [signature]
```

--------------------------------

### ToolResult Constructor Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/rtmt.md

Demonstrates creating ToolResult instances for different routing destinations.

```python
from rtmt import ToolResult, ToolResultDirection

# Result for the model only (e.g., internal search results)
result_to_server = ToolResult("[doc1]: content", ToolResultDirection.TO_SERVER)

# Result to show the user (e.g., citations)
result_to_client = ToolResult(
    json.dumps({"sources": [{"id": "1", "title": "Doc 1"}]}),
    ToolResultDirection.TO_CLIENT
)
```

--------------------------------

### Environment Configuration Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

A typical .env file used for configuring Azure OpenAI and Azure Search services.

```env
AZURE_OPENAI_ENDPOINT=https://myservice.openai.azure.com
AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview
AZURE_OPENAI_REALTIME_VOICE_CHOICE=alloy
AZURE_OPENAI_API_KEY=sk-...

AZURE_SEARCH_ENDPOINT=https://myservice.search.windows.net
AZURE_SEARCH_INDEX=documents
AZURE_SEARCH_SEMANTIC_CONFIGURATION=default
AZURE_SEARCH_API_KEY=...
AZURE_SEARCH_IDENTIFIER_FIELD=chunk_id
AZURE_SEARCH_CONTENT_FIELD=chunk
AZURE_SEARCH_EMBEDDING_FIELD=text_vector
AZURE_SEARCH_TITLE_FIELD=title
AZURE_SEARCH_USE_VECTOR_QUERY=true
```

--------------------------------

### Sample Log Output

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/errors.md

An example of log output showing development mode, authentication, and search activity.

```text
Running in development mode, loading from .env file
Using AzureKeyCredential
Realtime voice choice set to alloy
Searching for 'machine learning' in the knowledge base.
Grounding source: chunk_001 OR chunk_002
```

--------------------------------

### Parsing Example for ToolResult

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/types.md

Provides an example of how to parse the `tool_result` string from an `ExtensionMiddleTierToolResponse` and map it to `GroundingFile` objects.

```typescript
const message: ExtensionMiddleTierToolResponse = {...};
const result: ToolResult = JSON.parse(message.tool_result);
const groundingFiles: GroundingFile[] = result.sources.map(x => ({
    id: x.chunk_id,
    name: x.title,
    content: x.chunk
}));
```

--------------------------------

### RTMiddleTier Constructor Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/rtmt.md

Initializes an RTMiddleTier instance for managing realtime API communication.

```python
from azure.core.credentials import AzureKeyCredential
from rtmt import RTMiddleTier

credentials = AzureKeyCredential("your-api-key")
rtmt = RTMiddleTier(
    endpoint="https://myservice.openai.azure.com",
    deployment="gpt-4o-realtime-preview",
    credentials=credentials,
    voice_choice="alloy"
)
```

--------------------------------

### useAudioRecorder Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/frontend-hooks.md

Example of using the useAudioRecorder hook to handle microphone permissions, start/stop recording, and send recorded audio chunks to a backend.

```typescript
import useAudioRecorder from "@/hooks/useAudioRecorder";

function RecordingComponent() {
    const { start, stop } = useAudioRecorder({
        onAudioRecorded: (base64Audio) => {
            console.log("Audio chunk recorded, size:", base64Audio.length);
            // Send to realtime API
            addUserAudio(base64Audio);
        }
    });

    const handleStartRecording = async () => {
        try {
            await start();
            console.log("Recording started");
        } catch (error) {
            console.error("Failed to start recording:", error);
        }
    };

    const handleStopRecording = async () => {
        await stop();
        console.log("Recording stopped");
    };

    return (
        <div>
            <button onClick={handleStartRecording}>Record</button>
            <button onClick={handleStopRecording}>Stop</button>
        </div>
    );
}

```

--------------------------------

### Recorder Constructor Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Initializes a Recorder instance that captures microphone input and calls a callback when audio data is available.

```typescript
import { Recorder } from "@/components/audio/recorder";

const recorder = new Recorder((buffer) => {
    console.log("Audio data:", buffer);
});
```

--------------------------------

### Example Tool Definition

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/rtmt.md

Defines a simple 'multiply' tool with its schema and implementation, and registers it with RTMT.

```python
from rtmt import Tool, ToolResult, ToolResultDirection

schema = {
    "type": "function",
    "name": "multiply",
    "description": "Multiply two numbers",
    "parameters": {
        "type": "object",
        "properties": {
            "a": {"type": "number"},
            "b": {"type": "number"}
        },
        "required": ["a", "b"],
        "additionalProperties": False
    }
}

async def multiply_impl(args: dict) -> ToolResult:
    result = args["a"] * args["b"]
    return ToolResult(str(result), ToolResultDirection.TO_SERVER)

tool = Tool(target=multiply_impl, schema=schema)
rtmt.tools["multiply"] = tool
```

--------------------------------

### Player play Method Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Plays audio data through the audio device.

```typescript
const pcmData = new Int16Array([/* PCM samples */]);
player.play(pcmData);
```

--------------------------------

### Add a new dependency to requirements.txt

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Example of how to add a new package with a version pin to requirements.txt.

```text
# Add new package with version pin
new-package==1.2.3
```

--------------------------------

### Successful WebSocket Connection Example (Client-side JavaScript)

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

JavaScript code demonstrating how a client can establish a WebSocket connection and send an initial session update.

```javascript
// Client code
const ws = new WebSocket('ws://localhost:8765/realtime');

ws.onopen = () => {
    console.log('Connected');
    
    // Send session update
    ws.send(JSON.stringify({
        type: 'session.update',
        session: {
            turn_detection: { type: 'server_vad' }
        }
    }));
};

ws.onmessage = (event) => {
    const message = JSON.parse(event.data);
    console.log('Received:', message.type);
};

ws.onerror = (error) => {
    console.error('WebSocket error:', error);
};
```

--------------------------------

### Environment variables for local development

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Example .env file content for local development, including Azure service endpoints and deployment names.

```env
AZURE_TENANT_ID=<your-tenant-id>
AZURE_OPENAI_ENDPOINT=https://<your-openai-service>.openai.azure.com
AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview
AZURE_OPENAI_REALTIME_VOICE_CHOICE=alloy
AZURE_SEARCH_ENDPOINT=https://<your-search-service>.search.windows.net
AZURE_SEARCH_INDEX=<your-index-name>
AZURE_SEARCH_SEMANTIC_CONFIGURATION=default
AZURE_SEARCH_IDENTIFIER_FIELD=chunk_id
AZURE_SEARCH_CONTENT_FIELD=chunk
AZURE_SEARCH_TITLE_FIELD=title
AZURE_SEARCH_EMBEDDING_FIELD=text_vector
AZURE_SEARCH_USE_VECTOR_QUERY=true
```

--------------------------------

### Update version in requirements.txt

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Example of how to update the version of an existing dependency in requirements.txt.

```text
# Change version
package-name==2.0.0  # was 1.0.0
```

--------------------------------

### Check Python code style

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Installs Ruff and runs linting on the Python backend code.

```bash
# Install ruff if not already installed
pip install ruff

# Run linting
ruff check app/backend
```

--------------------------------

### RTMiddleTier system_message configuration

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/configuration.md

Python type hint and example for setting the system_message on the RTMiddleTier object.

```python
rtmt.system_message: Optional[str] = None
```

```python
rtmt.system_message = """
You are a helpful assistant. Always use the search tool first.
Always cite sources using report_grounding tool.
Keep answers short for audio.
""".strip()
```

--------------------------------

### Search Tool Argument Handling

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/errors.md

Example of how arguments are loaded and passed to a tool function, with a note on error handling.

```python
args = item["arguments"]
result = await tool.target(json.loads(args))
```

--------------------------------

### Set environment variable for voice choice

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Example command to set a specific environment variable for voice choice during deployment.

```bash
azd env set AZURE_OPENAI_REALTIME_VOICE_CHOICE shimmer
```

--------------------------------

### Run frontend tests

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to run frontend tests using npm.

```bash
cd app/frontend
npm test
```

--------------------------------

### Recorder stop Method Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Stops recording and releases all audio resources.

```typescript
await recorder.stop();
```

--------------------------------

### Attach RTMiddleTier to aiohttp Application Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/rtmt.md

Registers the RTMiddleTier WebSocket handler with an aiohttp Application.

```python
from aiohttp import web
from rtmt import RTMiddleTier

app = web.Application()
rtmt = RTMiddleTier(...)
rtmt.attach_to_app(app, "/realtime")
```

--------------------------------

### Development Server Execution

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

Command to run the development server directly from the app.py script.

```bash
cd app/backend
python app.py
```

--------------------------------

### ResourceExistsError Handling

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/errors.md

Handles cases where an indexer is already running, logging that it's not starting again.

```python
try:
    indexer_client.run_indexer(indexer_name)
    logger.info("Indexer started...")
except ResourceExistsError:
    logger.info("Indexer already running, not starting again")
```

--------------------------------

### Reuse existing index from azure-search-openai-demo

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/docs/existing_services.md

Commands to set environment variables for reusing an existing Azure Search index.

```bash
azd env set AZURE_SEARCH_REUSE_EXISTING true
```

```bash
azd env set AZURE_SEARCH_SERVICE_RESOURCE_GROUP <YOUR_RESOURCE_GROUP>
```

```bash
azd env set AZURE_SEARCH_ENDPOINT https://<YOUR_SEARCH_SERVICE>.search.windows.net
```

```bash
azd env set AZURE_SEARCH_SEMANTIC_CONFIGURATION default
```

```bash
azd env set AZURE_SEARCH_IDENTIFIER_FIELD id
```

```bash
azd env set AZURE_SEARCH_CONTENT_FIELD content
```

```bash
azd env set AZURE_SEARCH_TITLE_FIELD sourcepage
```

```bash
azd env set AZURE_SEARCH_EMBEDDING_FIELD embedding
```

```bash
azd env set AZURE_SEARCH_REUSE_EXISTING true
```

```bash
azd env set AZURE_SEARCH_INDEX gptkbindex
```

--------------------------------

### Create and activate Python virtual environment

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Commands to create and activate a Python virtual environment.

```bash
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
```

--------------------------------

### Player stop Method Example

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/audio-components.md

Stops playback by sending a null message to the worklet, signaling end-of-stream.

```typescript
player.stop();
```

--------------------------------

### Path Registration

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Attaches the RTMiddleTier to the aiohttp app, registering a GET handler at '/realtime' that upgrades the connection to WebSocket.

```python
rtmt.attach_to_app(app, "/realtime")
```

--------------------------------

### Python Dependencies

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/app/backend/requirements.txt

This file lists the Python packages and their versions required to run the backend application.

```text
aiohttp == 3.9.3
azure-identity == 1.18.0
azure-search-documents == 11.6.0b4
python-dotenv == 1.0.1
azure-storage-blob == 12.23.1
gunicorn
rich
```

--------------------------------

### Format frontend code

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Navigates to the frontend directory and formats the code using npm.

```bash
cd app/frontend
npm run format
```

--------------------------------

### Python Function Signature for Uploading Documents

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/setup_intvect.md

Defines the signature for the upload_documents function, outlining its parameters and return type.

```python
def upload_documents(
    azure_credential,
    indexer_name: str,
    azure_search_endpoint: str,
    azure_storage_endpoint: str,
    azure_storage_container: str
) -> None:

```

--------------------------------

### Logging Configuration

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

Configures basic logging at INFO level and initializes a logger for the 'voicerag' module.

```python
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("voicerag")
```

--------------------------------

### Reuse existing OpenAI real-time deployment

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/docs/existing_services.md

Commands to set environment variables for reusing an existing Azure OpenAI service.

```bash
azd env set AZURE_OPENAI_REUSE_EXISTING true
```

```bash
azd env set AZURE_OPENAI_RESOURCE_GROUP <YOUR_RESOURCE_GROUP>
```

```bash
azd env set AZURE_OPENAI_ENDPOINT https://<YOUR_OPENAI_SERVICE>.openai.azure.com
```

```bash
azd env set AZURE_OPENAI_REALTIME_DEPLOYMENT <YOUR_REALTIME_DEPLOYMENT_NAME>
```

--------------------------------

### Static Path Configuration

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/endpoints.md

Configures the root path '/' to serve 'static/index.html' and all other paths to serve files from the 'static/' directory.

```python
app.add_routes([web.get('/', lambda _: web.FileResponse(current_directory / 'static/index.html'))])
app.router.add_static('/', path=current_directory / 'static', name='static')
```

--------------------------------

### Production Server Execution with Gunicorn

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/app.md

Command to run the application in production using gunicorn with an async worker.

```bash
gunicorn -w 1 -k aiohttp.GunicornWebWorker app:create_app --bind 0.0.0.0:8765
```

--------------------------------

### Synchronize local .env file after deployment

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Scripts to update the local .env file after an azd deployment.

```bash
./scripts/write_env.sh  # Linux/Mac
pwsh ./scripts/write_env.ps1  # Windows
```

--------------------------------

### Run tests

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to run tests using pytest.

```bash
source .venv/bin/activate  # Ensure virtualenv is active
pytest -q
```

--------------------------------

### AudioChat Component

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/api-reference/frontend-hooks.md

Example of using the useRealTime hook to manage audio input, output, and interaction with a real-time API for a chat application.

```typescript
import useRealTime from "@/hooks/useRealtime";
import { ResponseAudioDelta, ExtensionMiddleTierToolResponse } from "@/types";

function AudioChat() {
    const { startSession, addUserAudio, inputAudioBufferClear } = useRealTime({
        enableInputAudioTranscription: true,
        
        onWebSocketOpen: () => {
            console.log("Connected to realtime API");
        },
        
        onReceivedResponseAudioDelta: (message: ResponseAudioDelta) => {
            console.log("Received audio delta:", message.delta);
            playAudio(message.delta);
        },
        
        onReceivedExtensionMiddleTierToolResponse: (message: ExtensionMiddleTierToolResponse) => {
            console.log("Tool result:", message.tool_name, message.tool_result);
            const result = JSON.parse(message.tool_result);
            displaySources(result.sources);
        },
        
        onReceivedError: (message) => {
            console.error("Error from API:", message);
        }
    });

    const handleStartChat = () => {
        startSession();
    };

    const handleSendAudio = (base64Audio: string) => {
        addUserAudio(base64Audio);
    };

    const handleStop = () => {
        inputAudioBufferClear();
    };

    return (
        <div>
            <button onClick={handleStartChat}>Start</button>
            <button onClick={handleStop}>Stop</button>
        </div>
    );
}

```

--------------------------------

### Microphone Permission Denied Trigger

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/_autodocs/errors.md

TypeScript code that attempts to get user media for audio recording, which can trigger a NotAllowedError if permission is denied.

```typescript
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
```

--------------------------------

### Run tests with coverage

Source: https://github.com/azure-samples/aisearch-openai-rag-audio/blob/main/AGENTS.md

Command to run tests with coverage reporting using pytest.

```bash
pytest --cov=app/backend --cov-report=term-missing
```