LLMRTC

https://www.llmrtc.org
Admin
LLMRTC is an open-source TypeScript SDK for building real-time voice and vision AI agents with...
Tokens:100,041
Snippets:1,042
Trust Score:3.5
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install and Run Minimal Example

Source: https://www.llmrtc.org/getting-started/overview

Navigate to the minimal example directory, install dependencies, and start the development server. This demonstrates the essential code for a bare-bones implementation.

```bash
cd examples/minimal  
npm install  
npm run dev  

```

--------------------------------

### Install and Run Project Dependencies

Source: https://www.llmrtc.org/recipes/observability-and-metrics

Installs project dependencies and starts the development server. This is a common starting point for running the example.

```bash
npm install  
npm run dev  
```

--------------------------------

### Setup Documentation Environment

Source: https://www.llmrtc.org/meta/contributing

Install dependencies and start the documentation server. This is for making changes to the project's documentation.

```bash
npm install --workspace docs
```

```bash
npm run --workspace docs start
```

--------------------------------

### Install and Run Vite Demo

Source: https://www.llmrtc.org/getting-started/overview

Navigate to the Vite demo directory, install dependencies, and start the development server. This provides a full-featured example with audio visualization and tool calling UI.

```bash
cd examples/vite-demo  
npm install  
npm run dev  

```

--------------------------------

### Install and Start LLMRTC Backend

Source: https://www.llmrtc.org/backend/cli

Install the package, set required environment variables, and start the LLMRTC backend server.

```bash
# Install the package  
npm install @llmrtc/llmrtc-backend  
  
# Set required environment variables  
export OPENAI_API_KEY=sk-...  
  
# Start the server  
npx llmrtc-backend  

```

--------------------------------

### Install and Run Minimal Voice Assistant

Source: https://www.llmrtc.org/recipes/minimal-voice-assistant

Install project dependencies and start the development server for both the frontend and backend. Access the frontend at http://localhost:5173 and check the backend health at http://localhost:8787/health.

```bash
npm install
npm run dev        # starts Vite frontend + backend  
# Frontend: http://localhost:5173  
# Backend:  http://localhost:8787/health  
```

--------------------------------

### Development Setup for LLMRTC Web Client

Source: https://www.llmrtc.org/web-client/installation

Instructions for setting up the development environment, including starting the backend and frontend servers. Assumes a monorepo structure with 'backend' and 'frontend' directories.

```bash
# Terminal 1: Start backend  
cd backend  
npm run dev  
  
# Terminal 2: Start frontend  
cd frontend  
npm run dev  

```

--------------------------------

### LLMRTCWebClient Basic Usage

Source: https://www.llmrtc.org/web-client/overview

Demonstrates the basic setup and usage of the LLMRTCWebClient, including creating a client instance, setting up event handlers, starting the connection, sharing audio, and cleaning up.

```APIDOC
## LLMRTCWebClient Basic Usage

This example shows how to initialize the `LLMRTCWebClient`, subscribe to various events, start the connection, and share audio.

### Method

```javascript
import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';

// Create client
const client = new LLMRTCWebClient({
  signallingUrl: 'wss://your-server.com'
});

// Set up event handlers
client.on('stateChange', (state) => {
  console.log('Connection state:', state);
});

client.on('transcript', (text) => {
  console.log('User said:', text);
});

client.on('llmChunk', (chunk) => {
  process.stdout.write(chunk);
});

client.on('ttsStart', () => {
  console.log('Assistant speaking...');
});

client.on('ttsComplete', () => {
  console.log('Assistant finished.');
});

// Start connection
await client.start();

// Share microphone
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const controller = client.shareAudio(stream);

// Later: cleanup
controller.stop();
await client.close();
```

### Description

This code snippet illustrates the fundamental steps to integrate the `LLMRTCWebClient` into a web application. It covers client instantiation with a signaling URL, event listener setup for connection state, user speech, LLM responses, and audio playback cues. It also demonstrates initiating the connection, capturing and sharing microphone audio, and properly closing the connection and associated resources.
```

--------------------------------

### Initialize and Use LLM Web Client

Source: https://www.llmrtc.org/web-client/overview

Demonstrates basic usage of the LLM Web Client, including client creation, event handler setup, starting the connection, sharing audio, and cleanup.

```javascript
import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';  
  
// Create client  
const client = new LLMRTCWebClient({  
  signallingUrl: 'wss://your-server.com'  
});  
  
// Set up event handlers  
client.on('stateChange', (state) => {  
  console.log('Connection state:', state);  
});  
  
client.on('transcript', (text) => {  
  console.log('User said:', text);  
});  
  
client.on('llmChunk', (chunk) => {  
  process.stdout.write(chunk);  
});  
  
client.on('ttsStart', () => {  
  console.log('Assistant speaking...');  
});  
  
client.on('ttsComplete', () => {  
  console.log('Assistant finished.');  
});  
  
// Start connection  
await client.start();  
  
// Share microphone  
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });  
const controller = client.shareAudio(stream);  
  
// Later: cleanup  
controller.stop();  
await client.close();  

```

--------------------------------

### Install Ollama on macOS

Source: https://www.llmrtc.org/getting-started/local-only-stack

Installs Ollama using Homebrew and starts the Ollama service.

```bash
# Install  
brew install ollama  
  
# Start Ollama service  
ollama serve  
```

--------------------------------

### Install Ollama on Linux

Source: https://www.llmrtc.org/getting-started/local-only-stack

Installs Ollama using a curl script and starts the Ollama service.

```bash
# Install  
curl -fsSL https://ollama.com/install.sh | sh  
  
# Start Ollama service  
ollama serve  
```

--------------------------------

### Basic LLMRTC Server Setup

Source: https://www.llmrtc.org/backend/library

Initialize and start the LLMRTC server with essential LLM, Speech-to-Text, and Text-to-Speech providers. Ensure your API keys are set as environment variables.

```typescript
import { LLMRTCServer } from '@llmrtc/llmrtc-backend';
import { OpenAILLMProvider, OpenAIWhisperProvider, OpenAITTSProvider } from '@llmrtc/llmrtc-backend';

const server = new LLMRTCServer({
  providers: {
    llm: new OpenAILLMProvider({
      apiKey: process.env.OPENAI_API_KEY!,
      model: 'gpt-5.2'
    }),
    stt: new OpenAIWhisperProvider({
      apiKey: process.env.OPENAI_API_KEY!
    }),
    tts: new OpenAITTSProvider({
      apiKey: process.env.OPENAI_API_KEY!,
      voice: 'nova'
    })
  },
  systemPrompt: 'You are a helpful voice assistant.',
  port: 8787
});

await server.start();

```

--------------------------------

### Install Dependencies

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Installs root and client dependencies for the application.

```bash
# Install root dependencies  
npm install  
  
# Install client dependencies  
cd client && npm install && cd ..  

```

--------------------------------

### Create React App Client Setup with LLMRTC Web Client

Source: https://www.llmrtc.org/web-client/installation

Example of setting up the LLMRTCWebClient in a Create React App project. Ensure REACT_APP_SIGNAL_URL is set in your environment.

```typescript
// src/client.ts  
import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';  
  
const client = new LLMRTCWebClient({  
  signallingUrl: process.env.REACT_APP_SIGNAL_URL!  
});  

```

--------------------------------

### Vite Client Setup with LLMRTC Web Client

Source: https://www.llmrtc.org/web-client/installation

Example of setting up the LLMRTCWebClient in a Vite project. Ensure your VITE_SIGNAL_URL environment variable is set.

```typescript
// src/client.ts  
import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';  
  
const client = new LLMRTCWebClient({  
  signallingUrl: import.meta.env.VITE_SIGNAL_URL  
});  

```

--------------------------------

### Setup Docker Compose for Faster-Whisper

Source: https://www.llmrtc.org/providers/local-faster-whisper

Downloads the Docker Compose configuration and starts the Faster-Whisper server using Docker Compose. Supports both GPU and CPU configurations.

```bash
curl -sO https://raw.githubusercontent.com/fedirz/faster-whisper-server/master/compose.yaml  

# For GPU  
docker compose up --detach faster-whisper-server-cuda  

# For CPU  
docker compose up --detach faster-whisper-server-cpu  

```

--------------------------------

### Initialize Project and Install Dependencies

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Set up a new project directory, initialize npm, set the module type, and install the LLMRTC backend package.

```bash
mkdir my-voice-server  
cd my-voice-server  
npm init -y  
npm pkg set type=module  
npm install @llmrtc/llmrtc-backend dotenv  
```

--------------------------------

### Combine and Configure Server Hooks

Source: https://www.llmrtc.org/core-sdk/hooks-and-metrics

Combine various hooks, including timing, connection management, LLM response guarding, and error reporting, for a comprehensive server setup. This example also shows initializing the server with providers and a custom metrics adapter.

```typescript
import {
  LLMRTCServer,
  createTimingHooks,
  type OrchestratorHooks,
  type ServerHooks
} from '@llmrtc/llmrtc-backend';

// Combine hooks
const hooks: OrchestratorHooks & ServerHooks = {
  ...createTimingHooks(),

  onConnection(sessionId, connectionId) {
    console.log(`[${sessionId}] Connected`);
  },

  onDisconnect(sessionId, timing) {
    console.log(`[${sessionId}] Disconnected after ${timing.durationMs}ms`);
  },

  onLLMEnd(ctx, result, timing) {
    // Guardrail: check for banned content
    if (result.fullText.includes('forbidden')) {
      throw new Error('Content blocked');
    }
  },

  onError(error, context) {
    // Report to error tracking
    sentry.captureException(error, { extra: context });
  }
};

const server = new LLMRTCServer({
  providers: { llm, stt, tts },
  hooks,
  metrics: new PrometheusMetrics()
});

```

--------------------------------

### Application Output Example

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Example output when the voice assistant application is running successfully.

```text
  Voice Assistant Server  
  ======================  
  Server:  http://127.0.0.1:8787  
  Client:  http://localhost:5173  
  
  Ready for connections!  


```

--------------------------------

### Start the Application

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Runs both the server and client components of the voice assistant.

```bash
# Run both server and client  
npm run dev  

```

--------------------------------

### Configure LLMRTC Server with Separate Vision Provider

Source: https://www.llmrtc.org/getting-started/local-only-stack

Example of initializing the LLMRTCServer with a text-only LLM and a dedicated LlavaVisionProvider. This setup is an alternative to native vision integration.

```javascript
import {
  LLMRTCServer,
  OllamaLLMProvider,
  LlavaVisionProvider,
  FasterWhisperProvider,
  PiperTTSProvider
} from '@llmrtc/llmrtc-backend';

const server = new LLMRTCServer({
  providers: {
    llm: new OllamaLLMProvider({
      baseUrl: 'http://localhost:11434',
      model: 'llama3.2'  // Text-only model
    }),
    stt: new FasterWhisperProvider({
      baseUrl: 'http://localhost:9000'
    }),
    tts: new PiperTTSProvider({
      baseUrl: 'http://localhost:5002'
    }),
    vision: new LlavaVisionProvider({
      baseUrl: 'http://localhost:11434',
      model: 'llava'
    })
  },
  systemPrompt: 'You are a helpful assistant that can see and hear.'
});
```

--------------------------------

### Start Ollama Server

Source: https://www.llmrtc.org/providers/local-llava

Command to start the Ollama server locally.

```bash
ollama serve  
```

--------------------------------

### Install Ollama

Source: https://www.llmrtc.org/providers/local-llava

Instructions for installing Ollama on macOS and Linux.

```bash
# macOS  
brew install ollama  
  
# Linux  
curl -fsSL https://ollama.com/install.sh | sh  
```

--------------------------------

### Create and Configure Project

Source: https://www.llmrtc.org/getting-started/web-client-quickstart

Sets up a new React project with Vite, installs the LLM RTC Web Client, and navigates into the project directory.

```bash
npm create vite@latest my-voice-client -- --template react-ts
cd my-voice-client
npm install @llmrtc/llmrtc-web-client

```

--------------------------------

### Start Ollama Server

Source: https://www.llmrtc.org/providers/local-ollama

Command to start the Ollama server process.

```bash
ollama serve
```

--------------------------------

### OpenAI-Only Stack Setup

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Use this snippet for the simplest setup using only OpenAI for LLM, Speech-to-Text, and Text-to-Speech.

```typescript
import {
  LLMRTCServer,
  OpenAILLMProvider,
  OpenAIWhisperProvider,
  OpenAITTSProvider
} from '@llmrtc/llmrtc-backend';

const server = new LLMRTCServer({
  providers: {
    llm: new OpenAILLMProvider({ apiKey: process.env.OPENAI_API_KEY }),
    stt: new OpenAIWhisperProvider({ apiKey: process.env.OPENAI_API_KEY }),
    tts: new OpenAITTSProvider({ apiKey: process.env.OPENAI_API_KEY })
  },
  systemPrompt: 'You are a helpful assistant.'
});

```

--------------------------------

### Run Development Server

Source: https://www.llmrtc.org/getting-started/web-client-quickstart

Starts the development server for the React application, allowing you to view the voice assistant in your browser.

```bash
npm run dev

```

--------------------------------

### Install LLM RTC Core SDK

Source: https://www.llmrtc.org/core-sdk/overview

Install the core SDK package using npm.

```bash
npm install @llmrtc/llmrtc-core  

```

--------------------------------

### Install Project Dependencies

Source: https://www.llmrtc.org/meta/contributing

Run this command at the repository root to install all necessary dependencies for the monorepo workspaces.

```bash
npm install
```

--------------------------------

### Install Ollama via Script (Linux)

Source: https://www.llmrtc.org/providers/local-ollama

Execute this script to install Ollama on Linux systems.

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

--------------------------------

### Server-side Setup (server.ts)

Source: https://www.llmrtc.org/tutorials/add-vision

Initializes and configures the LLMRTC server with necessary providers and settings.

```typescript
import { config } from 'dotenv';
config();

import {
  LLMRTCServer,
  AnthropicLLMProvider,
  OpenAIWhisperProvider,
  OpenAITTSProvider
} from '@llmrtc/llmrtc-backend';

if (!process.env.OPENAI_API_KEY) {
  console.error('Error: OPENAI_API_KEY is required');
  process.exit(1);
}
if (!process.env.ANTHROPIC_API_KEY) {
  console.error('Error: ANTHROPIC_API_KEY is required');
  process.exit(1);
}

const server = new LLMRTCServer({
  providers: {
    llm: new AnthropicLLMProvider({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: 'claude-sonnet-4-20250514'
    }),
    stt: new OpenAIWhisperProvider({
      apiKey: process.env.OPENAI_API_KEY
    }),
    tts: new OpenAITTSProvider({
      apiKey: process.env.OPENAI_API_KEY,
      voice: 'nova'
    })
  },
  port: 8787,
  streamingTTS: true,
  systemPrompt: `You are a helpful voice assistant. Keep your responses concise
and conversational - typically 1-2 sentences. Speak naturally as if having
a real conversation.`
});

server.on('listening', ({ host, port }) => {
  console.log(`Server running at http://${host}:${port}`);
});

await server.start();

```

--------------------------------

### Run Ollama Server with Docker

Source: https://www.llmrtc.org/providers/local-ollama

Start the Ollama server using Docker. Includes options for GPU support.

```bash
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

```bash
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
```

--------------------------------

### Install FFmpeg on Windows

Source: https://www.llmrtc.org/getting-started/installation

Install FFmpeg on Windows using Chocolatey. Alternatively, download from ffmpeg.org.

```bash
choco install ffmpeg  

```

--------------------------------

### Verify FFmpeg Installation

Source: https://www.llmrtc.org/getting-started/installation

Run this command to verify that FFmpeg has been installed correctly and check its version.

```bash
ffmpeg -version  
# Should show version info  

```

--------------------------------

### Create New LLMRTC Project

Source: https://www.llmrtc.org/getting-started/installation

Set up a new project directory, initialize package.json, set module type, and install the backend package.

```bash
# Create directory  
mkdir my-voice-assistant  
cd my-voice-assistant  
  
# Initialize package.json  
npm init -y  
  
# Set module type for ES imports  
npm pkg set type=module  
  
# Install packages  
npm install @llmrtc/llmrtc-backend  

```

--------------------------------

### Verify Prerequisites

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Check if Node.js and FFmpeg are installed and meet the version requirements.

```bash
node --version   # v20.x.x or higher  
ffmpeg -version  # FFmpeg 4.x or higher  
```

--------------------------------

### Initialize LMStudioLLMProvider

Source: https://www.llmrtc.org/providers/lmstudio

Instantiate the LMStudioLLMProvider with the base URL of the LMStudio server and the desired model. Ensure the model is loaded in LMStudio before starting.

```javascript
import { LMStudioLLMProvider } from '@llmrtc/llmrtc-provider-lmstudio';  
  
const llm = new LMStudioLLMProvider({
  baseUrl: 'http://localhost:1234/v1',
  model: 'llama-3.2-3b'
});
```

--------------------------------

### Install LLMRTC Web Client with npm

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC web client package using npm.

```bash
npm install @llmrtc/llmrtc-web-client  

```

--------------------------------

### Install FFmpeg

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Install FFmpeg, a required dependency for streaming TTS, on macOS or Ubuntu.

```bash
# macOS
brew install ffmpeg

# Ubuntu
sudo apt install ffmpeg

# Verify
ffmpeg -version

```

--------------------------------

### start()

Source: https://www.llmrtc.org/web-client/connection-lifecycle

Initiates the connection process for the LLMRTCWebClient. This method transitions the client from the 'disconnected' state to 'connecting' and subsequently to 'connected' upon successful establishment of the connection.

```APIDOC
## start()

### Description
Initiates the connection to the signaling server and begins the WebRTC handshake. This method should be called after client initialization.

### Method
`await client.start()`

### Usage
```javascript
const client = new LLMRTCWebClient({ signallingUrl: 'ws://localhost:8787' });
await client.start();
// Expected state transition: connecting -> connected
```
```

--------------------------------

### Install LLMRTC Backend with yarn

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC backend package using yarn.

```bash
yarn add @llmrtc/llmrtc-backend  

```

--------------------------------

### Tool Description Examples

Source: https://www.llmrtc.org/getting-started/tool-calling-quickstart

Illustrates good and bad examples of tool descriptions to help LLMs understand when to use a tool effectively.

```javascript
// Good: Specific and helpful
description: 'Get the current weather forecast including temperature, conditions, and humidity for any city worldwide'

// Bad: Vague
description: 'Weather stuff'
```

--------------------------------

### Install LLMRTC Web Client with yarn

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC web client package using yarn.

```bash
yarn add @llmrtc/llmrtc-web-client  

```

--------------------------------

### Run the Server

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Execute the server script using Node.js to start the LLMRTC voice server.

```bash
node server.js  
```

--------------------------------

### Install LLMRTC Backend with npm

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC backend package using npm.

```bash
npm install @llmrtc/llmrtc-backend  

```

--------------------------------

### REST API Usage Example

Source: https://www.llmrtc.org/providers/local-llava

Example of sending a request to the Ollama API to generate content based on an image.

```bash
curl http://localhost:11434/api/generate -d '{  
  "model": "llava",  
  "prompt": "Describe what you see",  
  "images": ["<base64-encoded-image>"]  
}'  
```

--------------------------------

### Install and Run Faster-Whisper Server with pip

Source: https://www.llmrtc.org/providers/local-faster-whisper

Installs the faster-whisper-server package using pip and then runs the server. Useful for development or environments where Docker is not preferred.

```bash
pip install faster-whisper-server  
faster-whisper-server --host 0.0.0.0 --port 8000  

```

--------------------------------

### Upgrade Node.js with nvm

Source: https://www.llmrtc.org/getting-started/installation

Install and switch to Node.js version 20 using nvm. Ensure nvm is installed first.

```bash
# Install nvm if needed: https://github.com/nvm-sh/nvm  
nvm install 20  
nvm use 20  

```

--------------------------------

### OpenAI Provider Setup

Source: https://www.llmrtc.org/providers/openai

Initialize the OpenAI LLM, STT, and TTS providers. Ensure your OPENAI_API_KEY is set in your environment variables.

```typescript
import { OpenAILLMProvider, OpenAIWhisperProvider, OpenAITTSProvider } from '@llmrtc/llmrtc-provider-openai';  

const llm = new OpenAILLMProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'gpt-5.2' });  
const stt = new OpenAIWhisperProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'whisper-1' });  
const tts = new OpenAITTSProvider({ apiKey: process.env.OPENAI_API_KEY, model: 'tts-1', voice: 'nova' });  
```

--------------------------------

### Install LLMRTC Backend with pnpm

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC backend package using pnpm.

```bash
pnpm add @llmrtc/llmrtc-backend  

```

--------------------------------

### Initialize and Start LLM-RTC Web Client

Source: https://www.llmrtc.org/web-client/installation

Instantiate the LLMRTCWebClient with your signaling server URL and start the connection. This snippet also includes basic event listeners for state changes and errors.

```javascript
import { LLMRTCWebClient } from '@llmrtc/llmrtc-web-client';  
  
const client = new LLMRTCWebClient({  
  signallingUrl: 'wss://your-server.com'  
});  
  
client.on('stateChange', (state) => {  
  console.log('State:', state);  
});  
  
client.on('error', (error) => {  
  console.error('Error:', error);  
});  
  
try {  
  await client.start();  
  console.log('Connected successfully!');  
} catch (error) {  
  console.error('Connection failed:', error);  
}  
```

--------------------------------

### Install LLMRTC Web Client with pnpm

Source: https://www.llmrtc.org/getting-started/installation

Use this command to install the LLMRTC web client package using pnpm.

```bash
pnpm add @llmrtc/llmrtc-web-client  

```

--------------------------------

### JavaScript/TypeScript Usage Example

Source: https://www.llmrtc.org/providers/local-llava

Example of using the Ollama library to send a chat request with an image.

```javascript
import Ollama from 'ollama';  
  
const response = await Ollama.chat({  
  model: 'llava',  
  messages: [{  
    role: 'user',  
    content: 'What do you see in this image?',  
    images: ['./photo.jpg']  // or base64 encoded  
  }]  
});  
  
console.log(response.message.content);  
```

--------------------------------

### Install FFmpeg on Ubuntu/Debian

Source: https://www.llmrtc.org/getting-started/installation

Install FFmpeg on Ubuntu or Debian systems using apt. This is required for streaming TTS.

```bash
sudo apt update  
sudo apt install ffmpeg  

```

--------------------------------

### Install and Run Faster-Whisper Server with pip

Source: https://www.llmrtc.org/getting-started/local-only-stack

Installs the Faster-Whisper server using pip and runs it on host 0.0.0.0 and port 9000.

```bash
# Install  
pip install faster-whisper-server  
  
# Run  
faster-whisper-server --host 0.0.0.0 --port 9000  
```

--------------------------------

### Verify Ollama Installation

Source: https://www.llmrtc.org/getting-started/local-only-stack

Tests a downloaded Ollama model with a sample prompt and checks the Ollama API.

```bash
# Test the model  
ollama run llama3.2 "Hello, how are you?"  
  
# Check API  
curl http://localhost:11434/api/tags  
```

--------------------------------

### Manual Installation and Run of Piper Server

Source: https://www.llmrtc.org/getting-started/local-only-stack

Downloads Piper, a voice model, and installs piper-http-server to run the TTS service locally.

```bash
# Download Piper  
wget https://github.com/rhasspy/piper/releases/latest/download/piper_linux_x86_64.tar.gz  
tar -xzf piper_linux_x86_64.tar.gz  
cd piper  
  
# Download a voice model  
wget https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/amy/medium/en_US-amy-medium.onnx  
wget https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/amy/medium/en_US-amy-medium.onnx.json  
  
# Run HTTP server (requires piper-http-server)  
pip install piper-http-server  
piper-http-server --port 5002 --model en_US-amy-medium.onnx  
```

--------------------------------

### Server Setup with LLMRTC

Source: https://www.llmrtc.org/tutorials/voice-with-tools

Initializes and configures the LLMRTC server with LLM, STT, and TTS providers, setting up API keys and port.

```typescript
import { config } from 'dotenv';
config();

import {
  LLMRTCServer,
  AnthropicLLMProvider,
  OpenAIWhisperProvider,
  OpenAITTSProvider
} from '@llmrtc/llmrtc-backend';

if (!process.env.OPENAI_API_KEY) {
  console.error('Error: OPENAI_API_KEY is required');
  process.exit(1);
}
if (!process.env.ANTHROPIC_API_KEY) {
  console.error('Error: ANTHROPIC_API_KEY is required');
  process.exit(1);
}

const server = new LLMRTCServer({
  providers: {
    llm: new AnthropicLLMProvider({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: 'claude-sonnet-4-20250514'
    }),
    stt: new OpenAIWhisperProvider({
      apiKey: process.env.OPENAI_API_KEY
    }),
    tts: new OpenAITTSProvider({
      apiKey: process.env.OPENAI_API_KEY,
      voice: 'nova'
    })
  },
  port: 8787,
  streamingTTS: true,
  systemPrompt: `You are a helpful voice assistant. Keep responses concise.`
});

server.on('listening', ({ host, port }) => {
  console.log(`Server running at http://${host}:${port}`);
});

await server.start();

```

--------------------------------

### Install FFmpeg for Different Operating Systems

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Install FFmpeg, a required tool for processing streaming TTS audio. Use the appropriate command for your operating system (macOS, Ubuntu/Debian, or Windows).

```bash
# Install FFmpeg
# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt-get install ffmpeg

# Windows (with chocolatey)
choco install ffmpeg

```

--------------------------------

### AWS Bedrock Provider Setup

Source: https://www.llmrtc.org/providers/aws-bedrock

Instantiate the BedrockLLMProvider with AWS credentials and specify the desired model. Ensure AWS environment variables are configured.

```typescript
import { BedrockLLMProvider } from '@llmrtc/llmrtc-provider-bedrock';  
  
const llm = new BedrockLLMProvider({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
  },
  model: 'anthropic.claude-sonnet-4-20250514-v1:0'
});

```

--------------------------------

### Custom Routes Example

Source: https://www.llmrtc.org/backend/library

Demonstrates how to use `getApp()` to access the underlying Express application and define custom HTTP endpoints for the LLMRTC server.

```APIDOC
## Custom Routes

Access the Express app to add custom HTTP endpoints:
```
const server = new LLMRTCServer({ /* config */ });
const app = server.getApp();

// Add REST endpoints
app.get('/api/sessions', (req, res) => {
  // Return session data
  res.json({ sessions: [] });
});

app.post('/api/sessions/:id/message', async (req, res) => {
  const { id } = req.params;
  const { text } = req.body;
  // Inject text message into session
  res.json({ ok: true });
});

await server.start();
```
```

--------------------------------

### Verify FFmpeg Installation and PATH

Source: https://www.llmrtc.org/getting-started/installation

Check if FFmpeg is installed and accessible in your system's PATH. Add it if necessary for your operating system.

```bash
# Check if ffmpeg is accessible  
which ffmpeg
```

```bash
# Add to PATH if needed (macOS/Linux)  
export PATH="/usr/local/bin:$PATH"
```

--------------------------------

### Create Project Directory and Navigate

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Set up the project structure by creating a new directory for your voice assistant and navigating into it.

```bash
mkdir voice-assistant
cd voice-assistant

```

--------------------------------

### Start Client Connection

Source: https://www.llmrtc.org/web-client/connection-lifecycle

Initiate the connection process for the LLMRTC client. The state transitions from 'connecting' to 'connected' upon successful connection.

```javascript
const client = new LLMRTCWebClient({ signallingUrl: 'ws://localhost:8787' });
await client.start();
// State: connecting → connected

```

--------------------------------

### OpenAI Stack Configuration

Source: https://www.llmrtc.org/backend/cli

Configure and start the LLMRTC backend using OpenAI for all services. Ensure your OpenAI API key is set.

```bash
export OPENAI_API_KEY=sk-...  
export SYSTEM_PROMPT="You are a helpful voice assistant."  
npx llmrtc-backend  

```

--------------------------------

### Server Setup and Event Handling

Source: https://www.llmrtc.org/tutorials/voice-with-tools

Sets up a server to handle client connections and disconnections, logging events with timestamps. This is essential for managing real-time communication in voice applications.

```typescript
import { Server } from "@huggingface/inference";

const server = new Server({
  // ... server options
});

server.on('connection', async ({ id }) => {
  console.log(`[${new Date().toLocaleTimeString()}] Client connected: ${id}`);
});

server.on('disconnect', ({ id }) => {
  console.log(`[${new Date().toLocaleTimeString()}] Client disconnected: ${id}`);
});

await server.start();
```

--------------------------------

### Create Client Directory

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Use this command to create the client directory and its source subdirectory.

```bash
mkdir -p client/src
```

--------------------------------

### Test WebSocket Connection

Source: https://www.llmrtc.org/getting-started/backend-quickstart

Install wscat and connect to the server's WebSocket endpoint to test incoming connections and message handling.

```bash
# Install wscat if needed  
npm install -g wscat  
  
# Connect to server  
wscat -c ws://localhost:8787  
```

--------------------------------

### Register Tools with ToolRegistry

Source: https://www.llmrtc.org/getting-started/tool-calling-quickstart

Create a `ToolRegistry` instance and use its `register` method to add custom tools. This example shows registering the previously defined weather tool and a new tool for getting the current time.

```typescript
import { ToolRegistry, defineTool } from '@llmrtc/llmrtc-core';

// Create registry
const registry = new ToolRegistry();

// Register weather tool
registry.register(getWeatherTool);

// Register more tools
registry.register(defineTool({
  name: 'get_time',
  description: 'Get the current time in a timezone',
  parameters: {
    type: 'object',
    properties: {
      timezone: {
        type: 'string',
        description: 'Timezone, e.g., "America/New_York" or "Asia/Tokyo"'
      }
    },
    required: ['timezone']
  }
}, async ({ timezone }) => {
  const time = new Date().toLocaleString('en-US', { timeZone: timezone });
  return { timezone, time };
}));

```

--------------------------------

### Start Camera Capture

Source: https://www.llmrtc.org/web-client/video-and-vision

Initiates camera capture with specified resolution and facing mode. The captured stream is shared with the client at the default interval.

```javascript
async function startCamera() {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      width: { ideal: 1280 },
      height: { ideal: 720 },
      facingMode: 'user'
    }
  });

  // Share with default interval (1000ms between frames)
  const controller = client.shareVideo(stream);

  return controller;
}

```

--------------------------------

### Start Docker Services and Pull Model

Source: https://www.llmrtc.org/getting-started/local-only-stack

Commands to start all services defined in docker-compose.yml in detached mode, pull an Ollama model, and start the LLMRTC server.

```bash
docker-compose up -d

# Pull Ollama model
docker exec -it ollama ollama pull llama3.2

# Start LLMRTC server
node server.js
```

--------------------------------

### LLMRTCServer with Custom ICE Servers

Source: https://www.llmrtc.org/backend/configuration

Configure the LLMRTCServer with custom STUN/TURN servers for NAT traversal. Includes examples for both STUN and TURN with authentication.

```typescript
const server = new LLMRTCServer({
  providers: { llm, stt, tts },
  iceServers: [
    { urls: 'stun:stun.example.com:3478' },
    {
      urls: 'turn:turn.example.com:3478',
      username: 'user',
      credential: 'pass'
    }
  ]
});
```

--------------------------------

### Basic Voice Playbook Setup with Tools

Source: https://www.llmrtc.org/backend/voice-playbook

Defines tools, a playbook structure with stages and transitions, and initializes the LLM/RTC server. Ensure LLM, STT, and TTS providers are correctly configured.

```typescript
import {
  LLMRTCServer,
  ToolRegistry,
  defineTool
} from '@llmrtc/llmrtc-backend';
import type { Playbook } from '@llmrtc/llmrtc-core';

// Define tools
const tools = new ToolRegistry();

tools.register(defineTool({
  name: 'check_balance',
  description: 'Check account balance',
  parameters: {
    type: 'object',
    properties: {
      accountId: { type: 'string' }
    },
    required: ['accountId']
  }
}, async ({ accountId }) => {
  // Implementation
  return { balance: 1500.00 };
}));

// Define playbook
const playbook: Playbook = {
  id: 'support-bot',
  initialStage: 'greeting',
  stages: [
    {
      id: 'greeting',
      name: 'Greeting',
      systemPrompt: 'Greet the customer and ask how you can help.'
    },
    {
      id: 'billing',
      name: 'Billing Support',
      systemPrompt: 'Help with billing inquiries. Use check_balance for account info.',
      tools: [tools.getTool('check_balance').definition]
    },
    {
      id: 'technical',
      name: 'Technical Support',
      systemPrompt: 'Help with technical issues.'
    },
    {
      id: 'farewell',
      name: 'Farewell',
      systemPrompt: 'Thank the customer and end the conversation.'
    }
  ],
  transitions: [
    { id: 't1', from: 'greeting', condition: { type: 'keyword', keywords: ['billing', 'payment', 'invoice'] }, action: { targetStage: 'billing' } },
    { id: 't2', from: 'greeting', condition: { type: 'keyword', keywords: ['technical', 'bug', 'error'] }, action: { targetStage: 'technical' } },
    { id: 't3', from: 'billing', condition: { type: 'llm_decision' }, action: { targetStage: 'farewell' } },
    { id: 't4', from: 'technical', condition: { type: 'llm_decision' }, action: { targetStage: 'farewell' } }
  ]
};

// Create server with playbook
const server = new LLMRTCServer({
  providers: { llm, stt, tts },
  playbook,
  toolRegistry: tools,
  playbookOptions: {
    maxToolCallsPerTurn: 10,
    phase1TimeoutMs: 60000,
    debug: false
  }
});

await server.start();

```

--------------------------------

### Install Workspace Dependencies in LLMRTC Monorepo

Source: https://www.llmrtc.org/getting-started/installation

Install all dependencies for all workspaces within the LLMRTC monorepo.

```bash
# Install all workspace dependencies  
npm install  
  
# Build all packages  
npm run build  

```

--------------------------------

### Install FFmpeg on macOS

Source: https://www.llmrtc.org/getting-started/installation

Install FFmpeg on macOS using Homebrew. This is required for streaming TTS.

```bash
# Using Homebrew  
brew install ffmpeg  

```

--------------------------------

### Start Local Services

Source: https://www.llmrtc.org/getting-started/local-only-stack

Commands to start the Ollama service and the Faster-Whisper Docker container in separate terminals.

```bash
# Terminal 1: Ollama  
ollama serve  
  
# Terminal 2: Faster-Whisper  
docker start faster-whisper  
```

--------------------------------

### Run Faster-Whisper Server (GPU)

Source: https://www.llmrtc.org/providers/local-faster-whisper

Starts the Faster-Whisper server with GPU acceleration using Docker. Requires NVIDIA Container Toolkit. Mounts the Hugging Face cache.

```bash
docker run -d \
  --gpus all \
  --name faster-whisper \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  fedirz/faster-whisper-server:latest-cuda  

```

--------------------------------

### Define a Support Playbook

Source: https://www.llmrtc.org/core-sdk/playbooks

This example defines a 'support' playbook with multiple stages, system prompts, and tool definitions. It includes transitions based on keywords, tool calls, and LLM decisions.

```typescript
import { Playbook } from '@llmrtc/llmrtc-core';

export const supportPlaybook: Playbook = {
  id: 'support',
  initialStage: 'greeting',
  globalSystemPrompt: 'You are a concise support agent.',

  stages: [
    { id: 'greeting', name: 'Greeting', systemPrompt: 'Greet and ask the issue.' },
    { id: 'triage', name: 'Triage', systemPrompt: 'Clarify the issue, collect order id.', tools: [lookupOrder.definition] },
    { id: 'resolution', name: 'Resolution', systemPrompt: 'Resolve or escalate.' },
    { id: 'farewell', name: 'Farewell', systemPrompt: 'Close politely.' }
  ],

  transitions: [
    { id: 'start-triage', from: 'greeting', condition: { type: 'keyword', keywords: ['order', 'refund', 'broken'] }, action: { targetStage: 'triage' } },
    { id: 'resolved', from: 'triage', condition: { type: 'tool_call', toolName: 'lookup_order' }, action: { targetStage: 'resolution' } },
    { id: 'wrapup', from: '*', condition: { type: 'llm_decision' }, action: { targetStage: 'farewell' } }
  ]
};

```

--------------------------------

### Configure LLMRTC Server with Native Vision

Source: https://www.llmrtc.org/getting-started/local-only-stack

Example of initializing the LLMRTCServer using OllamaLLMProvider with a vision-capable model like 'gemma3'. No separate vision provider is needed.

```javascript
import {
  LLMRTCServer,
  OllamaLLMProvider,
  FasterWhisperProvider,
  PiperTTSProvider
} from '@llmrtc/llmrtc-backend';

const server = new LLMRTCServer({
  providers: {
    llm: new OllamaLLMProvider({
      baseUrl: 'http://localhost:11434',
      model: 'gemma3'  // Vision-capable model handles images natively
    }),
    stt: new FasterWhisperProvider({
      baseUrl: 'http://localhost:9000'
    }),
    tts: new PiperTTSProvider({
      baseUrl: 'http://localhost:5002'
    })
    // No separate vision provider needed!
  },
  systemPrompt: 'You are a helpful assistant that can see and hear.'
});
```

--------------------------------

### Example Error Message

Source: https://www.llmrtc.org/protocol/error-codes

An example of a JSON object representing an error, specifically indicating an LLM timeout.

```json
{
  "type": "error",
  "code": "LLM_TIMEOUT",
  "message": "LLM response exceeded 30 second timeout"
}

```

--------------------------------

### Speech Start Detection Message

Source: https://www.llmrtc.org/protocol/message-types

Sent when Voice Activity Detection (VAD) detects that the user has started speaking.

```json
{
  type: 'speech-start'
}
```

--------------------------------

### Complete server.ts with Tools

Source: https://www.llmrtc.org/tutorials/voice-with-tools

This snippet shows the full server configuration for a voice assistant that can use custom tools. It includes setting up LLM, STT, and TTS providers, defining tools, and configuring the playbook.

```typescript
import { config } from 'dotenv';
config();

import {
  LLMRTCServer,
  AnthropicLLMProvider,
  OpenAIWhisperProvider,
  OpenAITTSProvider,
  defineTool,
  ToolRegistry
} from '@llmrtc/llmrtc-backend';
import type { Playbook, Stage } from '@llmrtc/llmrtc-backend';

// Validate environment
if (!process.env.OPENAI_API_KEY) {
  console.error('Error: OPENAI_API_KEY is required');
  process.exit(1);
}
if (!process.env.ANTHROPIC_API_KEY) {
  console.error('Error: ANTHROPIC_API_KEY is required');
  process.exit(1);
}

// ===========================================
// Define Tools
// ===========================================

const getWeatherTool = defineTool(
  {
    name: 'get_weather',
    description: 'Get the current weather for a city. Use when user asks about weather.',
    parameters: {
      type: 'object',
      properties: {
        city: { type: 'string', description: 'The city name' },
        units: { type: 'string', enum: ['celsius', 'fahrenheit'], description: 'Temperature units' }
      },
      required: ['city']
    }
  },
  async ({ city, units = 'celsius' }) => {
    console.log(`[Tool] get_weather called for ${city}`);

    const mockWeather: Record<string, { temp: number; condition: string }> = {
      'tokyo': { temp: 22, condition: 'Partly cloudy' },
      'new york': { temp: 18, condition: 'Sunny' },
      'london': { temp: 14, condition: 'Rainy' },
      'paris': { temp: 16, condition: 'Cloudy' },
      'sydney': { temp: 25, condition: 'Sunny' }
    };

    const data = mockWeather[city.toLowerCase()] || { temp: 20, condition: 'Clear' };
    const temp = units === 'fahrenheit' ? Math.round(data.temp * 9/5 + 32) : data.temp;

    return { city, temperature: temp, units, condition: data.condition };
  }
);

const getTimeTool = defineTool(
  {
    name: 'get_time',
    description: 'Get the current time. Use when user asks what time it is.',
    parameters: {
      type: 'object',
      properties: {},
      required: []
    }
  },
  async () => {
    console.log('[Tool] get_time called');
    const now = new Date();
    return {
      time: now.toLocaleTimeString('en-US', { hour: 'numeric', minute: '2-digit' }),
      date: now.toLocaleDateString('en-US', { weekday: 'long', month: 'long', day: 'numeric' })
    };
  }
);

// ===========================================
// Define Playbook
// ===========================================

const assistantStage: Stage = {
  id: 'assistant',
  name: 'Voice Assistant',
  description: 'A helpful assistant with tools',
  systemPrompt: `You are a helpful voice assistant with access to tools.  

Available tools:  
- get_weather: Use when asked about weather in any city  
- get_time: Use when asked about current time or date  

After using a tool, incorporate the result naturally into your response.  
Keep all responses concise and conversational - this is a voice interface.`,
  tools: [getWeatherTool.definition, getTimeTool.definition],
  toolChoice: 'auto',
  twoPhaseExecution: true
};

const playbook: Playbook = {
  id: 'voice-assistant',
  name: 'Voice Assistant with Tools',
  version: '1.0.0',
  stages: [assistantStage],
  transitions: [],
  initialStage: 'assistant'
};

// ===========================================
// Create Tool Registry
// ===========================================

const toolRegistry = new ToolRegistry();
toolRegistry.register(getWeatherTool);
toolRegistry.register(getTimeTool);

// ===========================================
// Create Server
// ===========================================

const server = new LLMRTCServer({
  providers: {
    llm: new AnthropicLLMProvider({
      apiKey: process.env.ANTHROPIC_API_KEY,
      model: 'claude-sonnet-4-20250514'
    }),
    stt: new OpenAIWhisperProvider({
      apiKey: process.env.OPENAI_API_KEY
    }),
    tts: new OpenAITTSProvider({
      apiKey: process.env.OPENAI_API_KEY,
      voice: 'nova'
    })
  },
  playbook,
  toolRegistry,
  port: 8787,
  streamingTTS: true
});

// ===========================================
// Server Events
// ===========================================

server.on('listening', ({ host, port }) => {
  console.log('');
  console.log('  Voice Assistant with Tools');
  console.log('  ==========================');
  console.log(`  Server:  http://${host}:${port}`);
  console.log(`  Client:  http://localhost:5173`);
  console.log('');
  console.log('  Try saying:');
  console.log('  - 
```

--------------------------------

### Install Ollama via Homebrew (macOS)

Source: https://www.llmrtc.org/providers/local-ollama

Use this command to install Ollama on macOS using the Homebrew package manager.

```bash
brew install ollama
```

--------------------------------

### Initialize React App Entry Point

Source: https://www.llmrtc.org/tutorials/build-voice-assistant

Sets up the main entry point for a React application using createRoot. Ensures the App component is rendered within StrictMode for development checks.

```typescript
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import App from './App';

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <App />
  </StrictMode>
);

```

--------------------------------

### Configuring ToolExecutor Options

Source: https://www.llmrtc.org/getting-started/tool-calling-quickstart

Demonstrates how to instantiate a ToolExecutor with custom configurations for timeout, concurrency, argument validation, and monitoring callbacks.

```javascript
import { ToolExecutor } from '@llmrtc/llmrtc-core';

const executor = new ToolExecutor(registry, {
  // Maximum execution time per tool (ms)
  timeout: 30000,

  // How many tools can run concurrently
  maxConcurrency: 5,

  // Validate arguments against JSON Schema
  validateArguments: true,

  // Callbacks for monitoring
  onToolStart: (name, args) => {
    console.log(`Starting ${name}`, args);
  },
  onToolEnd: (name, result, durationMs) => {
    console.log(`${name} completed in ${durationMs}ms`);
  },
  onToolError: (name, error) => {
    console.error(`${name} failed:`, error);
  }
});
```

--------------------------------

### Server Ready Message with Metered TURN

Source: https://www.llmrtc.org/protocol/message-types

An example of the 'ready' message including STUN and TURN server configurations, specifically demonstrating Metered TURN credentials.

```json
{
  "type": "ready",
  "id": "abc123",
  "protocolVersion": 1,
  "iceServers": [
    { "urls": "stun:stun.metered.ca:80" },
    { "urls": "turn:global.relay.metered.ca:80", "username": "abc", "credential": "xyz" },
    { "urls": "turn:global.relay.metered.ca:443?transport=tcp", "username": "abc", "credential": "xyz" }
  ]
}
```

--------------------------------

### TTS Synthesis Start Message

Source: https://www.llmrtc.org/protocol/message-types

Indicates that Text-to-Speech (TTS) synthesis is beginning, sent before the first audio chunk.

```json
{
  type: 'tts-start'
}
```
Tokens:
Raw