### Setup and Run Development Server Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/examples/directToLLMTransports/README.md Install dependencies, start the development server, and configure environment variables. Update the .env file with your API keys. ```bash npm i npm run dev cp env.example .env # update .env with API keys ``` -------------------------------- ### Basic Setup Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Example of how to set up the OpenAIRealTimeWebRTCTransport with basic configuration options. ```APIDOC ## Basic Setup ```javascript import { OpenAIRealTimeWebRTCTransport, OpenAIServiceOptions } from '@pipecat-ai/openai-realtime-webrtc-transport'; const options: OpenAIServiceOptions = { api_key: 'YOUR_API_KEY', session_config: { instructions: 'you are a confused jellyfish', } }; let PipecatConfig: PipecatClientOptions = { transport: new OpenAIRealTimeWebRTCTransport(options), ... }; ``` ``` -------------------------------- ### Installation Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Instructions on how to install the necessary packages for the OpenAI RealTime WebRTC Transport. ```APIDOC ## Installation ```bash copy npm install \ @pipecat-ai/client-js \ @pipecat-ai/openai-realtime-webrtc-transport ``` ``` -------------------------------- ### Install and Build Dependencies Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/examples/directToLLMTransports/README.md Install project dependencies and build the application. Run these commands from the base folder. ```bash # from base folder $ npm i $ npm run build ``` -------------------------------- ### Install Daily Transport Package Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/daily/README.md Install the necessary packages for @pipecat-ai/client-js and the daily-transport. ```bash npm install \ @pipecat-ai/client-js \ @pipecat-ai/daily-transport ``` -------------------------------- ### Install Gemini Live Websocket Transport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/gemini-live-websocket-transport/README.md Install the necessary packages for the Gemini Live Websocket Transport. ```bash npm install \ @pipecat-ai/client-js \ @pipecat-ai/real-time-websocket-transport \ @pipecat-ai/gemini-live-websocket-transport ``` -------------------------------- ### Install and Build Transport Libraries Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/README.md Commands to install project dependencies and build the transport libraries. Run these in your local development environment. ```bash $ npm i $ npm run build ``` -------------------------------- ### Install Websocket Transport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/websocket-transport/README.md Install the Websocket transport package along with the Pipecat client using npm. ```bash npm install \ @pipecat-ai/client-js \ @pipecat-ai/websocket-transport ``` -------------------------------- ### Basic Pipecat Client Setup with SmallWebRTCTransport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/small-webrtc-transport/README.md Initialize the Pipecat client with the SmallWebRTCTransport and connect to the server. Configure microphone and camera settings as needed. ```javascript import { PipecatClient } from "@pipecat-ai/client-js"; import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport"; const pcClient = new PipecatClient({ transport: new SmallWebRTCTransport(), enableCam: false, // Default camera off enableMic: true, // Default microphone on callbacks: { // Event handlers }, }); pcClient.connect({ connection_url: 'https://your.server/offer_endpoint' }); // OR... pcClient.connect({ endpoint: 'https://your-server/connect', // endpoint to return connection_url }); ``` -------------------------------- ### Basic Setup for GeminiLiveWebsocketTransport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/gemini-live-websocket-transport/README.md Set up the GeminiLiveWebsocketTransport with API key and generation configuration. Ensure PipecatClientOptions are also provided. ```javascript import { GeminiLiveWebsocketTransport, GeminiLLMServiceOptions } from '@pipecat-ai/gemini-live-websocket-transport'; const options: GeminiLLMServiceOptions = { api_key: 'YOUR_API_KEY', generation_config: { temperature: 0.7, maxOutput_tokens: 1000 } }; let pcConfig: PipecatClientOptions = { transport: new GeminiLiveWebsocketTransport(options), ... }; ``` -------------------------------- ### Install Pipecat Client JS and WebRTC Transport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/small-webrtc-transport/README.md Install the necessary packages for the Pipecat client and the small WebRTC transport. ```bash npm install \ @pipecat-ai/client-js \ @pipecat-ai/small-webrtc-transport ``` -------------------------------- ### Basic Pipecat Client Setup with DailyTransport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/daily/README.md Initialize the PipecatClient with the DailyTransport, configuring camera and microphone settings. Connect to a Daily.co room using its URL or a server endpoint. ```javascript import { PipecatClient } from "@pipecat-ai/client-js"; import { DailyTransport } from "@pipecat-ai/daily-transport"; const pcClient = new PipecatClient({ transport: new DailyTransport({ dailyFactoryOptions }), enableCam: false, // Default camera off enableMic: true, // Default microphone on callbacks: { // Event handlers }, // ... }); pcClient.connect({ url: 'https://your.daily.co/room' }); // OR... pcClient.connect({ endpoint: 'https://your-server/connect', // endpoint to return url }); ``` -------------------------------- ### Basic Websocket Transport Setup Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/websocket-transport/README.md Initialize the Pipecat client with the WebSocketTransport and connect to a WebSocket endpoint. Ensure to import necessary classes. ```javascript import { PipecatClient } from "@pipecat-ai/client-js"; import { WebSocketTransport } from "@pipecat-ai/websocket-transport"; const pcClient = new PipecatClient({ transport: new WebSocketTransport(), enableMic: true, // Default microphone on callbacks: { // Event handlers }, }); pcClient.connect({ ws_url: 'https://your.websocket/endpoint' }); // OR... pcClient.connect({ endpoint: 'https://your-server/connect', // endpoint to return ws_url }); ``` -------------------------------- ### Install OpenAI Realtime WebRTC Transport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Install the necessary packages for the OpenAI Realtime WebRTC Transport. This includes the client-js library and the specific transport package. ```bash npm install \ @pipecat-ai/client-js \ @pipecat-ai/openai-realtime-webrtc-transport ``` -------------------------------- ### Handling Request Data in startBotAndConnect Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/small-webrtc-transport/CHANGELOG.md Demonstrates how `startBotAndConnect` differentiates request data sent to `start` and `connect` endpoints. The `start` endpoint receives the full `requestData` object, while the `connect` endpoint only receives the `body` property. ```javascript requestData: { createDailyRoom: false, enableDefaultIceServers: true, body: { character_id: characterId } } ``` -------------------------------- ### Updating Session Configuration Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Example of how to dynamically update the session configuration for the transport after initialization. ```APIDOC ## Updating Session Configuration ```javascript pcClient.transport.updateSessionConfig({ instructions: 'you are a an over-sharing neighbor', input_audio_noise_reduction: { type: 'near_field' } }); ``` ``` -------------------------------- ### Basic OpenAI RealTime WebRTC Transport Setup Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Set up the OpenAIRealTimeWebRTCTransport with essential options including API key and session configuration. Ensure to replace 'YOUR_API_KEY' with your actual OpenAI API key. ```javascript import { OpenAIRealTimeWebRTCTransport, OpenAIServiceOptions } from '@pipecat-ai/openai-realtime-webrtc-transport'; const options: OpenAIServiceOptions = { api_key: 'YOUR_API_KEY', session_config: { instructions: 'you are a confused jellyfish', } }; let PipecatConfig: PipecatClientOptions = { transport: new OpenAIRealTimeWebRTCTransport(options), ... }; ``` -------------------------------- ### Add Message to Context Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Append a user message to the conversation context. This is typically done at setup time to provide initial context to the model. ```javascript // at setup time... pcClient.appendToContext({ role: "user", content: 'Hello OpenAI!' }); ``` -------------------------------- ### PipecatClient Initialization Options Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/examples/directToLLMTransports/README.md Configure the PipecatClient with transport, microphone, and camera settings. 'enableMic' is set to true for voice input. ```typescript let pcConfig: PipecatClientOptions = { transport, enableMic: true, enableCam: false, }; ``` -------------------------------- ### Enable Local Commitlint Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md Optionally, enable the Husky hook for local commit message validation to ensure compliance with Conventional Commits before pushing. ```bash git config --local hooks.commitlint true ``` -------------------------------- ### Clone the Repository Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md Clone the Pipecat Client Web Transports repository to your local machine to begin contributing. ```bash git clone https://github.com/your-username/pipecat-client-web-transports ``` -------------------------------- ### OpenAI LLM Service Configuration Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/examples/directToLLMTransports/README.md Configure the OpenAI LLM service with your API key. Ensure VITE_DANGEROUS_OPENAI_API_KEY is set in your environment variables. ```typescript const llm_service_options: OpenAIServiceOptions = { api_key: import.meta.env.VITE_DANGEROUS_OPENAI_API_KEY, // ... additional configuration }; ``` -------------------------------- ### Custom Offer URL Template for Small WebRTC Transport Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/small-webrtc-transport/CHANGELOG.md Configure a custom URL template for generating offer URLs. This is useful when the offer endpoint is not at the default `/api/offer` path. ```javascript transport: new SmallWebRTCTransport({ offerUrlTemplate: `${this.baseUrl}/sessions/:sessionId/api/offer` }), ``` -------------------------------- ### Create a New Branch Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md Create a new branch for your contribution to keep your changes organized. ```bash git checkout -b your-branch-name ``` -------------------------------- ### SmallWebRTCTransport Media Flow Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/README.md Illustrates the typical media flow when using a SmallWebRTCTransport between a client and a bot process. This diagram shows how RTVI messages and WebRTC media are exchanged. ```text ┌──────────────────────────────────────────────────┐ │ │ ┌─────────────────────────┐ │ Server ┌─────────┐ │ │ │ │ │Pipecat │ │ │ Client │ RTVI Messages │ │Pipeline │ │ │ │ & │ │ │ │ ┌────────────────────┐ │ WebRTC Media │ ┌────────────────────┐ media │ ┌─────┐ │ │ │ │SmallWebRTCTransport│◄─┼────────────────┼─►│SmallWebRTCTransport┼────────────┼─► STT │ │ │ │ └────────────────────┘ │ │ └───────▲────────────┘ in │ └──┬──┘ │ │ │ │ │ │ │ │ │ │ └─────────────────────────┘ │ │ │ ┌──▼──┐ │ │ │ │ │ │ LLM │ │ │ │ │ │ └──┬──┘ │ │ │ │ │ │ │ │ │ │ │ ┌──▼──┐ │ │ │ │ media │ │ TTS │ │ │ │ └─────────────────────────┼─┴─────┘ │ │ │ out └─────────┘ │ │ │ └──────────────────────────────────────────────────┘ ``` -------------------------------- ### Sending Messages Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Demonstrates how to send a user message to the OpenAI service within the transport. ```APIDOC ## Sending Messages ```javascript // at setup time... pcClient.appendToContext({ role: "user", content: 'Hello OpenAI!' }); ``` ``` -------------------------------- ### Gemini LLM Service Configuration Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/examples/directToLLMTransports/README.md Configure the Gemini LLM service with your API key and desired model. Ensure VITE_DANGEROUS_GEMINI_API_KEY is set in your environment. ```typescript const llm_service_options: GeminiLLMServiceOptions = { api_key: process.env.VITE_DANGEROUS_GEMINI_API_KEY, model: "models/gemini-2.0-flash-exp", // ... additional configuration }; ``` -------------------------------- ### Handling Events Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Information on how to handle events emitted by the transport, referencing the main Pipecat event handlers. ```APIDOC ## Handling Events The transport implements the various [Pipecat event handlers](https://docs.pipecat.ai/client/js/api-reference/callbacks). Check out the docs or samples for more info. ``` -------------------------------- ### OpenAI RealTime WebRTC Transport API Reference Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md This section outlines the available methods, states, and configuration options for the OpenAI RealTime WebRTC Transport. ```APIDOC ## OpenAI RealTime WebRTC Transport API Reference ### Methods - `initialize()`: Set up the transport and establish connection - `sendMessage(message)`: Send a text message - `handleUserAudioStream(data)`: Stream audio data to the model - `disconnectLLM()`: Close the connection - `sendReadyMessage()`: Signal ready state ### States The transport can be in one of the following states: - "disconnected" - "initializing" - "initialized" - "connecting" - "connected" - "ready" - "disconnecting" - "error" ### Configuration Options ```typescript /********************************** * OpenAI-specific types * types and comments below are based on: * gpt-4o-realtime-preview-2024-12-17 **********************************/ type JSONSchema = { [key: string]: any }; export type OpenAIFunctionTool = { type: "function"; name: string; description: string; parameters: JSONSchema; }; export type OpenAIServerVad = { type: "server_vad"; create_response?: boolean; // defaults to true interrupt_response?: boolean; // defaults to true prefix_padding_ms?: number; // defaults to 300ms silence_duration_ms?: number; // defaults to 500ms threshold?: number; // range (0.0, 1.0); defaults to 0.5 }; export type OpenAISemanticVAD = { type: "semantic_vad"; eagerness?: "low" | "medium" | "high" | "auto"; // defaults to "auto", equivalent to "medium" create_response?: boolean; // defaults to true interrupt_response?: boolean; // defaults to true }; export type OpenAISessionConfig = Partial<{ modalities?: string; instructions?: string; voice?: | "alloy" | "ash" | "ballad" | "coral" | "echo" | "sage" | "shimmer" | "verse"; input_audio_noise_reduction?: { type: "near_field" | "far_field"; } | null; // defaults to null/off input_audio_transcription?: { model: "whisper-1" | "gpt-4o-transcribe" | "gpt-4o-mini-transcribe"; language?: string; prompt?: string[] | string; // gpt-4o models take a string } | null; // we default this to gpt-4o-transcribe turn_detection?: OpenAIServerVad | OpenAISemanticVAD | null; // defaults to server_vad temperature?: number; max_tokens?: number | "inf"; tools?: Array; }>; export interface OpenAIServiceOptions { api_key: string; model?: string; initial_messages?: LLMContextMessage[]; settings?: OpenAISessionConfig; } ``` ### Error Handling The transport includes comprehensive error handling for: - Connection failures - WebRTC connection errors - API key validation - Message transmission errors ``` -------------------------------- ### Gemini Live Websocket Transport API Reference Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/gemini-live-websocket-transport/README.md This section details the methods, states, and configuration options available for the Gemini Live Websocket Transport. ```APIDOC ## Gemini Live Websocket Transport API Reference ### Methods - `initialize()`: Set up the transport and establish connection - `sendMessage(message)`: Send a text message - `handleUserAudioStream(data)`: Stream audio data to the model - `disconnectLLM()`: Close the connection - `sendReadyMessage()`: Signal ready state ### States The transport can be in one of the following states: - "disconnected" - "initializing" - "initialized" - "connecting" - "connected" - "ready" - "disconnecting" - "error" ### Configuration Options ```typescript interface GeminiLLMServiceOptions { api_key: string; // Required: Your Gemini API key initial_messages?: Array<{ // Optional: Initial conversation context content: string; role: string; }>; generation_config?: { candidate_count?: number; maxOutput_tokens?: number; temperature?: number; top_p?: number; top_k?: number; presence_penalty?: number; frequency_penalty?: number; response_modalities?: string; speech_config?: { voice_config?: { prebuilt_voice_config?: { voice_name: "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede"; }; }; }; }; } ``` ### Sending Messages ```javascript // at setup time... pcClient.appendToContext({ role: "user", content: 'Hello Gemini!' }); ``` ### Handling Events The transport implements the various [Pipecat event handlers](https://docs.pipecat.ai/client/js/api-reference/callbacks). Check out the docs or samples for more info. ``` -------------------------------- ### Update OpenAI Session Configuration Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Dynamically update the session configuration for the OpenAI transport. This allows for changes to instructions or audio processing settings during runtime. ```javascript pcClient.transport.updateSessionConfig({ instructions: 'you are a an over-sharing neighbor', input_audio_noise_reduction: { type: 'near_field' } }); ``` -------------------------------- ### Push Changes to Forked Repository Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md Push your committed changes from your local branch to your forked repository on GitHub. ```bash git push origin your-branch-name ``` -------------------------------- ### DailyTransport Media Flow Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/README.md Illustrates the typical media flow when using the DailyTransport, showing the connection between a client, the DailyTransport service, and the Pipecat pipeline for STT, LLM, and TTS. ```text ┌────────────────────────────────────────────┐ │ │ ┌───────────────────┐ │ Server ┌─────────┐ │ │ │ │ │Pipecat │ │ │ Client │ RTVI Messages │ │Pipeline │ │ │ │ & │ │ │ │ │ ┌──────────────┐ │ WebRTC Media │ ┌──────────────┐ media │ ┌─────┐ │ │ │ │DailyTransport│◄─┼────────────────┼─►│DailyTransport┼────────────┼─► STT │ │ │ │ └──────────────┘ │ │ └───────▲──────┘ in │ └──┬──┘ │ │ │ │ │ │ │ │ │ │ └───────────────────┘ │ │ │ ┌──▼──┐ │ │ │ │ │ │ LLM │ │ │ │ │ │ └──┬──┘ │ │ │ │ │ │ │ │ │ │ media │ ┌──▼──┐ │ │ │ └───────────────────┼─┴─────┘ │ │ │ out └─────────┘ │ │ │ └────────────────────────────────────────────┘ ``` -------------------------------- ### OpenAI Session Configuration Types Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/openai-realtime-webrtc-transport/README.md Defines TypeScript types for OpenAI's Realtime API session configuration, including function tools, server-side VAD, semantic VAD, and various other settings like voice, noise reduction, and transcription. ```typescript /* ********************************** * OpenAI-specific types * types and comments below are based on: * gpt-4o-realtime-preview-2024-12-17 **********************************/ type JSONSchema = { [key: string]: any }; export type OpenAIFunctionTool = { type: "function"; name: string; description: string; parameters: JSONSchema; }; export type OpenAIServerVad = { type: "server_vad"; create_response?: boolean; // defaults to true interrupt_response?: boolean; // defaults to true prefix_padding_ms?: number; // defaults to 300ms silence_duration_ms?: number; // defaults to 500ms threshold?: number; // range (0.0, 1.0); defaults to 0.5 }; export type OpenAISemanticVAD = { type: "semantic_vad"; eagerness?: "low" | "medium" | "high" | "auto"; // defaults to "auto", equivalent to "medium" create_response?: boolean; // defaults to true interrupt_response?: boolean; // defaults to true }; export type OpenAISessionConfig = Partial<{ modalities?: string; instructions?: string; voice?: | "alloy" | "ash" | "ballad" | "coral" | "echo" | "sage" | "shimmer" | "verse"; input_audio_noise_reduction?: { type: "near_field" | "far_field"; } | null; // defaults to null/off input_audio_transcription?: { model: "whisper-1" | "gpt-4o-transcribe" | "gpt-4o-mini-transcribe"; language?: string; prompt?: string[] | string; // gpt-4o models take a string } | null; // we default this to gpt-4o-transcribe turn_detection?: OpenAIServerVad | OpenAISemanticVAD | null; // defaults to server_vad temperature?: number; max_tokens?: number | "inf"; tools?: Array; }>; export interface OpenAIServiceOptions { api_key: string; model?: string; initial_messages?: LLMContextMessage[]; settings?: OpenAISessionConfig; } ``` -------------------------------- ### Conventional Commits Format Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md Follow the Conventional Commits format for at least one commit per Pull Request. This format helps in automating changelog generation and maintaining commit history. ```bash (): ``` -------------------------------- ### Append Message to Context Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/transports/gemini-live-websocket-transport/README.md Append a user message to the conversation context for the Gemini transport. ```javascript // at setup time... pcClient.appendToContext({ role: "user", content: 'Hello Gemini!' }); ``` -------------------------------- ### Disable Local Commitlint Source: https://github.com/pipecat-ai/pipecat-client-web-transports/blob/main/CONTRIBUTING.md To disable the local commit message validation, unset the Husky hook configuration. ```bash git config --local --unset hooks.commitlint ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.