### Setup Development Environment Source: https://github.com/pipecat-ai/pipecat/blob/main/CLAUDE.md Installs project dependencies and extras for development. Ensure gstreamer and local extras are included. ```bash uv sync --group dev --all-extras --no-extra gstreamer --no-extra local ``` -------------------------------- ### Run a Pipecat Example Source: https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md Execute a specific example script using the uv package manager. ```bash uv run python getting-started/01-say-one-thing.py ``` -------------------------------- ### Run Example with Daily Transport Source: https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md Execute a voice agent example configured to use the Daily transport. ```bash uv run getting-started/06-voice-agent.py -t daily ``` -------------------------------- ### setup Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_stop.md Initializes the strategy with a task manager. ```APIDOC ## setup ### Description Initialize the strategy with the given task manager. ### Parameters #### Parameters - **task_manager** ([BaseTaskManager](pipecat.utils.asyncio.task_manager.md#pipecat.utils.asyncio.task_manager.BaseTaskManager)) - Required - The task manager to be associated with this instance. ``` -------------------------------- ### Setup Pipeline Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.pipeline.pipeline.md Asynchronously sets up the pipeline and all its contained processors using the provided frame processor setup configuration. ```APIDOC ## POST /setup ### Description Set up the pipeline and all contained processors. ### Method POST ### Endpoint /setup ### Parameters #### Request Body - **setup** (FrameProcessorSetup) - Required - Configuration for frame processor setup. ``` -------------------------------- ### AzureLLMService Example Source: https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md Example of an Azure OpenAI compatible LLM service. Demonstrates extending the base OpenAILLMService. ```python from pipecat.services.openai.llm import OpenAILLMService class AzureLLMService(OpenAILLMService): pass ``` -------------------------------- ### Initialize KrispVivaIPUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.krisp_viva_ip_user_turn_start_strategy.md Configure the Krisp VIVA IP User Turn Start Strategy with model path and probability threshold. This strategy is designed to be used with other start strategies for fallback. ```python from pipecat.turns.user_start import KrispVivaIPUserTurnStartStrategy strategies = UserTurnStrategies( start=[ KrispVivaIPUserTurnStartStrategy( model_path="/path/to/ip_model.kef", threshold=0.5, ), TranscriptionUserTurnStartStrategy(), ], ) ``` -------------------------------- ### Install Development Dependencies Source: https://github.com/pipecat-ai/pipecat/blob/main/README.md Installs development and testing dependencies using uv, excluding 'gstreamer' and 'local' extras. ```bash uv sync --group dev --all-extras \ --no-extra gstreamer \ --no-extra local \ ``` -------------------------------- ### Set Up Environment Variables Source: https://github.com/pipecat-ai/pipecat/blob/main/README.md Copies the example environment file to .env for configuration. ```bash cp env.example .env ``` -------------------------------- ### Run Example with Twilio Transport Source: https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md Execute a voice agent example using Twilio, requiring an ngrok tunnel for local development. ```bash ngrok http 7860 ``` ```bash uv run getting-started/06-voice-agent.py -t twilio -x NGROK_HOST_NAME ``` -------------------------------- ### Configure Environment Variables Source: https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md Initialize the environment configuration file from the provided example template. ```bash cp env.example .env # Edit .env with your API keys ``` -------------------------------- ### Install Pre-commit Hooks Source: https://github.com/pipecat-ai/pipecat/blob/main/CLAUDE.md Installs pre-commit hooks to maintain code quality and consistency before commits. ```bash uv run pre-commit install ``` -------------------------------- ### Setup BaseUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.base_user_turn_start_strategy.md Initializes the strategy with a task manager. This method should be called before processing any frames. ```python async setup(task_manager: [BaseTaskManager](pipecat.utils.asyncio.task_manager.md#pipecat.utils.asyncio.task_manager.BaseTaskManager)) ``` -------------------------------- ### Initialize MetricsLogObserver Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.loggers.metrics_log_observer.md Examples showing how to instantiate the observer to log either all metrics or a specific subset. ```default observers = [MetricsLogObserver()] ``` ```default from pipecat.metrics.metrics import LLMUsageMetricsData, TTSUsageMetricsData observers = [ MetricsLogObserver( include_metrics={LLMUsageMetricsData, TTSUsageMetricsData} ) ] ``` -------------------------------- ### Tracing Setup and Configuration Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.utils.md Functions to initialize and verify the availability of the tracing system. ```APIDOC ## Tracing Setup ### Description Functions to manage the lifecycle and availability of tracing within the application. ### Functions - `is_tracing_available()`: Returns a boolean indicating if the tracing system is currently initialized and available. - `setup_tracing()`: Initializes the tracing infrastructure for the application. ``` -------------------------------- ### GrokLLMService Example Source: https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md Example of a Grok LLM service, showing how to override the base class where necessary. Useful for custom API integrations. ```python from pipecat.services.openai.llm import OpenAILLMService class GrokLLMService(OpenAILLMService): pass ``` -------------------------------- ### Extend default user turn start strategies Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_turn_strategies.md Use this pattern to combine custom strategies like wake phrases with the default set of start strategies. ```python start_strategies = [ WakePhraseUserTurnStartStrategy(phrases=["hey pipecat"]), *default_user_turn_start_strategies(), ] ``` -------------------------------- ### Install uv Package Manager Source: https://github.com/pipecat-ai/pipecat/blob/main/README.md Installs the uv package manager using a curl script. Refer to uv documentation for help. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` -------------------------------- ### BaseUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.base_user_turn_start_strategy.md Base class for strategies that determine when a user starts speaking. Subclasses should implement logic to detect the start of a user’s turn. ```APIDOC ## Class: BaseUserTurnStartStrategy ### Description Base class for strategies that determine when a user starts speaking. Subclasses should implement logic to detect the start of a user’s turn. This could be based on voice activity, number of words spoken, or other heuristics. Events triggered by user turn start strategies: - on_push_frame: Indicates the strategy wants to push a frame. - on_broadcast_frame: Indicates the strategy wants to broadcast a frame. - on_user_turn_started: Signals that a user turn has started. ### Methods #### `__init__` Initialize the base user turn start strategy. * **Parameters:** * **enable_interruptions** (bool) - If True, the user aggregator will emit an interruption frame when the user turn starts. Defaults to True. * **enable_user_speaking_frames** (bool) - If True, the user aggregator will emit frames indicating when the user starts speaking, as well as interruption frames. This is enabled by default, but you may want to disable it if another component (e.g., an STT service) is already generating these frames. Defaults to True. * **kwargs** - Additional keyword arguments. #### `setup` Initialize the strategy with the given task manager. * **Parameters:** * **task_manager** (BaseTaskManager) - The task manager to be associated with this instance. #### `cleanup` Cleanup the strategy. #### `reset` Reset the strategy to its initial state. #### `process_frame` Process an incoming frame. Subclasses should override this to implement logic that decides whether the user turn has started. * **Parameters:** * **frame** (Frame) - The frame to be processed. * **Returns:** A ProcessFrameResult indicating the outcome, or None (treated as CONTINUE for backward compatibility). #### `push_frame` Emit on_push_frame to push a frame using the user aggreagtor. * **Parameters:** * **frame** (Frame) - The frame to be pushed. * **direction** (FrameDirection) - What direction the frame should be pushed to. Defaults to FrameDirection.DOWNSTREAM. #### `broadcast_frame` Emit on_broadcast_frame to broadcast a frame using the user aggreagtor. * **Parameters:** * **frame_cls** (type) - The class of the frame to be broadcasted. * **kwargs** - Keyword arguments to be passed to the frame’s constructor. #### `trigger_user_turn_started` Trigger the on_user_turn_started event. #### `trigger_reset_aggregation` Trigger the on_reset_aggregation event. ### Properties #### `task_manager` Returns the configured task manager. ``` -------------------------------- ### KrispVivaIPUserTurnStartStrategy Methods Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.krisp_viva_ip_user_turn_start_strategy.md Core methods for the Krisp VIVA IP User Turn Start Strategy. ```APIDOC ## KrispVivaIPUserTurnStartStrategy Methods ### async cleanup() Release Krisp SDK resources. ### async reset() Reset the strategy to its initial state. ### async process_frame(frame: Frame) -> ProcessFrameResult Process a frame to detect genuine user interruptions. On `VADUserStartedSpeakingFrame`, begins collecting audio. On `InputAudioRawFrame`, feeds audio through the IP model and triggers a user turn if the interruption probability exceeds the threshold. On `VADUserStoppedSpeakingFrame` or `BotStoppedSpeakingFrame`, resets the candidate state. #### Parameters - **frame** (Frame) - Required - The incoming frame. #### Returns - **ProcessFrameResult** - STOP if a genuine interruption was detected, CONTINUE otherwise. ``` -------------------------------- ### Initialize BaseUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.base_user_turn_start_strategy.md Base class for user turn start detection. Subclasses implement logic based on voice activity or word count. Handles event emission for frame pushing, broadcasting, and user turn start signals. ```python pipecat.turns.user_start.base_user_turn_start_strategy.BaseUserTurnStartStrategy(, enable_interruptions: bool = True, enable_user_speaking_frames: bool = True, **kwargs) ``` -------------------------------- ### Initialize MinWords User Turn Start Strategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.md Initializes the MinWords strategy to start a user turn after a minimum number of words are spoken. Optionally uses interim transcriptions for earlier detection. ```python from pipecat.turns.user_start import MinWordsUserTurnStartStrategy strategies = UserTurnStrategies( start=[ MinWordsUserTurnStartStrategy( min_words=5, use_interim=True, ), ], ) ``` -------------------------------- ### SessionContinuationHelper.on_assistant_audio_started Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.services.aws.nova_sonic.md Callback when the assistant starts producing audio. ```APIDOC ## SessionContinuationHelper.on_assistant_audio_started() ### Description Callback when the assistant starts producing audio. ### Method (Not specified in source) ### Endpoint (Not specified in source) ### Parameters (No parameters specified in source) ### Request Example (No request example specified in source) ### Response #### Success Response (200) (No success response specified in source) #### Response Example (No response example specified in source) ``` -------------------------------- ### Customize Network Settings Source: https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md Override default host and port settings when running an example script. ```bash uv run python --host 0.0.0.0 --port 8080 ``` -------------------------------- ### User Turn Start Strategies Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.md Provides default strategies for determining when a user turn begins. ```APIDOC ## Function: default_user_turn_start_strategies() ### Description Returns a list of default strategies used to detect the start of a user turn. ### Returns - (list) - A list of callable strategies. ``` -------------------------------- ### KrispVivaFilter Start Method Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.audio.filters.krisp_viva_filter.md Initializes the Krisp processor with the transport's sample rate. ```APIDOC ## KrispVivaFilter Start ### Description Initializes the Krisp processor with the transport's sample rate. ### Method start ### Endpoint N/A (Method within a class) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters * **sample_rate** (int) - The sample rate of the input transport in Hz. ### Response #### Success Response (200) None #### Response Example None ``` -------------------------------- ### Initialize Service with Settings Source: https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md Demonstrates how to instantiate a service using default settings or by providing specific overrides. ```python # Uses all defaults svc = MyTTSService(api_key="sk-xxx") # Overrides just the voice — access Settings through the service class svc = MyTTSService( api_key="sk-xxx", settings=MyTTSService.Settings(voice="custom-voice"), ) ``` -------------------------------- ### Pipeline with Custom Processors Source: https://context7.com/pipecat-ai/pipecat/llms.txt Example of how to integrate custom FrameProcessors like TranscriptionLogger and TextTransformer into a Pipecat pipeline. This setup includes input/output transport, STT, LLM, TTS, and aggregators. ```python pipeline = Pipeline([ transport.input(), stt, TranscriptionLogger(log_prefix="[USER] "), user_aggregator, llm, TextTransformer(), tts, transport.output(), assistant_aggregator ]) ``` -------------------------------- ### UserTurnController Event Handlers Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_turn_controller.md Example event handlers for UserTurnController. These functions are decorated with @controller.event_handler to register them for specific events like turn start, stop, timeout, and frame operations. ```python @controller.event_handler("on_user_turn_started") async def on_user_turn_started(controller, strategy: BaseUserTurnStartStrategy, params: UserTurnStartedParams): ... ``` ```python @controller.event_handler("on_user_turn_stopped") async def on_user_turn_stopped(controller, strategy: BaseUserTurnStopStrategy, params: UserTurnStoppedParams): ... ``` ```python @controller.event_handler("on_user_turn_stop_timeout") async def on_user_turn_stop_timeout(controller): ... ``` ```python @controller.event_handler("on_push_frame") async def on_push_frame(controller, frame: Frame, direction: FrameDirection): ... ``` ```python @controller.event_handler("on_broadcast_frame") async def on_broadcast_frame(controller, frame_cls: Type[Frame], **kwargs): ... ``` -------------------------------- ### VADController Event Handlers Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.audio.vad.vad_controller.md Example of how to register event handlers for VADController events like speech start, stop, and activity. These handlers are called when specific speech-related events occur. ```python @vad_controller.event_handler("on_speech_started") async def on_speech_started(controller): ... ``` ```python @vad_controller.event_handler("on_speech_stopped") async def on_speech_stopped(controller): ... ``` ```python @vad_controller.event_handler("on_speech_activity") async def on_speech_activity(controller): ... ``` ```python @vad_controller.event_handler("on_push_frame") async def on_push_frame(controller, frame: Frame, direction: FrameDirection): ... ``` ```python @vad_controller.event_handler("on_broadcast_frame") async def on_broadcast_frame(controller, frame_cls: Type[Frame], **kwargs): ... ``` -------------------------------- ### Configure WakePhraseUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.wake_phrase_user_turn_start_strategy.md Examples showing how to initialize the strategy in timeout mode or single activation mode, along with event handler registration. ```python # Timeout mode (default): wake phrase unlocks interaction for 10s strategy = WakePhraseUserTurnStartStrategy( phrases=["hey pipecat", "ok pipecat"], timeout=10.0, ) # Single activation: wake phrase required before every turn strategy = WakePhraseUserTurnStartStrategy( phrases=["hey pipecat"], single_activation=True, ) @strategy.event_handler("on_wake_phrase_detected") async def on_wake_phrase_detected(strategy, phrase): ... @strategy.event_handler("on_wake_phrase_timeout") async def on_wake_phrase_timeout(strategy): ... ``` -------------------------------- ### Install Pipecat AI Claude Plugins Source: https://github.com/pipecat-ai/pipecat/blob/main/README.md Installs the Pipecat AI development skills for Claude Code from the plugin marketplace. ```bash claude plugin marketplace add pipecat-ai/pipecat claude plugin install pipecat-dev@pipecat-dev-skills ``` -------------------------------- ### Configure and Run Pipecat Pipeline Source: https://context7.com/pipecat-ai/pipecat/llms.txt Full setup including service initialization, tool registration, pipeline construction, and event handling. ```python async def main(): # Initialize services stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), settings=CartesiaTTSService.Settings(voice="71a7ad14-091c-4e8e-a314-022ece01c121") ) llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4.1", system_instruction="""You are a helpful voice assistant for scheduling. Keep responses brief and conversational.""" ) ) # Register functions llm.register_function("get_weather", get_weather) llm.register_function("book_appointment", book_appointment) # Speak while processing function calls @llm.event_handler("on_function_calls_started") async def on_function_calls(service, calls): await tts.queue_frame(TTSSpeakFrame("Let me check that for you.")) # Define tools tools = ToolsSchema(standard_tools=[ FunctionSchema( name="get_weather", description="Get weather for a location", properties={"location": {"type": "string"}}, required=["location"] ), FunctionSchema( name="book_appointment", description="Book an appointment", properties={ "date": {"type": "string", "description": "Date in YYYY-MM-DD"}, "time": {"type": "string", "description": "Time in HH:MM"} }, required=["date", "time"] ) ]) # Setup context and aggregators context = LLMContext(tools=tools) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()) ) # Setup transport transport = DailyTransport( room_url=os.getenv("DAILY_ROOM_URL"), token=os.getenv("DAILY_TOKEN"), bot_name="Assistant", params=DailyParams(audio_in_enabled=True, audio_out_enabled=True) ) # Build pipeline pipeline = Pipeline([ transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator ]) # Create task task = PipelineTask( pipeline, params=PipelineParams(enable_metrics=True, enable_usage_metrics=True), idle_timeout_secs=300 ) # Transport events @transport.event_handler("on_client_connected") async def on_connected(transport, client): logger.info("Client connected") context.add_message({"role": "developer", "content": "Greet the user warmly."}) await task.queue_frames([LLMRunFrame()]) @transport.event_handler("on_client_disconnected") async def on_disconnected(transport, client): logger.info("Client disconnected") await task.cancel() # Task events @task.event_handler("on_pipeline_error") async def on_error(task, frame): logger.error(f"Pipeline error: {frame}") # Run runner = PipelineRunner(handle_sigint=True) await runner.run(task) if __name__ == "__main__": asyncio.run(main()) ``` -------------------------------- ### LLMAutoContextSummarizationConfig Example Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.utils.context.llm_context_summarization.md Configuration for automatic context summarization, using nested summary settings. This is the recommended approach for new implementations. ```python LLMAutoContextSummarizationConfig( max_context_tokens=8000, max_unsummarized_messages=20, summary_config=LLMContextSummaryConfig( target_context_tokens=6000, min_messages_after_summary=4, ), ) ``` -------------------------------- ### Input Audio Buffer Speech Started Event Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.services.openai.realtime.events.md This event indicates that speech has been detected and has started within the input audio buffer. ```APIDOC ## Input Audio Buffer Speech Started Event ### Description Event indicating speech has started in the input audio buffer. ### Event Type `input.audio_buffer.speech_started` ### Parameters - **audio_start_ms** (int) - Required - Start time of speech in milliseconds. - **item_id** (str) - Required - ID of the associated conversation item. ### Response Example ```json { "type": "input.audio_buffer.speech_started", "audio_start_ms": 1678886400000, "item_id": "item_xyz789" } ``` ``` -------------------------------- ### Initialize and Use StartupTimingObserver Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.startup_timing_observer.md Instantiate the observer, register event handlers for startup timing and transport timing reports, and then attach it to a PipelineTask. This setup allows for monitoring the performance of pipeline startup. ```python observer = StartupTimingObserver() @observer.event_handler("on_startup_timing_report") async def on_report(observer, report): for t in report.processor_timings: print(f"{t.processor_name}: {t.duration_secs:.3f}s") @observer.event_handler("on_transport_timing_report") async def on_transport(observer, report): if report.bot_connected_secs is not None: print(f"Bot connected in {report.bot_connected_secs:.3f}s") print(f"Client connected in {report.client_connected_secs:.3f}s") task = PipelineTask(pipeline, observers=[observer]) ``` -------------------------------- ### setup_tracing Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.utils.tracing.setup.md Configures OpenTelemetry tracing with a user-provided exporter. ```APIDOC ## POST pipecat.utils.tracing.setup.setup_tracing ### Description Set up OpenTelemetry tracing with a user-provided exporter. ### Parameters #### Request Body - **service_name** (str) - Optional - The name of the service for traces. Defaults to 'pipecat'. - **exporter** (object) - Optional - A pre-configured OpenTelemetry span exporter instance. - **console_export** (bool) - Optional - Whether to also export traces to console (useful for debugging). ### Response - **Returns** (bool) - True if setup was successful, False otherwise. ``` -------------------------------- ### Initialize and use StartupTimingObserver Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.startup_timing_observer.md Configures the observer to track specific service types and registers event handlers to log timing reports. ```python observer = StartupTimingObserver( processor_types=(STTService, TTSService) ) @observer.event_handler("on_startup_timing_report") async def on_report(observer, report): for t in report.processor_timings: logger.info(f"{t.processor_name}: {t.duration_secs:.3f}s") @observer.event_handler("on_transport_timing_report") async def on_transport(observer, report): if report.bot_connected_secs is not None: logger.info(f"Bot connected in {report.bot_connected_secs:.3f}s") logger.info(f"Client connected in {report.client_connected_secs:.3f}s") task = PipelineTask(pipeline, observers=[observer]) ``` -------------------------------- ### Python Dataclass Docstring with Example Source: https://github.com/pipecat-ai/pipecat/blob/main/CONTRIBUTING.md Illustrates docstring conventions for a Python dataclass, including a description, an example of message format, and documentation of fields using the 'Parameters:' section. ```python @dataclass class MessageFrame: """Frame containing messages in OpenAI format. Supports both simple and content list message formats. Example:: [ {"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi there!"} ] Parameters: messages: List of messages in OpenAI format. """ messages: List[dict] ``` -------------------------------- ### KrispVivaIPUserTurnStartStrategy Initialization Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.krisp_viva_ip_user_turn_start_strategy.md Initializes the Krisp VIVA IP user turn start strategy with configurable parameters. ```APIDOC ## KrispVivaIPUserTurnStartStrategy ### Description User turn start strategy using Krisp VIVA Interruption Prediction. When VAD detects user speech, this strategy feeds audio frames into the Krisp VIVA IP model. The model outputs a probability indicating whether the speech is a genuine interruption (as opposed to a backchannel). A user turn is triggered only when this probability exceeds the configured threshold. This strategy is designed to work alongside other start strategies (e.g. `TranscriptionUserTurnStartStrategy` as a fallback) via the strategy list in `UserTurnStrategies`. ### Parameters #### Constructor Parameters - **model_path** (str | None) - Optional - Path to the Krisp VIVA IP model file (.kef). If None, uses the KRISP_VIVA_IP_MODEL_PATH environment variable. - **threshold** (float) - Optional - IP probability threshold (0.0 to 1.0). When the model’s output exceeds this value, the speech is classified as a genuine interruption. Defaults to 0.5. - **frame_duration_ms** (int) - Optional - Frame duration in milliseconds for IP processing. Supported values: 10, 15, 20, 30, 32. Defaults to 20. - **api_key** (str) - Optional - Krisp SDK API key. If empty, falls back to the KRISP_VIVA_API_KEY environment variable. - **kwargs** - Additional arguments passed to BaseUserTurnStartStrategy. ### Request Example ```python from pipecat.turns.user_start import KrispVivaIPUserTurnStartStrategy strategies = UserTurnStrategies( start=[ KrispVivaIPUserTurnStartStrategy( model_path="/path/to/ip_model.kef", threshold=0.5, ), TranscriptionUserTurnStartStrategy(), ], ) ``` ``` -------------------------------- ### MinWordsUserTurnStartStrategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.md User turn start strategy based on a minimum number of words spoken by the user. Signals the start of a user turn once the user has spoken at least a specified number of words. ```APIDOC ## MinWordsUserTurnStartStrategy ### Description User turn start strategy based on a minimum number of words spoken by the user. This strategy signals the start of a user turn once the user has spoken at least a specified number of words, as determined from transcription frames. Optionally, interim transcriptions can be used for earlier detection. ### Method __init__ ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **min_words** (int) - Minimum number of spoken words required to trigger the start of a user turn. - **use_interim** (bool) - Whether to consider interim transcription frames for earlier detection. Defaults to True. - **kwargs** - Additional keyword arguments. ### Request Example None ### Response #### Success Response (200) None #### Response Example None ## MinWordsUserTurnStartStrategy.reset() ### Description Reset the strategy to its initial state. ### Method async reset() ### Endpoint None ### Parameters None ### Request Example None ### Response #### Success Response (200) None #### Response Example None ## MinWordsUserTurnStartStrategy.process_frame() ### Description Process an incoming frame to detect the start of a user turn. This method updates internal state based on transcription frames and triggers the user turn once the minimum word count is reached. ### Method async process_frame(frame) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **frame** (Frame) - The frame to be analyzed. ### Request Example None ### Response #### Success Response (200) - **ProcessFrameResult** (ProcessFrameResult) - STOP if the minimum word count was reached, CONTINUE otherwise. #### Response Example None ``` -------------------------------- ### Initialize Sample Rate in Service Start Source: https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md Initialize sample rates in the start() method using the PipelineParams from the StartFrame. This ensures the correct sample rate is used for the service's output. ```python async def start(self, frame: StartFrame): """Start the service.""" await super().start(frame) self._settings.output_sample_rate = self.sample_rate await self._connect() ``` -------------------------------- ### Build and Run a Pipecat Pipeline Source: https://context7.com/pipecat-ai/pipecat/llms.txt Demonstrates initializing AI services, setting up context and aggregators, building a pipeline with various processors, and running it with a PipelineTask. Ensure API keys and model/voice settings are correctly configured. ```python import asyncio from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.services.openai.llm import OpenAILLMService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.frames.frames import LLMRunFrame async def main(): # Initialize services stt = DeepgramSTTService(api_key="DEEPGRAM_API_KEY") llm = OpenAILLMService( api_key="OPENAI_API_KEY", settings=OpenAILLMService.Settings( model="gpt-4.1", system_instruction="You are a helpful voice assistant." ) ) tts = CartesiaTTSService( api_key="CARTESIA_API_KEY", settings=CartesiaTTSService.Settings(voice="71a7ad14-091c-4e8e-a314-022ece01c121") ) # Create context and aggregators for turn management context = LLMContext() user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()) ) # Build the pipeline pipeline = Pipeline([ transport.input(), # Audio input from transport stt, # Speech-to-text user_aggregator, # Aggregate user turns llm, # LLM processing tts, # Text-to-speech transport.output(), # Audio output to transport assistant_aggregator # Aggregate assistant responses ]) # Create and run task task = PipelineTask( pipeline, params=PipelineParams( enable_metrics=True, enable_usage_metrics=True, audio_in_sample_rate=16000, audio_out_sample_rate=24000 ), idle_timeout_secs=300 ) # Event handlers @task.event_handler("on_pipeline_started") async def on_started(task, frame): print("Pipeline started") @task.event_handler("on_pipeline_finished") async def on_finished(task, frame): print("Pipeline finished") runner = PipelineRunner(handle_sigint=True) await runner.run(task) asyncio.run(main()) ``` -------------------------------- ### Initializing ServiceSwitcher with Manual Strategy Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.pipeline.service_switcher.md Instantiate ServiceSwitcher and specify ServiceSwitcherStrategyManual as the strategy type to enable manual service selection. The first service in the list becomes the initial active service. ```python stt_switcher = ServiceSwitcher( services=[stt_1, stt_2], strategy_type=ServiceSwitcherStrategyManual ) ``` -------------------------------- ### WakePhraseUserTurnStartStrategy Methods Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.wake_phrase_user_turn_start_strategy.md Core methods for setting up, processing, and cleaning up the strategy. ```APIDOC ## WakePhraseUserTurnStartStrategy Methods ### Description Provides methods for initializing, processing frames, and managing the lifecycle of the WakePhraseUserTurnStartStrategy. ### Methods #### setup * **Description**: Initialize the strategy with the given task manager. * **Parameters**: * **task_manager** ([BaseTaskManager]) - The task manager to be associated with this instance. #### cleanup * **Description**: Cleanup the strategy. #### reset * **Description**: Reset the strategy. In timeout mode, preserves state and refreshes timeout since reset means a turn started (activity). In single activation mode, does nothing — the keepalive timeout (started when the wake phrase was detected) handles the transition back to IDLE. #### process_frame * **Description**: Process an incoming frame for wake phrase detection or passthrough. * **Parameters**: * **frame** ([Frame]) - The frame to be processed. * **Returns**: * **STOP**: when the wake phrase is detected or when in IDLE state (blocks subsequent strategies). * **CONTINUE**: when in AWAKE state (allows subsequent strategies to proceed). ### Request Example ```python # Example for process_frame result = await strategy.process_frame(frame) ``` ### Response * **process_frame** returns a ProcessFrameResult (STOP or CONTINUE). ``` -------------------------------- ### SkipTagsAggregator Class Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.utils.text.skip_tags_aggregator.md Aggregator that prevents end of sentence matching between start/end tags. It buffers text until it finds an end of sentence or a start tag. If a start tag is found, it keeps aggregating text unconditionally until the corresponding end tag is found. ```APIDOC ## Class: SkipTagsAggregator ### Description Aggregator that prevents end of sentence matching between start/end tags. This aggregator buffers text until it finds an end of sentence or a start tag. If a start tag is found the aggregator will keep aggregating text unconditionally until the corresponding end tag is found. It's particularly useful for processing content with custom delimiters that should prevent text from being considered for end of sentence matching. The aggregator ensures that tags spanning multiple text chunks are correctly identified and that content within tags is never split at sentence boundaries. ### Parameters #### Constructor Parameters - **tags** (Sequence[tuple[str, str]]) - Required - Sequence of StartEndTags objects defining the tag pairs that should prevent sentence boundary detection. - **kwargs** - Optional - Additional arguments passed to SimpleTextAggregator (e.g. aggregation_type). ### Methods #### async aggregate(text: str) ##### Description Aggregate text while respecting tag boundaries. Processes the input text character-by-character, updates tag state, and uses the parent’s lookahead logic for sentence detection when not inside tags. In TOKEN mode, text is passed through immediately unless we’re inside a tag, in which case we buffer until the closing tag is found. ##### Parameters - **text** (str) - Required - Text to aggregate. ##### Yields Aggregation objects containing text up to a sentence boundary, marked as SENTENCE type (or TOKEN type in TOKEN mode). #### async handle_interruption() ##### Description Handle interruptions by clearing the buffer and tag state. Called when an interruption occurs in the processing pipeline, to reset the state and discard any partially aggregated text. #### async reset() ##### Description Clear the internally aggregated text and tag state. Resets the aggregator to its initial state, discarding any buffered text. ``` -------------------------------- ### SpeechTimeoutUserTurnStopStrategy.setup Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_stop.speech_timeout_user_turn_stop_strategy.md Initializes the SpeechTimeoutUserTurnStopStrategy with a task manager. ```APIDOC ## setup(task_manager) ### Description Initialize the strategy with the given task manager. ### Parameters * **task_manager** ([BaseTaskManager](pipecat.utils.asyncio.task_manager.md#pipecat.utils.asyncio.task_manager.BaseTaskManager)) - The task manager to be associated with this instance. ### Method async setup(task_manager: BaseTaskManager) ### Example ```python await strategy.setup(task_manager) ``` ``` -------------------------------- ### SessionContinuationHelper.on_user_content_started Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.services.aws.nova_sonic.md Callback when the user starts providing content. ```APIDOC ## SessionContinuationHelper.on_user_content_started() ### Description Callback when the user starts providing content. ### Method (Not specified in source) ### Endpoint (Not specified in source) ### Parameters (No parameters specified in source) ### Request Example (No request example specified in source) ### Response #### Success Response (200) (No success response specified in source) #### Response Example (No response example specified in source) ``` -------------------------------- ### SessionContinuationHelper.start_monitor Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.services.aws.nova_sonic.md Starts monitoring for session continuation events. ```APIDOC ## SessionContinuationHelper.start_monitor() ### Description Starts monitoring for session continuation events. ### Method (Not specified in source) ### Endpoint (Not specified in source) ### Parameters (No parameters specified in source) ### Request Example (No request example specified in source) ### Response #### Success Response (200) (No success response specified in source) #### Response Example (No response example specified in source) ``` -------------------------------- ### StartupTimingObserver Event Handlers Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.startup_timing_observer.md Documentation for the event handlers used to receive startup and transport timing reports. ```APIDOC ## Event: on_startup_timing_report ### Description Triggered when the pipeline startup timing report is generated, containing duration data for each processor. ### Response - **report** (StartupTimingReport) - Object containing start_time, total_duration_secs, and a list of processor_timings. ## Event: on_transport_timing_report ### Description Triggered when transport connection milestones are reached. ### Response - **report** (TransportTimingReport) - Object containing start_time, bot_connected_secs (optional), and client_connected_secs. ``` -------------------------------- ### Initialize DebugLogObserver Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.loggers.debug_log_observer.md Basic initialization to log all frames from all services. ```python observers = DebugLogObserver() ``` -------------------------------- ### BotStartedSpeakingFrame Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.frames.frames.md Frame indicating the bot started speaking. ```APIDOC ## BotStartedSpeakingFrame ### Description Frame indicating the bot started speaking. Emitted upstream and downstream by the BaseTransportOutput to indicate the bot started speaking. ``` -------------------------------- ### UserStartedSpeakingFrame Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.frames.frames.md Frame indicating that the user turn has started. ```APIDOC ## UserStartedSpeakingFrame ### Description Frame indicating that the user turn has started. Emitted when the user turn starts, which usually means that some transcriptions are already available. ``` -------------------------------- ### MetricsLogObserver Initialization Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.loggers.metrics_log_observer.md This snippet shows how to initialize the MetricsLogObserver. You can log all metrics by default or specify a set of metrics to include. ```APIDOC ## Initialize MetricsLogObserver ### Description Initializes the metrics log observer. It can be configured to log all metrics or only specific types of metrics. ### Method `__init__` ### Parameters #### Parameters - **include_metrics** (set | None) - Optional - Set of metrics types to include. If specified, only these metrics types will be logged. If None, all metrics are logged. - **kwargs** - Additional arguments passed to the parent class. ### Request Example ```python # Log all metrics types observers = [MetricsLogObserver()] # Log only LLM and TTS metrics from pipecat.metrics.metrics import LLMUsageMetricsData, TTSUsageMetricsData observers = [ MetricsLogObserver( include_metrics={LLMUsageMetricsData, TTSUsageMetricsData} ) ] ``` ``` -------------------------------- ### User Turn Completion Mixin Usage Example Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_turn_completion_mixin.md This example shows how to conditionally use the `_push_turn_text` method when turn completion is enabled, otherwise falling back to the standard `push_frame` method. This controls whether LLM responses are processed with turn completion logic. ```python if self._filter_incomplete_user_turns: await self._push_turn_text(chunk.text) else: await self.push_frame(LLMTextFrame(chunk.text)) ``` -------------------------------- ### FunctionCallsStartedFrame Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.frames.frames.md Frame signaling that function call execution is starting. ```APIDOC ## FunctionCallsStartedFrame ### Description A frame signaling that one or more function call execution is going to start. ### Parameters - **function_calls** (Sequence[FunctionCallFromLLM]) - Required - Sequence of function calls that will be executed. ``` -------------------------------- ### configure() Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.runner.daily.md Configures a Daily room URL and token, with optional support for SIP capabilities. It automatically creates a temporary room if no existing URL is provided. ```APIDOC ## configure() ### Description Configures a Daily room URL and token. It either uses an existing room URL from the environment or creates a new temporary room. ### Parameters - **aiohttp_session** (ClientSession) - Required - HTTP session for API requests. - **api_key** (str) - Optional - Daily API key. - **room_exp_duration** (float) - Optional - Room expiration in hours (default 2.0). - **token_exp_duration** (float) - Optional - Token expiration in hours (default 2.0). - **sip_caller_phone** (str) - Optional - Phone number for SIP display name. - **sip_enable_video** (bool) - Optional - Enable video for SIP. - **sip_num_endpoints** (int) - Optional - Number of allowed SIP endpoints. - **enable_dialout** (bool) - Optional - Enable outbound dialing. - **sip_codecs** (dict) - Optional - Codecs for audio/video. - **sip_provider** (str) - Optional - SIP provider name. - **room_geo** (str) - Optional - Geographic region for the room. - **room_properties** (DailyRoomProperties) - Optional - Custom room properties. - **token_properties** (DailyMeetingTokenProperties) - Optional - Custom token properties. ### Response - **DailyRoomConfig** (Object) - Returns an object containing room_url, token, and optional sip_endpoint. ``` -------------------------------- ### LLMThoughtStartFrame Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.frames.frames.md Represents a frame indicating the start of an LLM thought. ```APIDOC ## LLMThoughtStartFrame ### Description Frame indicating the start of an LLM thought process. ### Parameters - **append_to_context** (bool) - Optional - Whether the thought should be appended to the LLM context. - **llm** (str | None) - Optional - Identifier of the LLM provider for LLM-specific handling. ``` -------------------------------- ### GeminiFileAPI - Get File Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.services.google.gemini_live.md Retrieves a specific file by its ID. ```APIDOC ## GET /api/gemini/files/{file_id} ### Description Retrieves a specific file using its unique identifier. ### Method GET ### Endpoint /api/gemini/files/{file_id} ### Parameters #### Path Parameters - **file_id** (string) - Required - The unique identifier of the file to retrieve. ### Response #### Success Response (200) - **file_content** (string) - The content of the file. #### Response Example ```json { "file_content": "" } ``` ``` -------------------------------- ### Docstring Example for a Pipecat Service Source: https://github.com/pipecat-ai/pipecat/blob/main/CLAUDE.md Illustrates the Google-style docstring format for classes and their initialization methods, including event handler documentation. Use this pattern for new service implementations. ```python class MyService(LLMService): """Description of what the service does. More detailed description. Event handlers available: - on_connected: Called when we are connected Example:: @service.event_handler("on_connected") async def on_connected(service, frame): ... """ def __init__(self, param1: str, **kwargs): """Initialize the service. Args: param1: Description of param1. **kwargs: Additional arguments passed to parent. """ super().__init__(**kwargs) ``` -------------------------------- ### UserTurnStrategies Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.md Defines the interface for user turn start and stop strategies. ```APIDOC ## Class: UserTurnStrategies ### Description An abstract base class or interface for defining user turn detection strategies. ### Attributes #### `start` (property) - A collection of strategies to detect the start of a user turn. #### `stop` (property) - A collection of strategies to detect the end of a user turn. ``` -------------------------------- ### UserTurnController Initialization and Configuration Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_turn_controller.md Details on how to initialize and configure the UserTurnController, including its parameters and event handlers. ```APIDOC ## UserTurnController ### Description Controller for managing user turn lifecycle. This class manages user turn state (active/inactive), handles start and stop strategies, and emits events when user turns begin, end, or timeout occurs. ### Event Handlers Available: - on_user_turn_started: Emitted when a user turn starts. - on_user_turn_stopped: Emitted when a user turn stops. - on_user_turn_stop_timeout: Emitted if no stop strategy triggers before timeout. - on_push_frame: Emitted when a strategy wants to push a frame. - on_broadcast_frame: Emitted when a strategy wants to broadcast a frame. ### Example Event Handler Usage: ```python @controller.event_handler("on_user_turn_started") async def on_user_turn_started(controller, strategy: BaseUserTurnStartStrategy, params: UserTurnStartedParams): ... @controller.event_handler("on_user_turn_stopped") async def on_user_turn_stopped(controller, strategy: BaseUserTurnStopStrategy, params: UserTurnStoppedParams): ... @controller.event_handler("on_user_turn_stop_timeout") async def on_user_turn_stop_timeout(controller): ... @controller.event_handler("on_push_frame") async def on_push_frame(controller, frame: Frame, direction: FrameDirection): ... @controller.event_handler("on_broadcast_frame") async def on_broadcast_frame(controller, frame_cls: Type[Frame], **kwargs): ... ``` ### Method `__init__` ### Parameters #### Constructor Parameters - **user_turn_strategies** (UserTurnStrategies) - Required - Configured strategies for starting and stopping user turns. - **user_turn_stop_timeout** (float) - Optional - Timeout in seconds to automatically stop a user turn if no activity is detected. Defaults to 5.0. ### Property #### task_manager - **task_manager** (BaseTaskManager) - Read-only - Returns the configured task manager. ``` -------------------------------- ### process_frame Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.turns.user_start.transcription_user_turn_start_strategy.md Processes an incoming frame to detect the start of a user turn. ```APIDOC ## POST /api/turns/user_start/transcription/process_frame ### Description Processes an incoming frame to detect the start of a user turn. ### Method POST ### Endpoint /api/turns/user_start/transcription/process_frame ### Parameters #### Query Parameters - **strategy_id** (string) - Required - The unique identifier for the strategy. #### Request Body - **frame** (object) - Required - The frame to be processed. - **type** (string) - The type of the frame. - **data** (any) - The data contained within the frame. ### Response #### Success Response (200) - **result** (string) - Indicates whether to STOP or CONTINUE processing. STOP if a transcription was received, CONTINUE otherwise. #### Response Example { "result": "STOP" } ``` -------------------------------- ### StartupTimingObserver Initialization Source: https://github.com/pipecat-ai/pipecat/blob/main/docs/api/api/pipecat.observers.startup_timing_observer.md Initializes the StartupTimingObserver to measure processor startup times and transport timing. ```APIDOC ## StartupTimingObserver ### Description Observer that measures processor startup times during pipeline initialization. Tracks how long each processor's `start()` method takes and also measures transport timing. ### Method __init__ ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Parameters - **processor_types** (tuple[type] | None) - Optional. A tuple of processor types to measure. If None, all non-internal processors are measured. - **kwargs** - Additional arguments passed to the parent class. ### Request Example ```python from pipecat.observers.startup_timing_observer import StartupTimingObserver from pipecat.processors.services.stt_service import STTService from pipecat.processors.services.tts_service import TTSService observer = StartupTimingObserver( processor_types=(STTService, TTSService) ) ``` ### Response #### Success Response (200) None #### Response Example None ```