### Run HeyGenVideoService Example (Daily) Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Executes the HeyGenVideoService example with the Daily transport. This allows integration with Daily's video conferencing platform. ```shell uv run python examples/video-avatar/video-avatar-heygen-video-service.py --transport daily ``` -------------------------------- ### Install Dependencies and Run Agent Source: https://docs.liveavatar.com/docs/guides/livekit/custom-livekit-agent Install project dependencies using uv and run the LiveKit agent demo script. ```bash uv sync python src/liveavatar_hosted_demo.py ``` -------------------------------- ### Run HeyGenTransport Example Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Executes the HeyGenTransport example script. This setup manages the LiveKit room, and a room URL will be provided in the logs to connect a client. ```shell uv run python examples/video-avatar/video-avatar-heygen-transport.py ``` -------------------------------- ### Run HeyGenVideoService Example (WebRTC) Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Runs the HeyGenVideoService example using the WebRTC transport for a local browser-based test. A URL will be printed to connect. ```shell uv run python examples/video-avatar/video-avatar-heygen-video-service.py --transport webrtc ``` -------------------------------- ### Start the session Source: https://docs.liveavatar.com/docs/full-mode/overview Use the session token to start the session and initialize the WebRTC room. ```APIDOC ## Start the session ### Description Use the session token to start the session and initialize the WebRTC room. ### Method POST ### Endpoint https://api.liveavatar.com/v1/sessions/start ### Parameters #### Request Headers - **authorization** (string) - Required - Bearer token (e.g., "Bearer ") ### Response #### Success Response (200) - **livekit_url** (string) - The URL for the LiveKit room. - **livekit_client_token** (string) - The client token for connecting to LiveKit. ``` -------------------------------- ### Install LiveKit Agents with LiveAvatar Plugin Source: https://docs.liveavatar.com/docs/lite-mode/plugins/livekit Install the LiveKit agents package with the LiveAvatar plugin support using uv. Alternatively, use pip for installation. ```shell uv add "livekit-agents[liveavatar]~=1.5" ``` ```shell pip install "livekit-agents[liveavatar]~=1.5" ``` -------------------------------- ### Configure API Keys for Pipecat Examples Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Sets up the necessary API keys in a .env file for running Pipecat examples. Keys are required for HeyGen, Deepgram, Cartesia, Google, and optionally Daily. ```bash HEYGEN_LIVE_AVATAR_API_KEY=... DEEPGRAM_API_KEY=... CARTESIA_API_KEY=... GOOGLE_API_KEY=... DAILY_API_KEY=... ``` -------------------------------- ### Clone and Setup LiveKit Agent Starter Source: https://docs.liveavatar.com/docs/guides/livekit/custom-livekit-agent Clone the reference implementation repository and set up the local environment variables for LiveAvatar and LiveKit integration. ```bash git clone https://github.com/heygen-com/liveavatar-starter-livekit-agent-python cd liveavatar-starter-livekit-agent-python cp .env.example .env.local ``` ```bash LIVEAVATAR_API_KEY=... # from app.liveavatar.com AVATAR_ID=... # any avatar in your account; sandbox-compatible by default LIVEKIT_API_KEY=... # your LK Cloud project — used for the inference gateway only LIVEKIT_API_SECRET=... IS_SANDBOX=true # default; switch to false for production avatars / billed minutes ``` -------------------------------- ### Install Dependencies and Run LiveKit Agent Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Install project dependencies using uv and run the worker and dispatcher scripts in separate terminals. The worker registers against your LiveKit project, and the dispatcher drives a session, connecting the avatar to your room. ```bash uv sync # Terminal 1 — register the worker against your LK project python src/worker.py dev # Terminal 2 — drive a session python src/byo_livekit_demo.py ``` -------------------------------- ### Start a LiveAvatar Session Source: https://docs.liveavatar.com/docs/full-mode/overview Use the generated session token to start the session and initialize the WebRTC room. Requires the session token obtained in the previous step. ```bash curl -X POST https://api.liveavatar.com/v1/sessions/start \ -H "accept: application/json" \ -H "authorization: Bearer " ``` -------------------------------- ### Install LiveAvatar Web SDK Source: https://docs.liveavatar.com/docs/full-mode/overview Install the LiveAvatar Web SDK using npm for production integration. This SDK is used to connect from your frontend to the LiveKit room. ```bash npm install @heygen/liveavatar-web-sdk ``` -------------------------------- ### Install Pipecat HeyGen Plugin (pip) Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Use 'pip install' to install the pipecat-ai plugin with HeyGen support. ```shell pip install "pipecat-ai[heygen]" ``` -------------------------------- ### Install Pipecat HeyGen Plugin (uv) Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Use 'uv add' to install the pipecat-ai plugin with HeyGen support. ```shell uv add "pipecat-ai[heygen]" ``` -------------------------------- ### Install LiveAvatar Agent Skills Manually Source: https://docs.liveavatar.com/docs/agent-skills This method involves cloning the repository and then symlinking the skills to your personal skills directory. Use this if the CLI installation is not suitable. ```bash git clone https://github.com/heygen-com/liveavatar-agent-skills.git # Symlink all skills to personal skills directory (available in all projects) for skill in liveavatar-agent-skills/skills/*/; do ln -s "$(pwd)/$skill" ~/.claude/skills/$(basename "$skill") done ``` -------------------------------- ### Send user.start_push_to_talk Command Source: https://docs.liveavatar.com/docs/full-mode/events Signals to start capturing user audio for Push-to-Talk (PTT) sessions. Send this to the `agent-control` topic. ```json { "event_type": "user.start_push_to_talk" } ``` -------------------------------- ### Install LiveAvatar Agent Skills CLI Source: https://docs.liveavatar.com/docs/agent-skills Use this command to install the LiveAvatar Agent Skills using the skills CLI. This is the recommended method for integrating LiveAvatar with AI coding agents. ```bash npx skills add heygen-com/liveavatar-agent-skills ``` -------------------------------- ### Listen for user.push_to_talk_started Event Source: https://docs.liveavatar.com/docs/full-mode/events Confirms that Push-to-Talk (PTT) successfully started. Listen for this on the `agent-response` topic. ```json { "event_type": "user.push_to_talk_started" } ``` -------------------------------- ### Listen for user.speak_started Event Source: https://docs.liveavatar.com/docs/full-mode/events Indicates that the user has started sending audio. Listen for this on the `agent-response` topic. ```json { "event_type": "user.speak_started" } ``` -------------------------------- ### Deploy LiveKit Agent to LiveKit Cloud Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Use these commands for initial setup and subsequent deployments of your agent to LiveKit Cloud. Remember to commit changes to `livekit.toml` after running `lk agent create`. ```bash # First-time setup — writes subdomain + agent id back into livekit.toml lk agent create --secrets-file .env.local # Subsequent updates lk agent deploy # Inspect lk agent status lk agent logs ``` -------------------------------- ### Sequence Diagram: Starting a Full Mode Session Source: https://docs.liveavatar.com/docs/full-mode/lifecycle Illustrates the sequence of interactions between the end user, frontend, backend, LiveAvatar API, and the managed room when starting a FULL Mode session. Ensure the session token has 'mode' set to 'FULL'. ```mermaid sequenceDiagram participant User as End User participant Frontend as Developer Frontend participant Backend as Developer API participant API as LiveAvatar API participant Room as LiveAvatar Managed Room User->>Frontend: User visits site Frontend->>Backend: Start session flow Backend->>API: POST /v1/sessions/token API-->>Backend: Session-scoped token Backend-->>Frontend: Send token back User->>Frontend: User starts the session Frontend->>API: POST /v1/sessions/start (with token) API->>Room: Start room with avatar API-->>Frontend: Room client token + session details Frontend->>Room: Establish room connection with client token Room-->>Frontend: Conversation starts Frontend-->>User: User sees avatar conversation ``` -------------------------------- ### Listen for user.push_to_talk_start_failed Event Source: https://docs.liveavatar.com/docs/full-mode/events Indicates that Push-to-Talk (PTT) failed to start. Listen for this on the `agent-response` topic. ```json { "event_type": "user.push_to_talk_start_failed" } ``` -------------------------------- ### Join LiveKit Room for Testing Source: https://docs.liveavatar.com/docs/full-mode/overview For a quick test, open the provided LiveKit URL directly in your browser. This URL includes the livekit_url and livekit_client_token obtained after starting the session. ```text https://meet.livekit.io/custom?liveKitUrl=&token= ``` -------------------------------- ### LiveKit Agent Worker Subcommands Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent The worker script supports multiple subcommands for different operational modes. 'dev' is for local development with hot-reloading, 'start' is the production entrypoint, and 'download-files' pre-downloads model weights for faster cold starts. ```python src/worker.py ``` -------------------------------- ### Setup Push-to-Talk Session Source: https://docs.liveavatar.com/docs/full-mode/push-to-talk Configure the session token to enable Push-to-Talk mode by setting `interactivity_type` to `PUSH_TO_TALK`. ```json { "mode": "FULL", "interactivity_type": "PUSH_TO_TALK", "avatar_id": "", "avatar_persona": { "voice_id": "", "context_id": "" } } ``` -------------------------------- ### Listen for avatar.speak_started Event Source: https://docs.liveavatar.com/docs/full-mode/events Indicates that the avatar has started speaking. Listen for this on the `agent-response` topic. ```json { "event_type": "avatar.speak_started" } ``` -------------------------------- ### Start Session with ElevenLabs Agent Config Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent Configure a LITE mode session to use the ElevenLabs Agent Connector. Provide your ElevenLabs API secret ID and Agent ID. ```json { "mode": "LITE", "avatar_id": "", "elevenlabs_agent_config": { "secret_id": "", "agent_id": "" } } ``` -------------------------------- ### LITE Mode Session Lifecycle Diagram Source: https://docs.liveavatar.com/docs/lite-mode/lifecycle This diagram illustrates the sequence of events when starting a LITE Mode LiveAvatar session, from user interaction to the avatar appearing in a WebRTC room. ```mermaid sequenceDiagram participant User as End User participant Frontend as Developer Frontend participant Agent as Developer API / Agent participant Room as Session Room participant API as LiveAvatar API User->>Frontend: User visits site Frontend->>Agent: Start Session Flow Agent->>API: POST /v1/sessions/token API-->>Agent: Session scoped token Agent->>API: POST /v1/sessions/start API-->>Agent: Session Details Note right of API: LiveAvatar sends the
Avatar to Room Agent->>Room: Establish agent connection Frontend->>Room: Establish user connection Room-->>Frontend: Conversation Start Frontend-->>User: User sees Avatar conversation ``` -------------------------------- ### Start Session with Custom LLM Source: https://docs.liveavatar.com/docs/full-mode/custom-llm Initiate a session using your custom LLM by setting the `llm_configuration_id` in the session token. This JSON payload configures the session mode, avatar, and LLM details. ```json { "mode": "FULL", "avatar_id": "", "llm_configuration_id": "", "avatar_persona": { "voice_id": "", "context_id": "" } } ``` -------------------------------- ### Clone Pipecat Repository Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Clones the pipecat repository and installs development dependencies including GStreamer. ```shell git clone https://github.com/pipecat-ai/pipecat.git cd pipecat uv sync --group dev --all-extras --no-extra gstreamer --no-extra local ``` -------------------------------- ### Start Session with Custom Voice Source: https://docs.liveavatar.com/docs/full-mode/custom-tts Initiate a LiveAvatar FULL Mode session using your imported ElevenLabs voice. The `voice_id` specified here must correspond to the voice imported in the previous step. LiveAvatar will use your ElevenLabs credentials for audio generation. ```json { "mode": "FULL", "avatar_id": "", "avatar_persona": { "voice_id": "", "context_id": "" } } ``` -------------------------------- ### Clone and Configure LiveKit Agent Starter Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Clone the reference implementation repository and set up your local environment variables for LiveAvatar and LiveKit integration. Ensure all required credentials are provided in the .env.local file. ```bash git clone https://github.com/heygen-com/liveavatar-starter-livekit-agent-python cd liveavatar-starter-livekit-agent-python cp .env.example .env.local ``` ```bash LIVEAVATAR_API_KEY=... # from app.liveavatar.com AVATAR_ID=... # any avatar in your account; sandbox-compatible by default LIVEKIT_URL=wss://.livekit.cloud LIVEKIT_API_KEY=... # your LK Cloud project — used for the room AND inference LIVEKIT_API_SECRET=... IS_SANDBOX=true # default; switch to false for production avatars / billed minutes ``` -------------------------------- ### Initialize HeyGenVideoService Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Sets up the HeyGenVideoService for live avatar integration. Ensure HEYGEN_LIVE_AVATAR_API_KEY is set in your environment. Sandbox mode uses a fixed avatar ID and is_sandbox=True; remove these for production. ```python import os import aiohttp from pipecat.services.heygen import HeyGenVideoService from pipecat.services.heygen.client import ServiceType from pipecat.services.heygen.api_liveavatar import LiveAvatarNewSessionRequest async with aiohttp.ClientSession() as session: heygen = HeyGenVideoService( api_key=os.environ["HEYGEN_LIVE_AVATAR_API_KEY"], session=session, service_type=ServiceType.LIVE_AVATAR, session_request=LiveAvatarNewSessionRequest( # Sandbox mode only supports this fixed avatar_id and is_sandbox=True. # Remove both when moving to production and use your own avatar_id. avatar_id="dd73ea75-1218-4ef3-92ce-606d5f7fbc0a", is_sandbox=True, ), ) ``` -------------------------------- ### Send avatar.start_listening Command Source: https://docs.liveavatar.com/docs/full-mode/events Switches the avatar to a listening state from idle. Send this to the `agent-control` topic. ```json { "event_type": "avatar.start_listening" } ``` -------------------------------- ### Graph: Recommended Architecture for Session Management Source: https://docs.liveavatar.com/docs/full-mode/lifecycle Depicts the recommended architecture where the frontend acts as the primary controller for emitting events from the LiveKit room to minimize latency. Event data can be relayed to the backend for further processing. ```mermaid graph LR Frontend["Developer Frontend"] <-->|"send/receive events"| Room["LiveAvatar Managed Room"] Frontend <-->|"avatar video stream"| Room Frontend -->|"relay event data"| Backend["Developer API"] Agent["LiveAvatar Agent"] <--> Room ``` -------------------------------- ### ElevenLabs Agent Event Payload (LiveKit Data Channel) Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent Example structure of an `elevenlabs_agent_event` received via the LiveKit data channel. The `data` field contains the verbatim ElevenLabs payload. ```json { "event_id": "abc-123", "event_type": "elevenlabs_agent_event", "session_id": "session-456", "elevenlabs_event_type": "agent_response", "data": { ... } } ``` -------------------------------- ### Integrate LiveAvatarSession into LiveKit Agent Source: https://docs.liveavatar.com/docs/lite-mode/plugins/livekit Drop AvatarSession into your agent entrypoint. Import AvatarSession, construct it, and call avatar.start() before session.start(). ```python from livekit import agents from livekit.agents import AgentServer, AgentSession from livekit.plugins import liveavatar server = AgentServer() @server.rtc_session(agent_name="my-agent") async def my_agent(ctx: agents.JobContext): session = AgentSession( # ... your existing stt, llm, tts, vad, turn_detection ) avatar = liveavatar.AvatarSession( avatar_id="...", # or rely on LIVEAVATAR_AVATAR_ID ) # Avatar joins the room and hooks the session's TTS output. await avatar.start(session, room=ctx.room) # Now start the conversation loop. await session.start( # ... room, agent, room_options, etc. ) ``` -------------------------------- ### Dynamic Variables for Session Personalization Source: https://docs.liveavatar.com/docs/core-concepts/contexts Use dynamic variables to inject per-user information into a shared context at session start. Reference variables in your prompt or opening text using `${variable_name}` syntax and supply values in the `dynamic_variables` object when creating a session. ```json { "dynamic_variables": { "user_name": "Jordan", "plan": "Pro" } } ``` -------------------------------- ### Create Session Token in Sandbox Mode Source: https://docs.liveavatar.com/docs/sandbox-mode Use this `curl` command to create a session token for Sandbox Mode. Ensure you replace `` and `` with your actual values. The `is_sandbox` flag must be set to `true`. ```bash curl -X POST https://api.liveavatar.com/v1/sessions/token \ -H "X-API-KEY: " \ -H "content-type: application/json" \ -d '{ "mode": "FULL", "is_sandbox": true, "avatar_id": "dd73ea75-1218-4ef3-92ce-606d5f7fbc0a", "avatar_persona": { "voice_id": "", "language": "en" } }' ``` -------------------------------- ### Create a FULL Mode Session Token Source: https://docs.liveavatar.com/docs/full-mode/overview Generate a session token on your backend to define the avatar, voice, context, and session configuration for FULL Mode. Requires your API key. ```bash curl -X POST https://api.liveavatar.com/v1/sessions/token \ -H "X-API-KEY: " \ -H "accept: application/json" \ -H "content-type: application/json" \ -d '{ "mode": "FULL", "avatar_id": "", "avatar_persona": { "voice_id": "", "context_id": "", "language": "en" } }' ``` -------------------------------- ### Basic LITE Mode Configuration Source: https://docs.liveavatar.com/docs/lite-mode/configuration Configure LITE mode with a specific avatar and video quality settings. Ensure the avatar_id is valid. ```json { "mode": "LITE", "avatar_id": "", "video_settings": { "quality": "high", "encoding": "VP8" } } ``` -------------------------------- ### Import ElevenLabs Voice Source: https://docs.liveavatar.com/docs/full-mode/custom-tts Import your ElevenLabs voice into LiveAvatar using the previously obtained `secret_id` and your ElevenLabs `voice_id`. The imported voice can then be found in your LiveAvatar voice listing. ```bash curl -X POST https://api.liveavatar.com/v1/voices/third_party \ -H "X-API-KEY: " \ -H "content-type: application/json" \ -d '{ "secret_id": "", "voice_id": "" }' ``` -------------------------------- ### Configure LiveAvatar API Key Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Set the HEYGEN_LIVE_AVATAR_API_KEY environment variable to authenticate with LiveAvatar. Alternatively, pass the 'api_key' argument when constructing the relevant class. ```bash HEYGEN_LIVE_AVATAR_API_KEY=... ``` -------------------------------- ### Initialize HeyGenTransport for LiveAvatar Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat Use this when you want your transport to own the room and manage the LiveAvatar session directly. Ensure `service_type` is set to `ServiceType.LIVE_AVATAR` and provide a `LiveAvatarNewSessionRequest`. ```python import os import aiohttp from pipecat.transports.heygen import HeyGenTransport from pipecat.services.heygen.client import ServiceType from pipecat.services.heygen.api_liveavatar import LiveAvatarNewSessionRequest async with aiohttp.ClientSession() as session: transport = HeyGenTransport( session=session, api_key=os.environ["HEYGEN_LIVE_AVATAR_API_KEY"], service_type=ServiceType.LIVE_AVATAR, session_request=LiveAvatarNewSessionRequest( # Sandbox mode only supports this fixed avatar_id and is_sandbox=True. # Remove both when moving to production and use your own avatar_id. avatar_id="dd73ea75-1218-4ef3-92ce-606d5f7fbc0a", is_sandbox=True, ), ) ``` -------------------------------- ### Agent Dispatch Logic Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Illustrates the sequence of operations when running the Python demo script, from token minting to worker interaction. ```python 1. Mint room + tokens against your LK project. 2. Create a LiveAvatar session pinned to your room. 3. Start the session. 4. Dispatch the agent into the room. 5. Open the viewer. 6. Worker accepts the dispatch. 7. Clean up on exit. ``` -------------------------------- ### LITE Mode with Custom LiveKit Infrastructure Source: https://docs.liveavatar.com/docs/lite-mode/configuration Configure LITE mode to use your own LiveKit infrastructure by providing the LiveKit server URL and an agent token. This allows for custom ASR, LLM, and TTS stacks. ```json { "mode": "LITE", "avatar_id": "", "livekit_config": { "url": "wss://your-livekit-server.com", "token": "" } } ``` -------------------------------- ### Configure Custom LLM Endpoint Source: https://docs.liveavatar.com/docs/full-mode/custom-llm Create a custom LLM configuration by providing your endpoint details. This POST request uses the `secret_id` obtained from the previous step and specifies the `model_name` and `base_url`. The `base_url` is optional and defaults to OpenAI; your endpoint must be OpenAI-compatible. ```bash curl -X POST https://api.liveavatar.com/v1/llm_configurations \ -H "X-API-KEY: " \ -H "content-type: application/json" \ -d '{ "display_name": "My Custom LLM", "model_name": "gpt-4o-mini", "secret_id": "", "base_url": "https://api.openai.com" }' ``` -------------------------------- ### Register LLM API Key Source: https://docs.liveavatar.com/docs/full-mode/custom-llm Store your LLM API credentials securely. This POST request registers your API key with LiveAvatar, returning a `secret_id` for future use. Ensure you replace `` with your actual API key. ```bash curl -X POST https://api.liveavatar.com/v1/secrets \ -H "X-API-KEY: " \ -H "content-type: application/json" \ -d '{ "secret_type": "LLM_API_KEY", "secret_value": "", "secret_name": "My LLM Key" }' ``` -------------------------------- ### Send user.stop_push_to_talk Command Source: https://docs.liveavatar.com/docs/full-mode/events Signals to stop capturing user audio for Push-to-Talk (PTT) sessions. Send this to the `agent-control` topic. ```json { "event_type": "user.stop_push_to_talk" } ``` -------------------------------- ### Create a session token Source: https://docs.liveavatar.com/docs/full-mode/overview Generate a session token on your backend to define the avatar, voice, context, and session configuration for a FULL Mode session. ```APIDOC ## Create a session token ### Description Generate a session token on your backend. This defines the avatar, voice, context, and session configuration. ### Method POST ### Endpoint https://api.liveavatar.com/v1/sessions/token ### Parameters #### Request Body - **mode** (string) - Required - "FULL" - **avatar_id** (string) - Required - The ID of the avatar to use. - **avatar_persona** (object) - Required - Configuration for the avatar's persona. - **voice_id** (string) - Required - The ID of the voice to use. - **context_id** (string) - Required - The ID of the context to use. - **language** (string) - Required - The language for the session (e.g., "en"). ### Request Example ```json { "mode": "FULL", "avatar_id": "", "avatar_persona": { "voice_id": "", "context_id": "", "language": "en" } } ``` ### Response #### Success Response (200) - **session_id** (string) - The ID of the created session. - **session_token** (string) - The token to use for starting the session. ``` -------------------------------- ### Agent Dispatcher Configuration Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Explains the requirement for the worker to register under a specific name and the dispatcher to use the same name for `create_dispatch`. ```python `agent_name` dispatch is **not** automatic dispatch. The worker must register under a name (`@server.rtc_session(agent_name=AGENT_NAME)` in `worker.py`) and the dispatcher must call `create_dispatch` with the same name. Workers using LiveKit's default auto-dispatch won't receive these jobs. ``` -------------------------------- ### Listen for user.push_to_talk_stop_failed Event Source: https://docs.liveavatar.com/docs/full-mode/events Indicates that Push-to-Talk (PTT) failed to stop. Listen for this on the `agent-response` topic. ```json { "event_type": "user.push_to_talk_stop_failed" } ``` -------------------------------- ### LITE Mode Session Management Flow Source: https://docs.liveavatar.com/docs/lite-mode/lifecycle This graph visualizes the data flow during the management phase of a LITE Mode session, showing how user audio is processed and avatar video is streamed. ```mermaid graph LR User["User"] -->|"user audio"| Room["Session Room"] Room -->|"avatar video"| User Room -->|"audio sent to agent"| Agent["Developer Agent"] API["LiveAvatar API"] -->|"stream avatar video"| Room Agent -->|"ws command event"| API API -->|"ws server event"| Agent ``` -------------------------------- ### Listen for user.push_to_talk_stopped Event Source: https://docs.liveavatar.com/docs/full-mode/events Confirms that Push-to-Talk (PTT) successfully stopped. Listen for this on the `agent-response` topic. ```json { "event_type": "user.push_to_talk_stopped" } ``` -------------------------------- ### Listen for ElevenLabs Agent Events (Web SDK) Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent Subscribe to ElevenLabs agent events using the LiveAvatar Web SDK. This allows you to react to specific events from your ElevenLabs agent in real-time. ```typescript import { AgentEventsEnum } from "@heygen/liveavatar-web-sdk"; session.on(AgentEventsEnum.ELEVENLABS_AGENT_EVENT, (event) => { console.log("ElevenLabs event:", event.elevenlabs_event_type, event.data); }); ``` -------------------------------- ### Avatar Token Usage Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent Clarifies that `livekit_config.livekit_client_token` is for the avatar's participant token, and a separate viewer token is needed for the browser client. ```python `livekit_config.livekit_client_token` is the **avatar's** token, not the viewer's. The avatar uses it to join your room as a participant. Mint the viewer token separately for the browser client. ``` -------------------------------- ### Listen for user.transcription Event Source: https://docs.liveavatar.com/docs/full-mode/events Provides the transcription of the user's speech. Listen for this on the `agent-response` topic. ```json { "event_type": "user.transcription", "payload": {"text": "User's transcribed speech."} } ``` -------------------------------- ### Send avatar.speak_response Command Source: https://docs.liveavatar.com/docs/full-mode/events Instructs the avatar to generate an LLM response to the input text, then speak it. Send this to the `agent-control` topic. ```json { "event_type": "avatar.speak_response", "payload": {"text": "What is the weather today?"} } ``` -------------------------------- ### Graph: Full Mode Session Participants Source: https://docs.liveavatar.com/docs/full-mode/lifecycle Visualizes the participants in a FULL Mode session, including the End User, LiveAvatar Managed Room, Agent, and Avatar. The Agent and Avatar are managed by LiveAvatar and join the room on your behalf. ```mermaid graph LR User["End User"] subgraph LA["LiveAvatar Managed"] Room["LiveKit Room"] Agent["Agent"] Avatar["Avatar"] end User <-->|"audio, video, events"| Room Room <-->|"events"| Agent Room <-->|"avatar stream"| Avatar Agent -->|"drives"| Avatar ``` -------------------------------- ### Configure LiveAvatar API Key and Avatar ID Source: https://docs.liveavatar.com/docs/lite-mode/plugins/livekit Set the LIVEAVATAR_API_KEY and optionally LIVEAVATAR_AVATAR_ID environment variables for plugin configuration. The avatar_id can be overridden by a keyword argument in AvatarSession. ```bash LIVEAVATAR_API_KEY=... # required — your LiveAvatar key LIVEAVATAR_AVATAR_ID=... # optional — overridden by avatar_id= kwarg if set ```