### Clone and Install Dependencies

Source: https://docs.liveavatar.com/docs/guides/change-background

Clone the LiveAvatar web SDK repository, navigate to the directory, install dependencies using pnpm, and copy the example environment file.

```bash
git clone https://github.com/heygen-com/liveavatar-web-sdk.git
cd liveavatar-web-sdk
pnpm install
cp apps/bg-removal-demo/.env.example apps/bg-removal-demo/.env.local
# fill in API_KEY and defaults in apps/bg-removal-demo/.env.local
```

--------------------------------

### Install Dependencies and Run Agent

Source: https://docs.liveavatar.com/docs/guides/livekit/custom-livekit-agent

Install project dependencies using uv and run the LiveKit agent demo script.

```bash
uv sync
python src/liveavatar_hosted_demo.py
```

--------------------------------

### Run HeyGenTransport Example

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Execute the HeyGenTransport example script. This setup is ideal if LiveAvatar manages the LiveKit room. The script will output a room URL for connecting a LiveKit client.

```shell
uv run python examples/video-avatar/video-avatar-heygen-transport.py
```

--------------------------------

### Start LiveAvatar Agent

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/vision-agents

This snippet shows the basic setup for an Agent using LiveAvatar, Gemini LLM, Deepgram for TTS/STT, and GetStream for edge communication. It includes creating a call and sending an initial response.

```python
import asyncio
from uuid import uuid4
from dotenv import load_dotenv

from vision_agents.core import Agent, User
from vision_agents.plugins import deepgram, gemini, getstream, liveavatar

load_dotenv()


async def start_avatar_agent():
    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Avatar Agent", id="agent"),
        instructions="You're a friendly AI assistant. Keep responses short.",
        llm=gemini.LLM("gemini-3-flash-preview"),
        tts=deepgram.TTS(),
        stt=deepgram.STT(),
        avatar=liveavatar.Avatar(),
    )

    call = await agent.create_call("default", str(uuid4()))

    async with agent.join(call):
        await agent.simple_response("Hello! I'm your AI assistant with an avatar.")
        await agent.finish()


if __name__ == "__main__":
    asyncio.run(start_avatar_agent())
```

--------------------------------

### Run HeyGenVideoService Example with Daily Transport

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Execute the HeyGenVideoService example specifying the Daily transport. This integrates the avatar into a Daily.co room.

```shell
uv run python examples/video-avatar/video-avatar-heygen-video-service.py --transport daily
```

--------------------------------

### Start the Background Removal Demo Server

Source: https://docs.liveavatar.com/docs/guides/change-background

Run this command from the monorepo root to start the development server for the background removal demo. Open the provided URL in your browser to interact with the demo.

```bash
pnpm demo:bg-removal
```

--------------------------------

### Start the session

Source: https://docs.liveavatar.com/docs/full-mode/overview

Use the session token to start the session and initialize the WebRTC room.

```APIDOC
## Start the session

### Description
Use the session token to start the session and initialize the WebRTC room.

### Method
POST

### Endpoint
https://api.liveavatar.com/v1/sessions/start

### Parameters
#### Request Headers
- **authorization** (string) - Required - Bearer token (e.g., "Bearer <session_token>")

### Response
#### Success Response (200)
- **livekit_url** (string) - The URL for the LiveKit room.
- **livekit_client_token** (string) - The client token for connecting to LiveKit.
```

--------------------------------

### Install LiveAvatar Agent Skills Manually

Source: https://docs.liveavatar.com/docs/agent-skills

Manually install LiveAvatar agent skills by cloning the repository and creating symbolic links. This method is useful if the CLI is not available or for custom setups.

```bash
git clone https://github.com/heygen-com/liveavatar-agent-skills.git

# Symlink all skills to personal skills directory (available in all projects)
for skill in liveavatar-agent-skills/skills/*/; do
  ln -s "$(pwd)/$skill" ~/.claude/skills/$(basename "$skill")
done
```

--------------------------------

### Install LiveKit Agents with LiveAvatar

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/livekit

Install the LiveKit agents package with the LiveAvatar extra using uv. Alternatively, pip install works if you are not using uv.

```shell
uv add "livekit-agents[liveavatar]~=1.5"
```

```shell
pip install "livekit-agents[liveavatar]~=1.5"
```

--------------------------------

### Clone and Setup LiveKit Agent Starter

Source: https://docs.liveavatar.com/docs/guides/livekit/custom-livekit-agent

Clone the reference implementation repository and set up the local environment variables for LiveAvatar and LiveKit integration.

```bash
git clone https://github.com/heygen-com/liveavatar-starter-livekit-agent-python
cd liveavatar-starter-livekit-agent-python
cp .env.example .env.local
```

```bash
LIVEAVATAR_API_KEY=...   # from app.liveavatar.com
AVATAR_ID=...            # any avatar in your account; sandbox-compatible by default
LIVEKIT_API_KEY=...      # your LK Cloud project — used for the inference gateway only
LIVEKIT_API_SECRET=...
IS_SANDBOX=true          # default; switch to false for production avatars / billed minutes
```

--------------------------------

### Install Dependencies and Run LiveKit Agent

Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent

Install project dependencies using uv and run the worker and dispatcher scripts in separate terminals. The worker registers against your LiveKit project, and the dispatcher drives a session, connecting the avatar to your room.

```bash
uv sync

# Terminal 1 — register the worker against your LK project
python src/worker.py dev

# Terminal 2 — drive a session
python src/byo_livekit_demo.py
```

--------------------------------

### Install LiveAvatar Web SDK

Source: https://docs.liveavatar.com/docs/full-mode/overview

Install the LiveAvatar Web SDK using npm for production integration. This SDK is used to connect from your frontend to the LiveKit room.

```bash
npm install @heygen/liveavatar-web-sdk
```

--------------------------------

### Start a LiveAvatar Session

Source: https://docs.liveavatar.com/docs/full-mode/overview

Use the generated session token to start the session and initialize the WebRTC room. Requires the session token obtained in the previous step.

```bash
curl -X POST https://api.liveavatar.com/v1/sessions/start \
  -H "accept: application/json" \
  -H "authorization: Bearer <session_token>"
```

--------------------------------

### Sequence Diagram: Starting a LITE Mode Session

Source: https://docs.liveavatar.com/docs/lite-mode/lifecycle

Illustrates the flow for starting a LITE Mode session, from user interaction to API calls for token generation and session initiation.

```mermaid
sequenceDiagram
    participant User as End User
    participant Frontend as Developer Frontend
    participant Agent as Developer API / Agent
    participant Room as Session Room
    participant API as LiveAvatar API

    User->>Frontend: User visits site
    Frontend->>Agent: Start Session Flow
    Agent->>API: POST /v1/sessions/token
    API-->>Agent: Session scoped token
    Agent->>API: POST /v1/sessions/start
    API-->>Agent: Session Details
    Note right of API: LiveAvatar sends the<br/>Avatar to Room
    Agent->>Room: Establish agent connection
    Frontend->>Room: Establish user connection
    Room-->>Frontend: Conversation Start
    Frontend-->>User: User sees Avatar conversation
```

--------------------------------

### Send user.start_push_to_talk Command

Source: https://docs.liveavatar.com/docs/full-mode/events

Signals to start capturing user audio for Push-to-Talk (PTT) sessions. Send this to the `agent-control` topic.

```json
{
  "event_type": "user.start_push_to_talk"
}
```

--------------------------------

### Configure API Keys for Pipecat Examples

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Set up your environment variables by creating a .env file at the repository root with necessary API keys for LiveAvatar, Deepgram, Cartesia, Google, and optionally Daily.

```bash
HEYGEN_LIVE_AVATAR_API_KEY=...   # https://app.liveavatar.com → API settings
DEEPGRAM_API_KEY=...             # https://console.deepgram.com
CARTESIA_API_KEY=...             # https://play.cartesia.ai
GOOGLE_API_KEY=...               # https://aistudio.google.com/apikey
DAILY_API_KEY=...                # https://dashboard.daily.co (only for Daily transport)
```

--------------------------------

### Listen for user.push_to_talk_started Event

Source: https://docs.liveavatar.com/docs/full-mode/events

Confirms that Push-to-Talk (PTT) successfully started. Listen for this on the `agent-response` topic.

```json
{
  "event_type": "user.push_to_talk_started"
}
```

--------------------------------

### Install VisionAgents with LiveAvatar (pip)

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/vision-agents

Use 'pip install' to install the VisionAgents package with the LiveAvatar plugin. This command ensures all necessary components are included for LiveAvatar integration.

```shell
pip install "vision-agents[liveavatar]"
```

--------------------------------

### Setup Chroma Keying Toggle

Source: https://docs.liveavatar.com/docs/guides/change-background

Wire up a toggle to switch between raw video and a canvas with chroma keying applied. Ensure to import `setupChromaKey` and manage the processing state.

```typescript
import { setupChromaKey } from "./chromaKey";

const videoElement = document.getElementById("avatarVideo") as HTMLVideoElement;
const canvasElement = document.getElementById("avatarCanvas") as HTMLCanvasElement;
const chromaKeyToggle = document.getElementById("chromaKeyToggle") as HTMLInputElement;

let stopChromaKeyProcessing: (() => void) | null = null;

function updateChromaKeyState() {
  if (!videoElement.srcObject) return;

  if (stopChromaKeyProcessing) {
    stopChromaKeyProcessing();
    stopChromaKeyProcessing = null;
  }

  if (chromaKeyToggle.checked) {
    canvasElement.style.display = "block";
    videoElement.style.display = "none";

    stopChromaKeyProcessing = setupChromaKey(videoElement, canvasElement, {
      minHue: 60,
      maxHue: 180,
      minSaturation: 0.1,
      threshold: 1.0,
    });
  } else {
    videoElement.style.display = "block";
    canvasElement.style.display = "none";
  }
}

chromaKeyToggle.addEventListener("click", updateChromaKeyState);
```

--------------------------------

### Run HeyGenVideoService Example with WebRTC Transport

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Run the HeyGenVideoService example using the WebRTC transport for a local, browser-based test. The script will print a URL (default http://localhost:7860) to access the test.

```shell
uv run python examples/video-avatar/video-avatar-heygen-video-service.py --transport webrtc
```

--------------------------------

### Listen for user.speak_started Event

Source: https://docs.liveavatar.com/docs/full-mode/events

Indicates that the user has started sending audio. Listen for this on the `agent-response` topic.

```json
{
  "event_type": "user.speak_started"
}
```

--------------------------------

### Deploy LiveKit Agent to LiveKit Cloud

Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent

Use these commands for initial setup and subsequent deployments of your agent to LiveKit Cloud. Remember to commit changes to `livekit.toml` after running `lk agent create`.

```bash
# First-time setup — writes subdomain + agent id back into livekit.toml
lk agent create --secrets-file .env.local

# Subsequent updates
lk agent deploy

# Inspect
lk agent status
lk agent logs
```

--------------------------------

### Sequence Diagram: Starting a Full Mode Session

Source: https://docs.liveavatar.com/docs/full-mode/lifecycle

Illustrates the sequence of interactions between the end user, frontend, backend, LiveAvatar API, and the managed room when starting a FULL Mode session. Ensure the session token has 'mode' set to 'FULL'.

```mermaid
sequenceDiagram
    participant User as End User
    participant Frontend as Developer Frontend
    participant Backend as Developer API
    participant API as LiveAvatar API
    participant Room as LiveAvatar Managed Room

    User->>Frontend: User visits site
    Frontend->>Backend: Start session flow
    Backend->>API: POST /v1/sessions/token
    API-->>Backend: Session-scoped token
    Backend-->>Frontend: Send token back
    User->>Frontend: User starts the session
    Frontend->>API: POST /v1/sessions/start (with token)
    API->>Room: Start room with avatar
    API-->>Frontend: Room client token + session details
    Frontend->>Room: Establish room connection with client token
    Room-->>Frontend: Conversation starts
    Frontend-->>User: User sees avatar conversation
```

--------------------------------

### Listen for user.push_to_talk_start_failed Event

Source: https://docs.liveavatar.com/docs/full-mode/events

Indicates that Push-to-Talk (PTT) failed to start. Listen for this on the `agent-response` topic.

```json
{
  "event_type": "user.push_to_talk_start_failed"
}
```

--------------------------------

### Install VisionAgents with LiveAvatar (uv)

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/vision-agents

Use 'uv add' to install the VisionAgents package with the LiveAvatar plugin. This is a common method for managing Python dependencies.

```shell
uv add "vision-agents[liveavatar]"
```

--------------------------------

### Realtime LLM Variant Setup

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/vision-agents

Configure an Agent to use a Realtime LLM with LiveAvatar. This setup bypasses the Text-to-Speech (TTS) step by forwarding the LLM's audio output directly to the avatar.

```python
from vision_agents.plugins import gemini, getstream, liveavatar

agent = Agent(
    edge=getstream.Edge(),
    agent_user=User(name="Avatar Agent", id="agent"),
    instructions="You're a friendly AI assistant.",
    llm=gemini.Realtime(),
    avatar=liveavatar.Avatar(is_sandbox=False),
)
```

--------------------------------

### Join LiveKit Room for Testing

Source: https://docs.liveavatar.com/docs/full-mode/overview

For a quick test, open the provided LiveKit URL directly in your browser. This URL includes the livekit_url and livekit_client_token obtained after starting the session.

```text
https://meet.livekit.io/custom?liveKitUrl=<livekit_url>&token=<livekit_client_token>
```

--------------------------------

### Setup Push-to-Talk Session

Source: https://docs.liveavatar.com/docs/full-mode/push-to-talk

Configure the session token to enable Push-to-Talk mode by setting `interactivity_type` to `PUSH_TO_TALK`.

```json
{
  "mode": "FULL",
  "interactivity_type": "PUSH_TO_TALK",
  "avatar_id": "<avatar_id>",
  "avatar_persona": {
    "voice_id": "<voice_id>",
    "context_id": "<context_id>"
  }
}
```

--------------------------------

### Install Pipecat HeyGen Plugin (uv)

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Use this command to add the pipecat-ai plugin with HeyGen support using uv.

```shell
uv add "pipecat-ai[heygen]"
```

--------------------------------

### Install LiveAvatar Agent Skills CLI

Source: https://docs.liveavatar.com/docs/agent-skills

Use the skills CLI to add the LiveAvatar agent skills package. This is the recommended method for integrating LiveAvatar with AI coding agents.

```bash
npx skills add heygen-com/liveavatar-agent-skills
```

--------------------------------

### Initialize HeyGenVideoService for LiveAvatar

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

This example demonstrates the minimal configuration required to initialize HeyGenVideoService for use with the LiveAvatar service type. Ensure you have the necessary API key and aiohttp client session. For production, replace the sandbox avatar ID and set `is_sandbox` to `False`.

```python
import os
import aiohttp
from pipecat.services.heygen import HeyGenVideoService
from pipecat.services.heygen.client import ServiceType
from pipecat.services.heygen.api_liveavatar import LiveAvatarNewSessionRequest

async with aiohttp.ClientSession() as session:
    heygen = HeyGenVideoService(
        api_key=os.environ["HEYGEN_LIVE_AVATAR_API_KEY"],
        session=session,
        service_type=ServiceType.LIVE_AVATAR,
        session_request=LiveAvatarNewSessionRequest(
            # Sandbox mode only supports this fixed avatar_id and is_sandbox=True.
            # Remove both when moving to production and use your own avatar_id.
            avatar_id="dd73ea75-1218-4ef3-92ce-606d5f7fbc0a",
            is_sandbox=True,
        ),
    )
```

--------------------------------

### LiveKit Agent Worker Subcommands

Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent

The worker script supports multiple subcommands for different operational modes. 'dev' is for local development with hot-reloading, 'start' is the production entrypoint, and 'download-files' pre-downloads model weights for faster cold starts.

```python
src/worker.py
```

--------------------------------

### Install Pipecat HeyGen Plugin (pip)

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Use this command to add the pipecat-ai plugin with HeyGen support using pip.

```shell
pip install "pipecat-ai[heygen]"
```

--------------------------------

### Listen for avatar.speak_started Event

Source: https://docs.liveavatar.com/docs/full-mode/events

Indicates that the avatar has started speaking. Listen for this on the `agent-response` topic.

```json
{
  "event_type": "avatar.speak_started"
}
```

--------------------------------

### Start LiveAvatar Session

Source: https://docs.liveavatar.com/docs/faq/firewall

Initiate a LiveAvatar session by sending a POST request with your session token to the LiveAvatar API. A successful response provides the necessary URLs and tokens for establishing a WebRTC connection. Failures here indicate an issue with the session token or request, not network configuration.

```bash
curl -X POST https://api.liveavatar.com/v1/sessions/start \
  -H "accept: application/json" \
  -H "authorization: Bearer <session_token>"
```

--------------------------------

### Starting a Conversational AI Agent Session

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/agora

Initiates a conversational AI agent session by calling the Agora API. This includes joining an RTC channel and configuring the LiveAvatar. Ensure you have your Agora App ID and API key.

```bash
POST https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join
Authorization: Basic <your_basic_auth_header>
Content-Type: application/json

{
  "properties": {
    "avatar": {
      "api_key": "<your_agora_api_key>",
      "avatar_id": "<your_avatar_id>",
      "agora_uid": "<your_agora_uid>",
      "agora_token": "<your_agora_token>"
    }
  },
  "llm": {
    "provider": "openai",
    "model": "gpt-4o"
  },
  "asr": {
    "provider": "agora",
    "model": "en-US"
  },
  "tts": {
    "provider": "agora",
    "voice_id": "alloy"
  }
}
```

--------------------------------

### Start Session with Custom LLM

Source: https://docs.liveavatar.com/docs/full-mode/custom-llm

Initiate a session using your custom LLM by setting the `llm_configuration_id` in the session token. This JSON payload configures the session mode, avatar, and LLM details.

```json
{
  "mode": "FULL",
  "avatar_id": "<avatar_id>",
  "llm_configuration_id": "<llm_configuration_id>",
  "avatar_persona": {
    "voice_id": "<voice_id>",
    "context_id": "<context_id>"
  }
}
```

--------------------------------

### Start Lite Session with ElevenLabs Agent Config

Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent

Configure a LITE mode session to connect with an ElevenLabs Agent. Provide your avatar ID and the `secret_id` obtained after registering your ElevenLabs API key, along with the ElevenLabs Agent ID.

```json
{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "elevenlabs_agent_config": {
    "secret_id": "<secret_id>",
    "agent_id": "<elevenlabs_agent_id>"
  }
}
```

--------------------------------

### Start Session with Voice Agent and Overrides

Source: https://docs.liveavatar.com/docs/core-concepts/voice-agents

Layer per-session values over a voice agent's defaults for a specific session. Overrides like `language` and `dynamic_variables` are specific to the agent type and will be rejected if not applicable.

```json
{
  "mode": "FULL",
  "avatar_id": "<avatar_id>",
  "voice_agent": {
    "id": "<voice_agent_id>",
    "language": "en",
    "dynamic_variables": {
      "user_name": "Jordan",
      "plan": "Pro"
    }
  }
}
```

--------------------------------

### Start LITE Session with OpenAI Realtime Config

Source: https://docs.liveavatar.com/docs/lite-mode/connectors/openai-realtime

Initiate a LITE mode session, configuring it to use the OpenAI Realtime connector. This JSON payload specifies the avatar, the secret ID for your OpenAI API key, an optional context ID for system prompts, and desired voice and model parameters.

```json
{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "openai_realtime_config": {
    "secret_id": "<secret_id>",
    "context_id": "<context_id>",
    "voice": "alloy",
    "model": "gpt-realtime",
    "temperature": 0.8
  }
}
```

--------------------------------

### Start a LITE Session with Gemini Realtime Config

Source: https://docs.liveavatar.com/docs/lite-mode/connectors/gemini-realtime

Initiate a LITE mode session with the Gemini Realtime Connector. Ensure `secret_id`, `avatar_id`, and optionally `context_id` are correctly provided. The `gemini_realtime_config` specifies the Gemini Live API integration details.

```json
{
  "mode": "LITE",
  "avatar_id": "<avatar_id>",
  "gemini_realtime_config": {
    "secret_id": "<secret_id>",
    "context_id": "<context_id>",
    "voice": "Puck",
    "model": "gemini-3.1-flash-live-preview",
    "temperature": 0.8
  }
}
```

--------------------------------

### Clone and Configure LiveKit Agent Starter

Source: https://docs.liveavatar.com/docs/guides/livekit/byo-livekit-agent

Clone the reference implementation repository and set up your local environment variables for LiveAvatar and LiveKit integration. Ensure all required credentials are provided in the .env.local file.

```bash
git clone https://github.com/heygen-com/liveavatar-starter-livekit-agent-python
cd liveavatar-starter-livekit-agent-python
cp .env.example .env.local
```

```bash
LIVEAVATAR_API_KEY=...   # from app.liveavatar.com
AVATAR_ID=...            # any avatar in your account; sandbox-compatible by default
LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=...      # your LK Cloud project — used for the room AND inference
LIVEKIT_API_SECRET=...
IS_SANDBOX=true          # default; switch to false for production avatars / billed minutes
```

--------------------------------

### Start Session with Voice Agent

Source: https://docs.liveavatar.com/docs/core-concepts/voice-agents

Reference a voice agent by its ID when starting a session in FULL Mode. This is mutually exclusive with `avatar_persona`.

```json
{
  "mode": "FULL",
  "avatar_id": "<avatar_id>",
  "voice_agent": {
    "id": "<voice_agent_id>"
  }
}
```

--------------------------------

### Clone Pipecat Repository and Install Dependencies

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

Clone the official Pipecat repository and install development dependencies including extras for GStreamer and local development.

```shell
git clone https://github.com/pipecat-ai/pipecat.git
cd pipecat
uv sync --group dev --all-extras --no-extra gstreamer --no-extra local
```

--------------------------------

### Send avatar.start_listening Command

Source: https://docs.liveavatar.com/docs/full-mode/events

Switches the avatar to a listening state from idle. Send this to the `agent-control` topic.

```json
{
  "event_type": "avatar.start_listening"
}
```

--------------------------------

### Graph: Recommended Architecture for Session Management

Source: https://docs.liveavatar.com/docs/full-mode/lifecycle

Depicts the recommended architecture where the frontend acts as the primary controller for emitting events from the LiveKit room to minimize latency. Event data can be relayed to the backend for further processing.

```mermaid
graph LR
    Frontend["Developer Frontend"] <-->|"send/receive events"| Room["LiveAvatar Managed Room"]
    Frontend <-->|"avatar video stream"| Room
    Frontend -->|"relay event data"| Backend["Developer API"]
    Agent["LiveAvatar Agent"] <--> Room
```

--------------------------------

### Stopping a Conversational AI Agent Session

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/agora

Terminates an ongoing conversational AI agent session, including the associated LiveAvatar. Use the agent ID obtained when starting the session.

```bash
POST https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/agents/:agent_id/leave
Authorization: Basic <your_basic_auth_header>
```

--------------------------------

### Start Conversational AI Session with LiveAvatar

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/agora

Initiates a conversational AI session by joining an Agora RTC channel. This endpoint configures the AI agent and integrates LiveAvatar for avatar-based responses.

```APIDOC
## POST /join

### Description
Initiates a conversational AI session, configuring the AI agent and integrating LiveAvatar. The response provides an `agent_id` for future reference.

### Method
POST

### Endpoint
`https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join`

### Parameters
#### Path Parameters
- **appid** (string) - Required - Your Agora project application ID.

#### Request Body
- **properties.avatar** (object) - Required - Configuration block for LiveAvatar integration.
  - **api_key** (string) - Required - Your LiveAvatar API key.
  - **avatar_id** (string) - Required - The ID of the LiveAvatar to use.
  - **agora_uid** (string) - Required - The unique Agora user ID for the LiveAvatar.
  - **agora_token** (string) - Required - The Agora RTC token for the LiveAvatar.
- **llm** (object) - Required - Configuration for the Large Language Model.
- **asr** (object) - Required - Configuration for the Automatic Speech Recognition.
- **tts** (object) - Required - Configuration for the Text-to-Speech engine.

### Request Example
```json
{
  "properties": {
    "avatar": {
      "api_key": "YOUR_LIVEAVATAR_API_KEY",
      "avatar_id": "YOUR_AVATAR_ID",
      "agora_uid": "avatar_agora_uid",
      "agora_token": "avatar_rtc_token"
    }
  },
  "llm": { ... },
  "asr": { ... },
  "tts": { ... }
}
```

### Response
#### Success Response (200)
- **agent_id** (string) - The unique identifier for the started agent session.
```

--------------------------------

### HTML Structure for Video and Chroma Key Toggle

Source: https://docs.liveavatar.com/docs/guides/change-background

Sets up the necessary HTML elements: a video element for the stream, a canvas for the keyed output, and a checkbox to control the chroma key effect.

```html
<article style="width: fit-content;">
  <video id="avatarVideo" autoplay playsinline></video>
  <canvas id="avatarCanvas"></canvas>
</article>

<!-- Chroma Key Toggle -->
<div class="chromakey-toggle">
  <input type="checkbox" id="chromaKeyToggle" />
  <label for="chromaKeyToggle">Enable Chroma Keying</label>
</div>
```

--------------------------------

### ElevenLabs Agent Event Payload Structure

Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent

This is an example of the JSON payload structure for an ElevenLabs agent event received on a LiveKit data channel. It includes event metadata and the original ElevenLabs data.

```json
{
  "event_id": "abc-123",
  "event_type": "elevenlabs_agent_event",
  "session_id": "session-456",
  "elevenlabs_event_type": "agent_response",
  "data": { ... }
}
```

--------------------------------

### Integrate LiveAvatar into Agent Entrypoint

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/livekit

Drop `AvatarSession` into your agent entrypoint. This involves adding imports, constructing `AvatarSession`, and calling `avatar.start()` before `session.start()` to hook the TTS output to the avatar.

```python
from livekit import agents
from livekit.agents import AgentServer, AgentSession
from livekit.plugins import liveavatar

server = AgentServer()

@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: agents.JobContext):
    session = AgentSession(
        # ... your existing stt, llm, tts, vad, turn_detection
    )

    avatar = liveavatar.AvatarSession(
        avatar_id="...",  # or rely on LIVEAVATAR_AVATAR_ID
    )

    # Avatar joins the room and hooks the session's TTS output.
    await avatar.start(session, room=ctx.room)

    # Now start the conversation loop.
    await session.start(
        # ... room, agent, room_options, etc.
    )
```

--------------------------------

### Initialize HeyGenTransport for LiveAvatar

Source: https://docs.liveavatar.com/docs/lite-mode/plugins/pipecat

This snippet shows the minimal configuration required to initialize the HeyGenTransport for LiveAvatar. Ensure you set `service_type` to `ServiceType.LIVE_AVATAR` and provide a matching `LiveAvatarNewSessionRequest`. The `api_key` should be retrieved from environment variables.

```python
import os
import aiohttp
from pipecat.transports.heygen import HeyGenTransport
from pipecat.services.heygen.client import ServiceType
from pipecat.services.heygen.api_liveavatar import LiveAvatarNewSessionRequest

async with aiohttp.ClientSession() as session:
    transport = HeyGenTransport(
        session=session,
        api_key=os.environ["HEYGEN_LIVE_AVATAR_API_KEY"],
        service_type=ServiceType.LIVE_AVATAR,
        session_request=LiveAvatarNewSessionRequest(
            # Sandbox mode only supports this fixed avatar_id and is_sandbox=True.
            # Remove both when moving to production and use your own avatar_id.
            avatar_id="dd73ea75-1218-4ef3-92ce-606d5f7fbc0a",
            is_sandbox=True,
        ),
    )
```

--------------------------------

### Register ElevenLabs API Key

Source: https://docs.liveavatar.com/docs/lite-mode/connectors/elevenlabs-agent

Use this `curl` command to register your ElevenLabs API key with LiveAvatar. Ensure you replace placeholders with your actual API key and desired secret name. This step is required before starting a session.

```bash
curl -X POST https://api.liveavatar.com/v1/secrets \
  -H "X-API-KEY: <YOUR_API_KEY>" \
  -H "content-type: application/json" \
  -d '{
    "secret_type": "ELEVENLABS_API_KEY",
    "secret_value": "<elevenlabs_api_key>",
    "secret_name": "ElevenLabs Agent Key"
  }'
```

--------------------------------

### Create Session Token in Sandbox Mode

Source: https://docs.liveavatar.com/docs/sandbox-mode

Use this `curl` command to create a session token for Sandbox Mode. Ensure you replace `<YOUR_API_KEY>` and `<voice_id>` with your actual values. The `is_sandbox` flag must be set to `true`.

```bash
curl -X POST https://api.liveavatar.com/v1/sessions/token \
  -H "X-API-KEY: <YOUR_API_KEY>" \
  -H "content-type: application/json" \
  -d '{
    "mode": "FULL",
    "is_sandbox": true,
    "avatar_id": "dd73ea75-1218-4ef3-92ce-606d5f7fbc0a",
    "avatar_persona": {
      "voice_id": "<voice_id>",
      "language": "en"
    }
  }'
```