### Install Local TURN Server Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/guide/deployment.md Execute the setup script to install a local TURN server. Ensure firewall rules are updated to allow coturn ports. ```bash chmod 777 scripts/setup_coturn.sh scripts/setup_coturn.sh ``` -------------------------------- ### Install Dependencies Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/musetalk.md Run this command to install project dependencies using the specified configuration file. ```bash uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml ``` -------------------------------- ### Install and Run FlashHead Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/flashhead.md Execute these commands to install dependencies, download necessary models for FlashHead, and run the demo with a specified configuration. ```bash uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` ```bash uv run scripts/download_models.py --handler flashhead ``` ```bash uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` -------------------------------- ### Absolute Path Example for plugin_tools_cmd Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/beta/chat-agent.md An example demonstrating the required absolute path format for the `plugin_tools_cmd` parameter in the OAC configuration. This is crucial as OAC may start from a different working directory. ```yaml plugin_tools_cmd: "node /home/luyi/Code/openclaw/dist/mcp/plugin-tools-serve.js" ``` -------------------------------- ### OAC and OpenClaw Startup Commands Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/beta/chat-agent.md Sequence of bash commands to start OpenClaw gateway, install OAC dependencies, download models, and start the OAC Agent mode. Ensure both OAC and OpenClaw run on the same machine. ```bash # 1. Start OpenClaw Gateway (in Terminal A) cd ~/Code/openclaw openclaw gateway run # 2. Install OAC dependencies (in Terminal B) cd ~/Code/OpenAvatarChat uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead_duplex_agent.yaml # 3. Download models uv run scripts/download_models.py --handler flashhead # 4. Start OAC Agent mode uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead_duplex_agent.yaml ``` -------------------------------- ### Start FlashHead Application Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/getting-started/flashhead.md Launch the demo application using the specified configuration file. ```bash uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` -------------------------------- ### Install UV (PyPI) Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Installs the UV package manager using pip. This is an alternative to the standalone installer. ```bash pip install uv ``` -------------------------------- ### Windows Installation Commands Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/tts/cosyvoice-local.md Use these commands to set up the Conda environment, install necessary packages including pynini, and run the installation and demo scripts for the CosyVoice Local Inference Handler on Windows. ```bash conda create -n openavatarchat python=3.10 conda activate openavatarchat conda install -c conda-forge pynini==2.1.6 set VIRTUAL_ENV=%CONDA_PREFIX% uv run --active install.py --config config/chat_with_openai_compatible.yaml uv run --active src/demo.py --config config/chat_with_openai_compatible.yaml ``` -------------------------------- ### Run MuseTalk Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/musetalk.md Start the MuseTalk demo application with the provided configuration file. ```bash uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml ``` -------------------------------- ### Install Dependencies and Compile Flash-Attention Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/getting-started/flashhead.md Run this command to install dependencies, which includes automatically compiling flash-attention. This step may take a while on the first run. ```bash uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` -------------------------------- ### Install and Run LiteAvatar Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/liteavatar.md Use these commands to install dependencies, download models, and run the LiteAvatar demo with the specified configuration. Ensure the DASHSCOPE_API_KEY environment variable is set or a .env file is present. ```bash uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml ``` ```bash uv run scripts/download_models.py --handler liteavatar ``` ```bash uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml ``` -------------------------------- ### Install Dependencies and Run LAM Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/lam.md Execute these commands sequentially to install project dependencies, download necessary models for the LAM handler, and run the demonstration script with the specified configuration. ```bash uv run install.py --config config/chat_with_lam.yaml ``` ```bash uv run scripts/download_models.py --handler lam ``` ```bash uv run src/demo.py --config config/chat_with_lam.yaml ``` -------------------------------- ### Install UV (Standalone) Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Installs the UV package manager using the official standalone installer script for macOS and Linux. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` -------------------------------- ### Install Dependencies with install.py Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Installs project dependencies by parsing configuration YAML files. Use --config for specific files or --all for all handler dependencies. ```bash uv run install.py --config .yaml ``` ```bash uv run install.py --config config/a.yaml --config config/b.yaml ``` ```bash uv run install.py --all ``` -------------------------------- ### Clone Project and Install Dependencies Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/readme_en.md Clone the OpenAvatarChat repository, update submodules, install dependencies using a specific configuration, and download necessary models. ```bash # Clone the project git clone https://github.com/HumanAIGC-Engineering/OpenAvatarChat.git cd OpenAvatarChat git submodule update --init --recursive --depth 1 # Install dependencies (LiteAvatar + Bailian API example) uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml # Download models uv run scripts/download_models.py --handler liteavatar # Start uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml ``` -------------------------------- ### Absolute Path Example for plugin_tools_cmd Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/beta/chat-agent.md This example demonstrates the required format for the `plugin_tools_cmd` parameter, emphasizing the need for an absolute path to the OpenClaw build artifact. ```yaml plugin_tools_cmd: "node /home/luyi/Code/openclaw/dist/mcp/plugin-tools-serve.js" ``` -------------------------------- ### Run MuseTalk Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/musetalk.md Commands to install dependencies, download models, and run the MuseTalk demo with a specified configuration file. ```bash uv run install.py --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml ``` ```bash uv run scripts/download_models.py --handler musetalk ``` ```bash uv run src/demo.py --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml ``` -------------------------------- ### Install Git LFS and Update Submodules Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/community/faq.md Ensure git-lfs is installed and submodules are updated to resolve dependency installation issues. This is a common first step for setting up the project. ```bash sudo apt install git-lfs git lfs install git submodule update --init --recursive --depth 1 ``` -------------------------------- ### Example Configuration Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md A sample YAML configuration for AvatarMusetalk, specifying parameters like FPS, batch size, model paths, and debugging options. Ensure output audio sample rate is a factor of FPS and matches the upstream TTS sample rate. ```yaml AvatarMusetalk: module: avatar/musetalk/avatar_handler_musetalk fps: 25 # 会被自动校正为 output_audio_sample_rate 的因子 batch_size: 2 # 低延迟场景建议 2,高吞吐场景可提高(必须 ≥2) avatar_video_path: "resource/avatar/liteavatar/20250408/sample_data/bg_video_silence.mp4" avatar_model_dir: "models/musetalk/avatar_model" force_create_avatar: false # 首次使用新视频自动生成,之后加载缓存 debug: false multi_thread_inference: true # 将 UNet 和 VAE 拆分到独立线程,默认开启 output_audio_sample_rate: 24000 # 需与上游 TTS 采样率一致 ``` -------------------------------- ### Manager Configuration Example Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/manager/data-tool.md Configure the Manager handler by adding it to your YAML configuration file under `handler_configs`. Adjust buffer, preview, and binary preview settings as needed. ```yaml handler_configs: # ... other handler configs ... Manager: module: manager/handler_data_tool buffer_limit: 200 # optional, max records per session preview_bytes: 4096 # optional, binary data preview size preview_chars: 512 # optional, text payload preview size include_binary_preview: false # optional, include base64 preview for ndarray data ``` -------------------------------- ### Install Dependencies and Build OpenClaw Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/beta/chat-agent.md Navigate to OpenClaw's root directory, install dependencies using pnpm, which automatically links the oac-bridge workspace package. Then, build OpenClaw. This build step is crucial as the OAC-side 'plugin_tools_cmd' points to the compiled output. ```bash cd ~/Code/openclaw # Install dependencies (automatically links the oac-bridge workspace package) pnpm install # Build OpenClaw (oac-bridge requires the compiled output) pnpm build ``` -------------------------------- ### Download FlashHead Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/flashhead.md Use this command to download the necessary models for the FlashHead handler. Ensure you have uv installed. ```bash uv run scripts/download_models.py --handler flashhead ``` -------------------------------- ### Start OpenClaw Gateway Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/beta/chat-agent.md Run this command to start the OpenClaw Gateway. After a successful startup, you should see a confirmation message in the OpenClaw logs indicating that the HTTP route for the OAC Bridge has been registered. ```bash openclaw gateway run ``` -------------------------------- ### Start Docker Compose Services Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Use Docker Compose to start the Open Avatar Chat service along with the coturn service. The default configuration is chat_with_openai_compatible_bailian_cosyvoice.yaml. ```bash docker compose up ``` -------------------------------- ### Run Docker Container with LiteAvatar Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Start the Docker container using the LiteAvatar configuration. This is the default and lightweight option. ```bash bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml ``` -------------------------------- ### Install Git LFS Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Installs Git Large File Storage (LFS) which is required for handling submodules and model dependencies. ```bash sudo apt install git-lfs git lfs install ``` -------------------------------- ### Run Docker Container with FlashHead Avatar Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Start the Docker container with the FlashHead (high-quality avatar) configuration. This option is GPU-intensive and has a concurrent limit of 1. ```bash bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` -------------------------------- ### Install PyTorch with CUDA 12.8 Support Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/FAQ.md Use this command to install PyTorch and related packages compatible with CUDA 12.8, necessary for supporting 50 series GPUs. ```bash #https://pytorch.org/get-started/locally/ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 ``` -------------------------------- ### SmartTurnEOU Handler Configuration Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/vad/smart-turn.md YAML configuration for the SmartTurnEOU handler. Adjust the threshold and model path according to your setup. ```yaml SmartTurnEOU: module: vad/smart_turn_eou/eou_handler_smart_turn threshold: 0.8 model_path: "models/smart_turn/smart-turn-v3.1-cpu.onnx" ``` -------------------------------- ### Download Smart Turn EOU Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/vad/smart-turn.md Use this command to download the required models for the Smart Turn end-of-utterance handler. Ensure you have uv installed. ```bash uv run scripts/download_models.py --handler smart_turn_eou ``` -------------------------------- ### Manually Download wav2vec2-base-960h Model Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/lam.md Clone the wav2vec2-base-960h model repository using git. Ensure you have git installed and configured. ```bash git clone --depth 1 https://huggingface.co/facebook/wav2vec2-base-960h ./models/wav2vec2-base-960h ``` -------------------------------- ### Manually Download LAM_audio2exp Model Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/lam.md Download and extract the LAM_audio2exp_streaming model. This requires wget and tar utilities to be installed. ```bash wget https://huggingface.co/3DAIGC/LAM_audio2exp/resolve/main/LAM_audio2exp_streaming.tar -P ./models/LAM_audio2exp/ tar -xzvf ./models/LAM_audio2exp/LAM_audio2exp_streaming.tar -C ./models/LAM_audio2exp ``` -------------------------------- ### Run Docker Container with MuseTalk Avatar Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Start the Docker container with the MuseTalk (video-driven avatar) configuration. Note that PYTORCH_JIT is automatically set to 0, and concurrent limit is 1. ```bash bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml ``` -------------------------------- ### Run Docker Container with LAM Avatar Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Start the Docker container with the LAM (3D avatar) configuration. This option utilizes WebSocket clients. ```bash bash run_docker_cuda128.sh --config config/chat_with_lam.yaml ``` -------------------------------- ### Multi-Session Reuse Path Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/musetalk_review.md Outlines the process for ending one session and starting another, focusing on context destruction, processor management, and resetting runtime states for the new session. ```text Session A 结束: → destroy_context() → finish_current() playback → processor.stop() → _stop_event.set() → join threads (5s timeout) → _clear_queues → set_callbacks(None) → release processor to pool Session B 开始: → create_context() → acquire processor from pool (try/except 保护,异常则 release) → start_context() → init_playback_streamer() → processor.start() → 检查僵尸线程(extra join 3s) → _reset_runtime_state()(清 interrupted、清队列、重置统计) → 创建新线程 → start()(异常则 _session_running=False + re-raise) ``` -------------------------------- ### Get Handler Detail Configuration Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Defines inputs, outputs, and signal filters for a handler. Specifically configures handling for `AVATAR_AUDIO` and `AVATAR_VIDEO` streams, and filters for `STREAM_CANCEL` signals on `CLIENT_PLAYBACK`. ```python def get_handler_detail(self, session_context, context) -> HandlerDetail: inputs = { ChatDataType.AVATAR_AUDIO: HandlerDataInfo( type=ChatDataType.AVATAR_AUDIO, input_consume_mode=ChatDataConsumeMode.ONCE, # 每条数据只消费一次 ) } outputs = { ChatDataType.AVATAR_AUDIO: HandlerDataInfo(...), ChatDataType.AVATAR_VIDEO: HandlerDataInfo( ..., output_stream_config=ChatStreamConfig( cancelable=False, # 视频流不可被上游取消 auto_link_input=False, # 不自动关联输入流 ), ), } signal_filters = [ # 仅监听 CLIENT_PLAYBACK 的 STREAM_CANCEL 信号 SignalFilterRule(ChatSignalType.STREAM_CANCEL, None, ChatDataType.CLIENT_PLAYBACK), ] ``` -------------------------------- ### Manage OpenAvatarChat Service with Docker Compose Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/getting-started/docker.md Use Docker Compose to start or stop the OpenAvatarChat service along with coturn. The configuration file can be changed in the docker-compose.yml. ```bash # 拉起服务 docker compose up # 关闭服务 docker compose down ``` -------------------------------- ### Run OpenAvatarChat Demo Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Launches the OpenAvatarChat demo application. Requires a configuration file specified with the --config flag. ```bash uv run src/demo.py --config .yaml ``` -------------------------------- ### Handle Method Flow Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Outlines the step-by-step process for the `handle` method, including type checking, stream key change detection, input validation, audio slicing, and end-of-speech handling. ```pseudocode handle(context, inputs, output_definitions): │ ├── 1. 类型检查: inputs.type != AVATAR_AUDIO → 直接返回 │ ├── 2. stream_key 变化检测(线程安全,使用 _stream_key_lock) │ │ │ ├── 获取 stream_key_str (from inputs.stream_id) │ ├── 与 context._current_tts_stream_key 比较 │ ├── need_switch = (stream_key_str 存在 且 与当前不同) │ │ │ └── if need_switch: │ ├── flush 切片器残余音频(丢弃,不处理) │ ├── 关闭上一个 CLIENT_PLAYBACK 流 (streamer.finish_current()) │ └── 为新 stream 打开 CLIENT_PLAYBACK 流 (streamer.open_stream()) │ ├── 3. 输入验证 │ ├── 检查 sample_rate 是否匹配 output_audio_sample_rate(不匹配则返回) │ ├── 音频数据 dtype 转 float32(如需要) │ └── 音频数据为 None 时填充 1 秒静音(容错) │ ├── 4. 音频切片(SliceContext,按 output_audio_sample_rate 切为 1 秒段) │ 对每段: │ audio_segment → MuseTalkSpeechAudio(speech_id, end_of_speech=False, audio_data=bytes) │ → context.processor.add_audio(speech_audio) │ └── 5. speech_end 处理(inputs.is_last_data == True) ├── flush 切片器残余音频 │ ├── 有残余 → 使用残余数据 │ └── 无残余 → 用 2 帧长度的静音填充(确保 speech_end 信号有足够帧来传递) └── MuseTalkSpeechAudio(speech_id, end_of_speech=True, audio_data=...) → context.processor.add_audio(speech_audio) ``` -------------------------------- ### Download Models with download_models.py Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/index.md Downloads necessary models for the project. Specify configuration files with --config, use --all for all handler models, or set --source to 'modelscope' (for Chinese users) or 'huggingface' (for overseas users). ```bash uv run scripts/download_models.py --config .yaml ``` ```bash uv run scripts/download_models.py --all ``` ```bash uv run scripts/download_models.py --config .yaml --source modelscope ``` ```bash uv run scripts/download_models.py --config .yaml --source huggingface ``` -------------------------------- ### Run OpenAvatarChat Container with Different Configurations Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/getting-started/docker.md Execute the run script with the --config flag to launch OpenAvatarChat with various digital human configurations like LiteAvatar, LAM, MuseTalk, and FlashHead. Access the interface at https://localhost:8282. ```bash # LiteAvatar(轻量数字人,默认配置) bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice.yaml # LAM(3D 数字人) bash run_docker_cuda128.sh --config config/chat_with_lam.yaml # MuseTalk(视频驱动数字人) bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_musetalk.yaml # FlashHead(高质量数字人,GPU 资源消耗较大) bash run_docker_cuda128.sh --config config/chat_with_openai_compatible_bailian_cosyvoice_flashhead.yaml ``` -------------------------------- ### Configure Bailian API Key Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Create a .env file in the project root to set environment variables, such as the API key for Bailian services (LLM and CosyVoice TTS). ```bash echo "DASHSCOPE_API_KEY=sk-xxxxx" > .env ``` -------------------------------- ### Download MuseTalk Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/musetalk.md Execute this script to download the necessary models for the MuseTalk handler. ```bash uv run scripts/download_models.py --handler musetalk ``` -------------------------------- ### Clone Project and Download Submodules Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Clone the OpenAvatarChat repository and initialize all Git submodules. Ensure recursive cloning for all submodules. ```bash git clone https://github.com/HumanAIGC-Engineering/OpenAvatarChat.git && cd OpenAvatarChat git submodule update --init --recursive --depth 1 ``` -------------------------------- ### Build Docker Image for OpenAvatarChat Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/getting-started/docker.md Clone the repository, update submodules, download necessary models, and build the Docker image. Ensure you have a CUDA 12.8 compatible environment. ```bash git clone https://github.com/HumanAIGC-Engineering/OpenAvatarChat.git && cd OpenAvatarChat git submodule update --init --recursive --depth 1 uv run scripts/download_models.py --all bash build_cuda128.sh ``` -------------------------------- ### Generate Self-Signed SSL Certificates Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/guide/deployment.md Run this script to create SSL certificates required for non-localhost access to enable camera/microphone permissions. ```bash scripts/create_ssl_certs.sh ``` -------------------------------- ### Interrupt Path Trigger Chain Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/musetalk_review.md Details the sequence of events when an interrupt is triggered, starting from a STREAM_CANCEL signal and progressing through context interruption, stream finishing, processor interruption, and flushing input slicer. ```text on_signal(STREAM_CANCEL, CLIENT_PLAYBACK) [signal_distribute_thread] → AvatarMuseTalkContext.interrupt() → [锁] _current_tts_stream_key = None → _playback_streamer.finish_current() [幂等安全] → processor.interrupt() → [锁] _generation_id += 1 → _interrupted.set() → [锁] 清空所有队列 → input_slice_context.flush() [丢弃 slicer 余量] ``` -------------------------------- ### Frame Generator Worker Logic Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md The Frame Generator Worker processes batches to generate frames using UNet and VAE. It collects results and places them into a queue for further processing. Ensure CUDA is preheated before this worker starts. ```python def _frame_generator_worker(self): # CUDA 预热: dummy_whisper → generate_frames() (UNet+VAE 一体) while not _stop_event.is_set(): result = _collect_batch() if result is None: continue # UNet + VAE 一体推理 recon_idx_list = _avatar.generate_frames(whisper_batch, frame_ids[0], batch_size) if _interrupted.is_set(): continue # 逐帧放入 _compose_queue(只处理 valid_num 个) for i in range(valid_num): ... ``` -------------------------------- ### DWPose ONNX Session Initialization Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Python code demonstrating lazy loading for DWPose ONNX Runtime sessions and ensuring cuDNN availability. It prioritizes CUDAExecutionProvider and falls back to CPUExecutionProvider. ```python _dwpose_session = None # 懒加载 _cudnn_preloaded = False def _ensure_cudnn_available(): """处理 pip 安装的 nvidia-cudnn-cu12 库路径问题。 通过 ctypes 预加载 libcudnn*.so.9 并更新 LD_LIBRARY_PATH, 确保 onnxruntime 的 CUDAExecutionProvider 能找到 cuDNN。""" def _get_dwpose_session(): """首次调用时初始化 ONNX Runtime session。 优先使用 CUDAExecutionProvider,fallback 到 CPUExecutionProvider。""" ``` -------------------------------- ### Configure TURN Server Settings Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/guide/deployment.md Add this configuration to your settings to specify the TURN server details for NAT traversal. Replace placeholders with your actual server information. ```yaml default: chat_engine: handler_configs: RtcClient: turn_config: turn_provider: "turn_server" urls: ["turn:your-turn-server.com:3478", "turns:your-turn-server.com:5349"] username: "your-username" credential: "your-credential" ``` -------------------------------- ### Manually Download FlashHead Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/reference/handlers/avatar/flashhead.md Provides commands for manually downloading the SoulX-FlashHead-1_3B and wav2vec2-base-960h models using huggingface-cli and git clone. ```bash pip install "huggingface_hub[cli]" huggingface-cli download Soul-AILab/SoulX-FlashHead-1_3B --local-dir ./models/SoulX-FlashHead-1_3B ``` ```bash git clone --depth 1 https://huggingface.co/facebook/wav2vec2-base-960h ./models/wav2vec2-base-960h ``` -------------------------------- ### Manage TTS Audio Stream to Client Playback Lifecycle Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/musetalk_review.md Illustrates the lifecycle management of TTS audio streams transitioning to CLIENT_PLAYBACK streams. It covers detecting new stream keys, flushing, closing old streams, opening new ones, and handling audio data through a processing pipeline. ```text TTS handler 产生带 stream_key 的 AVATAR_AUDIO 数据 ↓ handle() 检测到新 stream_key_str(锁保护下与 _current_tts_stream_key 比较) ↓ flush slicer 余量(防止跨流混音) ↓ 关闭旧 CLIENT_PLAYBACK 流(finish_current, 如果 prev_key != None) ↓ 打开新 CLIENT_PLAYBACK 流(open_stream) ↓ 更新 _current_tts_stream_key = stream_key_str ↓ 音频通过 slice_data 切成 1 秒片段 → processor.add_audio() ↓ add_audio 携带 generation_id → 条件性清除 _interrupted ↓ processor 流水线:特征提取(generation_id 校验) → 推理 → 合成 → 按帧输出 ↓ speech_end=True 的最后一帧到达 _frame_collector_worker ↓ _notify_speech_end(speech_id) ↓ on_speech_end 回调(锁保护下校验 speech_id) → streamer.finish_current() ↓ _current_tts_stream_key = None ``` -------------------------------- ### Build Docker Image Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Build the Docker image with CUDA 12.8 support. This script bundles all handler dependencies into the image. ```bash bash build_cuda128.sh ``` -------------------------------- ### Download LAM Handler Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/lam.md Use this command to download models for the LAM handler. Specify --source modelscope for alternative model sources. ```bash uv run scripts/download_models.py --handler lam ``` ```bash uv run scripts/download_models.py --handler lam --source modelscope ``` -------------------------------- ### FlashHead Avatar Handler Configuration Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/flashhead.md Configure the FlashHead avatar handler by specifying module paths, model directories, and performance parameters. The `ckpt_dir` and `wav2vec_dir` should point to the downloaded model locations. Adjust `fps` and `model_type` for desired performance. ```yaml FlashHead: module: avatar/flashhead/avatar_handler_flashhead ckpt_dir: "models/SoulX-FlashHead-1_3B" wav2vec_dir: "models/wav2vec2-base-960h" model_type: "lite" cond_image_path: "resource/avatar/flashhead/default.png" fps: 25 base_seed: 42 use_face_crop: false ``` -------------------------------- ### Download Handler Models Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/getting-started/docker.md Download all necessary handler models using the provided Python script. This step is crucial for the functionality of various avatar types. ```bash uv run scripts/download_models.py --all ``` -------------------------------- ### Handler Details and Flow Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Details the get_handler_detail function, its inputs, outputs, and signal filters, along with a step-by-step breakdown of the handle() method's processing flow. ```APIDOC ## get_handler_detail() Function Details ### Description Provides detailed information about the handler's configuration, including input and output data types, consumption modes, and signal filtering rules. ### Method Signature `get_handler_detail(self, session_context, context) -> HandlerDetail` ### Inputs - **ChatDataType.AVATAR_AUDIO**: Consumed once per data item. ### Outputs - **ChatDataType.AVATAR_AUDIO**: Output stream configuration. - **ChatDataType.AVATAR_VIDEO**: Output stream configuration with `cancelable=False` and `auto_link_input=False`. ### Signal Filters - Listens for `ChatSignalType.STREAM_CANCEL` on `ChatDataType.CLIENT_PLAYBACK`. --- ## handle() Method Flow ### Description Describes the step-by-step process of the handle() method, including type checking, stream key switching, input validation, audio slicing, and end-of-speech handling. ### Steps 1. **Type Check**: Returns if `inputs.type` is not `AVATAR_AUDIO`. 2. **Stream Key Change Detection**: Detects changes in `stream_key` and switches the `CLIENT_PLAYBACK` stream accordingly, flushing residual audio and closing the previous stream. 3. **Input Validation**: Checks `sample_rate`, converts audio dtype to `float32`, and pads with silence if audio data is `None`. 4. **Audio Slicing**: Slices audio into 1-second segments based on `output_audio_sample_rate` and adds them to the processor. 5. **Speech End Handling**: Flushes residual audio or pads with silence for `end_of_speech=True` signal. ``` -------------------------------- ### Configure LAM Client Avatar Assets Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/client/lam-client.md Use this YAML configuration to set up the LAM client, specifying the module, asset path for avatar assets, and the concurrent connection limit. ```yaml LamClient: module: client/ws_lam_client/ws_lam_client_handler asset_path: "lam_samples/barbara.zip" concurrent_limit: 5 ``` -------------------------------- ### Configure LAM for CPU/Mac Execution Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/FAQ.md Steps to configure the LAM for running on a CPU or Mac by removing GPU-specific dependencies and modifying device settings. ```bash #运行chat_with_lam.yaml 所需额外步骤 #第一步:移除torchvision 依赖 #第二步:替换onnxruntime-gpu 为 onnxruntime 依赖 #修改 src/handlers/avatar/lam/LAM_Audio2Expression/engines/infer.py ,删除所有.cuda()方法的调用 #按照readme 按照依赖,并运行即可 ``` -------------------------------- ### Signal Handling and Context Destruction Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Explains how the system handles stream cancellation signals and the process for destroying the context, ensuring resources are properly released. ```APIDOC ## on_signal() Method ### Description Handles incoming signals, specifically `STREAM_CANCEL` signals related to `CLIENT_PLAYBACK` streams, by interrupting the current context. ### Logic - If `signal.type` is `ChatSignalType.STREAM_CANCEL` and `signal.related_stream.data_type` is `ChatDataType.CLIENT_PLAYBACK`, then `context.interrupt()` is called. - There is no direct `INTERRUPT` signal handling; interruption is managed indirectly via `CLIENT_PLAYBACK` stream cancellation. --- ## destroy_context() Method ### Description Safely destroys the provided context, ensuring all associated resources like playback streams and processors are properly cleaned up. ### Steps 1. **Close Playback Stream**: If `_playback_streamer` exists, `finish_current()` is called to close the stream. 2. **Stop Processor**: The associated `processor` is stopped, its callbacks are cleared, and it's released back to the pool. 3. **Clear Context**: The context itself is cleared of its resources. ``` -------------------------------- ### Processor Pool Acquisition and Release Logic Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/src/handlers/avatar/musetalk/ARCHITECTURE.md Describes the linear scanning approach used by `acquire()` to find an idle processor and `release()` to mark a processor as available. `acquire()` returns `None` if all processors are busy. ```text - `acquire()` 线性扫描 `_active` 列表,找到第一个 `False` 的位置 - `release()` 线性扫描 `_processors` 列表,找到匹配的实例并标记为 `False` - 当所有 Processor 都被占用时,`acquire()` 返回 `None`(不阻塞等待) ``` -------------------------------- ### LiteAvatar Configuration Source: https://github.com/humanaigc-engineering/openavatarchat/blob/main/docs/en/reference/handlers/avatar/liteavatar.md Configure the LiteAvatar handler by specifying the module, avatar name, frame rate, and GPU usage in your configuration file. ```yaml LiteAvatar: module: avatar/liteavatar/avatar_handler_liteavatar avatar_name: 20250408/sample_data fps: 25 use_gpu: true ```