### Install Dolphin Dependencies

Source: https://github.com/kaleido-lab/dolphin/blob/main/README.md

Use conda to manage the environment and install project dependencies. Python 3.8 is recommended.

```bash
conda create -n dolphin python=3.8
conda activate dolphin

git clone https://github.com/BUAA-PrismGroup/dolphin.git
cd dolphin

pip install -r requirements.txt
```

--------------------------------

### Install Dependencies and Launch Dolphin

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Install necessary dependencies using conda and pip. Launch the application with default settings, specific models, or CPU-only mode. You can also specify a custom configuration file and port.

```bash
# Install dependencies (Python 3.8, CUDA 11.3)
conda create -n dolphin python=3.8
conda activate dolphin
pip install -r requirements.txt
```

```bash
# Start with all models on 4 GPUs (uses configs/backends.yaml)
python video_chatgpt.py
```

```bash
# Start with only specific models to save GPU memory
# Format: ModelName_device, comma-separated
python video_chatgpt.py --load VideoCaptioning_cuda:0,MoviepyInterface_cpu,ModelscopeT2V_cpu
```

```bash
# CPU-only mode
python video_chatgpt.py --load VideoCaptioning_cpu,ImageCaptioning_cpu,ModelscopeT2V_cpu
```

```bash
# Custom config file and port
python video_chatgpt.py --cfg ./configs/backends.yaml --port 7890
```

--------------------------------

### Run Dolphin on CPU

Source: https://github.com/kaleido-lab/dolphin/blob/main/README.md

Start the Dolphin application with specific models loaded on the CPU. This is recommended for users without a GPU.

```bash
python video_chatgpt.py --load VideoCaptioning_cpu,ImageCaptioning_cpu,ModelscopeT2V_cpu
```

--------------------------------

### Conditioned Video Generation

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates new videos guided by a structural control signal (canny edges, pose skeleton, or depth map) plus a text prompt. Input is a comma-separated string of control video path and text prompt.

```python
from modules.text2video_zero import CannyText2Video, PoseText2Video, DepthText2Video

# Generate video from canny edges + text
canny2vid = CannyText2Video(device="cuda:2")
result = canny2vid.inference("video/edge_clip.mp4, a robot dancing in a futuristic city")
print(result)  # "video/xxxx_canny2video_....mp4"

# Generate video from pose skeleton + text
pose2vid = PoseText2Video(device="cuda:2")
result = pose2vid.inference("video/pose_clip.mp4, an astronaut performing ballet")
print(result)  # "video/xxxx_pose2video_....mp4"

# Generate video from depth map + text
depth2vid = DepthText2Video(device="cuda:2")
result = depth2vid.inference("video/depth_clip.mp4, a golden retriever running through snow")
print(result)  # "video/xxxx_depth2video_....mp4"
```

--------------------------------

### Conditioned Video Generation

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates new videos guided by a structural control signal (canny edges, pose skeleton, or depth map) plus a text prompt, using Text2Video-Zero with ControlNet.

```APIDOC
## CannyText2Video, PoseText2Video, DepthText2Video

### Description
Generate new videos guided by a structural control signal (canny edges, pose skeleton, or depth map) plus a text prompt, using Text2Video-Zero with ControlNet. Input is a comma-separated string of the control video path and the text prompt.

### Methods
- `CannyText2Video(device)`: Initializes the Canny-guided video generator. `device` can be a string like "cuda:0".
- `PoseText2Video(device)`: Initializes the Pose-guided video generator. `device` can be a string like "cuda:0".
- `DepthText2Video(device)`: Initializes the Depth-guided video generator. `device` can be a string like "cuda:0".

### Inference
- `canny2vid.inference(input_string)`: Generates a video from canny edges and a text prompt. `input_string` format: "video/edge_clip.mp4, a robot dancing in a futuristic city".
- `pose2vid.inference(input_string)`: Generates a video from pose skeleton and a text prompt. `input_string` format: "video/pose_clip.mp4, an astronaut performing ballet".
- `depth2vid.inference(input_string)`: Generates a video from a depth map and a text prompt. `input_string` format: "video/depth_clip.mp4, a golden retriever running through snow".

### Returns
All inference methods return the path to the generated video file (e.g., "video/xxxx_canny2video_....mp4").
```

--------------------------------

### Configure New Video Foundation Model in backends.yaml

Source: https://github.com/kaleido-lab/dolphin/blob/main/README.md

This YAML configuration defines a new video foundation model, 'FaceText2Video', specifying its target class path and initialization parameters. It also includes a tool definition for this model, detailing its name, description, instance, and inference function.

```yaml
model_zoos:
  FaceText2Video:  # foundation model class
    target: modules.face2video.FaceText2Video # path of the class in project
    params: # params passed to the class
      device: cuda:0
tools:
  # - name: tool name
  #   desc: description about new tool
  #   instance: keep the name consistent with the one in the model_zoos section mentioned above
  #   func: inference function in foundation model class
  - name: Generate Video Condition On Face Video
    desc: "useful when you want to generate a new video from both the user description and a facial keypoints video. / 
      like: generate a new video of a human face from this human face video, or can you generate a video based on both the text 'A boy is playing basketball.' and this face video. / 
      The input to this tool should be a comma separated string of two, representing the video_path and the user description. "
    instance: FaceText2Video
    func: inference
```

--------------------------------

### Initialize Text-to-Video Generation with Control

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes various text-to-video models that utilize control mechanisms like Canny, Pose, and Depth, along with a VideoPix2Pix model. All require a CUDA-enabled device.

```python
from modules.text2video_zero import CannyText2Video, PoseText2Video, DepthText2Video, VideoPix2Pix

pose_text2video = PoseText2Video("cuda:5").inference  # input: f"{video_path}, {prompt}", output: new_video_path
canny_text2video = CannyText2Video("cuda:5").inference  # input: f"{video_path}, {prompt}", output: new_video_path
depth_text2video = DepthText2Video("cuda:5").inference  # input: f"{video_path}, {prompt}", output: new_video_path
video_pix2pix = VideoPix2Pix("cuda:5").inference # input: f"{video_path}, {prompt}", output: new_video_path
```

--------------------------------

### Text-to-Video Generation (ModelScope)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a short video from a text prompt using the damo-vilab/text-to-video-ms-1.7b diffusion model. Offers a basic inference method and a lower-level generate_video method for more control.

```python
from modules.modelscope_t2v import ModelscopeT2V

t2v = ModelscopeT2V(device="cuda:3")

# Basic text-to-video
video_path = t2v.inference("A panda eating bamboo in a misty forest")
print(video_path)  # "video/f3a1b2c4.mp4"

# Use the lower-level method for more control
custom_path = t2v.generate_video(
    prompt="A spaceship landing on Mars at sunset",
    seed=42,
    num_frames=16,
    num_inference_steps=25,
    out_file="video/mars_landing.mp4",
)
print(custom_path)  # "video/mars_landing.mp4"
```

--------------------------------

### Load and Preprocess Video Frames

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

The `prepare_video` function loads video files using decord, resamples frames to a target FPS, resizes them, and optionally normalizes pixel values. It returns a float tensor and the output FPS.

```python
from video_utils import prepare_video
import torch

# Load full video at native FPS, resize to 512px, normalized
video_tensor, fps = prepare_video(
    video_path="video/sample.mp4",
    resolution=512,
    device=torch.device("cuda:0"),
    dtype=torch.float16,
    normalize=True,   # pixel values in [-1, 1]
    start_t=0,
    end_t=-1,         # -1 means full duration
    output_fps=-1,    # -1 uses native FPS
)
print(video_tensor.shape)  # torch.Size([48, 3, 512, 288])  (F, C, H, W)
print(fps)                 # 24

# Load a 5-second clip without normalization (for annotators)
clip, fps = prepare_video("video/sample.mp4", 512, "cpu", normalize=False, start_t=3.0, end_t=8.0)
print(clip.shape)   # torch.Size([120, 3, 512, 288])
print(clip.min(), clip.max())  # tensor(0.) tensor(255.)
```

--------------------------------

### Pose-Guided Text-to-Video Generation

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a video based on a human pose and a text description. Requires paths to the pose video and a text prompt.

```python
print("./" + pose_text2video("video/0d50_human-pose_1759_v0006.mp4, an astronaut dancing on the moon"))
```

--------------------------------

### Run Dolphin with GPU Acceleration

Source: https://github.com/kaleido-lab/dolphin/blob/main/README.md

Launch the Dolphin application, allowing it to utilize available GPUs. Model and device assignments can be configured in `configs/backend.yaml` for optimal performance.

```bash
python video_chatgpt.py
```

--------------------------------

### Initialize BLIP2 Image Captioning

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes the BLIP2 model for image captioning, which can be used on video frames. Requires a CUDA-enabled device.

```python
from modules.blip import ImageCaptioning

tool_blip2 = ImageCaptioning("cuda:0")
frames_captioning = tool_blip2.inference # input: f"{video_path}", output: caption (str)
```

--------------------------------

### Initialize Modelscope Text-to-Video Generation

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes the ModelscopeT2V model for generating videos from text descriptions. Requires a CUDA-enabled device.

```python
from modules.modelscope_t2v import ModelscopeT2V

mst2v = ModelscopeT2V("cuda:1")
text2video = mst2v.inference # input: f"{description}", output: new_video_path
```

--------------------------------

### Configure Model Zoo and Tools in YAML

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Define foundation models and their exposure as agent tools in the `configs/backends.yaml` file. Specify the Python class path and constructor parameters for models, and map natural language names and descriptions to tool instances and their methods.

```yaml
model_zoos:
  VideoCaptioning:
    target: modules.mplug.VideoCaptioning
    params:
      device: cuda:0

  ModelscopeT2V:
    target: modules.modelscope_t2v.ModelscopeT2V
    params:
      device: cuda:1

  # Add a custom model:
  FaceText2Video:
    target: modules.face2video.FaceText2Video
    params:
      device: cuda:2

tools:
  - name: Get Video Description
    desc: "useful when you want to know what the video shows. The input should be the video_path string."
    instance: VideoCaptioning
    func: inference

  - name: Generate Video Condition On Text
    desc: "useful when you want to generate a video from text. The input should be a text string."
    instance: ModelscopeT2V
    func: inference

  # Register the custom tool:
  - name: Generate Video Condition On Face Video
    desc: "useful when you want to generate a new video from a face video and a text description.
      The input should be a comma-separated string of video_path and user description."
    instance: FaceText2Video
    func: inference
```

--------------------------------

### Initialize mPLUG Video Captioning

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes the mPLUG model for video captioning. Requires a CUDA-enabled device.

```python
from modules.mplug import VideoCaptioning

tool_vc = VideoCaptioning("cuda:7")
video_captioning = tool_vc.inference # input: f"{video_path}", output: caption (str)
```

--------------------------------

### Dynamically Instantiate Models from Config

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Use `utils.instantiate_from_config` to load and instantiate Python classes based on a YAML configuration. This is the primary method for building model zoo objects at startup. Ensure the configuration file is loaded using `OmegaConf.load`.

```python
from omegaconf import OmegaConf
import utils

# Load config and instantiate a single model programmatically
cfg = OmegaConf.load("configs/backends.yaml")

video_caption_cfg = cfg.model_zoos["VideoCaptioning"]
# {"target": "modules.mplug.VideoCaptioning", "params": {"device": "cuda:0"}}

model = utils.instantiate_from_config(video_caption_cfg)
# Equivalent to: from modules.mplug import VideoCaptioning; model = VideoCaptioning(device="cuda:0")

result = model.inference("video/sample.mp4")
print(result)
# "A dog is running in a grassy park near a lake."
```

--------------------------------

### Initialize Text2Audio from Bark

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes the Text2Audio tool from the Bark module and exposes its text-to-audio and text-to-music functions. Use these functions for audio generation from text.

```python
from modules.bark import Text2Audio

tool_t2a = Text2Audio()
text2audio = tool_t2a.text2audio   # input: f"{text}", output: audio_path
text2music = tool_t2a.text2music   # input: f"{text}", output: audio_path
```

--------------------------------

### Video Captioning with mPLUG

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a single natural-language description for an entire video using the mPLUG model. Downloads model weights on first use.

```python
from modules.mplug import VideoCaptioning

model = VideoCaptioning(device="cuda:0")
# Prints: "Initializing mPLUG for VideoCaptioning"

caption = model.inference("video/skateboard_clip.mp4")
print(caption)
```

--------------------------------

### ModelscopeT2V

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a short video from a text prompt using the `damo-vilab/text-to-video-ms-1.7b` diffusion model.

```APIDOC
## ModelscopeT2V

### Description
Generates a short video (16 frames at 8 fps) from a text prompt using the `damo-vilab/text-to-video-ms-1.7b` diffusion model with DPM-Solver++ scheduling and VAE slicing enabled.

### Method
- `ModelscopeT2V(device)`: Initializes the text-to-video generation model. `device` can be a string like "cuda:0".

### Inference
- `t2v.inference(prompt)`: Generates a video from a text prompt.
- `t2v.generate_video(prompt, seed, num_frames, num_inference_steps, out_file)`: A lower-level method for more control over video generation parameters.

### Parameters for `generate_video`
- `prompt` (string): The text description for the video.
- `seed` (int): The random seed for generation.
- `num_frames` (int): The number of frames in the generated video.
- `num_inference_steps` (int): The number of inference steps.
- `out_file` (string): The path to save the output video.

### Returns
Both `inference` and `generate_video` return the path to the generated video file (e.g., "video/f3a1b2c4.mp4").
```

--------------------------------

### Depth-Guided Text-to-Video Generation

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a video based on depth information and a text description. Requires paths to the depth video and a text prompt.

```python
print("./" + depth_text2video("video/7574_depth_235ee6b2_235ee6b2.mp4, a tiger"))
```

--------------------------------

### VideoCaptioning (mPLUG)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a single natural-language description for an entire video using the mPLUG model. It downloads necessary model weights on first use.

```APIDOC
## VideoCaptioning (mPLUG) — Video-Level Caption Generation

Wraps the mPLUG large model to produce a single natural-language description of an entire video. On first use it downloads `ViT-L-14.tar` and `mplug_large.pth` from Aliyun OSS into `model_zoo/mplug/`.

```python
from modules.mplug import VideoCaptioning

model = VideoCaptioning(device="cuda:0")
# Prints: "Initializing mPLUG for VideoCaptioning"

caption = model.inference("video/skateboard_clip.mp4")
print(caption)
# "A young man performs tricks on a skateboard in a city plaza while bystanders watch."
```
```

--------------------------------

### Initialize LangChain Agent with ConversationBot

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Instantiate the `ConversationBot` class, which orchestrates the model zoo, a LangChain agent, and conversation memory. Call `init_agent` with an OpenAI API key to activate the agent, specifying the desired language.

```python
from omegaconf import OmegaConf
from video_chatgpt import ConversationBot

cfg = OmegaConf.load("configs/backends.yaml")
bot = ConversationBot(cfg)

# Initialize the agent (English mode)
bot.init_agent(openai_api_key="sk-ինչ", lang="English")
```

--------------------------------

### Run Video Analysis with ConversationBot

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

The `run_video` method uploads a video, generates descriptions using captioning models, and updates the agent's memory. It returns the updated state, a video path, and text.

```python
import types
from video_chatgpt import ConversationBot
from omegaconf import OmegaConf

cfg = OmegaConf.load("configs/backends.yaml")
bot = ConversationBot(cfg)
bot.init_agent(openai_api_key="sk-...", lang="English")

# Simulate a Gradio NamedString upload object
fake_upload = types.SimpleNamespace(name="/tmp/my_clip.mp4")

state = []
state, _, updated_txt = bot.run_video(fake_upload, state, txt="", lang="English")
# state[-1] == ('<video controls autoplay ...>*video/a1b2c3d4.mp4*', 'Received.  ')
# updated_txt == " video/a1b2c3d4.mp4 "

# Ask about the uploaded video
state, _ = bot.run_text(f"What is the main activity in {updated_txt.strip()}?", state)
print(state[-1][1])
```

--------------------------------

### Initialize Video Analysis Annotators

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes annotator tools for extracting Canny edges, poses, and depth information from video frames. Pose and Depth require a CUDA-enabled device.

```python
from modules.annotator import Video2Canny, Video2Pose, Video2Depth

tool_v2c = Video2Canny()
tool_v2p = Video2Pose("cuda:1")
tool_v2d = Video2Depth("cuda:1")

video2canny = tool_v2c.inference # input: f"{video_path}", output: new_video_path
video2pose = tool_v2p.inference # input: f"{video_path}", output: new_video_path
video2depth = tool_v2d.inference # input: f"{video_path}", output: new_video_path
```

--------------------------------

### ConversationBot.run_video

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Uploads a video file, generates descriptions using video and image captioning models, and injects them into the agent's memory for contextual follow-up queries.

```APIDOC
## ConversationBot.run_video — Upload and Register a Video

When a video file is uploaded, `run_video` saves it to the `video/` directory, runs `VideoCaptioning` (mPLUG) and `ImageCaptioning` (BLIP-2) to generate descriptions, and injects them into the agent's memory buffer so the LLM has full context for follow-up queries.

```python
import types
from video_chatgpt import ConversationBot
from omegaconf import OmegaConf

cfg = OmegaConf.load("configs/backends.yaml")
bot = ConversationBot(cfg)
bot.init_agent(openai_api_key="sk-...", lang="English")

# Simulate a Gradio NamedString upload object
fake_upload = types.SimpleNamespace(name="/tmp/my_clip.mp4")

state = []
state, _, updated_txt = bot.run_video(fake_upload, state, txt="", lang="English")
# state[-1] == ('<video controls autoplay ...>*video/a1b2c3d4.mp4*', 'Received.  ')
# updated_txt == " video/a1b2c3d4.mp4 "

# Ask about the uploaded video
state, _ = bot.run_text(f"What is the main activity in {updated_txt.strip()}?", state)
print(state[-1][1])
# "The main activity is a person playing tennis on an outdoor court."
```
```

--------------------------------

### Text-to-Speech and Music Generation (Bark)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Synthesizes speech audio or music from text using Suno's Bark model. `text2audio` generates speech, while `text2music` uses music note markers to trigger music generation.

```python
from modules.bark import Text2Audio

tts = Text2Audio()
# Prints: "Loading bark models for text2audio..."

# Generate speech
speech_path = tts.text2audio(
    "Hello! My name is Dolphin, a general video interaction platform."
)
print(speech_path)  # "audio/3c2d1e0f.wav"

# Generate music
music_path = tts.text2music("In the jungle, the mighty jungle, the lion roars tonight")
print(music_path)  # "audio/7a8b9c0d.wav"
# Internally sends: "♪ In the jungle, the mighty jungle, the lion roars tonight ♪"
```

--------------------------------

### Music Generation with Text2Music

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a music file from a text prompt using the text2music model. The output is a file path to the generated music.

```python
print("./" + text2music("a goldendoodle playing in a park by a lake"))
```

--------------------------------

### Generate 3D Model Video from Text using Shap-E

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Utilizes OpenAI's Shap-E model for text-to-3D model video generation. Outputs are saved to the 'video/' directory. Supports both text-to-model and image-to-model modes.

```python
from modules.shap_e import Shap_E

shap_e = Shap_E(device="cuda:3")

# Single text prompt
shap_e.inference("a wooden chair with four legs")
# Output saved to video/ directory as a rotating 3D model video

# Batch from a file of prompts (one per line)
shap_e.cond_type = "t2m"
shap_e.inference("prompts.txt")
# Processes each line: "a red apple", "a coffee mug", etc.
```

--------------------------------

### Text-to-Video Generation

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a video based on a text description. Returns the path to the generated video file.

```python
print("./" + text2video("a goldendoodle playing in a park by a lake"))
```

--------------------------------

### Text2Audio (Bark)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Uses Suno's Bark model to synthesize either speech audio or music from text.

```APIDOC
## Text2Audio (Bark)

### Description
Uses Suno's Bark model to synthesize either speech audio or music from text. `text2audio` generates natural speech; `text2music` wraps the prompt in music note markers to trigger Bark's music generation mode.

### Method
- `Text2Audio()`: Initializes the Text-to-Audio model.

### Methods
- `tts.text2audio(text)`: Generates speech audio from the given text.
- `tts.text2music(prompt)`: Generates music from the given prompt, by internally formatting it for Bark's music generation.

### Returns
Both methods return the path to the generated audio file (e.g., "audio/3c2d1e0f.wav").
```

--------------------------------

### Video Generation with Depth2Video

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a video from a text prompt using the depth2video model. The output is a file path to the generated video.

```python
print("./" + video_pix2pix("video/235ee6b2.mp4, make it snowy"))
```

--------------------------------

### ImageCaptioning (BLIP-2)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Provides per-frame captioning for videos and visual question answering for individual images using the BLIP-2 model.

```APIDOC
## ImageCaptioning (BLIP-2) — Per-Frame Video Captioning and VQA

Uses BLIP-2 (`Salesforce/blip2-opt-2.7b`) to caption each second of a video individually (`frames_captioning`) or to answer questions about a single PIL image (`image_captioning` with `is_vqa=True`).

```python
from modules.blip import ImageCaptioning
from PIL import Image

model = ImageCaptioning(device="cuda:1")

# Caption every second of a video
frame_captions = model.inference("video/cooking_demo.mp4")
print(frame_captions)
# "Second 0: a chef chops vegetables on a cutting board. Second 1: the chef stirs a pot on the stove. ..."

# Visual question answering on a single image
img = Image.open("image/frame_001.png")
answer = model.image_captioning(img, prompt="What color is the chef's apron?", is_vqa=True)
print(answer)
# "white"
```
```

--------------------------------

### Frame-by-Frame Video Captioning

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates captions for each second of a video. Requires the path to the video file as input.

```python
print(frames_captioning("video/v0006.mp4"))
```

--------------------------------

### Audio Generation with Text2Audio

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates an audio file from a text prompt using the text2audio model. The output is a file path to the generated audio.

```python
print("./" + text2audio("a goldendoodle playing in a park by a lake"))
```

--------------------------------

### video_utils.prepare_video

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Loads and preprocesses a video file. It resamples frames to a target FPS within a specified time range, resizes frames to fit within given resolution, and optionally normalizes pixel values. Returns a float tensor of frames and the output FPS.

```APIDOC
## `video_utils.prepare_video` — Video Loading and Preprocessing

Reads a video file using `decord`, resamples frames to a target FPS within a time range, resizes to fit within `resolution` pixels (rounded to multiples of 64), and optionally normalizes pixel values to `[-1, 1]`. Returns a `(F, C, H, W)` float tensor and the output FPS.

```python
from video_utils import prepare_video
import torch

# Load full video at native FPS, resize to 512px, normalized
video_tensor, fps = prepare_video(
    video_path="video/sample.mp4",
    resolution=512,
    device=torch.device("cuda:0"),
    dtype=torch.float16,
    normalize=True,   # pixel values in [-1, 1]
    start_t=0,
    end_t=-1,         # -1 means full duration
    output_fps=-1,    # -1 uses native FPS
)
print(video_tensor.shape)  # torch.Size([48, 3, 512, 288])  (F, C, H, W)
print(fps)                 # 24

# Load a 5-second clip without normalization (for annotators)
clip, fps = prepare_video("video/sample.mp4", 512, "cpu", normalize=False, start_t=3.0, end_t=8.0)
print(clip.shape)   # torch.Size([120, 3, 512, 288])
print(clip.min(), clip.max())  # tensor(0.) tensor(255.)
```
```

--------------------------------

### BarkVoiceClone

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Uses ModelScope's SambertHifigan model to synthesize speech with a cloned human voice profile.

```APIDOC
## BarkVoiceClone

### Description
Uses ModelScope's SambertHifigan model to synthesize speech with a cloned human voice profile (voice `F7`). Requires a pre-trained model directory at `modules/bark_voice_clone/pretrain_work_dir/`.

### Method
- `BarkVoiceClone()`: Initializes the voice cloning model.

### Inference
- `vc.inference(text)`: Synthesizes speech from the given text using the cloned voice profile.

### Returns
Returns the path to the generated audio file (e.g., "audio/4e5f6a7b.wav"). The output is a 18050 Hz mono 16-bit WAV file using the F7 voice profile.
```

--------------------------------

### Video Captioning with Dolphin

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Generates a caption for an entire video. Requires the path to the video file as input.

```python
print(video_captioning("video/v0006.mp4"))
```

--------------------------------

### Image Captioning and VQA with BLIP-2

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Utilizes BLIP-2 for per-frame video captioning or visual question answering on single images. Requires specifying the device and optionally setting `is_vqa=True` for question answering.

```python
from modules.blip import ImageCaptioning
from PIL import Image

model = ImageCaptioning(device="cuda:1")

# Caption every second of a video
frame_captions = model.inference("video/cooking_demo.mp4")
print(frame_captions)

# Visual question answering on a single image
img = Image.open("image/frame_001.png")
answer = model.image_captioning(img, prompt="What color is the chef's apron?", is_vqa=True)
print(answer)
```

--------------------------------

### Save Frames as MP4 Video

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

The `create_video` function takes a list of numpy frame arrays, optionally rescales them from `[-1, 1]` to `[0, 255]`, and writes them to an MP4 file using imageio.

```python
from video_utils import create_video, prepare_video
from einops import rearrange
import numpy as np

# Load, process and re-save
video, fps = prepare_video("video/input.mp4", 512, "cpu", normalize=False)
frames = rearrange(video, "f c h w -> f h w c").numpy().astype(np.uint8)
frames_list = [frames[i] for i in range(len(frames))]

output_path = create_video(frames_list, fps=fps, path="video/output.mp4", rescale=False)
print(output_path)  # "video/output.mp4"

# With rescale=True (input in [-1, 1] range)
norm_video, fps = prepare_video("video/input.mp4", 512, "cpu", normalize=True)
norm_frames = [rearrange(norm_video[i], "c h w -> h w c").numpy() for i in range(len(norm_video))]
output_path = create_video(norm_frames, fps=fps, path="video/output_rescaled.mp4", rescale=True)
```

--------------------------------

### Simulated Human Voice TTS (BarkVoiceClone)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Synthesizes speech with a cloned human voice profile (F7) using ModelScope's SambertHifigan model. Requires a pre-trained model directory at modules/bark_voice_clone/pretrain_work_dir/.

```python
from modules.bark_voice_clone import BarkVoiceClone

vc = BarkVoiceClone()

audio_path = vc.inference("Welcome to the Dolphin video platform. How can I help you today?")
print(audio_path)  # "audio/4e5f6a7b.wav"
# Output is a 18050 Hz mono 16-bit WAV file using the F7 voice profile
```

--------------------------------

### Text-Instructed Video Style Editing

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Edits an existing video according to a natural-language instruction using InstructPix2Pix applied frame-by-frame. Input is a comma-separated string of video path and instruction text.

```python
from modules.text2video_zero import VideoPix2Pix

pix2pix = VideoPix2Pix(device="cuda:2")

# Change visual style
result = pix2pix.inference("video/park_scene.mp4, make it look like a watercolor painting")
print(result)  # "video/xxxx_pix2pix_....mp4"

# Replace objects
result = pix2pix.inference("video/dog_running.mp4, replace the dog with a cat")
print(result)  # "video/xxxx_pix2pix_....mp4"

# Change weather/environment
result = pix2pix.inference("video/street_scene.mp4, make it snowy")
print(result)  # "video/xxxx_pix2pix_....mp4"
```

--------------------------------

### Text2Image (Stable Diffusion)

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a single image from a text prompt using Stable Diffusion v1.5 with PNDM scheduling. The output image is saved to the `image/` directory.

```APIDOC
## `Text2Image` (Stable Diffusion) — Text-to-Image Generation

Wraps Stable Diffusion v1.5 with PNDM scheduling to generate a single image from a text prompt. The output image is saved to the `image/` directory.

```python
from modules.stable_diffusion import Text2Image

t2i = Text2Image()

image_path = t2i.image_generation("An astronaut riding a horse on Mars, photorealistic")
print(image_path)  # "image/b1c2d3e4.png"
```
```

--------------------------------

### Shap_E

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Generates a rotating 3D model video from a text description using OpenAI's Shap-E. Supports both text-to-model (`t2m`) and image-to-model (`i2m`) modes. Outputs are saved to the `video/` directory.

```APIDOC
## `Shap_E` — Text-to-3D Model Video Generation

Uses OpenAI's Shap-E to generate a rotating 3D model video from a text description. Outputs are saved to the `video/` directory. Supports both text-to-model (`t2m`) and image-to-model (`i2m`) modes.

```python
from modules.shap_e import Shap_E

shap_e = Shap_E(device="cuda:3")

# Single text prompt
shap_e.inference("a wooden chair with four legs")
# Output saved to video/ directory as a rotating 3D model video

# Batch from a file of prompts (one per line)
shap_e.cond_type = "t2m"
shap_e.inference("prompts.txt")
# Processes each line: "a red apple", "a coffee mug", etc.
```
```

--------------------------------

### utils.get_new_video_name

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Constructs a new output filename that encodes the operation name and lineage of source files, enabling traceability of multi-step processing chains. Also provides functions to generate fresh random filenames for videos, audio, and images.

```APIDOC
## `utils.get_new_video_name` — Derived Output Filename Generator

Constructs a new output filename that encodes the operation name and lineage of source files, enabling traceability of multi-step processing chains.

```python
from utils import get_new_video_name, generate_video_name_mp4, generate_audio_name, generate_image_name

# Derive a new filename from an existing one
new_name = get_new_video_name("video/abc123.mp4", func_name="edge")
print(new_name)  # "video/xxxx_edge_abc123_abc123.mp4"  (xxxx = random 4-char UUID)

# Chained operations preserve original file lineage
step1 = get_new_video_name("video/abc123.mp4", func_name="subclip")
# "video/1a2b_subclip_abc123_abc123.mp4"
step2 = get_new_video_name(step1, func_name="subtitles")
# "video/3c4d_subtitles_1a2b_abc123.mp4"  <- preserves original "abc123" at the end

# Generate fresh random filenames
print(generate_video_name_mp4())  # "video/f3a1b2c4.mp4"
print(generate_audio_name())      # "audio/7e8f9a0b.wav"
print(generate_image_name())      # "image/1c2d3e4f.png"
```
```

--------------------------------

### Synthesize Talking Head Video from Audio and Image using SadTalker

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Employs the SadTalker model to create a realistic talking-head video by animating a portrait image with a driven audio file. The input format is 'audio_path, image_path'.

```python
from modules.sadtalker import Sadtalker
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
sd = Sadtalker(device=device)

# Generate a talking head video
sd.inference("audio/speech.wav, image/portrait.png")
# Output video is written to video/sadtalker/

# Test directly (as in test.py)
sd.inference("audio/ac9fc7da.wav,image/test.png")
```

--------------------------------

### Initialize Moviepy Interface for Video Editing

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Initializes the Moviepy interface for various video editing tasks such as subclip extraction, adding subtitles, concatenation, and audio manipulation.

```python
from modules.video_moviepy import MoviepyInterface

movie_interface = MoviepyInterface()
video_subclip = movie_interface.intercept_fragments # input: f"{video_path}, {begin_second}, {end_second}", output: new_video_path
add_subtitles = movie_interface.add_subtitles # input: f"{video_path}, {start_time}, {duration}, {instruct_text}", output: new_video_path
concat_videos = movie_interface.concat_videos # input: f"{video_path1}, {video_path2}...", output: new_video_path
extract_audio = movie_interface.extract_audio # input: f"{video_path}", output: audio_path
add_audio_to_video = movie_interface.add_audio_to_video # input: f"{video_path}, {audio_path}", output: new_video_path
```

--------------------------------

### Video Preprocessing Annotators

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Extracts structured control signals frame-by-frame from a video. Canny edges require no extra model or GPU. Pose and Depth estimation require significant GPU memory.

```python
from modules.annotator import Video2Canny, Video2Pose, Video2Depth

# Edge detection (no GPU required)
canny = Video2Canny()
edge_video = canny.inference("video/dance_clip.mp4")
print(edge_video)  # "video/xxxx_edge_dance_dance.mp4"

# Pose estimation (~1.3 GB GPU)
pose = Video2Pose(device="cuda:1")
pose_video = pose.inference("video/dance_clip.mp4")
print(pose_video)  # "video/xxxx_pose_dance_dance.mp4"

# Depth estimation (~1.5 GB GPU)
depth = Video2Depth(device="cuda:1")
depth_video = depth.inference("video/dance_clip.mp4")
print(depth_video)  # "video/xxxx_depth_dance_dance.mp4"
```

--------------------------------

### Generate Derived Output Filenames

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

The `get_new_video_name` function generates new output filenames that encode the operation name and lineage of source files for traceability. Utility functions for generating fresh random filenames are also provided.

```python
from utils import get_new_video_name, generate_video_name_mp4, generate_audio_name, generate_image_name

# Derive a new filename from an existing one
new_name = get_new_video_name("video/abc123.mp4", func_name="edge")
print(new_name)  # "video/xxxx_edge_abc123_abc123.mp4"  (xxxx = random 4-char UUID)

# Chained operations preserve original file lineage
step1 = get_new_video_name("video/abc123.mp4", func_name="subclip")
# "video/1a2b_subclip_abc123_abc123.mp4"
step2 = get_new_video_name(step1, func_name="subtitles")
# "video/3c4d_subtitles_1a2b_abc123.mp4"  <- preserves original "abc123" at the end

# Generate fresh random filenames
print(generate_video_name_mp4())  # "video/f3a1b2c4.mp4"
print(generate_audio_name())      # "audio/7e8f9a0b.wav"
print(generate_image_name())      # "image/1c2d3e4f.png"
```

--------------------------------

### Video to Pose Extraction

Source: https://github.com/kaleido-lab/dolphin/blob/main/inference.ipynb

Extracts human pose information from a video. Requires the path to the video file as input and returns the path to the output video.

```python
print("./" + video2pose("video/v0006.mp4"))
```

--------------------------------

### Dolphin Project Citation

Source: https://github.com/kaleido-lab/dolphin/blob/main/README.md

This is the BibTeX entry for citing the Dolphin project. It includes author information, year, a link to the GitHub repository, and the project title.

```bibtex
@misc{dolphin,
    Author = {Zehuan Huang, Haoran Feng, Chongzhi Zhang, Lu Sheng, Ziwei Liu, Jing Shao},
    Year = {2023},
    Note = {https://github.com/kaleido-lab/dolphin},
    Title = {Dolphin: General Video Interaction Platform Based on LLMs}
}
```

--------------------------------

### Sadtalker

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Synthesizes a realistic talking-head video by animating a source portrait image with a driven audio file using the SadTalker model. Input is expected as a comma-separated string of `audio_path, image_path`. Output video is written to `video/sadtalker/`.

```APIDOC
## `Sadtalker` — Audio-Driven Talking Head Video Synthesis

Generates a realistic talking-head video by animating a source portrait image with a driven audio file, using the SadTalker model. Input is `"audio_path, image_path"`.

```python
from modules.sadtalker import Sadtalker
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
sd = Sadtalker(device=device)

# Generate a talking head video
sd.inference("audio/speech.wav, image/portrait.png")
# Output video is written to video/sadtalker/

# Test directly (as in test.py)
sd.inference("audio/ac9fc7da.wav,image/test.png")
```
```

--------------------------------

### Video Preprocessing Annotators

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Extracts structured control signals frame-by-frame from a video: Canny edges, OpenPose skeleton joints, and MiDaS depth maps. All return the path to a new video file.

```APIDOC
## Video2Canny, Video2Pose, Video2Depth

### Description
These annotator classes extract structured control signals frame-by-frame from a video: Canny edges (CPU, no extra model), OpenPose skeleton joints, and MiDaS depth maps. All return the path to a new video file.

### Methods
- `Video2Canny()`: Initializes the Canny edge detector.
- `Video2Pose(device)`: Initializes the Pose estimator. `device` can be a string like "cuda:0".
- `Video2Depth(device)`: Initializes the Depth estimator. `device` can be a string like "cuda:0".

### Inference
- `canny.inference(video_path)`: Processes a video to extract Canny edges.
- `pose.inference(video_path)`: Processes a video to extract pose skeletons.
- `depth.inference(video_path)`: Processes a video to extract depth maps.

### Returns
All inference methods return the path to the generated video file (e.g., "video/xxxx_edge_dance_dance.mp4").
```

--------------------------------

### video_utils.create_video

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Saves a list of numpy frame arrays as an MP4 video file. Optionally rescales pixel values from `[-1, 1]` to `[0, 255]` before saving. Uses `imageio` for writing.

```APIDOC
## `video_utils.create_video` — Save Frames as MP4

Takes a list of numpy frame arrays (H, W, C), optionally rescales from `[-1, 1]` to `[0, 255]`, and writes them to an MP4 file using `imageio`.

```python
from video_utils import create_video, prepare_video
from einops import rearrange
import numpy as np

# Load, process and re-save
video, fps = prepare_video("video/input.mp4", 512, "cpu", normalize=False)
frames = rearrange(video, "f c h w -> f h w c").numpy().astype(np.uint8)
frames_list = [frames[i] for i in range(len(frames))]

output_path = create_video(frames_list, fps=fps, path="video/output.mp4", rescale=False)
print(output_path)  # "video/output.mp4"

# With rescale=True (input in [-1, 1] range)
norm_video, fps = prepare_video("video/input.mp4", 512, "cpu", normalize=True)
norm_frames = [rearrange(norm_video[i], "c h w -> h w c").numpy() for i in range(len(norm_video))]
output_path = create_video(norm_frames, fps=fps, path="video/output_rescaled.mp4", rescale=True)
```
```

--------------------------------

### Generate Image from Text using Stable Diffusion

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Uses the Text2Image module to generate a single image from a text prompt. The output image is saved to the 'image/' directory.

```python
from modules.stable_diffusion import Text2Image

t2i = Text2Image()

image_path = t2i.image_generation("An astronaut riding a horse on Mars, photorealistic")
print(image_path)  # "image/b1c2d3e4.png"
```

--------------------------------

### VideoPix2Pix

Source: https://context7.com/kaleido-lab/dolphin/llms.txt

Edits an existing video according to a natural-language instruction using InstructPix2Pix applied frame-by-frame.

```APIDOC
## VideoPix2Pix

### Description
Edits an existing video according to a natural-language instruction (e.g., change style, replace objects, alter background) using InstructPix2Pix applied frame-by-frame. Input is a comma-separated string of the video path and the instruction text.

### Method
- `VideoPix2Pix(device)`: Initializes the video style editing model. `device` can be a string like "cuda:0".

### Inference
- `pix2pix.inference(input_string)`: Applies the instruction to the video. `input_string` format: "video/park_scene.mp4, make it look like a watercolor painting".

### Returns
Returns the path to the edited video file (e.g., "video/xxxx_pix2pix_....mp4").
```