### Usage example for DPO training Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Shows how to load a JSON dataset, create a DPOText2AudioDataset, and use it with a DataLoader. The example unpacks the batch into text, chosen paths, rejected paths, durations, and indices. ```python from datasets import load_dataset from torch.utils.data import DataLoader from tangoflux.utils import DPOText2AudioDataset raw = load_dataset("json", data_files={"train": "data/train_dpo.json"}) train_dataset = DPOText2AudioDataset( raw["train"], prefix="", text_column="captions", audio_w_column="chosen", audio_l_column="reject", duration="duration", num_examples=-1, ) loader = DataLoader(train_dataset, shuffle=True, batch_size=4, collate_fn=train_dataset.collate_fn) text, chosen_paths, rejected_paths, durations, indices = next(iter(loader)) ``` -------------------------------- ### Example: decode and save audio Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-nodes.md Example usage of the play method to decode and save audio to the output directory. It prints the UI audios result. ```python node = TangoFluxVAEDecodeAndPlay() result = node.play(vae, latents, filename_prefix="my_audio", format="wav", save_output=True) print(result["ui"]["audios"]) ``` -------------------------------- ### Installation: Install requirements Source: https://github.com/declare-lab/tangoflux/blob/main/comfyui/README.md Changes into the comfyui subdirectory and runs the install.py script to install dependencies. ```bash cd ComfyUI-TangoFlux/comfyui python install.py ``` -------------------------------- ### Installation: Navigate to custom_nodes directory Source: https://github.com/declare-lab/tangoflux/blob/main/comfyui/README.md First step of the installation process. Change to the ComfyUI custom_nodes directory before cloning the repository. ```bash cd ComfyUI/custom_nodes ``` -------------------------------- ### HTTP request example for GET /tangoflux/playaudio Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/endpoints.md Shows the URL query parameters for requesting an audio file. The filename must be a valid output file name; type and subfolder are optional. ```http GET /tangoflux/playaudio?filename=TangoFlux_00001_.wav&type=output&subfolder= ``` -------------------------------- ### Install TangoFlux from GitHub Source: https://github.com/declare-lab/tangoflux/blob/main/Demo.ipynb Installs the TangoFlux package directly from the GitHub repository. ```python !pip install git+https://github.com/declare-lab/TangoFlux.git ``` -------------------------------- ### TangoFluxSampler usage example Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-nodes.md Shows how to call the sample method with a prompt, steps, guidance_scale, duration, and seed, and prints the duration from the returned latents dict. ```python latents, = sampler.sample(model, "Rain on a tin roof", steps=50, guidance_scale=4.5, duration=10, seed=7) print(latents["duration"]) # 10 ``` -------------------------------- ### DPOText2AudioDataset.__len__ Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Returns the number of examples in the dataset. ```APIDOC ## DPOText2AudioDataset.__len__ ### Description Returns the number of examples in the dataset. ### Method `__len__` ### Endpoint `DPOText2AudioDataset.__len__` ### Parameters None ### Request Example ```python len(dataset) ``` ### Response #### Success Response - **int** - `len(self.inputs)`. #### Response Example ```python 1000 ``` ``` -------------------------------- ### Usage example for SFT training with Text2AudioDataset Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Shows how to load a JSON dataset, create a Text2AudioDataset with column names 'captions', 'location', and 'duration', and build a DataLoader with the dataset's collate_fn. The example is from tangoflux/train.py. ```python from datasets import load_dataset from torch.utils.data import DataLoader from tangoflux.utils import Text2AudioDataset raw = load_dataset("json", data_files={"train": "data/train.json"}) train_dataset = Text2AudioDataset( raw["train"], "", "captions", "location", "duration", num_examples=-1 ) train_dataloader = DataLoader( train_dataset, shuffle=True, batch_size=4, collate_fn=train_dataset.collate_fn, ) text, audio_paths, durations, _ = next(iter(train_dataloader)) ``` -------------------------------- ### Direct usage example of TangoFluxLoader Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-nodes.md Shows how to instantiate TangoFluxLoader and call load_tangoflux with TeaCache enabled and a custom threshold, returning the model and VAE. ```python from comfyui.nodes import TangoFluxLoader loader = TangoFluxLoader() model, vae = loader.load_tangoflux( enable_teacache=True, rel_l1_thresh=0.3, device="cuda" ) ``` -------------------------------- ### Launch TangoFlux DPO training with accelerate Source: https://github.com/declare-lab/tangoflux/blob/main/README.md Use this command to start DPO training. The training files must contain 'chosen', 'reject', 'caption', and 'duration' fields. The command uses the same accelerate and config files as standard training. ```bash CUDA_VISIBLE_DEVICES=0,1 accelerate launch --config_file='configs/accelerator_config.yaml' tangoflux/train_dpo.py --checkpointing_steps="best" --save_every=5 --config='configs/tangoflux_config.yaml' ``` -------------------------------- ### DPOText2AudioDataset __len__ method Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Returns the number of examples in the dataset, which is the length of the internal inputs list. ```python def __len__(self): ``` -------------------------------- ### Launch TangoFlux training with accelerate Source: https://github.com/declare-lab/tangoflux/blob/main/README.md Use this command to start standard training of TangoFlux. It requires an accelerate configuration file and a TangoFlux config file specifying training data paths. The command sets CUDA_VISIBLE_DEVICES to 0,1 and uses the provided config files. ```bash CUDA_VISIBLE_DEVICES=0,1 accelerate launch --config_file='configs/accelerator_config.yaml' tangoflux/train.py --checkpointing_steps="best" --save_every=5 --config='configs/tangoflux_config.yaml' ``` -------------------------------- ### Installation: Clone the repository Source: https://github.com/declare-lab/tangoflux/blob/main/comfyui/README.md Clones the TangoFlux repository into the ComfyUI-TangoFlux directory. Note the space in the command between the repository URL and the target directory name. ```bash git clone https://github.com/declare-lab/TangoFlux ComfyUI-TangoFlux ``` -------------------------------- ### Read WAV file and convert to fake stereo Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Example usage from the training scripts: reads a WAV file with a 30-second duration and repeats mono waveforms to create fake stereo. ```python wav = read_wav_file("data/test.wav", duration_sec=30) if wav.shape[0] == 1: wav = wav.repeat(2, 1) # "fake stereo" ``` -------------------------------- ### SFT dataset row example (JSON) Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/types.md Each element in data/train.json must contain captions, location, and duration fields. The captions field is the text prompt, location is the audio file path, and duration is the target duration in seconds. The column names can be configured via --text_column and --audio_column. ```json {"captions": "Rhythmic wooden table tapping...", "location": "data/test.wav", "duration": 10.0} ``` -------------------------------- ### Python client fetch for GET /tangoflux/playaudio Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/endpoints.md Equivalent client-side fetch using requests. Expects a 200 status and audio/wav content type, then saves the response body to a local file. ```python # Equivalent client-side fetch (aiohttp / requests) import requests r = requests.get( "http://127.0.0.1:8188/tangoflux/playaudio", params={"filename": "TangoFlux_00001_.wav", "type": "output"}, ) assert r.status_code == 200 assert r.headers["Content-Type"] == "audio/wav" open("TangoFlux_00001_.wav", "wb").write(r.content) ``` -------------------------------- ### Run tangoflux-demo basic Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Basic usage of the `tangoflux-demo` command to launch the Gradio web interface with default settings. ```bash tangoflux-demo ``` -------------------------------- ### Run tangoflux-demo with sharing Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Usage of `tangoflux-demo` with custom host, port, and sharing enabled to create a public share link. ```bash tangoflux-demo --host 0.0.0.0 --port 7860 --share ``` -------------------------------- ### DurationEmbedder usage example Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-model.md Shows how to instantiate DurationEmbedder and embed a tensor of duration values. The example asserts the output shape is (2, 1, 1024). ```python import torch from tangoflux.model import DurationEmbedder embedder = DurationEmbedder(number_embedding_dim=1024, min_value=0, max_value=30) emb = embedder(torch.tensor([10.0, 27.5])) # shape (2, 1, 1024) assert emb.shape == (2, 1, 1024) ``` -------------------------------- ### Define tangoflux-demo CLI command Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Defines the CLI entry point for `tangoflux-demo` with options `--host` (default '127.0.0.1'), `--port` (default None), and `--share` flag. Launches `demo.queue().launch(server_name=host, server_port=port, share=share)`. ```python @click.command() @click.option('--host', default='127.0.0.1', help='Host to bind to') @click.option('--port', default=None, help='Port to bind to') @click.option('--share', is_flag=True, help='Enable sharing via Gradio') def main(host, port, share): ``` -------------------------------- ### DPOText2AudioDataset.__init__ Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Initializes the dataset with the given parameters, preparing internal lists for captions, chosen and rejected audio paths, durations, and indices. ```APIDOC ## DPOText2AudioDataset.__init__ ### Description Initializes the dataset with the given parameters, preparing internal lists for captions, chosen and rejected audio paths, durations, and indices. ### Method Constructor ### Endpoint `DPOText2AudioDataset.__init__` ### Parameters #### Constructor Parameters - **dataset** (datasets.Dataset or IterableDataset) - Required - HF dataset (from `tangoflux/train_dpo.py`: `load_dataset("json", ...)["train"]`). - **prefix** (str) - Required - String prepended to every caption (`prefix + inp`). - **text_column** (str) - Required - Column name holding captions. - **audio_w_column** (str) - Required - Column name holding "chosen" (`w` = winner) audio paths. - **audio_l_column** (str) - Required - Column name holding "reject" (`l` = loser) audio paths. - **duration** (str) - Required - Column name holding durations. - **num_examples** (int) - Optional - Default `-1` - Truncates all internal lists to this length when `!= -1`; number of usable examples otherwise. ### Request Example ```python DPOText2AudioDataset( dataset=raw["train"], prefix="", text_column="captions", audio_w_column="chosen", audio_l_column="reject", duration="duration", num_examples=-1, ) ``` ### Response #### Success Response No return value. #### Response Example No response body. ``` -------------------------------- ### Basic inference_flow example Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-model.md Basic usage of inference_flow with a single prompt. Returns latents of shape (1, audio_seq_len, 64). ```python import torch from tangoflux import TangoFluxInference model = TangoFluxInference() latents = model.model.inference_flow( "Hammer slowly hitting the wooden table", num_inference_steps=50, guidance_scale=4.5, duration=10, ) assert latents.shape == (1, model.model.audio_seq_len, 64) ``` -------------------------------- ### Register console scripts in setup.py Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Registers the two console scripts `tangoflux` and `tangoflux-demo` in `setup.py`, mapping them to the `main` functions in `tangoflux/cli.py` and `tangoflux/demo.py` respectively. ```python entry_points={ "console_scripts": [ "tangoflux=tangoflux.cli:main", "tangoflux-demo=tangoflux.demo:main", ], }, ``` -------------------------------- ### GET /tangoflux/playaudio Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/endpoints.md Serves an audio file produced by the TangoFluxVAEDecodeAndPlay node. The filename, type, and subfolder query parameters determine which file is returned. No authentication is required. ```APIDOC ## GET /tangoflux/playaudio ### Description Serves an audio file produced by the TangoFluxVAEDecodeAndPlay node. The filename, type, and subfolder query parameters determine which file is returned. No authentication is required. ### Method GET ### Endpoint /tangoflux/playaudio ### Parameters #### Query Parameters - **filename** (str) - Required - Output file name; resolved via `folder_paths.annotated_filepath(filename)`. Leading-`/` values and any value containing `..` are rejected (403). After resolution, `os.path.basename` is applied again before joining. - **type** (str) - Optional - Directory type when the annotated path has no output dir: `folder_paths.get_directory_by_type(file_type)` — typically `"output"` or `"temp"` (the `TangoFluxVAEDecodeAndPlay` node writes `type="output"` or `type="temp"` into its `ui.audios` entries). Default: `"output"`. - **subfolder** (str) - Optional - Optional subfolder under the output dir; combined via `os.path.join(output_dir, subfolder)`; the result is checked with `os.path.commonpath` to stay inside `output_dir` (otherwise 403). Default: `None`. ### Request Example GET /tangoflux/playaudio?filename=my_audio_00001_.wav&type=output&subfolder= ### Response #### Success Response (200) - **Content-Type** - Derived from file extension (see table below). - **Body** - Raw bytes of the audio file. #### Response Example HTTP/1.1 200 OK Content-Type: audio/wav #### Content-Type Map | Extension | Content-Type | |-----------|--------------| | .wav | `audio/wav` | | .mp3 | `audio/mpeg` | | .flac | `audio/flac` | | .aac | `audio/aac` | | .wma | `audio/x-ms-wma` | #### Error Responses | Status | Condition | |--------|-----------| | 404 | `filename` query param missing or resolved `file_path` is not a file. | | 403 | `filename[0] == "/"` or `".." in filename`; subfolder escape — `os.path.commonpath((os.path.abspath(full_output_dir), output_dir)) != output_dir`. | | 400 | no output dir resolved — `output_dir is None` after `annotated_filepath`/`get_directory_by_type`; unsupported extension — `content_type is None`. | | 500 | `open(file_path, "rb").read()` raises anything. | All error responses are empty-bodied `web.Response(status=N)`. ### Usage Example GET /tangoflux/playaudio?filename=TangoFlux_00001_.wav&type=output ```python import requests r = requests.get( "http://127.0.0.1:8188/tangoflux/playaudio", params={"filename": "TangoFlux_00001_.wav", "type": "output"}, ) assert r.status_code == 200 assert r.headers["Content-Type"] == "audio/wav" open("TangoFlux_00001_.wav", "wb").write(r.content) ``` ``` -------------------------------- ### Load TangoFlux model and generate audio Source: https://github.com/declare-lab/tangoflux/blob/main/Inference.ipynb Imports torchaudio, TangoFluxInference, and IPython Audio. Creates a TangoFluxInference instance with the 'declare-lab/TangoFlux' model, generates audio from the prompt 'Hammer slowly hitting the wooden table' with 50 steps and 10 seconds duration, and displays the audio in the notebook. ```python import torchaudio from tangoflux import TangoFluxInference from IPython.display import Audio model = TangoFluxInference(name="declare-lab/TangoFlux") audio = model.generate("Hammer slowly hitting the wooden table", steps=50, duration=10) Audio(data=audio, rate=44100) ``` -------------------------------- ### Route definition for GET /tangoflux/playaudio Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/endpoints.md Registers the route on ComfyUI's PromptServer instance. No authentication is required. The handler parses query parameters from the request URL. ```python @server.PromptServer.instance.routes.get("/tangoflux/playaudio") async def play_audio(request): ``` -------------------------------- ### Launch tangoflux/train.py with accelerate Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-training.md Command to launch the training script tangoflux/train.py using accelerate with a specified accelerator config and training config. Uses --checkpointing_steps="best" and --save_every=5. ```bash accelerate launch --config_file='configs/accelerator_config.yaml' tangoflux/train.py \ --checkpointing_steps="best" --save_every=5 --config='configs/tangoflux_config.yaml' ``` -------------------------------- ### DPO usage in tangoflux/train_dpo.py Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-model.md Usage of the forward method for DPO objective (sft=False). Requires explicit reference-model setup: deep copy of transformer, freezing, and setting requires_grad=False. Returns loss, raw_model_loss, raw_ref_loss, implicit_acc. ```python import copy model.ref_transformer = copy.deepcopy(model.transformer) model.ref_transformer.requires_grad_ = False model.ref_transformer.eval() for param in model.ref_transformer.parameters(): param.requires_grad = False loss, raw_model_loss, raw_ref_loss, implicit_acc = model( audio_latent, text, duration=duration, sft=False ) ``` -------------------------------- ### Export nodes and server from comfyui/__init__.py Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-nodes.md Re-exports the node class mappings and server module, and sets the web directory for the ComfyUI extension. ```python from .nodes import NODE_CLASS_MAPPINGS from .server import * WEB_DIRECTORY = "./comfyui/web" __all__ = ["NODE_CLASS_MAPPINGS", "WEB_DIRECTORY"] ``` -------------------------------- ### tangoflux_config.yaml sample configuration Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/configuration.md Sample YAML configuration for training TangoFlux. Contains paths, training, and model sections. Loaded by train.py and train_dpo.py via --config (default tangoflux_config.yaml). ```yaml paths: train_file: "data/train.json" val_file: "data/val.json" test_file: "data/val.json" resume_from_checkpoint: "" output_dir: "outputs/" training: per_device_batch_size: 4 learning_rate: 5e-4 gradient_accumulation_steps: 1 num_train_epochs: 80 num_warmup_steps: 1000 max_audio_duration: 30 model: num_layers: 6 num_single_layers: 18 in_channels: 64 attention_head_dim: 128 joint_attention_dim: 1024 num_attention_heads: 8 audio_seq_len: 645 max_duration: 30 uncondition: false text_encoder_name: "google/flan-t5-large" ``` -------------------------------- ### Manual enablement of TeaCache (comfyui.teacache) Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-teacache.md Shows how to enable TeaCache without the loader: import teacache_forward, assign it to FluxTransformer2DModel.forward, and set the required class attributes (enable_teacache, cnt, num_steps, rel_l1_thresh, accumulated_rel_l1_distance, previous_modulated_input, previous_residual). After setup, call the model normally; the function runs the cached forward when enable_teacache is true. ```python from diffusers import FluxTransformer2DModel from comfyui.teacache import teacache_forward model = FluxTransformer2DModel(...) FluxTransformer2DModel.forward = teacache_forward model.__class__.enable_teacache = True model.__class__.cnt = 0 model.__class__.num_steps = 50 model.__class__.rel_l1_thresh = 0.25 model.__class__.accumulated_rel_l1_distance = 0 model.__class__.previous_modulated_input = None model.__class__.previous_residual = None out = model(hidden_states=..., encoder_hidden_states=..., timestep=..., img_ids=..., txt_ids=..., return_dict=False)[0] ``` -------------------------------- ### Initialize TangoFlux Inference Model Source: https://github.com/declare-lab/tangoflux/blob/main/Demo.ipynb Imports required libraries and initializes the TangoFlux inference model using the pretrained 'declare-lab/TangoFlux' checkpoint. ```python import IPython import torchaudio from tangoflux import TangoFluxInference from IPython.display import Audio model = TangoFluxInference(name='declare-lab/TangoFlux') ``` -------------------------------- ### Run tangoflux basic generation Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Basic usage of the `tangoflux` command to generate audio from a text prompt, saving to `output.wav` with specified duration and steps. ```bash tangoflux "Hammer slowly hitting the wooden table" output.wav --duration 10 --steps 50 ``` -------------------------------- ### Import datasets and audio utilities from tangoflux.utils Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Imports the dataset classes and audio helper functions used by the training scripts. ```python from tangoflux.utils import Text2AudioDataset, DPOText2AudioDataset from tangoflux.utils import read_wav_file, pad_wav, normalize_wav ``` -------------------------------- ### DPOText2AudioDataset class definition Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-utils.md Constructor signature for the DPOText2AudioDataset class. It takes a Hugging Face dataset, a prefix for captions, column names for text, chosen and rejected audio paths, and a duration column. The num_examples parameter truncates internal lists when not -1. ```python class DPOText2AudioDataset(Dataset): def __init__( self, dataset, prefix, text_column, audio_w_column, audio_l_column, duration, num_examples=-1, ): ``` -------------------------------- ### main function signature for generate_crpo_dataset.py Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-crpo.md Main CLI entry point for the candidate generation script. It parses arguments, validates GPU count, samples prompts, splits across GPUs, and runs generate_audio_chunk processes. ```python def main(): ``` -------------------------------- ### Define tangoflux CLI command with click Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-cli.md Defines the CLI entry point for `tangoflux` with positional arguments `prompt` and `output_file`, and options `--duration` (default 10) and `--steps` (default 50). Validation raises `click.BadParameter` (exit code 2) when duration is outside [1,30] or steps outside [10,100]. ```python @click.command() @click.argument('prompt') @click.argument('output_file') @click.option('--duration', default=10, type=int, help='Duration in seconds (1-30)') @click.option('--steps', default=50, type=int, help='Number of inference steps (10-100)') def main(prompt: str, output_file: str, duration: int, steps: int): ``` -------------------------------- ### TangoFluxLoader __init__ method Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/comfyui-nodes.md Initializes the loader with default values for model, vae, enable_teacache, rel_l1_thresh, and stores the original forward method of FluxTransformer2DModel for potential restoration. ```python def __init__(self): self.model = None self.vae = None self.enable_teacache = False self.rel_l1_thresh = 0.25 self.original_forward = FluxTransformer2DModel.forward ``` -------------------------------- ### TangoFluxInference constructor Source: https://github.com/declare-lab/tangoflux/blob/main/_autodocs/api-reference/tangoflux-inference.md Initializes the inference wrapper. Downloads the model snapshot from the Hugging Face Hub, loads the VAE and model weights, and moves them to the specified device. The default device is CUDA if available, otherwise CPU. ```python def __init__( self, name="declare-lab/TangoFlux", device="cuda" if torch.cuda.is_available() else "cpu", ): ```