### Install IndexTTS Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/00_START_HERE.txt Install the package via pip or from the source repository. ```bash pip install indextts ``` ```bash git clone https://github.com/asr-pub/index-tts-lora.git cd index-tts-lora pip install -e . ``` -------------------------------- ### Multi-Speaker Fine-Tuning Setup Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Initializes a FinetuneDataset with multiple manifest files and demonstrates passing speaker IDs during the training loop. ```python from indextts.data_utils import FinetuneDataset import json # Load speaker info with open("speaker_info.json") as f: speaker_info = json.load(f) # Prepare datasets for all speakers manifest_files = [item["train_jsonl"] for item in speaker_info] speaker_ids = [item["speaker"] for item in speaker_info] # Create combined dataset dataset = FinetuneDataset( manifest_files=manifest_files, bpe_path="checkpoints/bpe_model.model", speaker_ids=speaker_ids, config=config ) # In training loop, use speaker_ids to condition the model for batch in loader: speaker_ids = batch['speaker_ids'] # List of speaker ID strings # Pass to model (e.g., embed speaker IDs and inject into model) output = model( ..., speaker_ids=speaker_ids, ... ) ``` -------------------------------- ### Implement Training Loop for UnifiedVoice Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Demonstrates the full training pipeline including model initialization, optimizer setup, data loading, and the iterative training loop with checkpointing. ```python import torch import torch.nn as nn from torch.optim import Adam from indextts.gpt.model import UnifiedVoice from indextts.data_utils import FinetuneDataset from omegaconf import OmegaConf # Configuration config = OmegaConf.load("config.yaml") device = "cuda:0" if torch.cuda.is_available() else "cpu" epochs = 10 learning_rate = 1e-4 # Load model model = UnifiedVoice(**config.gpt).to(device) model.train() # Load checkpoint (optional) checkpoint = torch.load("checkpoints/gpt_weights.pth") model.load_state_dict(checkpoint, strict=False) # Create optimizer optimizer = Adam(model.parameters(), lr=learning_rate, weight_decay=0.01) # Create dataset and dataloader dataset = FinetuneDataset( manifest_files=["data/speaker/metadata_train.jsonl"], bpe_path=config.dataset.bpe_model, speaker_ids=["speaker_id"], config=config ) from torch.utils.data import DataLoader loader = DataLoader(dataset, batch_size=4, shuffle=True) # Training loop for epoch in range(epochs): for batch_idx, batch in enumerate(loader): # Move to device text_ids = batch['text_ids'].to(device) codes = batch['codes'].to(device) mels = batch['mels'].to(device) # Forward pass optimizer.zero_grad() output = model( speech_conditioning_latent=mels, text_inputs=text_ids, mel_codes=codes, # Add other parameters as needed ) # Compute loss (example: token prediction loss) if isinstance(output, tuple): logits = output[0] else: logits = output loss = nn.CrossEntropyLoss()( logits.view(-1, logits.size(-1)), codes.view(-1) ) # Backward pass loss.backward() optimizer.step() if batch_idx % 10 == 0: print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}") # Save checkpoint torch.save(model.state_dict(), f"checkpoints/gpt_weights_epoch{epoch}.pth") print("Training completed!") ``` -------------------------------- ### Manifest File JSONL Format Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Example of the required JSONL format where each line represents a single training sample. ```json {"text": "训练文本示例1", "codes": "speaker/sample_0001.codes.npy", "mels": "speaker/sample_0001.mels.npy", "condition": "speaker/sample_0001.condition.npy", "duration": 3.5} {"text": "Sample 2 text", "codes": "speaker/sample_0002.codes.npy", "mels": "speaker/sample_0002.mels.npy", "condition": null, "duration": 4.2} ``` -------------------------------- ### Text-to-Speech Inference Examples Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Demonstrates basic usage, generation control with sampling parameters, and multi-speaker inference. ```python # Basic usage output_file = tts.infer( audio_prompt="/path/to/reference.wav", text="这是一个合成的文本示例。", output_path="output.wav" ) # With generation control output_file = tts.infer( audio_prompt="/path/to/reference.wav", text="Hello, this is a test.", output_path="output.wav", do_sample=True, temperature=0.8, top_p=0.9, max_mel_tokens=800 ) # Multi-speaker inference output_file = tts.infer( audio_prompt="/path/to/speaker1_reference.wav", text="Synthesize with speaker 1", output_path="speaker1_output.wav", speaker_id="speaker_1_id" ) ``` -------------------------------- ### Load Speaker Info Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Example of loading and parsing a speaker information JSON file. ```python import json with open("speaker_info.json") as f: speaker_info = json.load(f) speaker_ids = [item["speaker"] for item in speaker_info] # ['kaishu_30min'] ``` -------------------------------- ### Install PyTorch dependencies Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Use this command to resolve ImportError when PyTorch is missing. ```bash # Install PyTorch pip install torch torchaudio ``` -------------------------------- ### Define Token List Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Example of a list of BPE token strings from a SentencePiece tokenizer. ```python tokens = ["▁这", "是", "一", "个", "▁测", "试"] ``` -------------------------------- ### Bucket Metadata Structure Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Example of the dictionary structure used for sentence metadata in batch organization. ```json { "idx": 0, # int: original sentence index "sent": ["▁这", "是"], # list[str]: tokenized sentence "len": 2 # int: token count } ``` -------------------------------- ### Speaker Information JSON Structure Source: https://github.com/asr-pub/index-tts-lora/blob/main/README.md Example of the speaker_info.json file generated after processing audio data. ```json [ { "speaker": "kaishu_30min", "avg_duration": 6.6729, "sample_num": 270, "total_duration_in_seconds": 1801.696, "total_duration_in_minutes": 30.028, "total_duration_in_hours": 0.500, "train_jsonl": "/path/to/kaishu_30min/metadata_train.jsonl", "valid_jsonl": "/path/to/kaishu_30min/metadata_valid.jsonl", "medoid_condition": "/path/to/kaishu_30min/medoid_condition.npy" } ] ``` -------------------------------- ### Perform accelerated inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Examples of using infer_fast for standard and low-memory GPU configurations. ```python # Fast synthesis of long text output_file = tts.infer_fast( audio_prompt="/path/to/reference.wav", text="Very long text with many sentences...", output_path="output.wav", max_text_tokens_per_sentence=100, sentences_bucket_max_size=4 ) # Tune for low-memory GPUs output_file = tts.infer_fast( audio_prompt="/path/to/reference.wav", text="Long text...", output_path="output.wav", max_text_tokens_per_sentence=80, # Smaller buckets sentences_bucket_max_size=2 # Fewer parallel sentences ) ``` -------------------------------- ### Define Token IDs Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Example of integer indices represented as a PyTorch tensor. ```python text_ids = torch.tensor([[100, 245, 389, 42]], dtype=torch.int32) # Shape: [batch=1, time=4] ``` -------------------------------- ### Instantiate FinetuneDataset for Training Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Demonstrates initializing the dataset for both single-speaker and multi-speaker configurations. ```python from omegaconf import OmegaConf from indextts.data_utils import FinetuneDataset config = OmegaConf.load("config.yaml") # Single speaker dataset = FinetuneDataset( manifest_files=["data/speaker/metadata_train.jsonl"], bpe_path="checkpoints/bpe_model.model", speaker_ids=["speaker_id"], config=config ) # Multi-speaker dataset = FinetuneDataset( manifest_files=[ "data/speaker1/metadata_train.jsonl", "data/speaker2/metadata_train.jsonl" ], bpe_path="checkpoints/bpe_model.model", speaker_ids=["speaker1_id", "speaker2_id"], config=config ) print(f"Dataset size: {len(dataset)}") sample = dataset[0] print(f"Sample keys: {sample.keys()}") ``` -------------------------------- ### Initialize FinetuneDataset Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Load configuration and speaker metadata to instantiate the training dataset. ```python from indextts.data_utils import FinetuneDataset from omegaconf import OmegaConf import json # Load configuration config = OmegaConf.load("config.yaml") # Load speaker info with open("finetune_data/processed_data/speaker_info.json") as f: speaker_info = json.load(f) # Extract manifest paths and speaker IDs manifest_files = [item["train_jsonl"] for item in speaker_info] speaker_ids = [item["speaker"] for item in speaker_info] # Create dataset dataset = FinetuneDataset( manifest_files=manifest_files, bpe_path=config.dataset.bpe_model, speaker_ids=speaker_ids, config=config ) print(f"Loaded {len(dataset)} training samples") ``` -------------------------------- ### Register CLI Entry Point Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md Defines the console script entry point in setup.py to map the indextts command to the main function. ```python entry_points={ "console_scripts": [ "indextts = indextts.cli:main", ] } ``` -------------------------------- ### Handle DeepSpeed fallback Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md DeepSpeed is optional; the system automatically falls back to standard inference if it is not installed. ```python # Fallback is automatic; inference still works without DeepSpeed tts = IndexTTS(is_fp16=True, device="cuda:0") # Uses standard inference even without DeepSpeed ``` -------------------------------- ### Configure FinetuneDataset Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Initializes the dataset for fine-tuning by providing a list of manifest files and corresponding speaker IDs. ```python from omegaconf import OmegaConf from indextts.data_utils import FinetuneDataset config = OmegaConf.load("config.yaml") dataset = FinetuneDataset( manifest_files=[ "data/speaker_1/metadata_train.jsonl", "data/speaker_2/metadata_train.jsonl" ], bpe_path=config.dataset.bpe_model, speaker_ids=["speaker_1_id", "speaker_2_id"], config=config ) ``` -------------------------------- ### Initialize TextTokenizer Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Create a TextTokenizer instance with a BPE model file and an optional normalizer. ```python from indextts.utils.front import TextTokenizer, TextNormalizer normalizer = TextNormalizer() normalizer.load() tokenizer = TextTokenizer( vocab_file="checkpoints/bpe_model.model", normalizer=normalizer ) ``` -------------------------------- ### FinetuneDataset.__init__ Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Initializes the dataset for loading multi-speaker fine-tuning data from manifest files. ```APIDOC ## FinetuneDataset.__init__(manifest_files, bpe_path, speaker_ids, config) ### Description Initializes the FinetuneDataset with manifest files, BPE vocabulary, and speaker identifiers. It filters samples based on duration (1.0s to 20.0s). ### Parameters - **manifest_files** (list[str]) - Required - List of JSONL manifest file paths. - **bpe_path** (str) - Required - Path to SentencePiece BPE vocabulary model file. - **speaker_ids** (list[str]) - Required - List of speaker ID strings. - **config** (DictConfig) - Required - OmegaConf configuration with preprocessing settings. ### Raises - ValueError: If manifest_files and speaker_ids lengths don't match. - FileNotFoundError: If manifest file or BPE model doesn't exist. - RuntimeError: If text normalizer fails to load. ``` -------------------------------- ### Initialize and run IndexTTS inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/00_START_HERE.txt Instantiate the main engine and perform text-to-speech synthesis. ```python tts = IndexTTS(cfg_path="config.yaml", device="cuda:0") output = tts.infer(audio_prompt="ref.wav", text="Text...", output_path="out.wav") ``` -------------------------------- ### Initialize IndexTTS with Configuration Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Automatically load configuration during IndexTTS initialization by providing the path to the YAML file and the model directory. ```python from indextts.infer import IndexTTS # Configuration is loaded automatically in __init__ tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints" ) # Access loaded config print(tts.cfg.gpt.max_mel_tokens) print(tts.cfg.dataset.sample_rate) print(tts.stop_mel_token) # Convenience attribute ``` -------------------------------- ### Initialize and access FinetuneDataset Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Load multi-speaker training data from manifest files and retrieve individual samples. ```python def __init__( self, manifest_files: list[str], bpe_path: str, speaker_ids: list[str], config: DictConfig ) ``` ```python from omegaconf import OmegaConf from indextts.data_utils import FinetuneDataset config = OmegaConf.load("config.yaml") dataset = FinetuneDataset( manifest_files=[ "data/speaker1/metadata_train.jsonl", "data/speaker2/metadata_train.jsonl" ], bpe_path="checkpoints/bpe_model.model", speaker_ids=["speaker1", "speaker2"], config=config ) sample = dataset[0] # Returns dict with text_ids, codes, mels, speaker_id ``` ```python def __len__(self) -> int ``` ```python def __getitem__(self, index: int) -> dict ``` -------------------------------- ### Load configuration with OmegaConf Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Demonstrates loading a YAML configuration file into a DictConfig object. ```python from omegaconf import OmegaConf, DictConfig config = OmegaConf.load("checkpoints/config.yaml") # type: DictConfig ``` -------------------------------- ### CLI Synthesis with Quality Control Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Demonstrates how to toggle between high-quality and fast inference modes using CLI flags. ```bash # High quality (slower) indextts "Text to synthesize" \ --voice reference.wav \ --output_path output_hq.wav \ --device cuda:0 \ --fp16 # Fast inference indextts "Text to synthesize" \ --voice reference.wav \ --output_path output_fast.wav \ --device cuda:0 ``` -------------------------------- ### Initialize FinetuneDataset Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/00_START_HERE.txt Configure the PyTorch dataset for model fine-tuning. ```python dataset = FinetuneDataset( manifest_files=["train.jsonl"], bpe_path="bpe_model.model", speaker_ids=["speaker_id"], config=config ) ``` -------------------------------- ### Initialize IndexTTS Instance Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Demonstrates initializing the IndexTTS class for both single-speaker and multi-speaker inference scenarios. ```python from indextts.infer import IndexTTS # Single-speaker inference tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", device="cuda:0", is_fp16=True ) # Multi-speaker inference tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", device="cuda:0", is_fp16=True, speaker_info_path="finetune_data/processed_data/speaker_info.json" ) ``` -------------------------------- ### IndexTTS.__init__ Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Initializes the IndexTTS model by loading the configuration, GPT language model, BigVGAN vocoder, and text processing components. It supports optional multi-speaker mode and device configuration. ```APIDOC ## IndexTTS.__init__ ### Description Initializes the IndexTTS model by loading the configuration, GPT language model, BigVGAN vocoder, and text processing components. Automatically detects and configures the device (CUDA, MPS, or CPU). ### Parameters - **cfg_path** (str) - Optional - Path to the YAML configuration file containing model hyperparameters and settings. Default: "checkpoints/config.yaml" - **model_dir** (str) - Optional - Directory path containing pre-trained model weights (GPT, BigVGAN, BPE). Default: "checkpoints" - **is_fp16** (bool) - Optional - Enable FP16 (half-precision) inference for reduced memory and faster computation. Default: True - **device** (str) - Optional - Device to run inference on: 'cuda:0', 'cuda:1', 'cpu', 'mps'. Auto-detected if None. - **use_cuda_kernel** (bool) - Optional - Use BigVGAN's custom fused activation CUDA kernel for improved speed. Only effective on CUDA devices. - **speaker_info_path** (str) - Optional - Path to speaker_info.json file for multi-speaker support. ### Example ```python from indextts.infer import IndexTTS # Single-speaker inference tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", device="cuda:0", is_fp16=True ) ``` ``` -------------------------------- ### Search Documentation via Command Line Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/00_START_HERE.txt Use standard shell commands to search or analyze documentation files. ```bash grep -l "keyword" *.md ``` ```bash wc -l *.md ``` ```bash grep -r "search_term" . ``` -------------------------------- ### CLI Synthesis with Custom Configuration Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Executes synthesis with specific model paths, device selection, and performance flags. ```bash indextts "Very long text that needs synthesis..." \ --voice reference_speaker.wav \ --output_path output.wav \ --config checkpoints/config.yaml \ --model_dir checkpoints \ --device cuda:0 \ --fp16 \ --force ``` -------------------------------- ### IndexTTS Constructor Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Initializes the IndexTTS model with configuration and device settings. ```APIDOC ## IndexTTS Constructor ### Description Initializes the IndexTTS model instance. ### Parameters - **cfg_path** (string) - Required - Path to the configuration YAML file. - **model_dir** (string) - Required - Directory containing model weights. - **is_fp16** (boolean) - Optional - Whether to use FP16 precision. - **device** (string) - Optional - Device to run on (e.g., 'cuda:0', 'cpu'). - **use_cuda_kernel** (boolean) - Optional - Whether to use CUDA kernels. - **speaker_info_path** (string) - Optional - Path to speaker information for multi-speaker mode. ``` -------------------------------- ### Initialize IndexTTS Model Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Configure the IndexTTS constructor with model paths, precision settings, and device preferences. ```python IndexTTS( cfg_path="checkpoints/config.yaml", # Configuration file path model_dir="checkpoints", # Model weights directory is_fp16=True, # Use FP16 precision device=None, # Auto-detect device use_cuda_kernel=None, # Auto-detect CUDA kernel availability speaker_info_path=None # Multi-speaker mode (optional) ) ``` -------------------------------- ### Access Dataset Samples and DataLoader Integration Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Shows how to access individual samples and integrate the dataset with a PyTorch DataLoader. ```python dataset = FinetuneDataset(...) sample = dataset[0] print(f"Text IDs shape: {sample['text_ids'].shape}") print(f"Codes shape: {sample['codes_npy'].shape}") print(f"Mels shape: {sample['mels_npy'].shape}") print(f"Speaker: {sample['speaker_id']}") # Use with DataLoader from torch.utils.data import DataLoader loader = DataLoader( dataset, batch_size=4, shuffle=True, collate_fn=custom_collate_fn # For padding variable-length sequences ) for batch in loader: # batch['text_ids']: [B, max_text_len] # batch['codes_npy']: [B, max_code_len] # ... pass ``` -------------------------------- ### Handle Missing Audio Prompt File Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Verify the existence of the reference audio file before passing it to the inference engine. ```python import os from indextts.infer import IndexTTS audio_file = "/path/to/reference.wav" if not os.path.exists(audio_file): print(f"Error: Audio file not found at {audio_file}") # Fix: provide correct path or download reference audio else: tts = IndexTTS() output = tts.infer( audio_prompt=audio_file, text="Text to synthesize", output_path="output.wav" ) ``` -------------------------------- ### Create and Evaluate Validation Dataset Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Initializes a validation dataset from manifest files and performs an evaluation loop using a DataLoader. ```python # Load validation data val_dataset = FinetuneDataset( manifest_files=["data/speaker/metadata_valid.jsonl"], bpe_path="config.dataset.bpe_model", speaker_ids=["speaker_id"], config=config ) # Use for evaluation val_loader = DataLoader(val_dataset, batch_size=4) model.eval() with torch.no_grad(): for batch in val_loader: # Evaluate on validation set text_ids = batch['text_ids'].to(device) mels = batch['mels'].to(device) codes = batch['codes'].to(device) output = model( speech_conditioning_latent=mels, text_inputs=text_ids, mel_codes=codes ) # Compute validation loss # ... ``` -------------------------------- ### Handle Missing Configuration File Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Catch FileNotFoundError when the specified configuration file path is incorrect or missing. ```python from indextts.infer import IndexTTS import os cfg_path = "checkpoints/config.yaml" try: tts = IndexTTS(cfg_path=cfg_path, model_dir="checkpoints") except FileNotFoundError: print(f"Config not found at {cfg_path}") # Verify directory structure and config location ``` -------------------------------- ### Initialize TextNormalizer Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Instantiate and load the TextNormalizer to prepare language-specific normalization rules. ```python from indextts.utils.front import TextNormalizer normalizer = TextNormalizer() normalizer.load() # Load language-specific normalizers ``` -------------------------------- ### Tokenize text with TextTokenizer Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Demonstrates tokenization, encoding to IDs, decoding back to text, and checking vocabulary size. ```python from indextts.utils.front import TextTokenizer, TextNormalizer normalizer = TextNormalizer() normalizer.load() tokenizer = TextTokenizer("checkpoints/bpe_model.model", normalizer) # Tokenize text text = "这是一个分词示例。" # Get tokens as strings tokens = tokenizer.tokenize(text) print(f"Tokens: {tokens}") # Get token IDs token_ids = tokenizer.encode(text) print(f"Token IDs: {token_ids}") # Convert back to text decoded = tokenizer.decode(token_ids) print(f"Decoded: {decoded}") # Check vocabulary size print(f"Vocabulary size: {tokenizer.vocab_size}") ``` -------------------------------- ### Handle Device Compatibility Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Check for available hardware devices before initializing the IndexTTS model to prevent runtime errors. ```python from indextts.infer import IndexTTS import torch # Check available devices device = "cuda:0" if torch.cuda.is_available() else "cpu" tts = IndexTTS(device=device) ``` -------------------------------- ### Train the Model Source: https://github.com/asr-pub/index-tts-lora/blob/main/README.md Execute the training process for the LoRA fine-tuning. ```shell python train.py ``` -------------------------------- ### Import paths for training Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md Use these imports for dataset handling, text normalization, and configuration management. ```python # Dataset from indextts.data_utils import FinetuneDataset # Text processing (same as inference) from indextts.utils.front import TextNormalizer, TextTokenizer # Configuration from omegaconf import OmegaConf ``` -------------------------------- ### Check System Information Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Prints current CUDA availability, version, device name, and total GPU memory. ```python import torch import torchaudio from indextts.infer import IndexTTS # Show device info print(f"CUDA available: {torch.cuda.is_available()}") print(f"CUDA version: {torch.version.cuda}") print(f"Device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU'}") # Show memory if torch.cuda.is_available(): print(f"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB") ``` -------------------------------- ### Run Inference via Command-Line Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Execute synthesis using CLI arguments that override configuration file settings. ```bash indextts "Text" \ --voice reference.wav \ --output_path output.wav \ --config checkpoints/config.yaml \ --model_dir checkpoints \ --device cuda:0 \ --fp16 \ --force ``` -------------------------------- ### Perform Basic Text-to-Speech Synthesis Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Initializes the IndexTTS engine and generates audio from a single sentence. ```python from indextts.infer import IndexTTS # Initialize TTS engine tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", device="cuda:0" ) # Synthesize single sentence output_path = tts.infer( audio_prompt="reference_speaker.wav", text="这是一个示例文本。", output_path="output.wav" ) print(f"Generated audio saved to: {output_path}") ``` -------------------------------- ### Initialize FinetuneDataset Constructor Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Defines the signature for the FinetuneDataset constructor used to load training data from manifest files. ```python def __init__( self, manifest_files: list[str], bpe_path: str, speaker_ids: list[str], config: DictConfig ) ``` -------------------------------- ### Configure PyTorch Environment Variables Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Set environment variables to control GPU performance, device visibility, and debugging behavior. ```bash # Enable/disable CuDNN benchmarking (affects GPU performance) export CUDNN_BENCHMARK=1 # Set CUDA device visibility export CUDA_VISIBLE_DEVICES=0,1 # Control floating-point precision export CUDA_LAUNCH_BLOCKING=1 # For debugging ``` -------------------------------- ### Synthesize Audio with Control Parameters Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Adjusts generation quality using parameters like temperature, top_p, top_k, and beam search settings. ```python from indextts.infer import IndexTTS tts = IndexTTS(device="cuda:0", is_fp16=True) # Control generation quality output = tts.infer( audio_prompt="reference.wav", text="Long text with multiple sentences. Each sentence will be processed.", output_path="output_quality.wav", # Quality tuning temperature=0.7, # Lower = more deterministic top_p=0.85, # Nucleus sampling threshold top_k=50, # Keep top K tokens num_beams=5, # Beam search width repetition_penalty=8.0, # Penalize repetition max_mel_tokens=800 # Allow longer output ) ``` -------------------------------- ### Performance Optimization Configuration Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/README.md Settings to improve inference speed and memory efficiency. ```python is_fp16=True ``` ```python infer_fast() ``` -------------------------------- ### Validate Required Model Files Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Ensures all necessary configuration and weight files exist before initializing the model. ```python import os from indextts.infer import IndexTTS def validate_files(cfg_path, model_dir, audio_prompt): """Validate all required files exist.""" errors = [] if not os.path.exists(cfg_path): errors.append(f"Config not found: {cfg_path}") if not os.path.exists(audio_prompt): errors.append(f"Audio not found: {audio_prompt}") required_files = ["gpt_weights.pth", "bigvgan_weights.pth", "bpe_model.model"] for fname in required_files: fpath = os.path.join(model_dir, fname) if not os.path.exists(fpath): errors.append(f"Model file not found: {fpath}") if errors: raise FileNotFoundError("\n".join(errors)) return True # Usage try: validate_files("checkpoints/config.yaml", "checkpoints", "reference.wav") tts = IndexTTS() except FileNotFoundError as e: print(f"Setup error: {e}") ``` -------------------------------- ### Command Line Inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/README.md Run text-to-speech synthesis directly from the command line. ```bash indextts "Text to synthesize" --voice reference.wav --output_path output.wav ``` -------------------------------- ### Basic CLI Synthesis Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Performs basic text-to-speech synthesis using the command-line interface. ```bash indextts "这是命令行合成示例。" \ --voice reference.wav \ --output_path output.wav ``` -------------------------------- ### Verify Model Checkpoints Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Checks for the existence of required weight files in the specified directory before loading. ```python import os model_dir = "checkpoints" required = ["gpt_weights.pth", "bigvgan_weights.pth"] for fname in required: if not os.path.exists(os.path.join(model_dir, fname)): print(f"Missing: {fname}") print("Download from: https://github.com/index-tts/index-tts") ``` -------------------------------- ### File Validation for Synthesis Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Ensures configuration files, audio prompts, and model weights exist before initializing the IndexTTS engine. ```python import os from indextts.infer import IndexTTS def validate_and_synthesize(cfg_path, model_dir, audio_prompt, text, output_path): """Validate all files before synthesis.""" # Check config file if not os.path.exists(cfg_path): raise FileNotFoundError(f"Config not found: {cfg_path}") # Check audio prompt if not os.path.exists(audio_prompt): raise FileNotFoundError(f"Audio prompt not found: {audio_prompt}") # Check model files required_files = [ "gpt_weights.pth", "bigvgan_weights.pth", "bpe_model.model" ] for fname in required_files: fpath = os.path.join(model_dir, fname) if not os.path.exists(fpath): raise FileNotFoundError(f"Model file not found: {fpath}") # All checks passed, proceed with synthesis tts = IndexTTS(cfg_path=cfg_path, model_dir=model_dir) return tts.infer( audio_prompt=audio_prompt, text=text, output_path=output_path ) # Usage try: output = validate_and_synthesize( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", audio_prompt="reference.wav", text="Text to synthesize", output_path="output.wav" ) print(f"Generated: {output}") except FileNotFoundError as e: print(f"Setup error: {e}") ``` -------------------------------- ### VQ-VAE Module Directory Structure Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md Displays the file organization within the vqvae directory. ```text indextts/vqvae/ ├── __init__.py └── xtts_dvae.py ``` -------------------------------- ### Import Core Index-TTS Components Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/README.md Standard import statements for accessing the primary inference, utility, and dataset classes. ```python from indextts.infer import IndexTTS, set_seed from indextts.utils.front import TextNormalizer, TextTokenizer from indextts.utils.feature_extractors import MelSpectrogramFeatures from indextts.data_utils import FinetuneDataset from indextts.cli import main ``` -------------------------------- ### Dataset Configuration Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Specifies training, preprocessing, and codec settings for the dataset. ```yaml dataset: sample_rate: 24000 bpe_model: "bpe/zh_punc.model" num_mels: 100 # Data filtering min_duration: 1.0 max_duration: 20.0 # Codec configuration codec: "hubert" codec_sample_rate: 16000 ``` -------------------------------- ### Run Inference via Python API Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Generate audio using the infer method with specific generation parameters and overrides. ```python # Generation parameters (passed to infer/infer_fast) tts.infer( audio_prompt="reference.wav", text="Text to synthesize", output_path="output.wav", max_text_tokens_per_sentence=120, # Override max_text_tokens from config # Generation kwargs do_sample=True, top_p=0.8, top_k=30, temperature=1.0, num_beams=3, repetition_penalty=10.0, max_mel_tokens=600 # Override config.gpt.max_mel_tokens ) ``` -------------------------------- ### Perform Batch Synthesis Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Iterates through a list of texts to generate multiple audio files sequentially. ```python from indextts.infer import IndexTTS import glob tts = IndexTTS(device="cuda:0") # Synthesize multiple texts texts = [ "第一个样本文本。", "第二个样本文本。", "第三个样本文本。" ] for idx, text in enumerate(texts): output_path = f"output_{idx}.wav" tts.infer( audio_prompt="reference.wav", text=text, output_path=output_path ) print(f"Saved: {output_path}") ``` -------------------------------- ### indextts CLI Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Command-line interface for synthesizing text into speech using a reference audio file. ```APIDOC ## indextts ### Description Synthesizes text to speech using a reference audio file. ### Arguments - **text** (str) - Required - Text to synthesize - **--voice** (str) - Required - Reference audio file path (WAV format) - **--output_path** (str) - Optional - Output WAV file path (Default: "gen.wav") - **--config** (str) - Optional - Configuration file path (Default: "checkpoints/config.yaml") - **--model_dir** (str) - Optional - Model directory path (Default: "checkpoints") - **--device** (str) - Optional - Device: 'cuda', 'cpu', 'mps' (Default: auto) - **--fp16** (flag) - Optional - Use FP16 precision (Default: True) - **--force** (flag) - Optional - Overwrite existing output file (Default: False) ### Example indextts "这是一个测试。" --voice reference.wav --output_path output.wav --device cuda:0 --fp16 ``` -------------------------------- ### Define Audio List Format Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md The input file requires tab-separated paths and transcripts. ```text /path/to/audio1.wav 训练文本第一个示例 /path/to/audio2.wav Training text example 2 /path/to/audio3.wav 另一个样本 ``` -------------------------------- ### FinetuneDataset.__getitem__ Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Retrieves a single training sample by index. ```APIDOC ## FinetuneDataset.__getitem__(index) ### Description Loads a single training sample, tokenizing text and loading pre-extracted acoustic features from disk. ### Parameters - **index** (int) - Required - Sample index (0 to len(dataset)-1). ### Returns - **dict** - A dictionary containing: - text_ids (Tensor): Text token IDs. - codes_npy (ndarray): Acoustic codes. - mels_npy (ndarray): Mel-spectrogram. - condition_npy (ndarray): Condition embedding (or None). - speaker_id (str): Speaker identifier. ``` -------------------------------- ### Module dependency graph Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md Visual representation of the internal module dependencies and external library requirements. ```text indextts.infer (IndexTTS) ├── indextts.gpt.model (UnifiedVoice) ├── indextts.BigVGAN.models (BigVGAN) ├── indextts.utils.front (TextNormalizer, TextTokenizer) ├── indextts.utils.feature_extractors (MelSpectrogramFeatures) ├── indextts.utils.checkpoint (load_checkpoint) └── torch, torchaudio, omegaconf indextts.cli.main() └── indextts.infer.IndexTTS indextts.data_utils.FinetuneDataset ├── indextts.utils.front (TextNormalizer, TextTokenizer) ├── torch └── numpy indextts.gpt.model ├── transformers (GPT2PreTrainedModel, GenerationMixin) ├── indextts.gpt.conformer_encoder ├── indextts.gpt.perceiver └── torch indextts.BigVGAN.models ├── torch ├── indextts.BigVGAN.activations ├── indextts.BigVGAN.alias_free_activation (CUDA or torch) └── indextts.BigVGAN.utils ``` -------------------------------- ### Define __getitem__ Method Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Defines the signature for retrieving a single training sample by index. ```python def __getitem__(self, index: int) -> dict ``` -------------------------------- ### Define Model Checkpoint Paths in YAML Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Specify paths to pre-trained model weights and versioning at the root level of the configuration file. ```yaml # Model weights paths gpt_checkpoint: "gpt_weights.pth" bigvgan_checkpoint: "bigvgan_weights.pth" dvae_checkpoint: "dvae_weights.pth" # Optional: VQ-VAE for debugging # Version tracking version: 1.5 ``` -------------------------------- ### Compare Multiple Speakers with Python API Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Iterates through available speakers to generate audio output for each using the same input text. ```python from indextts.infer import IndexTTS import json with open("speaker_info.json") as f: speaker_info = json.load(f) tts = IndexTTS(speaker_info_path="speaker_info.json") text_to_synthesize = "这是一个多说话人示例。" # Generate with each speaker for info in speaker_info: speaker_id = info["speaker"] ref_audio = f"data/{speaker_id}/reference.wav" output_path = f"output_{speaker_id}.wav" tts.infer( audio_prompt=ref_audio, text=text_to_synthesize, output_path=output_path, speaker_id=speaker_id ) print(f"Generated for speaker: {speaker_id}") ``` -------------------------------- ### Extract Audio Tokens and Speaker Conditions Source: https://github.com/asr-pub/index-tts-lora/blob/main/README.md Run the extraction script to process audio files and generate speaker information. The input file must contain audio paths and transcripts separated by tabs. ```shell # Extract tokens and speaker conditions python tools/extract_codec.py --audio_list ${audio_list} --extract_condition # audio_list format: audio_path + transcript, separated by \t /path/to/audio.wav 小朋友们,大家好,我是凯叔,今天我们讲一个龟兔赛跑的故事。 ``` -------------------------------- ### Configure Logging Level Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Suppress library-level warnings by setting the logging level. ```bash # Suppress TensorFlow/other library warnings export TF_CPP_MIN_LOG_LEVEL=3 ``` -------------------------------- ### Enable Verbose Logging Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/ERROR_HANDLING.md Enables detailed output during the inference process for troubleshooting. ```python from indextts.infer import IndexTTS tts = IndexTTS() output = tts.infer( audio_prompt="reference.wav", text="Text", output_path="output.wav", verbose=True # Print detailed logs ) ``` -------------------------------- ### CLI Synthesis Command Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/INDEX_TTS_API_REFERENCE.md Synthesizes audio from text using a reference WAV file via the command line. ```bash indextts "Your text here" --voice /path/to/reference.wav --output_path output.wav [--config checkpoints/config.yaml] [--model_dir checkpoints] [--device cuda:0] [--fp16] [--force] ``` ```bash indextts "这是一个测试。" --voice reference.wav --output_path output.wav --device cuda:0 --fp16 ``` -------------------------------- ### BigVGAN Directory Structure Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md The file system layout for the BigVGAN vocoder implementation. ```text indextts/BigVGAN/ ├── __init__.py # Exports ├── models.py # Generator and discriminator ├── bigvgan.py # BigVGAN configuration ├── activations.py # Activation functions ├── ECAPA_TDNN.py # Speaker encoder ├── utils.py # Utility functions ├── alias_free_activation/ # Anti-aliasing components │ ├── cuda/ # CUDA kernels │ └── torch/ # Torch implementation ├── alias_free_torch/ # Pure torch anti-aliasing ├── nnet/ # Neural network blocks │ ├── CNN.py │ ├── linear.py │ └── normalization.py └── [other components] ``` -------------------------------- ### Configure IndexTTS with YAML Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/README.md Defines the essential parameters for the GPT encoder, BigVGAN vocoder, and dataset settings. Ensure checkpoint paths point to valid model files. ```yaml gpt: max_mel_tokens: 600 max_text_tokens: 400 stop_mel_token: 50 stop_text_token: 0 layers: 30 model_dim: 1024 bigvgan: n_mels: 100 sample_rate: 24000 dataset: sample_rate: 24000 bpe_model: "bpe_model.model" gpt_checkpoint: "gpt_weights.pth" bigvgan_checkpoint: "bigvgan_weights.pth" version: 1.5 ``` -------------------------------- ### Define Generation Parameters Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TYPES.md Dictionary of keyword arguments passed to infer() or infer_fast() to control generation behavior. ```python generation_kwargs = { "do_sample": True, # bool "top_p": 0.8, # float, range [0, 1] "top_k": 30, # int, > 0 "temperature": 1.0, # float, > 0 "length_penalty": 0.0, # float "num_beams": 3, # int, >= 1 "repetition_penalty": 10.0, # float, >= 1 "max_mel_tokens": 600 # int } ``` -------------------------------- ### Configure High Quality Inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Use this configuration for higher quality output at the cost of slower generation speeds. ```yaml gpt: layers: 30 model_dim: 1024 num_heads: 16 generation_kwargs: do_sample: false # Greedy decoding num_beams: 5 # Larger beam width repetition_penalty: 10.0 max_mel_tokens: 800 # Allow longer outputs max_text_tokens_per_sentence: 150 ``` -------------------------------- ### Multi-Speaker Synthesis with Python API Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/USAGE_EXAMPLES.md Initializes IndexTTS with speaker information and performs inference using a specific speaker ID and reference audio. ```python from indextts.infer import IndexTTS import json # Load speaker info with open("speaker_info.json") as f: speaker_info = json.load(f) print("Available speakers:") for info in speaker_info: print(f" - {info['speaker']}: {info['total_duration_in_seconds']:.1f}s") # Initialize with multi-speaker support tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", device="cuda:0", speaker_info_path="speaker_info.json" ) # Synthesize with specific speaker speaker_1 = speaker_info[0]["speaker"] speaker_1_ref = f"data/{speaker_1}/reference.wav" output = tts.infer( audio_prompt=speaker_1_ref, text="Synthesize with speaker 1", output_path="speaker1_output.wav", speaker_id=speaker_1 ) ``` -------------------------------- ### Configure Balanced Inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md The default configuration providing a balance between generation speed and output quality. ```yaml gpt: layers: 30 model_dim: 1024 generation_kwargs: do_sample: true top_p: 0.8 top_k: 30 temperature: 1.0 num_beams: 3 max_mel_tokens: 600 max_text_tokens_per_sentence: 120 ``` -------------------------------- ### Configure Text Processing Cache Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Define the directory for WeText normalization cache. ```bash # Control WeText cache (for text normalization) export WETEXT_CACHE_DIR=/path/to/cache ``` -------------------------------- ### Loading .codes.npy Files Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/TRAINING_INTERFACE.md Loads acoustic token indices from a .codes.npy file, which contains a 1D array of int32 values. ```python import numpy as np codes = np.load("speaker/sample.codes.npy") print(codes.shape) # (time_frames,) print(codes.dtype) # int32 print(codes.min()) # 0 print(codes.max()) # 51 ``` -------------------------------- ### Tokenize text input Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/00_START_HERE.txt Convert raw text into token IDs using a BPE model. ```python tokenizer = TextTokenizer("bpe_model.model") tokens = tokenizer.tokenize("Text") token_ids = tokenizer.encode("Text") ``` -------------------------------- ### Import paths for model architecture Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/MODULE_STRUCTURE.md Advanced imports for accessing core GPT models, BigVGAN vocoders, and checkpoint loading utilities. ```python # GPT model from indextts.gpt.model import GPT2InferenceModel, UnifiedVoice # BigVGAN vocoder from indextts.BigVGAN.models import BigVGAN # Feature extraction from indextts.utils.checkpoint import load_checkpoint ``` -------------------------------- ### Configure GPT Model Parameters Source: https://github.com/asr-pub/index-tts-lora/blob/main/_autodocs/CONFIGURATION.md Defines the GPT-based text-to-acoustic-tokens model architecture, tokenizer settings, and training hyperparameters in YAML format. ```yaml gpt: # Tokenizer and vocabulary settings max_mel_tokens: 604 max_text_tokens: 402 start_text_token: 260 stop_text_token: 0 stop_mel_token: 50 # Model architecture layers: 30 model_dim: 1024 num_heads: 16 hidden_size: 4096 dropout: 0.1 # Speaker conditioning num_speakers: 1 speaker_embedding_dim: 512 # Training hyperparameters learning_rate: 0.0001 weight_decay: 0.01 ``` -------------------------------- ### Run Inference Source: https://github.com/asr-pub/index-tts-lora/blob/main/README.md Perform speech synthesis using the trained model. ```shell python indextts/infer.py ```