### Launch Web UI for Interactive Text-to-Speech Synthesis Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Start a Gradio-based web interface for interactive TTS synthesis with default or custom configuration. Supports audio upload, microphone recording, text input, and speaker selection. Accessible via browser at specified host and port. ```bash # Start web UI with default settings python webui.py # Custom configuration python webui.py \ --host 0.0.0.0 \ --port 8080 \ --model_dir checkpoints \ --verbose ``` -------------------------------- ### Initialize and Configure IndexTTS Engine in Python Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Set up the IndexTTS text-to-speech engine with configuration file, model directory, and hardware settings. Enables FP16 precision for faster inference and CUDA kernel optimization. Returns an initialized TTS object ready for inference operations. ```python from indextts.infer import IndexTTS import torch # Initialize TTS engine tts = IndexTTS( cfg_path="checkpoints/config.yaml", model_dir="checkpoints", is_fp16=True, device="cuda:0", use_cuda_kernel=True ) ``` -------------------------------- ### Custom Training Configuration with YAML Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Defines parameters for LoRA fine-tuning, including dataset properties, GPT model architecture, and training settings. This YAML configuration file specifies learning rates, LoRA parameters (rank, alpha, dropout), target modules for LoRA adaptation, and checkpoint paths. ```yaml # finetune_models/config.yaml # Dataset configuration dataset: bpe_model: bpe.model sample_rate: 24000 mel: n_fft: 1024 hop_length: 256 win_length: 1024 n_mels: 100 # GPT model configuration gpt: model_dim: 1280 max_mel_tokens: 800 max_text_tokens: 600 heads: 20 layers: 24 condition_type: "conformer_perceiver" # Training configuration train: finetune_model_dir: "finetune_models" seed: 91 epochs: 15 max_grad_norm: 1.0 text_weight: 0.1 # Balance between text and mel loss data_path: "finetune_data/processed_data" optimizer: learning_rate: 5.0e-5 weight_decay: 0.01 warmup_ratio: 0.1 loraplus_lr_ratio: 8.0 # LoRA+ B matrix LR multiplier lora: r: 16 # LoRA rank lora_alpha: 32 # LoRA scaling factor lora_dropout: 0.1 target_modules: - "attn.c_attn" # Attention query/key/value - "attn.c_proj" # Attention output projection - "mlp.c_fc" # MLP first layer - "mlp.c_proj" # MLP second layer # Model checkpoints dvae_checkpoint: dvae.pth gpt_checkpoint: gpt.pth bigvgan_checkpoint: bigvgan_generator.pth version: 1.5 ``` -------------------------------- ### Training Data Preparation Pipeline with Python Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Automates the preparation of audio datasets for fine-tuning using a Python subprocess. It requires an audio list file and outputs processed data including VQ-VAE codes, mel spectrograms, conditioning latents, and metadata splits. The script 'tools/extract_codec.py' is used for this process. ```python # Create audio list file (audio_list.txt): # /data/audio/sample1.wav 这是第一个训练样本的文本内容。 # /data/audio/sample2.wav 这是第二个训练样本的文本内容。 # Run extraction script import subprocess result = subprocess.run([ 'python', 'tools/extract_codec.py', '--audio_list', 'data/kaishu_audio_list.txt', '--extract_condition', '--output_dir', 'finetune_data/processed_data', '--model_path', 'checkpoints/gpt.pth', '--device', 'cuda', '--distance_metric', 'euclidean' ], capture_output=True, text=True) print(result.stdout) # Generated files structure: # finetune_data/processed_data/kaishu_audio_list_20250119_143022/ # ├── sample1.wav_codes.npy # VQ-VAE codes [T//2] # ├── sample1.wav_mel.npy # Mel spectrogram [100, T] # ├── sample1.wav_condition.npy # Conditioning latent [32, 512] # ├── metadata.jsonl # All samples metadata # ├── metadata_train.jsonl # 90% training split # ├── metadata_valid.jsonl # 10% validation split # ├── medoid_condition.npy # Representative speaker condition # ├── medoid_info.json # Medoid statistics # └── distance_matrix.npy # Sample similarity matrix # speaker_info.json format: # [ # { # "speaker": "kaishu_30min", # "avg_duration": 6.6729, # "sample_num": 270, # "total_duration_in_minutes": 30.028, # "train_jsonl": "/path/to/metadata_train.jsonl", # "valid_jsonl": "/path/to/metadata_valid.jsonl", # "medoid_condition": "/path/to/medoid_condition.npy" # } # ] ``` -------------------------------- ### Configure and Train Multi-Speaker LoRA Fine-Tuned Model Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Set up fine-tuning parameters including learning rate, LoRA rank, and training epochs in configuration file, then execute model training. Supports multi-speaker scenarios with speaker-specific embeddings. Generates trained model checkpoint with LoRA weights. ```bash # Step 2: Configure fine-tuning parameters # Edit finetune_models/config.yaml: # train: # finetune_model_dir: "finetune_models" # epochs: 15 # optimizer: # learning_rate: 5.0e-5 # loraplus_lr_ratio: 8.0 # lora: # r: 16 # lora_alpha: 32 # Step 3: Train the model python train.py ``` -------------------------------- ### Command Line Text-to-Speech Inference with Audio Prompt Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Execute text-to-speech synthesis from command line using audio prompts with configurable parameters. Supports both basic usage with default settings and advanced configurations including FP16 precision, custom devices, and forced processing. Output is saved as WAV audio files. ```bash # Basic usage with audio prompt indextts "小朋友们，大家好，我是凯叔，今天我们讲一个龟兔赛跑的故事。" \ -v /path/to/reference_audio.wav \ -o output.wav \ --config checkpoints/config.yaml \ --model_dir checkpoints # Advanced usage with custom parameters indextts "A thin man lies against the side of the street with his shirt and a shoe off and bags nearby." \ -v reference_english.wav \ -o generated_english.wav \ --fp16 \ --device cuda:0 \ --force ``` -------------------------------- ### Load Fine-Tuned Multi-Speaker Model for Inference Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Initialize IndexTTS with fine-tuned model configuration and speaker information to enable multi-speaker inference. Loads speaker embeddings and LoRA weights from training. Allows speaker selection via speaker_id parameter during synthesis. ```python from indextts.infer import IndexTTS # Load fine-tuned model with multi-speaker support tts = IndexTTS( cfg_path="finetune_models/config_finetuned.yaml", model_dir="finetune_models", is_fp16=True, device="cuda:0", speaker_info_path="finetune_data/processed_data/speaker_info.json" ) # Generate speech with specific speaker output = tts.infer( audio_prompt="prompts/kaishu_sample.wav", text="这是一个测试句子，使用凯叔的声音。", output_path="output/kaishu_output.wav", speaker_id="kaishu_30min", verbose=True ) ``` -------------------------------- ### Python Custom Training Loop for Index-TTS-LoRA Source: https://context7.com/asr-pub/index-tts-lora/llms.txt This Python code demonstrates the custom training loop for Index-TTS-LoRA. It loads configuration, prepares training and validation datasets using DataLoader, initializes a custom Trainer, and initiates the training process. The training includes automatic LoRA adapter injection, speaker condition management, gradient accumulation, learning rate scheduling, validation, checkpointing, and early stopping. Checkpoints are saved in the 'finetune_models/checkpoints/' directory. ```python from omegaconf import OmegaConf from torch.utils.data import DataLoader from indextts.data_utils import load_finetune_datasets, collate_finetune_fn import torch # Load configuration config = OmegaConf.load('finetune_models/config.yaml') # Prepare datasets bpe_model_path = f"{config.train.finetune_model_dir}/{config.dataset.bpe_model}" train_ds, valid_ds = load_finetune_datasets(config, bpe_model_path) train_loader = DataLoader( train_ds, batch_size=16, shuffle=True, collate_fn=collate_finetune_fn, num_workers=4 ) valid_loader = DataLoader( valid_ds, batch_size=8, shuffle=False, collate_fn=collate_finetune_fn, num_workers=2 ) # Initialize trainer from train import Trainer trainer = Trainer(config) # Training loop with automatic: # - LoRA adapter injection # - Speaker condition management # - Gradient accumulation # - Learning rate scheduling # - Validation and checkpointing # - Early stopping trainer.train(train_loader, valid_loader) # Generated checkpoints: # finetune_models/checkpoints/ # ├── gpt_epoch_1.pth # Epoch checkpoints # ├── gpt_epoch_2.pth # ├── ... # ├── gpt_best.pth # Best validation loss # └── gpt_finetuned.pth # Final model # Each checkpoint contains: # - model: merged LoRA + base model state_dict # - speakers: list of speaker IDs # - speaker_conditions: learned conditioning vectors per speaker ``` -------------------------------- ### Audio Feature Extraction with Python Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Extracts mel spectrograms, VQ-VAE codes, and conditioning latents from audio files using Python. It requires torch, torchaudio, and specific modules from the indextts library. Outputs include mel spectrograms, discrete codes, and conditioning latents. ```python from indextts.utils.feature_extractors import MelSpectrogramFeatures from indextts.vqvae.xtts_dvae import DiscreteVAE import torch import torchaudio # Initialize feature extractors mel_config = { 'sample_rate': 24000, 'n_fft': 1024, 'hop_length': 256, 'win_length': 1024, 'n_mels': 100, 'mel_fmin': 0, 'normalize': False } mel_extractor = MelSpectrogramFeatures(**mel_config) # Load VQ-VAE for discrete code extraction dvae = DiscreteVAE( channels=100, num_tokens=8192, hidden_dim=512, num_resnet_blocks=3, codebook_dim=512 ) dvae.load_state_dict(torch.load('checkpoints/dvae.pth')['model']) dvae.eval() # Process audio file audio, sr = torchaudio.load('input_audio.wav') if sr != 24000: resampler = torchaudio.transforms.Resample(sr, 24000) audio = resampler(audio) # Extract mel spectrogram mel = mel_extractor(audio) print(f"Mel shape: {mel.shape}") # (1, 100, T) # Get discrete codes with torch.no_grad(): codes = dvae.get_codebook_indices(mel) print(f"Codes shape: {codes.shape}") # (1, T//2) # Extract conditioning latent with UnifiedVoice from indextts.gpt.model import UnifiedVoice model = UnifiedVoice(model_dim=1280, heads=20, layers=24) model.load_state_dict(torch.load('checkpoints/gpt.pth')['model']) model.eval() mel_length = torch.tensor([mel.shape[-1]]) with torch.no_grad(): condition = model.get_conditioning(mel, mel_length) print(f"Condition shape: {condition.shape}") # (1, 32, 512) ``` -------------------------------- ### Standard Text-to-Speech Inference in Python Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Perform standard single-pass inference for short to medium text using an audio prompt. Configurable sampling parameters control output quality and diversity including temperature, top-p, top-k, and penalties. Returns path to generated audio file. ```python # Standard inference for short text output_path = tts.infer( audio_prompt="prompts/speaker_sample.wav", text="老宅的钟表停在午夜三点，灰尘中浮现一串陌生脚印。", output_path="output/generated.wav", verbose=True, max_text_tokens_per_sentence=120, do_sample=True, top_p=0.8, top_k=30, temperature=1.0, repetition_penalty=10.0, max_mel_tokens=600 ) print(f"Audio saved to: {output_path}") ``` -------------------------------- ### Extract Audio Codec Features for Multi-Speaker Fine-Tuning Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Extract and process audio features including codec codes, mel-spectrograms, and speaker conditions from training audio files. Generates preprocessed data files and metadata for subsequent LoRA fine-tuning. Creates speaker information JSON for multi-speaker model training. ```bash # Extract audio features and speaker conditions python tools/extract_codec.py \ --audio_list data/audio_list.txt \ --extract_condition \ --output_dir finetune_data/processed_data \ --model_path checkpoints/gpt.pth \ --device cuda ``` -------------------------------- ### Fast Batch Text-to-Speech Inference for Long Text Source: https://context7.com/asr-pub/index-tts-lora/llms.txt Execute optimized batch inference for long-form text content with automatic sentence bucketing and parallel processing. Achieves 2-10x performance improvements over standard inference by intelligently grouping sentences. Ideal for synthesizing multiple paragraphs or documents. ```python # Fast batch inference for long text long_text = """月光下，南瓜突然长出笑脸，藤蔓扭动着推开花园栅栏。 小女孩踮起脚，听见蘑菇在哼唱古老的摇篮曲。 那么Java里面中级还要学，M以及到外部前端的应用系统开发。""" output_path = tts.infer_fast( audio_prompt="prompts/speaker_sample.wav", text=long_text, output_path="output/batch_generated.wav", verbose=True, max_text_tokens_per_sentence=100, sentences_bucket_max_size=4, do_sample=True, top_p=0.8, temperature=1.0, max_mel_tokens=600 ) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.