### Install FullSubNet and Dependencies Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Setup conda environment and install all required dependencies for FullSubNet, including PyTorch with CUDA support, TensorBoard, and other Python packages. Optional ffmpeg installation is also included. ```shell # Clone repository git clone https://github.com/haoxiangsnr/FullSubNet cd FullSubNet # Create conda environment (Python 3.10) conda create --name FullSubNet python=3.10 conda activate FullSubNet # Install PyTorch with CUDA support conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch conda install tensorboard joblib matplotlib -c conda-forge # Install pip packages pip install Cython pip install librosa tbb tensorboard joblib matplotlib pesq pystoi tqdm toml torch_complex rich pip install https://github.com/vBaiCai/python-pesq/archive/master.zip # Optional: ffmpeg for mp3 datasets conda install -c conda-forge ffmpeg ``` -------------------------------- ### Install Pip Packages (Dependencies) Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/prerequisites.md Install Python packages from PyPI, including essential libraries for audio processing, machine learning, and visualization. Cython may need to be installed first. ```shell # you may need to install Cython firstly pip install Cython ``` ```shell pip install librosa tbb tensorboard joblib matplotlib pesq pystoi tqdm toml torch_complex rich ``` ```shell # install pypesq directly from the GitHub repository pip install https://github.com/vBaiCai/python-pesq/archive/master.zip ``` -------------------------------- ### Clone FullSubNet Repository Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/prerequisites.md Clone the FullSubNet repository and navigate into the project directory. This is the first step to get the project files. ```shell git clone https://github.com/haoxiangsnr/FullSubNet cd FullSubNet ``` -------------------------------- ### Install Conda Packages (PyTorch, etc.) Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/prerequisites.md Install core packages like PyTorch, torchvision, torchaudio, and cudatoolkit using conda. This ensures compatibility with CUDA. ```shell conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch ``` ```shell conda install tensorboard joblib matplotlib -c conda-forge ``` -------------------------------- ### Install FFmpeg for MP3 Support Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/prerequisites.md If your dataset contains MP3 audio files, install FFmpeg using conda to enable their processing. ```shell # (Optional) if there are "mp3" format audio files in your dataset, you need to install ffmpeg conda install -c conda-forge ffmpeg ``` -------------------------------- ### Visualize Training Logs with TensorBoard Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/getting_started.md Launch TensorBoard to visualize training logs, including loss curves, spectrogram figures, and audio files. The log directory is specified by the 'save_dir' key in the training configuration. ```shell tensorboard --logdir ~/Experiments/FullSubNet/train ``` -------------------------------- ### Visualize Training Logs with Tensorboard Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Visualize training progress, including loss curves, spectrograms, and audio files, using Tensorboard. Ensure the log directory is correctly specified. ```shell tensorboard --logdir ~/Experiments/FullSubNet/FullSubNet/train ``` -------------------------------- ### Train FullSubNet Model Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Use this command to train the FullSubNet model with a default configuration and two GPUs. Ensure you are in the correct directory. ```shell cd FullSubNet/recipes/dns_interspeech_2020 # Use a default config and two GPUs to train the FullSubNet model CUDA_VISIABLE_DEVICES=0,1 python train.py -C fullsubnet/train.toml -N 2 ``` -------------------------------- ### Resume FullSubNet Training Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Resume a previously interrupted experiment using the '-R' parameter. This command requires the same configuration file and number of GPUs. ```shell # Resume the experiment using "-R" parameter CUDA_VISIABLE_DEVICES=0,1 python train.py -C fullband_baseline/train.toml -N 2 -R ``` -------------------------------- ### FullSubNet Training Configuration Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt TOML configuration file for training FullSubNet. Specifies meta-parameters, acoustics, optimizer, model arguments, and trainer settings including validation and visualization options. ```toml # recipes/dns_interspeech_2020/fullsubnet/train.toml (key sections) [meta] save_dir = "~/Experiments/FullSubNet/" seed = 0 use_amp = true # automatic mixed precision cudnn_enable = false [acoustics] n_fft = 512 win_length = 512 sr = 16000 hop_length = 256 [optimizer] lr = 0.001 beta1 = 0.9 beta2 = 0.999 [model.args] num_freqs = 257 look_ahead = 2 sequence_model = "LSTM" fB_model_hidden_size = 512 sB_model_hidden_size = 384 norm_type = "offline_laplace_norm" num_groups_in_drop_band = 2 [trainer.validation] save_max_metric_score = true validation_interval = 2 # validate every 2 epochs [trainer.visualization] metrics = ["WB_PESQ", "NB_PESQ", "STOI", "SI_SDR"] ``` -------------------------------- ### Run FullSubNet Inference Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/getting_started.md Execute inference using the FullSubNet model. This command requires the path to the inference configuration file, the model checkpoint, and the output directory. ```shell cd FullSubNet/recipes/dns_interspeech_2020 python inference.py \ -C fullsubnet/inference.toml \ -M /path/to/your/checkpoint_dir/best_model.tar \ -O /path/to/your/enhancement/dir ``` -------------------------------- ### Instantiate FullSubNet Model Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Instantiates the FullSubNet model, combining full-band and sub-band sequence models. Configure parameters like number of frequencies, look-ahead frames, sequence model type (LSTM/GRU), neighbor frequencies for context, activation functions, hidden sizes, normalization type, and weight initialization. ```python import torch from recipes.dns_interspeech_2020.fullsubnet.model import Model # Instantiate the FullSubNet model (257 freq bins = n_fft/2 + 1 for n_fft=512) model = Model( num_freqs=257, look_ahead=2, # frames of look-ahead for causal processing sequence_model="LSTM", # "LSTM" or "GRU" fb_num_neighbors=0, # neighbor freqs from fullband output fed to subband sb_num_neighbors=15, # neighbor freqs from noisy spectrogram fed to subband fb_output_activate_function="ReLU", sb_output_activate_function=False, fb_model_hidden_size=512, sb_model_hidden_size=384, norm_type="offline_laplace_norm", # normalization type, see BaseModel.norm_wrapper num_groups_in_drop_band=2, # drop_band groups to reduce training cost weight_init=True, ) # Forward pass: input is noisy magnitude spectrogram [B, 1, F, T] with torch.no_grad(): noisy_mag = torch.rand(1, 1, 257, 63) # batch=1, channels=1, freqs=257, frames=63 output = model(noisy_mag) print(output.shape) # torch.Size([1, 2, 257, 63]) — real & imag parts of cIRM ``` -------------------------------- ### Train Fullband Baseline Model Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Train the Fullband baseline model using a default configuration and a single GPU. Navigate to the specified directory first. ```shell # Use default config and one GPU to train the Fullband baseline model CUDA_VISIABLE_DEVICES=0 python train.py -C fullband_baseline/train.toml -N 1 ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/prerequisites.md Create a new conda environment named 'FullSubNet' with Python 3.10 and activate it. Optionally, remove an existing environment with the same name first. ```shell # (Optional) if you already have an environment with the same name, you may remove it first conda env remove --name FullSubNet # create a new environment conda create --name FullSubNet python=3.10 conda activate FullSubNet ``` -------------------------------- ### Visualize Training Logs on a Specific Port Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Visualize training logs using Tensorboard on a custom port. This is useful if the default port is already in use. ```shell # specify a port tensorboard --logdir ~/Experiments/FullSubNet/FullSubNet/train --port 45454 ``` -------------------------------- ### Multi-GPU Training with torchrun Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Initiate multi-GPU distributed training using torchrun. Configure model, dataset, and training parameters via a TOML file. Supports resuming from checkpoints and validation-only modes. ```shell cd FullSubNet/recipes/dns_interspeech_2020 # Train FullSubNet on 2 GPUs CUDA_VISIBLE_DEVICES=0,1 \ torchrun --standalone --nnodes=1 --nproc_per_node=2 \ train.py -C fullsubnet/train.toml # Train Fullband Baseline on 1 GPU CUDA_VISIBLE_DEVICES=0 \ torchrun --standalone --nnodes=1 --nproc_per_node=1 \ train.py -C fullband_baseline/train.toml # Resume from latest checkpoint CUDA_VISIBLE_DEVICES=0,1 \ torchrun --standalone --nnodes=1 --nproc_per_node=2 \ train.py -C fullsubnet/train.toml -R # Validation-only mode (debug, uses 1 GPU) CUDA_VISIBLE_DEVICES=0 \ torchrun --standalone --nnodes=1 --nproc_per_node=1 \ train.py -C fullsubnet/train.toml -V # Visualize training logs with TensorBoard tensorboard --logdir ~/Experiments/FullSubNet/train --port 6006 ``` -------------------------------- ### Calculate Audio Metrics from Directory Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Use this script to calculate various audio quality metrics between reference and enhanced audio files in specified directories. It supports exporting results to Excel. ```bash python tools/calculate_metrics.py \ -R /path/to/clean/ \ -E /path/to/enhanced/ \ -M WB_PESQ,NB_PESQ,STOI \ -S DNS_2 ``` ```bash python tools/calculate_metrics.py \ -R /path/to/reference/ \ -E /path/to/estimation/ \ -M SI_SDR,STOI,WB_PESQ \ --sr 16000 ``` ```bash python tools/calculate_metrics.py \ -R /path/to/reference/ \ -E /path/to/estimation/ \ -M WB_PESQ,STOI \ -D /path/to/export_dir/ ``` -------------------------------- ### Calculate Performance Metrics Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/getting_started.md Calculate various audio enhancement metrics such as SI_SDR, STOI, and PESQ. Ensure you provide the correct paths to the reference and enhanced audio files. ```shell # Switch path cd FullSubNet # DNS-INTERSPEECH-2020 python tools/calculate_metrics.py \ -R /path/to/reference/ \ -E /path/to/enhancement/ \ -M SI_SDR,STOI,WB_PESQ,NB_PESQ \ -S DNS_1 ``` -------------------------------- ### Configure Inference Dataset Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/getting_started.md This TOML snippet shows how to configure the dataset directory and sample rate for inference. Replace '/path/to/your/dataset_X/' with actual paths. ```toml [dataset.args] dataset_dir_list = [ "/path/to/your/dataset_1/", "/path/to/your/dataset_2/", "others..." ] sr = 16000 ``` -------------------------------- ### Resume Fullband Baseline Training Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/usage/getting_started.md Use this command to resume training for the Fullband baseline model using the '-R' parameter. Ensure you are in the dataset directory. ```shell CUDA_VISIBLE_VIZ=0,1 torchrun --standalone --nnodes=1 --nproc_per_node=2 train.py -C fullband_baseline/train.toml -R ``` -------------------------------- ### STFT and ISTFT for Single and Multi-channel Audio Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Wrappers for torch.stft and torch.istft supporting single-channel [B, T] and multi-channel [B, C, T] inputs. Demonstrates reconstruction from complex spectrum, real/imag tuple, and magnitude/phase tuple. ```python import torch from audio_zen.acoustics.feature import stft, istft # Parameters matching the default FullSubNet config n_fft, hop_length, win_length = 512, 256, 512 # Single-channel: [B, T] waveform = torch.randn(2, 16000) # 2 utterances, 1 second at 16 kHz mag, phase, real, imag = stft(waveform, n_fft, hop_length, win_length) print(mag.shape) # torch.Size([2, 257, 126]) — [B, F, T_frames] # Multi-channel: [B, C, T] mc_waveform = torch.randn(2, 4, 16000) # 2 utterances, 4 channels mc_mag, mc_phase, mc_real, mc_imag = stft(mc_waveform, n_fft, hop_length, win_length) print(mc_mag.shape) # torch.Size([2, 4, 257, 126]) — [B, C, F, T_frames] # Reconstruct from complex spectrum complex_spec = torch.complex(real, imag) # [B, F, T] reconstructed = istft(complex_spec, n_fft, hop_length, win_length, length=16000, input_type="complex") print(reconstructed.shape) # torch.Size([2, 16000]) # Reconstruct from (real, imag) tuple reconstructed_ri = istft((real, imag), n_fft, hop_length, win_length, length=16000, input_type="real_imag") # Reconstruct from (magnitude, phase) tuple reconstructed_mp = istft((mag, phase), n_fft, hop_length, win_length, length=16000, input_type="mag_phase") ``` -------------------------------- ### Calculate DNS-INTERSPEECH-2020 Metrics Source: https://github.com/audio-westlakeu/fullsubnet/wiki/Getting-Started Calculate performance metrics for the DNS-INTERSPEECH-2020 dataset. This command requires paths to the reference and enhanced audio files, and specifies the desired metrics. ```shell # DNS-INTERSPEECH-2020 python tools/calculate_metrics.py \ -R ~/Datasets/DNS-Challenge-INTERSPEECH/datasets/test_set/synthetic/no_reverb/clean \ -E ~/Enhancement/fullsubnet_dns_interspeech_no_reverb/enhanced_1155 \ -M SI_SDR,STOI,WB_PESQ,NB_PESQ \ -S DNS_1 ``` -------------------------------- ### Calculate Speech Enhancement Metrics Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Compute speech quality and intelligibility metrics using a command-line tool. Compares reference (clean) and estimated (enhanced) audio directories. Supports parallel processing and Excel export. ```shell cd FullSubNet # Compute all standard metrics on DNS INTERSPEECH 2020 test set python tools/calculate_metrics.py \ -R /path/to/clean_reference/ \ -E /path/to/enhanced_output/ \ -M SI_SDR,STOI,WB_PESQ,NB_PESQ \ -S DNS_1 ``` -------------------------------- ### Audio Pre/Post-processing Utility Functions Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Utilize these helper functions for common audio processing tasks such as amplitude normalization, dB-FS scaling, clipping detection, aligned subsampling, activity detection, and overlap-add concatenation. ```python import numpy as np import librosa from audio_zen.acoustics.feature import ( norm_amplitude, tailor_dB_FS, is_clipped, aligned_subsample, subsample, overlap_cat, activity_detector ) import torch # Load audio clean, _ = librosa.load("clean.wav", sr=16000) noisy, _ = librosa.load("noisy.wav", sr=16000) # Normalize amplitude to range [-1, 1] clean_normed, scalar = norm_amplitude(clean) print(f"Scalar: {scalar:.4f}") # Scale to a target dB FS level (e.g., -25 dBFS, as used in DNS training) clean_scaled, rms, scale = tailor_dB_FS(clean, target_dB_FS=-25) print(f"RMS before: {rms:.4f}, scale applied: {scale:.4f}") # Check if audio is clipped (any sample > 0.999) clipped = is_clipped(clean_scaled) print(f"Clipped: {clipped}") # False # Take aligned fixed-length subsample from a clean/noisy pair (3.072s @ 16kHz = 49152 samples) sub_clean, sub_noisy = aligned_subsample(clean, noisy, sub_sample_length=49152) print(f"Subsample lengths: {len(sub_clean)}, {len(sub_noisy)}") # 49152, 49152 # Activity detection: percentage of frames above energy threshold activity_ratio = activity_detector(clean, fs=16000, activity_threshold=0.13, target_level=-25) print(f"Activity: {activity_ratio:.2%}") # e.g., Activity: 73.45% # Overlap-add concatenation of enhanced chunks (50% overlap) chunk1 = torch.rand(1, 16000) chunk2 = torch.rand(1, 16000) chunk3 = torch.rand(1, 16000) result = overlap_cat([chunk1, chunk2, chunk3], dim=-1) print(result.shape) # torch.Size([1, 40000]) — 3 chunks with 50% overlap ``` -------------------------------- ### Load PyTorch Checkpoint Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Loads a PyTorch checkpoint file for inspection. Use map_location='cpu' to load on CPU. ```python import torch checkpoint = torch.load( "~/Experiments/FullSubNet/fullsubnet/checkpoints/best_model.tar", map_location="cpu" ) print(f"Best epoch: {checkpoint['epoch']}") print(f"Best score: {checkpoint['best_score']:.4f}") ``` -------------------------------- ### BaseTrainer Checkpoint Management Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt The BaseTrainer manages experiment checkpoints, saving the full state (model, optimizer, etc.) as 'latest_model.tar' and 'best_model.tar', and model weights only as 'model_XXXX.pth'. ```text # Checkpoint directory layout after training: # ~/Experiments/FullSubNet/fullsubnet/ # ├── checkpoints/ # │ ├── latest_model.tar # full state: epoch, best_score, optimizer, scaler, model # │ ├── best_model.tar # full state at best validation score epoch # │ ├── model_0002.pth # model weights only, epoch 2 # │ ├── model_0004.pth # model weights only, epoch 4 # │ ├── ... # ├── logs/ # TensorBoard event files # │ ├── train_validation_WB_PESQ/ # noisy vs enhanced PESQ curves ``` -------------------------------- ### Initialize and Use Sub-band Sequence Model Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Defines a SequenceModel for sub-band processing with specific input and output sizes. Demonstrates forward pass with random data. ```python sb_model = SequenceModel( input_size=31, # sb_num_neighbors=15 -> 15*2+1=31; fb_num_neighbors=0 -> 1 output_size=2, # predict real & imag components of cIRM hidden_size=384, num_layers=2, bidirectional=False, sequence_model="LSTM", output_activate_function=False, ) with torch.no_grad(): x = torch.rand(4, 257, 100) # [B=4, F=257, T=100] out = fb_model(x) print(out.shape) # torch.Size([4, 257, 100]) sb_x = torch.rand(4 * 257, 31, 100) # B*F sub-band units per forward pass sb_out = sb_model(sb_x) print(sb_out.shape) # torch.Size([1028, 2, 100]) ``` -------------------------------- ### Calculate Speech Quality Metrics using Metrics API Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Directly call speech quality metric functions from the audio_zen.metrics module. Ensure numpy arrays are used and specify the sample rate (sr) when necessary. ```python import numpy as np import librosa from audio_zen.metrics import SI_SDR, STOI, WB_PESQ, NB_PESQ, REGISTERED_METRICS # Load a clean reference and an enhanced estimate ref, _ = librosa.load("/path/to/clean.wav", sr=16000) est, _ = librosa.load("/path/to/enhanced.wav", sr=16000) # Scale-Invariant Signal-to-Distortion Ratio (higher is better, in dB) si_sdr = SI_SDR(ref, est) print(f"SI-SDR: {si_sdr:.2f} dB") # e.g., SI-SDR: 18.34 dB # Short-Time Objective Intelligibility (range 0-1, higher is better) stoi_score = STOI(ref, est, sr=16000) print(f"STOI: {stoi_score:.4f}") # e.g., STOI: 0.9213 # Wideband PESQ (range -0.5 to 4.5, higher is better) wb_pesq = WB_PESQ(ref, est, sr=16000) print(f"WB-PESQ: {wb_pesq:.4f}") # e.g., WB-PESQ: 3.1420 # Narrowband PESQ (range 1 to 4.5, higher is better) nb_pesq = NB_PESQ(ref, est, sr=16000) print(f"NB-PESQ: {nb_pesq:.4f}") # e.g., NB-PESQ: 3.5012 # Using the registry (e.g., in evaluation loops) for metric_name, metric_fn in REGISTERED_METRICS.items(): score = metric_fn(ref, est) print(f"{metric_name}: {score:.4f}") ``` -------------------------------- ### FullSubNet Inference Configuration Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt TOML configuration for speech enhancement inference. Defines acoustics, inferencer settings, dataset paths, and model arguments for the enhancement process. ```toml # recipes/dns_interspeech_2020/fullsubnet/inference.toml [acoustics] sr = 16000 n_fft = 512 win_length = 512 hop_length = 256 [inferencer] path = "inferencer.Inferencer" type = "full_band_crm_mask" # inference method name in the Inferencer class [inferencer.args] n_neighbor = 15 [dataset.args] dataset_dir_list = [ "/path/to/your/noisy_speech_dir/" ] sr = 16000 [model.args] num_freqs = 257 look_ahead = 2 sequence_model = "LSTM" fb_num_neighbors = 0 sb_num_neighbors = 15 fb_model_hidden_size = 512 sb_model_hidden_size = 384 norm_type = "offline_laplace_norm" ``` -------------------------------- ### Audio Utility Functions Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Helper functions for audio pre/post-processing, including amplitude normalization, dB-FS level adjustment, activity detection, aligned sub-sampling, and overlap-add concatenation of enhanced chunks. ```APIDOC ## Audio Utility Functions (`audio_zen/acoustics/feature.py`) Helper functions for audio pre/post-processing used in dataset preparation and training: amplitude normalization, dB-FS level adjustment, activity detection, aligned sub-sampling for paired clean/noisy clips, and overlap-add concatenation of enhanced chunks. ```python import numpy as np import librosa from audio_zen.acoustics.feature import ( norm_amplitude, tailor_dB_FS, is_clipped, aligned_subsample, subsample, overlap_cat, activity_detector ) import torch # Load audio clean, _ = librosa.load("clean.wav", sr=16000) noisy, _ = librosa.load("noisy.wav", sr=16000) # Normalize amplitude to range [-1, 1] clean_normed, scalar = norm_amplitude(clean) print(f"Scalar: {scalar:.4f}") # Scale to a target dB FS level (e.g., -25 dBFS, as used in DNS training) clean_scaled, rms, scale = tailor_dB_FS(clean, target_dB_FS=-25) print(f"RMS before: {rms:.4f}, scale applied: {scale:.4f}") # Check if audio is clipped (any sample > 0.999) clipped = is_clipped(clean_scaled) print(f"Clipped: {clipped}") # False # Take aligned fixed-length subsample from a clean/noisy pair (3.072s @ 16kHz = 49152 samples) sub_clean, sub_noisy = aligned_subsample(clean, noisy, sub_sample_length=49152) print(f"Subsample lengths: {len(sub_clean)}, {len(sub_noisy)}") # 49152, 49152 # Activity detection: percentage of frames above energy threshold activity_ratio = activity_detector(clean, fs=16000, activity_threshold=0.13, target_level=-25) print(f"Activity: {activity_ratio:.2%}") # e.g., Activity: 73.45% # Overlap-add concatenation of enhanced chunks (50% overlap) chunk1 = torch.rand(1, 16000) chunk2 = torch.rand(1, 16000) chunk3 = torch.rand(1, 16000) result = overlap_cat([chunk1, chunk2, chunk3], dim=-1) print(result.shape) # torch.Size([1, 40000]) — 3 chunks with 50% overlap ``` ``` -------------------------------- ### BaseModel Spectral Normalization Methods Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Provides static spectral normalization methods for speech enhancement, including offline Laplace, online cumulative Laplace, forgetting normalization, and cumulative layer normalization. Demonstrates usage with a 4D spectrogram tensor. ```python import torch from audio_zen.model.base_model import BaseModel spec = torch.rand(2, 1, 257, 100) # [B=2, C=1, F=257, T=100] # Offline utterance-level Laplace normalization (used in default FullSubNet config) normed_offline = BaseModel.offline_laplace_norm(spec) print(normed_offline.shape) # torch.Size([2, 1, 257, 100]) # Online cumulative Laplace normalization (causal, suitable for streaming inference) normed_cumulative = BaseModel.cumulative_laplace_norm(spec) # Online forgetting normalization (exponential moving average over time) normed_forgetting = BaseModel.forgetting_norm(spec, sample_length=192) # Online cumulative layer normalization (zero-mean, unit-variance, causal) normed_cln = BaseModel.cumulative_layer_norm(spec) # Offline Gaussian (z-score) normalization normed_gaussian = BaseModel.offline_gaussian_norm(spec) # Use norm_wrapper inside a custom model class: class MyModel(BaseModel): def __init__(self, norm_type="cumulative_laplace_norm"): super().__init__() self.norm = self.norm_wrapper(norm_type) # returns the selected static method def forward(self, x): return self.norm(x) ``` -------------------------------- ### Metrics API Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Provides functions to calculate speech quality and intelligibility metrics. These can be called directly or used with BaseTrainer for TensorBoard logging. All functions accept numpy arrays of shape [..., T]. ```APIDOC ## Metrics API (`audio_zen/metrics.py`) Four speech quality and intelligibility metric functions registered in `REGISTERED_METRICS` dict. Can be called directly or used via `BaseTrainer.metrics_visualization()` for TensorBoard logging during validation. All functions take numpy arrays of shape `[..., T]`. ```python import numpy as np import librosa from audio_zen.metrics import SI_SDR, STOI, WB_PESQ, NB_PESQ, REGISTERED_METRICS # Load a clean reference and an enhanced estimate ref, _ = librosa.load("/path/to/clean.wav", sr=16000) est, _ = librosa.load("/path/to/enhanced.wav", sr=16000) # Scale-Invariant Signal-to-Distortion Ratio (higher is better, in dB) si_sdr = SI_SDR(ref, est) print(f"SI-SDR: {si_sdr:.2f} dB") # e.g., SI-SDR: 18.34 dB # Short-Time Objective Intelligibility (range 0-1, higher is better) stoi_score = STOI(ref, est, sr=16000) print(f"STOI: {stoi_score:.4f}") # e.g., STOI: 0.9213 # Wideband PESQ (range -0.5 to 4.5, higher is better) wb_pesq = WB_PESQ(ref, est, sr=16000) print(f"WB-PESQ: {wb_pesq:.4f}") # e.g., WB-PESQ: 3.1420 # Narrowband PESQ (range 1 to 4.5, higher is better) nb_pesq = NB_PESQ(ref, est, sr=16000) print(f"NB-PESQ: {nb_pesq:.4f}") # e.g., NB-PESQ: 3.5012 # Using the registry (e.g., in evaluation loops) for metric_name, metric_fn in REGISTERED_METRICS.items(): score = metric_fn(ref, est) print(f"{metric_name}: {score:.4f}") ``` ``` -------------------------------- ### stft / istft Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Thin wrappers around torch.stft and torch.istft supporting both single-channel [B, T] and multi-channel [B, C, T] inputs. `stft` returns magnitude, phase, real, and imaginary components. `istft` accepts complex, real/imag tuple, or mag/phase tuple inputs via the `input_type` argument. ```APIDOC ## stft / istft (`audio_zen/acoustics/feature.py`) Thin wrappers around `torch.stft` and `torch.istft` supporting both single-channel `[B, T]` and multi-channel `[B, C, T]` inputs. `stft` returns magnitude, phase, real, and imaginary components. `istft` accepts complex, real/imag tuple, or mag/phase tuple inputs via the `input_type` argument. ```python import torch from audio_zen.acoustics.feature import stft, istft # Parameters matching the default FullSubNet config n_fft, hop_length, win_length = 512, 256, 512 # Single-channel: [B, T] waveform = torch.randn(2, 16000) # 2 utterances, 1 second at 16 kHz mag, phase, real, imag = stft(waveform, n_fft, hop_length, win_length) print(mag.shape) # torch.Size([2, 257, 126]) — [B, F, T_frames] # Multi-channel: [B, C, T] mc_waveform = torch.randn(2, 4, 16000) # 2 utterances, 4 channels mc_mag, mc_phase, mc_real, mc_imag = stft(mc_waveform, n_fft, hop_length, win_length) print(mc_mag.shape) # torch.Size([2, 4, 257, 126]) — [B, C, F, T_frames] # Reconstruct from complex spectrum complex_spec = torch.complex(real, imag) # [B, F, T] reconstructed = istft(complex_spec, n_fft, hop_length, win_length, length=16000, input_type="complex") print(reconstructed.shape) # torch.Size([2, 16000]) # Reconstruct from (real, imag) tuple reconstructed_ri = istft((real, imag), n_fft, hop_length, win_length, length=16000, input_type="real_imag") # Reconstruct from (magnitude, phase) tuple reconstructed_mp = istft((mag, phase), n_fft, hop_length, win_length, length=16000, input_type="mag_phase") ``` ``` -------------------------------- ### Speech Enhancement Inference Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Run speech enhancement on WAV files using a trained checkpoint. Output enhanced files are saved to a timestamped subdirectory. Configure dataset paths and sample rate in the inference TOML. ```shell cd FullSubNet/recipes/dns_interspeech_2020 # Enhance noisy speech using a trained checkpoint python inference.py \ -C fullsubnet/inference.toml \ -M ~/Experiments/FullSubNet/fullsubnet/checkpoints/best_model.tar \ -O ~/Experiments/FullSubNet/enhanced_output/ # Use a pre-trained model downloaded from the Releases page python inference.py \ -C fullsubnet/inference.toml \ -M /path/to/downloaded/fullsubnet_best_model_58epochs.tar \ -O ~/output/enhanced/ # Output: ~/output/enhanced/enhanced_0058/ (enhanced wavs) # ~/output/enhanced/noisy/ (copy of noisy wavs) ``` -------------------------------- ### Angular Commit Style Format Source: https://github.com/audio-westlakeu/fullsubnet/blob/main/docs/source/reference/contributing.md Use this format for commit messages to ensure consistency and enable automated version bumping. Specify the type and a short summary at a minimum. ```shell (optional scope): short summary in present tense (optional body: explains motivation for the change) (optional footer: note BREAKING CHANGES here, and issues to be closed) ``` -------------------------------- ### SequenceModel Wrapper for LSTM/GRU Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt A reusable SequenceModel wrapper for PyTorch LSTM/GRU. It standardizes the tensor convention to `[B, F, T]` (frequency as feature dim, time as sequence length), applies an optional linear projection, and supports various output activation functions. This is used as the backbone for both full-band and sub-band models in FullSubNet. ```python import torch from audio_zen.model.module.sequence_model import SequenceModel # Full-band sequence model: processes all 257 frequencies at once as a feature vector fb_model = SequenceModel( input_size=257, output_size=257, hidden_size=512, num_layers=2, bidirectional=False, sequence_model="LSTM", output_activate_function="ReLU", ) ``` -------------------------------- ### BaseModel Normalization Methods Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt BaseModel provides static spectral normalization methods for real-time and offline speech enhancement. These methods operate on 4D tensors [B, C, F, T] and can be selected via `norm_wrapper(norm_type)`. ```APIDOC ## BaseModel Normalization Methods (`audio_zen/model/base_model.py`) `BaseModel` provides a suite of static spectral normalization methods suitable for real-time (causal/online) and offline speech enhancement. Selected via `norm_wrapper(norm_type)` which returns the appropriate callable. Normalization operates on 4D tensors `[B, C, F, T]`. ```python import torch from audio_zen.model.base_model import BaseModel spec = torch.rand(2, 1, 257, 100) # [B=2, C=1, F=257, T=100] # Offline utterance-level Laplace normalization (used in default FullSubNet config) normed_offline = BaseModel.offline_laplace_norm(spec) print(normed_offline.shape) # torch.Size([2, 1, 257, 100]) # Online cumulative Laplace normalization (causal, suitable for streaming inference) normed_cumulative = BaseModel.cumulative_laplace_norm(spec) # Online forgetting normalization (exponential moving average over time) normed_forgetting = BaseModel.forgetting_norm(spec, sample_length=192) # Online cumulative layer normalization (zero-mean, unit-variance, causal) normed_cln = BaseModel.cumulative_layer_norm(spec) # Offline Gaussian (z-score) normalization normed_gaussian = BaseModel.offline_gaussian_norm(spec) # Use norm_wrapper inside a custom model class: class MyModel(BaseModel): def __init__(self, norm_type="cumulative_laplace_norm"): super().__init__() self.norm = self.norm_wrapper(norm_type) # returns the selected static method def forward(self, x): return self.norm(x) ``` ``` -------------------------------- ### Drop Band Frequencies in BaseModel Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Use BaseModel.drop_band to reduce computational cost during training by distributing frequency subsets across batch samples. Each group processes a fraction of frequencies, enabling larger effective batch sizes. ```python import torch from audio_zen.model.base_model import BaseModel # Sub-band input after freq_unfold and permute: [B, F_features, F, T] sb_input = torch.rand(12, 31, 256, 100) # batch=12, feature_dim=31, freqs=256, time=100 # Drop band with num_groups=2: each sample sees half the frequencies output = BaseModel.drop_band(sb_input, num_groups=2) print(output.shape) # torch.Size([12, 31, 128, 100]) — F reduced from 256 to 128, batch unchanged # With num_groups=3: output3 = BaseModel.drop_band(sb_input, num_groups=3) print(output3.shape) # torch.Size([12, 31, 85, 100]) — floor(256/3) = 85 frequencies per group ``` -------------------------------- ### BaseModel.freq_unfold for Sub-band Processing Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Splits a 4D spectrogram into overlapping sub-band units along the frequency axis. Demonstrates usage with different numbers of neighbors for sub-band windowing. ```python import torch from audio_zen.model.base_model import BaseModel # Spectrogram: [B=2, C=1, F=257, T=100] spec = torch.rand(2, 1, 257, 100) # Extract sub-band units with 15 neighbors on each side (sub-band window = 31 freqs) sub_bands = BaseModel.freq_unfold(spec, num_neighbors=15) print(sub_bands.shape) # torch.Size([2, 257, 1, 31, 100]) # [B, N=num_freqs, C, F_subband=31, T] # With 0 neighbors: just a permute/reshape, each frequency independently sub_bands_single = BaseModel.freq_unfold(spec, num_neighbors=0) print(sub_bands_single.shape) # torch.Size([2, 257, 1, 1, 100]) ``` -------------------------------- ### BaseModel.freq_unfold Source: https://context7.com/audio-westlakeu/fullsubnet/llms.txt Splits a 4D spectrogram into overlapping sub-band units along the frequency axis using `F.unfold`. Each output unit contains one center frequency plus `num_neighbors` frequencies on each side (with reflect padding at boundaries). This is the key operation that enables per-frequency sub-band processing. ```APIDOC ## BaseModel.freq_unfold (`audio_zen/model/base_model.py`) Splits a 4D spectrogram into overlapping sub-band units along the frequency axis using `F.unfold`. Each output unit contains one center frequency plus `num_neighbors` frequencies on each side (with reflect padding at boundaries). This is the key operation that enables per-frequency sub-band processing. ```python import torch from audio_zen.model.base_model import BaseModel # Spectrogram: [B=2, C=1, F=257, T=100] spec = torch.rand(2, 1, 257, 100) # Extract sub-band units with 15 neighbors on each side (sub-band window = 31 freqs) sub_bands = BaseModel.freq_unfold(spec, num_neighbors=15) print(sub_bands.shape) # torch.Size([2, 257, 1, 31, 100]) # [B, N=num_freqs, C, F_subband=31, T] # With 0 neighbors: just a permute/reshape, each frequency independently sub_bands_single = BaseModel.freq_unfold(spec, num_neighbors=0) print(sub_bands_single.shape) # torch.Size([2, 257, 1, 1, 100]) ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.