### Install nnAudio Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Install nnAudio using pip. You can install a specific version or from the source repository. ```bash pip install nnaudio==0.3.4 # or from source pip install git+https://github.com/KinWaiCheuk/nnAudio.git#subdirectory=Installation ``` -------------------------------- ### Install nnAudio from GitHub Source: https://github.com/kinwaicheuk/nnaudio/blob/master/README.md Use this command to install the latest version of nnAudio directly from its GitHub repository. ```bash pip install git+https://github.com/KinWaiCheuk/nnAudio.git#subdirectory=Installation ``` -------------------------------- ### Install nnAudio specific version Source: https://github.com/kinwaicheuk/nnaudio/blob/master/README.md Install a specific version of nnAudio, such as 0.3.4, using pip. ```bash pip install nnaudio==0.3.4 ``` -------------------------------- ### End-to-End GPU Pipeline with Custom nn.Module Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt An example of an end-to-end model pipeline using nnAudio layers within a custom `nn.Module` subclass. This demonstrates loading audio, computing spectrograms on the GPU, and passing them through pooling and a linear layer for classification. ```python import torch.nn as nn from nnAudio import features # Assuming device and x are already defined as in the previous snippet # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # x = torch.tensor(np.random.randn(16, 22050), dtype=torch.float32).to(device) class E2EModel(nn.Module): def __init__(self): super().__init__() self.spec = features.CQT(sr=22050, hop_length=512, trainable=True, verbose=False) self.pool = nn.AdaptiveAvgPool1d(1) self.head = nn.Linear(84, 10) def forward(self, raw_audio): cqt_spec = self.spec(raw_audio) # (B, 84, T) on same device pooled = self.pool(cqt_spec).squeeze(-1) # (B, 84) return self.head(pooled) # (B, 10) model = E2EModel().to(device) out = model(x) print(f"Model output shape: {out.shape}, device: {out.device}") ``` -------------------------------- ### GPU Acceleration and DataParallel with nnAudio Layers Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Move nnAudio layers to GPU using `.to(device)` and wrap them in `nn.DataParallel` for multi-GPU training. This example shows basic GPU usage and DataParallel integration. ```python import torch import numpy as np from nnAudio import features device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') x = torch.tensor(np.random.randn(16, 22050), dtype=torch.float32).to(device) # Move layer to GPU mel = features.MelSpectrogram(sr=22050, n_mels=128, verbose=False).to(device) spec = mel(x) # computation happens on GPU # spec.device => cuda:0 # Multi-GPU with DataParallel if torch.cuda.device_count() > 1: mel_parallel = torch.nn.DataParallel(mel) spec_parallel = mel_parallel(x) ``` -------------------------------- ### Griffin_Lim Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Implements the Griffin-Lim algorithm for phase reconstruction from magnitude spectrograms, with examples for different iteration counts and quality estimation. ```APIDOC ## Griffin_Lim ### Description Implements the fast Griffin-Lim algorithm to reconstruct waveforms from magnitude-only spectrograms. Uses iterative STFT/iSTFT cycles with a momentum parameter to speed up convergence. Requires that `n_fft`, `hop_length`, `win_length`, `window`, and `center` match the STFT used to produce the input magnitude spectrogram. ### Usage ```python import torch import numpy as np from nnAudio import features sr = 22050 device = 'cpu' x = torch.tensor(np.random.randn(3, sr), dtype=torch.float32) # 1. Create magnitude spectrogram via STFT n_fft = 2048 hop_length = 512 stft_layer = features.STFT(n_fft=n_fft, hop_length=hop_length, output_format='Magnitude', verbose=False) mag_spec = stft_layer(x) # mag_spec.shape => (3, 1025, time_steps) # 2. Reconstruct waveform with Griffin-Lim gl = features.Griffin_Lim(n_fft=n_fft, n_iter=64, hop_length=hop_length, window='hann', center=True, momentum=0.99, device=device) recon = gl(mag_spec) # recon.shape => (3, ~22050) # Faster but less accurate (fewer iterations) gl_fast = features.Griffin_Lim(n_fft=n_fft, n_iter=16, hop_length=hop_length, device=device) recon_fast = gl_fast(mag_spec) # Quality estimate via signal-to-noise ratio mse = torch.mean((x - recon[:, :sr])**2).item() signal_power = torch.mean(x**2).item() snr_db = 10 * np.log10(signal_power / (mse + 1e-10)) print(f"Reconstruction SNR: {snr_db:.1f} dB (n_iter=64)") ``` ``` -------------------------------- ### Generate Spectrograms with nnAudio STFT Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/index.md Use nnAudio's STFT layer to compute spectrograms from audio waveforms. Ensure PyTorch and scipy are installed, and the audio file is loaded and preprocessed into a CUDA-enabled float tensor. ```python from nnAudio import features from scipy.io import wavfile import torch sr, song = wavfile.read('./Bach.wav') # Loading your audio x = song.mean(1) # Converting Stereo to Mono x = torch.tensor(x, device='cuda:0').float() # casting the array into a PyTorch Tensor spec_layer = features.STFT(n_fft=2048, freq_bins=None, hop_length=512, window='hann', freq_scale='linear', center=True, pad_mode='reflect', fmin=50,fmax=11025, sr=sr) # Initializing the model spec = spec_layer(x) # Feed-forward your waveform to get the spectrogram ``` -------------------------------- ### Import nnAudio Spectrogram Modules (Legacy and Preferred) Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Shows how to import audio feature extraction classes from nnAudio. Demonstrates both the legacy `nnAudio.Spectrogram` namespace and the current preferred `nnAudio.features` path. ```python import torch import numpy as np # Legacy import (still supported) from nnAudio import Spectrogram sr = 22050 x = torch.tensor(np.random.randn(2, sr), dtype=torch.float32) stft = Spectrogram.STFT(n_fft=2048, verbose=False) mel = Spectrogram.MelSpectrogram(sr=sr, verbose=False) mfcc = Spectrogram.MFCC(sr=sr, verbose=False) cqt = Spectrogram.CQT(sr=sr, verbose=False) cqt2 = Spectrogram.CQT2010v2(sr=sr, verbose=False) vqt = Spectrogram.VQT(sr=sr, verbose=False) gt = Spectrogram.Gammatonegram(sr=sr, verbose=False) # New preferred import from nnAudio import features as F stft_new = F.STFT(n_fft=2048, verbose=False) ``` -------------------------------- ### Configure Gammatonegram with 128 Bins and Trainable Parameters Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Demonstrates creating a Gammatonegram with a higher resolution (128 bins) and a trainable configuration for both bins and STFT parameters. Requires PyTorch for training. ```python import torch from nnAudio import features sr = 22050 x = torch.randn(2, sr) # Higher resolution with 128 bins gammatone_hi = features.Gammatonegram(sr=sr, n_fft=2048, hop_length=256, n_bins=128, fmin=50.0, fmax=8000, power=1.0, verbose=False) gtgram_hi = gammatone_hi(x) # Trainable Gammatonegram gammatone_train = features.Gammatonegram(sr=sr, n_bins=64, trainable_bins=True, trainable_STFT=True, verbose=False) optimizer = torch.optim.Adam(gammatone_train.parameters(), lr=1e-4) loss = gammatone_train(x).mean() loss.backward() print(f"Gammatonegram shape: {gtgram.shape}") ``` -------------------------------- ### Trainable CQT for learning pitch-related kernels Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Initializes a Constant-Q Transform module that can be trained to learn pitch-related kernels. Requires setting up an optimizer and performing backpropagation. ```python cqt_trainable = features.CQT(sr=sr, trainable=True, output_format='Magnitude', verbose=False) optimizer = torch.optim.Adam(cqt_trainable.parameters(), lr=1e-4) loss = cqt_trainable(x).mean() loss.backward() optimizer.step() ``` -------------------------------- ### Run nnAudio Unit Tests Source: https://github.com/kinwaicheuk/nnaudio/blob/master/README.md Instructions for running the unit tests for nnAudio. Ensure you have sufficient GPU memory (at least 1931 MiB) before running. ```bash cd Installation pytest ``` -------------------------------- ### Trainable VQT module Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Initializes a Variable-Q Transform (VQT) module that is set up for training, allowing its parameters to be learned. Requires defining an optimizer and performing a backward pass. ```python vqt_train = features.VQT(sr=sr, trainable=True, verbose=False) optimizer = torch.optim.Adam(vqt_train.parameters(), lr=1e-4) loss = vqt_train(x).mean() loss.backward() print(f"VQT output shape: {spec_cqt.shape}") ``` -------------------------------- ### VQT equivalent to CQT with gamma=0 Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Initializes a Variable-Q Transform (VQT) with gamma set to 0, making it functionally equivalent to a standard Constant-Q Transform. Useful for comparing VQT behavior to CQT. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) vqt_cqt = features.VQT(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=12, gamma=0, verbose=False) spec_cqt = vqt_cqt(x) # spec_cqt.shape => (4, 84, time_steps) ``` -------------------------------- ### Legacy Spectrogram Module Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Illustrates how to use the legacy `nnAudio.Spectrogram` module for backward compatibility, alongside the preferred `nnAudio.features` import path. ```APIDOC ## Legacy `nnAudio.Spectrogram` Module ### Description Prior to version 0.3.0, all classes lived under `nnAudio.Spectrogram`. This namespace still works for backward compatibility. From 0.3.0 onward, `nnAudio.features` is the preferred import path. ### Usage ```python import torch import numpy as np # Legacy import (still supported) from nnAudio import Spectrogram sr = 22050 x = torch.tensor(np.random.randn(2, sr), dtype=torch.float32) stft = Spectrogram.STFT(n_fft=2048, verbose=False) mel = Spectrogram.MelSpectrogram(sr=sr, verbose=False) mfcc = Spectrogram.MFCC(sr=sr, verbose=False) cqt = Spectrogram.CQT(sr=sr, verbose=False) cqt2 = Spectrogram.CQT2010v2(sr=sr, verbose=False) vqt = Spectrogram.VQT(sr=sr, verbose=False) gt = Spectrogram.Gammatonegram(sr=sr, verbose=False) # New preferred import from nnAudio import features as F stft_new = F.STFT(n_fft=2048, verbose=False) ``` ``` -------------------------------- ### CQT2010v2: Efficient CQT with early downsampling Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Implements an efficient Constant-Q Transform using octave-wise resampling, suitable for low frequencies. Enables early downsampling for further speedup and supports normalized output. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr * 2), dtype=torch.float32) cqt2010 = features.CQT2010v2(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=12, earlydownsample=True, norm=True, verbose=False) spec = cqt2010(x) # spec.shape => (4, 84, time_steps) ``` -------------------------------- ### Trainable iSTFT Kernels and Window Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Instantiates a trainable Inverse Short-Time Fourier Transform (iSTFT) layer. Both the STFT kernels and the window function can be trained. ```python istft_trainable = features.iSTFT(n_fft=1024, hop_length=256, trainable_kernels=True, trainable_window=True, verbose=False) print(list(istft_trainable.parameters())) # kernel_sin, kernel_cos, window_mask ``` -------------------------------- ### VQT with perceptually motivated time-frequency trade-off Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Configures a Variable-Q Transform (VQT) with a non-zero gamma value to achieve a perceptually motivated trade-off between time and frequency resolution, particularly for low-frequency bins. ```python vqt_perceptual = features.VQT(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=36, gamma=20, verbose=False) spec_perceptual = vqt_perceptual(x) ``` -------------------------------- ### High-resolution CQT with 36 bins per octave Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Generates a Constant-Q Transform with high frequency resolution. Requires specifying the number of bins per octave. ```python cqt_hires = features.CQT(sr=sr, hop_length=512, fmin=32.70, n_bins=252, bins_per_octave=36, verbose=False) cqt_hires_out = cqt_hires(x) ``` -------------------------------- ### Move nnAudio-integrated Model to GPU Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/intro.md If nnAudio's features are part of a larger PyTorch model, move the entire model to the target device using `.to(device)`. This ensures all components, including the STFT layer, are on the GPU. ```python net = Model() net.to(device) ``` -------------------------------- ### nnAudio.features.CQT2010v2 Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt An alternative CQT implementation based on the Schörkhuber 2010 resampling algorithm. It is memory-efficient for low-frequency bins and supports early downsampling for speedup. ```APIDOC ## CQT2010v2 ### Description An alternative CQT implementation based on the Schörkhuber 2010 resampling algorithm. Instead of one large kernel spanning the full spectrum, it uses a small top-octave kernel and iteratively downsamples the signal by 2× per octave. This is more memory-efficient for low-frequency bins (below ~40 Hz) and supports early downsampling (`earlydownsample=True`) for additional speedup. Shares the same forward interface and output formats as `CQT1992v2`. ### Parameters - `sr` (int): Sample rate of the input audio. - `hop_length` (int): The number of samples between successive frames. - `fmin` (float): The lowest frequency of the CQT kernel. - `n_bins` (int): The total number of frequency bins. - `bins_per_octave` (int): The number of bins per octave. - `earlydownsample` (bool): If True, enables early downsampling for speedup. - `norm` (int or None): Normalization factor. If 1, uses L1 norm. - `output_format` (str): The format of the output. Can be 'Magnitude', 'Complex', or 'Spectrogram'. - `verbose` (bool): If True, prints verbose output. ### Usage Examples #### Efficient CQT2010v2 ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr * 2), dtype=torch.float32) cqt2010 = features.CQT2010v2(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=12, earlydownsample=True, norm=True, verbose=False) spec = cqt2010(x) # spec.shape => (4, 84, time_steps) ``` #### Convolutional normalization ```python spec_conv = cqt2010(x, normalization_type='convolutional') ``` #### Complex output ```python cqt2010_cplx = features.CQT2010v2(sr=sr, output_format='Complex', verbose=False) spec_cplx = cqt2010_cplx(x) # spec_cplx.shape => (4, 84, time_steps, 2) print(f"CQT2010v2 magnitude shape: {spec.shape}") ``` ``` -------------------------------- ### Reconstruct Waveform from Magnitude Spectrogram using Griffin-Lim Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Implements the Griffin-Lim algorithm to reconstruct waveforms from magnitude-only spectrograms. Ensure STFT parameters match the input spectrogram. The number of iterations affects accuracy and speed. ```python import torch import numpy as np from nnAudio import features sr = 22050 device = 'cpu' x = torch.tensor(np.random.randn(3, sr), dtype=torch.float32) # 1. Create magnitude spectrogram via STFT n_fft = 2048 hop_length = 512 stft_layer = features.STFT(n_fft=n_fft, hop_length=hop_length, output_format='Magnitude', verbose=False) mag_spec = stft_layer(x) # mag_spec.shape => (3, 1025, time_steps) # 2. Reconstruct waveform with Griffin-Lim gl = features.Griffin_Lim(n_fft=n_fft, n_iter=64, hop_length=hop_length, window='hann', center=True, momentum=0.99, device=device) recon = gl(mag_spec) # recon.shape => (3, ~22050) # Faster but less accurate (fewer iterations) gl_fast = features.Griffin_Lim(n_fft=n_fft, n_iter=16, hop_length=hop_length, device=device) recon_fast = gl_fast(mag_spec) # Quality estimate via signal-to-noise ratio MSE = torch.mean((x - recon[:, :sr])**2).item() signal_power = torch.mean(x**2).item() snr_db = 10 * np.log10(signal_power / (mse + 1e-10)) print(f"Reconstruction SNR: {snr_db:.1f} dB (n_iter=64)") ``` -------------------------------- ### Custom Class Documentation Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/_templates/custom-class-template.rst This section details the members and methods of the custom class, including special methods like __call__, __add__, and __mul__. It also lists any methods inherited by the class. ```APIDOC ## Class: {{ fullname }} .. currentmodule:: {{ module }} .. autoclass:: {{ objname }} :members: :show-inheritance: :no-inherited-members: :special-members: __call__, __add__, __mul__ {% block methods %} {% if methods %} .. rubric:: Methods .. autosummary:: :nosignatures: {% for item in methods %} {%- if item not in inherited_members %} ~{{ name }}.{{ item }} {%- endif -%} {%- endfor %} {% endif %} {% endblock %} ``` -------------------------------- ### STFT to iSTFT Round-trip Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Demonstrates a full STFT to iSTFT conversion cycle. Ensure the `iSTFT=True` parameter is set during STFT instantiation to enable the `inverse()` method. ```python # --- Round-trip: STFT → iSTFT --- reconstructed = stft_cplx.inverse(spec_complex, onesided=True, length=sr) # reconstructed.shape => (4, 22050) print(f"Reconstruction error: {torch.mean((x - reconstructed)**2).item():.6f}") ``` -------------------------------- ### Standard Constant-Q Transform (CQT) Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes a standard Constant-Q Transform (CQT) using the recommended `CQT1992v2` implementation. This version is efficient and directly uses `conv1d`. The output format is magnitude by default. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) # Standard CQT: 84 bins (C1 to B7), 12 bins per octave cqt = features.CQT(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=12, verbose=False) cqt_mag = cqt(x) # cqt_mag.shape => (4, 84, time_steps) ``` -------------------------------- ### Complex CQT output with librosa-compatible normalization Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes a Complex Constant-Q Transform, suitable for phase analysis, with normalization compatible with librosa. The output includes a channel for phase information. ```python cqt_cplx = features.CQT(sr=sr, hop_length=256, output_format='Complex', verbose=False) cqt_complex = cqt_cplx(x, normalization_type='librosa') # cqt_complex.shape => (4, 84, time_steps, 2) ``` -------------------------------- ### nnAudio.features.CQT Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Provides a Constant-Q Transform (CQT) implementation. It supports various configurations including high-resolution bins, complex output, trainable kernels, and frequency range specification by fmax. ```APIDOC ## CQT ### Description Provides a Constant-Q Transform (CQT) implementation. It supports various configurations including high-resolution bins, complex output, trainable kernels, and frequency range specification by fmax. ### Parameters - `sr` (int): Sample rate of the input audio. - `hop_length` (int): The number of samples between successive frames. - `fmin` (float): The lowest frequency of the CQT kernel. - `n_bins` (int): The total number of frequency bins. - `bins_per_octave` (int): The number of bins per octave. - `output_format` (str): The format of the output. Can be 'Magnitude', 'Complex', or 'Spectrogram'. - `trainable` (bool): If True, the CQT kernels are trainable. - `fmax` (float): The highest frequency of the CQT kernel. If specified, `n_bins` is ignored. - `normalization_type` (str): Type of normalization to apply. E.g., 'librosa'. - `verbose` (bool): If True, prints verbose output. ### Usage Examples #### High-resolution CQT ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) cqt_hires = features.CQT(sr=sr, hop_length=512, fmin=32.70, n_bins=252, bins_per_octave=36, verbose=False) cqt_hires_out = cqt_hires(x) ``` #### Complex output with librosa-compatible normalization ```python cqt_cplx = features.CQT(sr=sr, hop_length=256, output_format='Complex', verbose=False) cqt_complex = cqt_cplx(x, normalization_type='librosa') # cqt_complex.shape => (4, 84, time_steps, 2) ``` #### Trainable CQT ```python cqt_trainable = features.CQT(sr=sr, trainable=True, output_format='Magnitude', verbose=False) optimizer = torch.optim.Adam(cqt_trainable.parameters(), lr=1e-4) loss = cqt_trainable(x).mean() loss.backward() optimizer.step() ``` #### CQT with fmax specified ```python cqt_fmax = features.CQT(sr=sr, fmin=55.0, fmax=4186.0, bins_per_octave=24, verbose=False) print(cqt_fmax(x).shape) ``` ``` -------------------------------- ### Trainable STFT Kernels Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Enables gradient-based learning of STFT kernels by setting `trainable=True`. This allows the Fourier kernels to be optimized alongside the rest of the neural network. ```python # --- Trainable STFT (kernels as nn.Parameter) --- stft_trainable = features.STFT(n_fft=512, trainable=True, output_format='Magnitude', verbose=False) optimizer = torch.optim.Adam(stft_trainable.parameters(), lr=1e-3) loss = stft_trainable(x).mean() loss.backward() # gradients flow into Fourier kernels optimizer.step() ``` -------------------------------- ### Gammatonegram Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Demonstrates the creation and usage of Gammatonegram features with different configurations, including higher resolution and trainable parameters. ```APIDOC ## Gammatonegram ### Description Extracts Gammatone filterbank spectrograms from audio signals. ### Usage ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(2, sr), dtype=torch.float32) # Higher resolution with 128 bins gammatone_hi = features.Gammatonegram(sr=sr, n_fft=2048, hop_length=256, n_bins=128, fmin=50.0, fmax=8000, power=1.0, verbose=False) gtgram_hi = gammatone_hi(x) # Trainable Gammatonegram gammatone_train = features.Gammatonegram(sr=sr, n_bins=64, trainable_bins=True, trainable_STFT=True, verbose=False) optimizer = torch.optim.Adam(gammatone_train.parameters(), lr=1e-4) loss = gammatone_train(x).mean() loss.backward() print(f"Gammatonegram shape: {gtgram_hi.shape}") ``` ``` -------------------------------- ### CQT2010v2 with convolutional normalization Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Applies convolutional normalization to the output of the CQT2010v2 transform. This is an alternative normalization method for spectral analysis. ```python spec_conv = cqt2010(x, normalization_type='convolutional') ``` -------------------------------- ### Move nnAudio STFT Layer to GPU Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/intro.md To utilize GPU acceleration, transfer the initialized STFT layer to the desired device using the `.to(device)` method. This is applicable when using the STFT layer standalone. ```python spec_layer = features.STFT().to(device) ``` -------------------------------- ### Process Waveforms with nnAudio-integrated Model Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/intro.md After integrating nnAudio into a PyTorch model, you can directly pass waveform tensors to the model for processing. The model handles the spectrogram extraction internally during the forward pass. ```python waveforms = torch.randn(4,44100) model(waveforms) # automatically convert waveforms into spectrograms ``` -------------------------------- ### VQT complex output for phase analysis Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Generates a Complex Variable-Q Transform (VQT) output, which includes phase information alongside magnitude. This is useful for applications requiring phase-sensitive spectral analysis. ```python vqt_cplx = features.VQT(sr=sr, hop_length=256, gamma=10, output_format='Complex', verbose=False) spec_cplx = vqt_cplx(x) # spec_cplx.shape => (4, 84, time_steps, 2) ``` -------------------------------- ### nnAudio.features.VQT Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Variable-Q Transform (VQT) extends CQT with a `gamma` parameter for adjustable time-frequency resolution per bin, mimicking human auditory perception. ```APIDOC ## VQT ### Description Extends CQT with a `gamma` parameter that controls the trade-off between time and frequency resolution per bin. When `gamma=0`, VQT is equivalent to CQT. Larger `gamma` values widen the frequency resolution at the cost of time resolution for low-frequency bins, mimicking a perceptually motivated auditory filterbank. Shares the same resampling strategy as `CQT2010v2`. ### Parameters - `sr` (int): Sample rate of the input audio. - `hop_length` (int): The number of samples between successive frames. - `fmin` (float): The lowest frequency of the VQT kernel. - `n_bins` (int): The total number of frequency bins. - `bins_per_octave` (int): The number of bins per octave. - `gamma` (float): Controls the time-frequency resolution trade-off. `gamma=0` is equivalent to CQT. - `output_format` (str): The format of the output. Can be 'Magnitude', 'Complex', or 'Spectrogram'. - `trainable` (bool): If True, the VQT kernels are trainable. - `verbose` (bool): If True, prints verbose output. ### Usage Examples #### VQT equivalent to CQT (gamma=0) ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) vqt_cqt = features.VQT(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=12, gamma=0, verbose=False) spec_cqt = vqt_cqt(x) # spec_cqt.shape => (4, 84, time_steps) ``` #### VQT with perceptually motivated time-frequency trade-off ```python vqt_perceptual = features.VQT(sr=sr, hop_length=512, fmin=32.70, n_bins=84, bins_per_octave=36, gamma=20, verbose=False) spec_perceptual = vqt_perceptual(x) ``` #### Complex output ```python vqt_cplx = features.VQT(sr=sr, hop_length=256, gamma=10, output_format='Complex', verbose=False) spec_cplx = vqt_cplx(x) # spec_cplx.shape => (4, 84, time_steps, 2) ``` #### Trainable VQT ```python vqt_train = features.VQT(sr=sr, trainable=True, verbose=False) optimizer = torch.optim.Adam(vqt_train.parameters(), lr=1e-4) loss = vqt_train(x).mean() loss.backward() print(f"VQT output shape: {spec_cqt.shape}") ``` ``` -------------------------------- ### Basic STFT Magnitude Spectrogram Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes the STFT of audio clips and outputs the magnitude spectrogram. Ensure the input tensor shape is compatible with the STFT layer. ```python import torch import numpy as np from nnAudio import features # Simulate a batch of 4 audio clips at 22050 Hz, 1 second each sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) # --- Basic magnitude spectrogram --- stft = features.STFT(n_fft=2048, hop_length=512, window='hann', freq_scale='no', sr=sr, output_format='Magnitude', verbose=False) mag = stft(x) # mag.shape => (4, 1025, 44) # (batch, n_fft//2+1, time_steps) ``` -------------------------------- ### CQT2010v2 complex output for phase analysis Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Generates a Complex Constant-Q Transform (CQT2010v2) for detailed phase analysis. The output format includes both magnitude and phase information. ```python cqt2010_cplx = features.CQT2010v2(sr=sr, output_format='Complex', verbose=False) spec_cplx = cqt2010_cplx(x) # spec_cplx.shape => (4, 84, time_steps, 2) print(f"CQT2010v2 magnitude shape: {spec.shape}") ``` -------------------------------- ### Implement Combined Frequency Periodicity (CFP) Feature Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Utilizes the Combined Frequency Periodicity (CFP) feature for multipitch estimation, combining frequency and periodicity information. The `g` parameter controls the number of non-linear activation layers. Outputs include the combined feature, raw STFT, frequency feature, and periodicity feature. ```python import torch import numpy as np from nnAudio import features fs = 16000 x = torch.tensor(np.random.randn(2, fs * 3), dtype=torch.float32) # Full CFP with all four outputs cfp = features.Combined_Frequency_Periodicity( fr=2, # frequency resolution fs=fs, # sample rate hop_length=320, window_size=2049, fc=80, # starting frequency (Hz) tc=1/1000, # inverse of ending frequency g=[0.24, 0.6, 1], # 3-layer non-linear activation NumPerOct=48 ) Z, tfrL0, tfrLF, tfrLQ = cfp(x) # Z.shape => (2, freq_bins, time_steps) — combined feature # tfrL0.shape => (2, freq_bins, time_steps) — STFT magnitude # tfrLF.shape => (2, freq_bins, time_steps) — frequency feature # tfrLQ.shape => (2, freq_bins, time_steps) — periodicity feature # Simplified CFP variant (returns only Z, aligned timesteps) cfp_simple = features.CFP(fr=2, fs=fs, hop_length=320, window_size=2049, fc=80, tc=1/1000, g=[0.24, 0.6, 1]) Z_only = cfp_simple(x) print(f"CFP Z shape: {Z.shape}, CFP simplified shape: {Z_only.shape}") ``` -------------------------------- ### MFCC with Custom Spectrogram Settings Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Calculates 40 MFCCs using custom Mel spectrogram parameters, including HTK scale and a specific frequency range. The output shape depends on the input audio length and STFT parameters. ```python # 40-coefficient MFCC with custom spectrogram settings mfcc_40 = features.MFCC(sr=sr, n_mfcc=40, n_fft=1024, hop_length=256, n_mels=80, fmin=80.0, fmax=7600, htk=True, verbose=False) mfcc_40_out = mfcc_40(x) # mfcc_40_out.shape => (4, 40, time_steps) print(f"MFCC shape: {mfcc_40_out.shape}") ``` -------------------------------- ### CQT with frequency range specified by fmax Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Configures a Constant-Q Transform where the frequency range is defined by a maximum frequency (fmax) instead of the total number of bins. Useful for setting a specific upper limit for spectral analysis. ```python cqt_fmax = features.CQT(sr=sr, fmin=55.0, fmax=4186.0, bins_per_octave=24, verbose=False) print(cqt_fmax(x).shape) ``` -------------------------------- ### Integrate nnAudio STFT into a PyTorch Model Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/intro.md Shows how to embed the nnAudio STFT layer within a PyTorch nn.Module for on-the-fly spectrogram extraction during model training or inference. The model processes raw waveforms and converts them to spectrograms internally. ```python from nnAudio import features import torch import torch.nn as nn class Model(torch.nn.Module): def __init__(self, n_fft, output_dim): super().__init__() self.epsilon=1e-10 # Getting Mel Spectrogram on the fly self.spec_layer = features.STFT(n_fft=n_fft, freq_bins=None, hop_length=512, window='hann', freq_scale='no', center=True, pad_mode='reflect', fmin=50, fmax=6000, sr=22050, trainable=False, output_format='Magnitude') self.n_bins = n_fft//2 # Creating CNN Layers self.CNN_freq_kernel_size=(128,1) self.CNN_freq_kernel_stride=(2,1) k_out = 128 k2_out = 256 self.CNN_freq = nn.Conv2d(1,k_out, kernel_size=self.CNN_freq_kernel_size,stride=self.CNN_freq_kernel_stride) self.CNN_time = nn.Conv2d(k_out,k2_out, kernel_size=(1,3),stride=(1,1)) self.region_v = 1 + (self.n_bins-self.CNN_freq_kernel_size[0])//self.CNN_freq_kernel_stride[0] self.linear = torch.nn.Linear(k2_out*self.region_v, output_dim, bias=False) def forward(self,x): z = self.spec_layer(x) z = torch.log(z+self.epsilon) z2 = torch.relu(self.CNN_freq(z.unsqueeze(1))) z3 = torch.relu(self.CNN_time(z2)).mean(-1) y = self.linear(torch.relu(torch.flatten(z3,1))) return torch.sigmoid(y) ``` -------------------------------- ### Standard Gammatonegram with 64 bins Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes a standard Gammatonegram using specified STFT parameters and a fixed number of bins. The 'norm=1' parameter indicates normalization by the number of FFT points. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) gammatone = features.Gammatonegram(sr=sr, n_fft=2048, hop_length=512, n_bins=64, fmin=0.0, fmax=None, norm=1, verbose=False) gtgram = gammatone(x) # gtgram.shape => (4, 64, time_steps) ``` -------------------------------- ### MFCC Layer within an End-to-End Speech Model Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Demonstrates integrating an MFCC feature extraction layer into a PyTorch speech recognition model. The extracted features are permuted for RNN input, and the output is passed through an RNN and a linear layer. ```python import torch.nn as nn class SpeechModel(nn.Module): def __init__(self): super().__init__() self.mfcc = features.MFCC(sr=16000, n_mfcc=13, verbose=False) self.rnn = nn.GRU(input_size=13, hidden_size=64, batch_first=True) self.fc = nn.Linear(64, 5) def forward(self, wav): feat = self.mfcc(wav).permute(0, 2, 1) # (B, T, n_mfcc) _, h = self.rnn(feat) return self.fc(h.squeeze(0)) ``` -------------------------------- ### Standalone iSTFT Module Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Uses the `iSTFT` module independently to convert complex spectrograms back to waveforms. All STFT parameters must match the forward STFT layer for accurate reconstruction. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(2, sr), dtype=torch.float32) # Forward STFT stft_layer = features.STFT(n_fft=1024, hop_length=256, window='hann', center=True, output_format='Complex', verbose=False) spec = stft_layer(x) # spec.shape => (2, 513, time_steps, 2) # Standalone iSTFT with matching parameters istft_layer = features.iSTFT(n_fft=1024, hop_length=256, window='hann', center=True, verbose=False) recon = istft_layer(spec, onesided=True, length=sr) # recon.shape => (2, 22050) ``` -------------------------------- ### nnAudio.features.STFT Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes the Short-Time Fourier Transform (STFT) of audio waveforms. Supports various frequency scales, output formats (Magnitude, Complex, Phase), and trainable kernels. ```APIDOC ## nnAudio.features.STFT — Short-Time Fourier Transform Computes the STFT of a batch of waveforms using a 1D convolutional neural network layer. Supports linear, logarithmic, and log2 frequency spacing between bins. The output format can be 'Magnitude', 'Complex' (real + imaginary stored as last-dim pair), or 'Phase'. Setting `trainable=True` allows the Fourier kernels to be updated via backpropagation. Setting `iSTFT=True` also registers inverse kernels so `layer.inverse()` can be called directly without instantiating a separate `iSTFT` object. ### Parameters - **n_fft** (int) - The number of FFT points. - **hop_length** (int) - The number of samples between successive frames. - **window** (str or torch.Tensor) - The window function to apply. Can be 'hann', 'hamming', 'bartlett', 'blackman', or a custom tensor. - **freq_scale** (str) - Frequency scale for the spectrogram. Options: 'no' (linear), 'log', 'log2'. - **fmin** (int) - Minimum frequency for log-scaled spectrograms. - **fmax** (int) - Maximum frequency for log-scaled spectrograms. - **sr** (int) - Sampling rate of the audio. - **output_format** (str) - Format of the output spectrogram. Options: 'Magnitude', 'Complex', 'Phase'. - **iSTFT** (bool) - If True, registers inverse kernels for direct use of `layer.inverse()`. - **trainable** (bool) - If True, Fourier kernels are trainable `nn.Parameter` objects. - **center** (bool) - If True, pads the signal so that the center of the signal is at the beginning of the first frame. - **verbose** (bool) - If True, prints information about the layer. ### Request Example ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) # Basic magnitude spectrogram stft = features.STFT(n_fft=2048, hop_length=512, window='hann', freq_scale='no', sr=sr, output_format='Magnitude', verbose=False) mag = stft(x) # Log-frequency spectrogram stft_log = features.STFT(n_fft=2048, hop_length=512, freq_scale='log', fmin=50, fmax=8000, sr=sr, output_format='Magnitude', verbose=False) log_mag = stft_log(x) # Complex output stft_cplx = features.STFT(n_fft=1024, hop_length=256, output_format='Complex', iSTFT=True, verbose=False) spec_complex = stft_cplx(x) # Round-trip: STFT -> iSTFT reconstructed = stft_cplx.inverse(spec_complex, onesided=True, length=sr) # Trainable STFT stft_trainable = features.STFT(n_fft=512, trainable=True, output_format='Magnitude', verbose=False) optimizer = torch.optim.Adam(stft_trainable.parameters(), lr=1e-3) loss = stft_trainable(x).mean() loss.backward() optimizer.step() ``` ### Response #### Success Response (200) - **spectrogram** (torch.Tensor) - The computed spectrogram. Shape depends on `output_format` and parameters. #### Response Example ```json { "example": "(4, 1025, 44) for Magnitude, (4, 513, time_steps, 2) for Complex" } ``` ``` -------------------------------- ### Standard MFCC Calculation Source: https://context7.com/kinwaicheuk/nnaudio/llms.txt Computes Mel-Frequency Cepstral Coefficients (MFCCs) with 20 coefficients using default settings. The `norm='ortho'` parameter applies orthogonal DCT-II normalization. ```python import torch import numpy as np from nnAudio import features sr = 22050 x = torch.tensor(np.random.randn(4, sr), dtype=torch.float32) # Standard 20-coefficient MFCC mfcc_layer = features.MFCC(sr=sr, n_mfcc=20, norm='ortho', n_fft=2048, hop_length=512, n_mels=128, top_db=80.0, verbose=False) mfcc = mfcc_layer(x) # mfcc.shape => (4, 20, 44) # (batch, n_mfcc, time_steps) ``` -------------------------------- ### BibTeX Citation for nnAudio Source: https://github.com/kinwaicheuk/nnaudio/blob/master/Sphinx/source/citing.md Use this BibTeX entry in your LaTeX documents to cite the nnAudio paper. ```tex @ARTICLE{9174990, author={K. W. {Cheuk} and H. {Anderson} and K. {Agres} and D. {Herremans}}, journal={IEEE Access}, title={nnAudio: An on-the-Fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolutional Neural Networks}, year={2020}, volume={8}, number={}, pages={161981-162003}, doi={10.1109/ACCESS.2020.3019084}} ```