### Install FunASR

Source: https://github.com/ddlbojack/emotion2vec/blob/main/README.md

Install the FunASR package. This is the first step for using the emotion2vec model through FunASR.

```bash
pip install -U funasr

```

--------------------------------

### Install fairseq and clone emotion2vec

Source: https://github.com/ddlbojack/emotion2vec/blob/main/README.md

Install fairseq and clone the emotion2vec repository. This is the first step for extracting features from your dataset using the source code installation method.

```bash
pip install fairseq
git clone https://github.com/ddlBoJack/emotion2vec.git

```

--------------------------------

### Install FunASR for emotion2vec+ inference

Source: https://github.com/ddlbojack/emotion2vec/blob/main/README.md

Installs the FunASR package, which is required for running emotion2vec+ inference.

```bash
pip install -U funasr
```

--------------------------------

### Bash example for running emotion2vec_speech_features.py

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/feature-extraction-scripts.md

Example command to run the feature extraction script. Sets PYTHONPATH to fairseq root, uses CUDA_VISIBLE_DEVICES=0, and passes arguments for data, model, split, checkpoint, save-dir, and layer. The wrapper script iemocap_downstream/scripts/emotion2vec_extract_features.sh runs this with PYTHONPATH=$1 and iterates layers {11..11} (last layer), logging true_layer = layer + 1.

```bash
export PYTHONPATH=/path/to/fairseq:$PYTHONPATH
CUDA_VISIBLE_DEVICES=0 python scripts/emotion2vec_speech_features.py \
    --data $manifest_path \
    --model /path/to/emotion2vec/upstream \
    --split=train \
    --checkpoint=/path/to/emotion2vec_base.pt \
    --save-dir=$feat_path \
    --layer=11
```

--------------------------------

### wav.scp input format for FunASR

Source: https://github.com/ddlbojack/emotion2vec/blob/main/README.md

Example wav.scp file format for batch inference with FunASR, listing audio file names and paths.

```text
wav_name1 wav_path1.wav
wav_name2 wav_path2.wav
...
```

--------------------------------

### BaseModel usage example

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Shows how to instantiate BaseModel and run inference on random features. It creates a label dictionary for the four IEMOCAP emotions, generates a batch of 100 frames, and prints the predicted label.

```python
import torch
from model import BaseModel

label_dict = {'ang': 0, 'hap': 1, 'neu': 2, 'sad': 3}
model = BaseModel(input_dim=768, output_dim=len(label_dict))

feat = torch.randn(1, 100, 768)          # 100 frames
padding_mask = torch.zeros(1, 100).bool()
outputs = model(feat, padding_mask)      # (1, 4) logits

_, predict = torch.max(outputs.data, dim=1)
idx2label = {v: k for k, v in label_dict.items()}
print(idx2label[predict.item()])
```

--------------------------------

### Example command for extract_features.py

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/feature-extraction-scripts.md

Example bash command to run the single-file extraction script with CUDA_VISIBLE_DEVICES=0. It specifies source wav, target npy, model directory, checkpoint, and granularity. The output shape is (768,) for utterance or (T, 768) for frame.

```bash
CUDA_VISIBLE_DEVICES=0 python extract_features.py \
  --source_file='/abs/path/scripts/test.wav' \
  --target_file='/abs/path/scripts/test.npy' \
  --model_dir='/abs/path/emotion2vec/upstream' \
  --checkpoint_dir='/abs/path/emotion2vec_base.pt' \
  --granularity='utterance'
```

--------------------------------

### extract_features method definition and usage example

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/data2vec-multi-model.md

First block shows the extract_features method definition that calls forward with features_only=True and remove_extra_tokens=True. Second block shows a usage example, where the returned dict keys are 'x', 'padding_mask', 'layer_results', 'mask'; for emotion2vec_base, layer_results has 12 entries and x shape is (1, T', 768).

```python
    def extract_features(
        self, source, mode=None, padding_mask=None, mask=False, remove_extra_tokens=True
    ):
        res = self.forward(
            source,
            mode=mode,
            padding_mask=padding_mask,
            mask=mask,
            features_only=True,
            remove_extra_tokens=remove_extra_tokens,
        )
        return res
```

```python
res = model.extract_features(source, padding_mask=None, remove_extra_tokens=True)
feats = res["x"]          # (1, T', 768); T' = conv-output length
layer_results = res["layer_results"]   # 12 entries for emotion2vec_base
```

--------------------------------

### AudioEncoder

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/modality-specific-encoder.md

Example of directly driving the AudioEncoder with a D2vAudioConfig to obtain encoded features and padding mask.

```APIDOC
## [AudioEncoder] [AudioEncoder]

### Description
Directly drive the AudioEncoder with a D2vAudioConfig to obtain encoded features and padding mask.

### Method
Constructor call

### Endpoint
`AudioEncoder(modality_cfg, embed_dim, make_block, norm_layer, layer_norm_first, alibi_biases, task)`

### Parameters
#### Constructor Parameters
- **modality_cfg** (D2vAudioConfig) - Required - Configuration for the audio modality.
- **embed_dim** (int) - Required - Embedding dimension of the encoder output.
- **make_block** (callable) - Required - Factory function to create transformer blocks (e.g., AltBlock).
- **norm_layer** (callable) - Required - Normalization layer factory (e.g., `lambda dim: torch.nn.LayerNorm(dim, eps=1e-6)`).
- **layer_norm_first** (bool) - Required - Whether to apply layer norm before the transformer blocks.
- **alibi_biases** (dict) - Required - ALiBi biases for attention.
- **task** (object) - Required - Task object (can be `None`).

### Request Example
```python
import torch
from upstream.models.audio import AudioEncoder, D2vAudioConfig

cfg = D2vAudioConfig()
encoder = AudioEncoder(
    modality_cfg=cfg,
    embed_dim=768,
    make_block=lambda dpr, dim=None, heads=None: ...,   # AltBlock factory
    norm_layer=lambda dim: torch.nn.LayerNorm(dim, eps=1e-6),
    layer_norm_first=False,
    alibi_biases={},
    task=None,
)

src = torch.randn(2, 16000)
pad = torch.zeros(2, 16000, dtype=torch.bool)
out = encoder(src, pad, mask=False, remove_masked=False)
```

### Response
#### Success Response
- **x** (Tensor) - Shape `(batch, conv_len, embed_dim)` - Encoded features.
- **padding_mask** (Tensor) - Shape `(batch, conv_len)` - Padding mask.

#### Response Example
```python
print(out["x"].shape)          # (2, conv_len, 768)
print(out["padding_mask"].shape)
```
```

--------------------------------

### forward

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/data2vec-multi-model.md

Documentation for the `forward` method of the emotion2vec model, including parameters, return values, and usage examples.

```APIDOC
## `forward`

### Description
Processes raw audio waveforms through the emotion2vec model. When `features_only=True`, returns extracted features; otherwise (pretraining mode) it is a no-op returning `None`.

### Method
`forward`

### Signature
```python
def forward(
    self,
    source,
    target=None,
    id=None,
    mode=None,
    padding_mask=None,
    mask=True,
    features_only=False,
    force_remove_masked=False,
    remove_extra_tokens=True,
    precomputed_mask=None,
    **kwargs,
):
```

### Parameters
- `source` (`torch.Tensor`) - Required - Raw waveform, shape `(B, T)` (float). Passed to the `AudioEncoder`.
- `target` (`torch.Tensor`) - Optional - Accepted for API compatibility; unused in the body.
- `id` (any) - Optional - Accepted for API compatibility; unused in the body.
- `mode` (any) - Optional - Accepted for API compatibility; unused in the body.
- `padding_mask` (`torch.Tensor`) - Optional - Boolean mask over input time steps, shape `(B, T)`; `True` denotes padding. Converted to the conv feature space by the encoder.
- `mask` (`bool`) - Optional - Enable the masked-prediction path (compute/apply input masks and clone batches) inside the encoder. Set `False` for pure feature extraction.
- `features_only` (`bool`) - Optional - If `True`, return extracted features; if `False` (pretraining mode), the method is a no-op returning `None`.
- `force_remove_masked` (`bool`) - Optional - When `True` with `features_only`, still remove masked positions from the sequence (`remove_masked=not features_only or force_remove_masked`).
- `remove_extra_tokens` (`bool`) - Optional - When `features_only`, strip the first `num_extra_tokens` columns from `x` and `padding_mask` (extra tokens default to 0 in the audio config).
- `precomputed_mask` (`torch.Tensor`) - Optional - Precomputed boolean mask passed straight to the encoder's `compute_mask`; skips mask generation.
- `**kwargs` - Optional - Silently ignored.

### Return
When `features_only=True`, returns a dict:
- `x` (`torch.Tensor`) - Encoded representations, shape `(B, T', embed_dim)`; `T'` excludes extra tokens when `remove_extra_tokens=True`.
- `padding_mask` (`torch.Tensor`) - Adjusted boolean padding mask aligned to `x`.
- `layer_results` (`list`) - One entry per surviving block (all blocks in eval mode; blocks dropped by layerdrop in training are skipped). Each entry is the `(x, t)` tuple returned by `AltBlock.forward`.
- `mask` (`MaskInfo` or `None`) - The `MaskInfo` namedtuple produced by the encoder when masking ran, else `None`.

When `features_only=False` (pretraining path): returns `None`.

### Throws
No custom exceptions. GPU OOM / shape mismatches surface as torch `RuntimeError`s. If `source` is not 2-D, the conv encoder raises a torch error.

### Example — extract utterance features from a loaded checkpoint
```python
import fairseq, torch

model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task(
    ["/path/to/emotion2vec_base.pt"]
)
model = model[0].eval().cuda()

# (B, T) float waveform at 16 kHz
source = torch.randn(1, 16000).cuda()

res = model.extract_features(source, padding_mask=None)
feats = res["x"]                 # (1, num_frames, 768)
padding_mask = res["padding_mask"]
print(feats.shape)

# frame-level -> utterance-level by mean pooling
utterance_feat = feats.squeeze(0).mean(dim=0)
```

### Example — masked pretraining-mode call (returns None)
```python
model.train()
out = model(
    source,
    mask=True,
    features_only=False,
    force_remove_masked=False,
)
assert out is None  # pretraining path returns None in this codebase
```
```

--------------------------------

### AudioEncoder forward pass example

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/audio-encoder.md

Constructs an AudioEncoder with a custom AltBlock block factory and runs a forward pass on a random 1-second waveform. Expects output shape torch.Size([1, 49, 768]) for the default conv spec.

```python
import torch
from functools import partial
from fairseq.models.wav2vec import ConvFeatureExtractionModel
from upstream.models.audio import AudioEncoder, D2vAudioConfig
from upstream.models.modules import AltBlock

cfg = D2vAudioConfig()
embed_dim = 768

def make_block(drop_path, dim=None, heads=None):
    return AltBlock(
        embed_dim if dim is None else dim,
        12 if heads is None else heads,
        cfg.mlp_ratio if hasattr(cfg, "mlp_ratio") else 4.0,
        qkv_bias=True,
        drop=0.1, attn_drop=0.1, mlp_drop=0.0,
        drop_path=drop_path,
        norm_layer=partial(torch.nn.LayerNorm, eps=1e-6, elementwise_affine=True),
        layer_norm_first=False,
        ffn_targets=True,
    )

encoder = AudioEncoder(
    modality_cfg=cfg,
    embed_dim=embed_dim,
    make_block=make_block,
    norm_layer=partial(torch.nn.LayerNorm, eps=1e-6, elementwise_affine=True),
    layer_norm_first=False,
    alibi_biases={},
    task=None,
)
encoder.eval()

src = torch.randn(1, 16000)
pad = torch.zeros(1, 16000, dtype=torch.bool)
with torch.no_grad():
    out = encoder(src, pad, mask=False, remove_masked=False)
print(out["x"].shape)          # torch.Size([1, 49, 768]) for the default conv spec
print(out["padding_mask"].shape)
```

--------------------------------

### setup_task

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Class method to set up the task (e.g., load dictionaries). It is the fairseq entry point for task construction.

```APIDOC
## setup_task

### Description
Class method to set up the task (e.g., load dictionaries). It is the fairseq entry point for task construction.

### Method
setup_task

### Signature
```python
@classmethod
setup_task(cls, cfg: Emotion2vecPretrainingConfig, **kwargs)
```

### Parameters
- **cfg** (Emotion2vecPretrainingConfig) - Required - Task configuration.
- **kwargs** - Optional - Ignored.

### Returns
A new `Emotion2vecPretrainingTask` instance.
```

--------------------------------

### setup_task method

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Classmethod that sets up the task by returning a new Emotion2vecPretrainingTask instance. It accepts the task configuration and ignores any additional keyword arguments.

```python
    @classmethod
    def setup_task(cls, cfg: Emotion2vecPretrainingConfig, **kwargs):
        """Setup the task (e.g., load dictionaries)."""
        return cls(cfg)
```

--------------------------------

### Emotion2vecPretrainingTask class with setup_task

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Defines the fairseq task class for emotion2vec pre-training, registered with the name "emotion2vec_pretraining" and using the Emotion2vecPretrainingConfig dataclass. The setup_task classmethod is the entry point for task construction, returning a new instance of the task.

```python
@register_task("emotion2vec_pretraining", dataclass=Emotion2vecPretrainingConfig)
class Emotion2vecPretrainingTask(FairseqTask):
    """ """

    cfg: Emotion2vecPretrainingConfig

    @classmethod
    def setup_task(cls, cfg: Emotion2vecPretrainingConfig, **kwargs):
        """Setup the task (e.g., load dictionaries).

        Args:
            cfg (Emotion2vecPretrainingConfig): configuration of this task
        """
        return cls(cfg)
```

--------------------------------

### build_model method

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Builds the model via the base implementation and copies w2v_args from the built model's cfg back onto model_cfg for legacy wav2vec checkpoint compatibility.

```python
    def build_model(self, model_cfg: FairseqDataclass, from_checkpoint=False):
        model = super().build_model(model_cfg, from_checkpoint)
        actualized_cfg = getattr(model, "cfg", None)
        if actualized_cfg is not None:
            # if "w2v_args" in actualized_cfg:
            if hasattr(actualized_cfg, "w2v_args"):
                model_cfg.w2v_args = actualized_cfg.w2v_args
        return model
```

--------------------------------

### Catch AssertionError for missing alibi_max_pos

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/errors.md

Example of catching the AssertionError raised when constructing an AudioEncoder with learned_alibi=True and alibi_max_pos=None. The error message suggests setting modality_cfg.alibi_max_pos.

```python
try:
    enc = AudioEncoder(...)
except AssertionError as e:
    print("set modality_cfg.alibi_max_pos when learned_alibi=True:", e)
```

--------------------------------

### IEMOCAP manifest generation with iemocap_manifest.py

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Creates a manifest file (train.tsv) for IEMOCAP by reading a label file (train.emo) and deriving audio paths from filenames. Requires soundfile and os modules. Assumes filenames like 'Ses01F_impro01_F000' where the 5th character is the session digit. Throws AssertionError/RuntimeError from soundfile.info if a wav is missing.

```python
def get_parser():
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", metavar="DIR", default='/path/to/IEMOCAP_full_release',
                        help="root directory containing audio files to index")
    parser.add_argument("--dest", default="/path/to/manifest", type=str, metavar="DIR", help="output directory")
    parser.add_argument("--label_path", default="/path/to/train.emo")
    return parser

def main(args):
    if not os.path.exists(args.dest):
        os.makedirs(args.dest)
    root = os.path.join(args.root, 'Session{}')
    with open(args.label_path) as rf, open(args.dest + '/train.tsv', 'w') as wf:
        print(args.root, file=wf)
        for line in rf.readlines():
            fname = line.split('\t')[0]
            session = fname[4]
            folder = fname.rsplit('_', 1)[0]
            fname = os.path.join(root.format(session), 'sentences/wav', folder, fname + '.wav')
            frames = soundfile.info(fname).frames
            suffix = fname.replace(args.root, '', 1).lstrip('/')
            print(suffix, frames, sep='\t', file=wf)
```

--------------------------------

### Catch AssertionError for post_save copy failure

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/errors.md

Example of catching the AssertionError raised when post_save fails to copy a checkpoint. The error message includes the source and destination paths.

```python
try:
    task.post_save("checkpoint_last.pt", 1000)
except AssertionError as e:
    print("post_save copy failed:", e)
```

--------------------------------

### Emotion2vecFeatureReader.__init__

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/feature-extraction-scripts.md

Initializes the feature reader by loading the model ensemble and task, and preparing the model for inference on CUDA.

```APIDOC
## Emotion2vecFeatureReader.__init__

### Description
Initializes the feature reader by importing the user module, loading the model ensemble and task, and putting the model in eval mode on CUDA.

### Method
Constructor

### Endpoint
`Emotion2vecFeatureReader(model_file, checkpoint, layer)`

### Parameters
#### Parameters
- `model_file` (str) - Required - User-module directory (`upstream/`).
- `checkpoint` (str) - Required - `.pt` checkpoint path.
- `layer` (int) - Required - Layer selector (currently unused in `get_feats`).

### Request Example
```python
reader = Emotion2vecFeatureReader(model_file='upstream', checkpoint='model.pt', layer=0)
```

### Response
#### Success Response
- No return value; initializes the reader object.

#### Response Example
No response body.

### Error Handling
- Fairseq loading errors (missing checkpoint file → `FileNotFoundError`).
- `AssertionError` from `read_audio` for non-16 kHz or non-mono files.

```

--------------------------------

### Prepare features and labels for IEMOCAP

Source: https://github.com/ddlbojack/emotion2vec/blob/main/iemocap_downstream/README.md

Sets environment variables and runs two shell scripts to generate manifest/labels and extract features using emotion2vec. The output files train.npy, train.lengths, and train.emo are placed in the feature path.

```bash
# set your own path here
IEMOCAP_ROOT=/path/to/IEMOCAP_full_release
manifest_path=/path/to/manifest
checkpoint_path=/path/to/emotion2vec_ckpt
feat_path=/path/to/feats
fairseq_root=/path/to/fairseq_root
model_path=../upstream # /path/to/emotion2vec/upstream

# generate manifest and labels from raw dataset
bash scripts/iemocap_manifest_and_labels.sh $IEMOCAP_ROOT $manifest_path

# generate features with emotion2vec
bash scripts/emotion2vec_extract_features.sh $fairseq_root $manifest_path $model_path $checkpoint_path $feat_path

cp $manifest_path/train.emo $feat_path
```

--------------------------------

### Train IEMOCAP with Hydra (main.py)

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Hydra entry point for IEMOCAP training. Seeds torch, then for each of 5 folds uses the fold-th session (per n_samples boundaries) as test and the rest for train/val; trains BaseModel(768, 4) with RMSprop(lr, momentum=0.9) and CyclicLR(base_lr=cfg.optimization.lr, max_lr=1e-3, step_size_up=10) stepped each epoch; criterion is nn.CrossEntropyLoss(); saves the best-validation checkpoint per fold at {hydra_run_dir}/model_{fold+1}.pth, then reloads it (strict=True) to score the test fold. Logs per-fold and averaged WA/UA/F1 (percentages). Also defines count_parameters(model) printing every named parameter's element count and the total.

```python
@hydra.main(config_path='config', config_name='default.yaml')
def train_iemocap(cfg: DictConfig):
    torch.manual_seed(cfg.common.seed)
    label_dict={'ang': 0, 'hap': 1, 'neu': 2, 'sad': 3}
    n_samples = [1085, 1023, 1151, 1031, 1241] # Session1, 2, 3, 4, 5
    idx_sessions = [0, 1, 2, 3, 4]
    ...
```

--------------------------------

### iemocap_manifest.py

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Generates a train.tsv manifest file from IEMOCAP audio files and a label file. It reads label filenames, derives the corresponding WAV paths, and writes the root directory and each file's relative path and frame count to the output TSV.

```APIDOC
## `iemocap_manifest.py`

### Description
Generates a `train.tsv` manifest file from IEMOCAP audio files and a label file. It reads label filenames, derives the corresponding WAV paths, and writes the root directory and each file's relative path and frame count to the output TSV.

### Method
CLI script

### Endpoint
`python iemocap_manifest.py [--root DIR] [--dest DIR] [--label_path FILE]`

### Parameters
#### Command-Line Arguments
- `--root` (str) - Optional - Default: `/path/to/IEMOCAP_full_release` - IEMOCAP root; `Session{N}/sentences/wav/{folder}/{name}.wav` paths are derived from label filenames.
- `--dest` (str) - Optional - Default: `/path/to/manifest` - Output directory; writes `{dest}/train.tsv`.
- `--label_path` (str) - Optional - Default: `/path/to/train.emo` - Label file (`name<TAB>emotion` per line).

### Request Example
```bash
python iemocap_manifest.py --root /path/to/IEMOCAP_full_release --dest /path/to/manifest --label_path /path/to/train.emo
```

### Response
#### Success Response
- **Return**: `None`
- **Output**: Writes `{dest}/train.tsv` containing the root directory on the first line, followed by lines with relative WAV path and frame count separated by a tab.

#### Response Example
```
/path/to/IEMOCAP_full_release
Ses01F/impro01/F001.wav	12345
Ses01F/impro01/F002.wav	67890
```

### Error Handling
- **Throws**: `AssertionError`/`RuntimeError` from `soundfile.info` if a WAV file is missing.
- **KeyError-safe**: `fname[4]` assumes filenames like `Ses01F_impro01_F000` (5th character = session digit).
```

--------------------------------

### Run emotion2vec+ inference with FunASR AutoModel

Source: https://github.com/ddlbojack/emotion2vec/blob/main/README.md

Runs emotion2vec+ inference on a test audio file using the FunASR AutoModel API. You can switch models by changing model_id (e.g., iic/emotion2vec_plus_base or iic/emotion2vec_base_finetuned). The hub parameter selects the download source: "ms" for China mainland, "hf" for elsewhere. The result contains feats, labels, and scores, with 9-class emotion labels. The model is downloaded automatically.

```python
'''
Using the finetuned emotion recognization model

rec_result contains {'feats', 'labels', 'scores'}
	extract_embedding=False: 9-class emotions with scores
	extract_embedding=True: 9-class emotions with scores, along with features

9-class emotions: 
iic/emotion2vec_plus_seed, iic/emotion2vec_plus_base, iic/emotion2vec_plus_large (May. 2024 release)
iic/emotion2vec_base_finetuned (Jan. 2024 release)
    0: angry
    1: disgusted
    2: fearful
    3: happy
    4: neutral
    5: other
    6: sad
    7: surprised
    8: unknown
'''

from funasr import AutoModel

# model="iic/emotion2vec_base"
# model="iic/emotion2vec_base_finetuned"
# model="iic/emotion2vec_plus_seed"
# model="iic/emotion2vec_plus_base"
model_id = "iic/emotion2vec_plus_large"

model = AutoModel(
    model=model_id,
    hub="ms",  # "ms" or "modelscope" for China mainland users; "hf" or "huggingface" for other overseas users
)

wav_file = f"{model.model_path}/example/test.wav"
rec_result = model.generate(wav_file, output_dir="./outputs", granularity="utterance", extract_embedding=False)
print(rec_result)
```

--------------------------------

### IEMOCAP downstream config (config/default.yaml)

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Configuration file for IEMOCAP downstream training. Only common.seed, dataset.feat_path, dataset.batch_size, dataset.eval_is_test, optimization.lr, and optimization.epoch are read by main.py; other fields are declared but unused.

```yaml
common:
  seed: 42

dataset:
  _name: IEMOCAP
  feat_path: /path/to/feat_path
  test_ratio: 0.2
  batch_size: 128
  fold: 5
  eval_is_test: False

optimization:
  epoch: 100
  lr: 5e-4
  weight_decay: 1e-5
  label_smooth: 0.0

lr_scheduler:
  _name: ??
  warmup_updates: ??

model:
  _name: BaseModel

```

--------------------------------

### BaseModel.__init__

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Initializes the BaseModel with a linear pre-net, ReLU activation, and a linear post-net. The input dimension defaults to 768 (emotion2vec frame dim) and output dimension to 4 (number of emotion classes).

```APIDOC
## BaseModel.__init__

### Description
Initializes the BaseModel with a linear pre-net, ReLU activation, and a linear post-net. The input dimension defaults to 768 (emotion2vec frame dim) and output dimension to 4 (number of emotion classes).

### Method
Constructor

### Parameters
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `input_dim` | `int` | no | `768` | Feature dimension (emotion2vec frame dim). |
| `output_dim` | `int` | no | `4` | Number of emotion classes. |

### Returns
None
```

--------------------------------

### Drive the AudioEncoder directly with D2vAudioConfig

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/modality-specific-encoder.md

Shows how to instantiate the AudioEncoder with a D2vAudioConfig and run a forward pass on random input. The output 'x' has shape (2, conv_len, 768) and the padding mask shape is also printed.

```python
import torch
from upstream.models.audio import AudioEncoder, D2vAudioConfig

cfg = D2vAudioConfig()
encoder = AudioEncoder(
    modality_cfg=cfg,
    embed_dim=768,
    make_block=lambda dpr, dim=None, heads=None: ... ,   # AltBlock factory
    norm_layer=lambda dim: torch.nn.LayerNorm(dim, eps=1e-6),
    layer_norm_first=False,
    alibi_biases={},
    task=None,
)

src = torch.randn(2, 16000)
pad = torch.zeros(2, 16000, dtype=torch.bool)
out = encoder(src, pad, mask=False, remove_masked=False)
print(out["x"].shape)          # (2, conv_len, 768)
print(out["padding_mask"].shape)
```

--------------------------------

### Assembling an encoder block stack with AltBlock

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/transformer-modules.md

Shows how to instantiate an AltBlock encoder block with typical hyperparameters and run a forward pass with a random input and padding mask. The block returns both the output and a target tensor, both with shape (2, 64, 768).

```python
import torch
from upstream.models.modules import AltBlock, BlockEncoder

block = AltBlock(
    dim=768,
    num_heads=12,
    mlp_ratio=4.0,
    qkv_bias=True,
    drop=0.1,
    attn_drop=0.1,
    mlp_drop=0.0,
    drop_path=0.1,
    norm_layer=torch.nn.LayerNorm,
    layer_norm_first=False,
    ffn_targets=True,
)

x = torch.randn(2, 64, 768)
pad = torch.zeros(2, 64, dtype=torch.bool)
out, target = block(x, padding_mask=pad, alibi_bias=None)
print(out.shape, target.shape)   # both torch.Size([2, 64, 768])
```

--------------------------------

### AltBlock

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/transformer-modules.md

Documentation for the AltBlock transformer block class, including constructor parameters and forward method signature.

```APIDOC
## `AltBlock`

### Description
A transformer encoder block with configurable pre/post layer norm, optional cosine attention, and optional ffn targets for self-supervised learning.

### Constructor

```python
class AltBlock(nn.Module):
    def __init__(
        self,
        dim,
        num_heads,
        mlp_ratio=4.0,
        qkv_bias=False,
        qk_scale=None,
        drop=0.0,
        attn_drop=0.0,
        mlp_drop=0.0,
        post_mlp_drop=0.0,
        drop_path=0.0,
        act_layer=nn.GELU,
        norm_layer=nn.LayerNorm,
        layer_norm_first=True,
        ffn_targets=False,
        cosine_attention=False,
    ):
```

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `dim` | `int` | yes | — | Model dimension. |
| `num_heads` | `int` | yes | — | Attention heads. |
| `mlp_ratio` | `float` | no | `4.0` | MLP hidden dim = `int(dim * mlp_ratio)`. |
| `qkv_bias` | `bool` | no | `False` | Bias on the qkv projection. |
| `qk_scale` | `float` or `None` | no | `None` | Attention scale override; defaults to `head_dim ** -0.5`. |
| `drop` | `float` | no | `0.0` | Dropout on attention output projection. |
| `attn_drop` | `float` | no | `0.0` | Dropout on attention weights. |
| `mlp_drop` | `float` | no | `0.0` | Dropout inside the MLP (timm `Mlp`). |
| `post_mlp_drop` | `float` | no | `0.0` | Dropout applied after the MLP, before the residual. |
| `drop_path` | `float` | no | `0.0` | DropPath rate; `nn.Identity` when `<= 0`. |
| `act_layer` | activation class | no | `nn.GELU` | MLP activation. |
| `norm_layer` | norm class | no | `nn.LayerNorm` | Normalization used by `norm1`/`norm2`. |
| `layer_norm_first` | `bool` | no | `True` | Pre-norm (attention+MLP inside residual) vs post-norm. |
| `ffn_targets` | `bool` | no | `False` | When `True`, the returned target `t` is the pre-drop residual MLP output `r`; when `False`, `t` equals the block output `x`. |
| `cosine_attention` | `bool` | no | `False` | Use cosine-similarity attention with a learned logit scale. |

### Method: `forward`

```python
def forward(self, x, padding_mask=None, alibi_bias=None):
```

#### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `x` | `torch.Tensor` | yes | — | Input, `(B, T, dim)`. |
| `padding_mask` | `torch.Tensor` or `None` | no | `None` | Boolean mask passed to attention; masked positions get `-inf` logits. |
| `alibi_bias` | `torch.Tensor` or `None` | no | `None` | ALiBi bias, `(B, H, T, T)` or `(H, T, T)`; added to attention logits. |

#### Returns

- **tuple** `(x, t)` — the block output and the target tensor used for SSL targets (`ffn_targets` selects whether `t` is the post-norm MLP residual or the final output). Uses `timm.models.vision_transformer.DropPath` and `Mlp` (imported lazily inside `__init__`).
```

--------------------------------

### load_ssl_features: Load IEMOCAP SSL features

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Loads IEMOCAP SSL features from a base path using `load_dataset`, maps emotion labels to integer ids via `label_dict`, and returns a dict with keys `feats`, `sizes`, `offsets`, `labels`, and `num`. The `max_speech_seq_len` parameter is forwarded to `load_dataset` as `max_length`.

```python
def load_ssl_features(feature_path, label_dict, max_speech_seq_len=None):
    data, sizes, offsets, labels = load_dataset(feature_path, labels='emo', min_length=1, max_length=max_speech_seq_len)
    labels = [ label_dict[elem] for elem in labels ]
    num = len(labels)
    iemocap_data = {
        "feats": data,
        "sizes": sizes,
        "offsets": offsets,
        "labels": labels,
        "num": num
    }
    return iemocap_data
```

--------------------------------

### Run emotion2vec pretraining with fairseq-hydra-train

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Invokes the emotion2vec pretraining task via the fairseq CLI. The task and model are registered under the names `emotion2vec_pretraining` and `data2vec_multi`; `Data2VecMultiModel` is built through `task.build_model` (fairseq default wiring). Hydra paths like `model.modalities.audio.feature_encoder_spec` are referenced by `AudioMaskingConfig`.

```bash
fairseq-hydra-train \
    task=emotion2vec_pretraining \
    task.data=/path/to/data \
    model=data2vec_multi \
    --config-dir /path/to/emotion2vec/upstream \
    --config-name your_data2vec_config
```

--------------------------------

### AltBlock transformer block with forward method

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/transformer-modules.md

Defines the AltBlock transformer block. The constructor accepts dim, num_heads, and optional flags like layer_norm_first, ffn_targets, and cosine_attention. The forward method returns a tuple (x, t) where t is the SSL target; when ffn_targets is True, t is the pre-drop residual MLP output r, otherwise it equals the block output x. Uses timm's DropPath and Mlp, imported lazily inside __init__.

```python
class AltBlock(nn.Module):
    def __init__(
        self,
        dim,
        num_heads,
        mlp_ratio=4.0,
        qkv_bias=False,
        qk_scale=None,
        drop=0.0,
        attn_drop=0.0,
        mlp_drop=0.0,
        post_mlp_drop=0.0,
        drop_path=0.0,
        act_layer=nn.GELU,
        norm_layer=nn.LayerNorm,
        layer_norm_first=True,
        ffn_targets=False,
        cosine_attention=False,
    ):
```

```python
    def forward(self, x, padding_mask=None, alibi_bias=None):
        if self.layer_norm_first:
            x = x + self.drop_path(self.attn(self.norm1(x), padding_mask, alibi_bias))
            r = x = self.mlp(self.norm2(x))
            t = x
            x = r + self.drop_path(self.post_mlp_dropout(x))
            if not self.ffn_targets:
                t = x
        else:
            x = x + self.drop_path(self.attn(x, padding_mask, alibi_bias))
            r = x = self.norm1(x)
            x = self.mlp(x)
            t = x
            x = self.norm2(r + self.drop_path(self.post_mlp_dropout(x)))
            if not self.ffn_targets:
                t = x

        return x, t
```

--------------------------------

### AltAttention

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/transformer-modules.md

Documentation for the AltAttention class, including its constructor and forward method parameters and return value.

```APIDOC
## `AltAttention`

### Description
Multi-head self-attention module with optional cosine attention and ALiBi bias support. It computes attention over an input tensor of shape `(B, N, C)` and returns an output of the same shape.

### Constructor

```python
class AltAttention(nn.Module):
    def __init__(
        self,
        dim,
        num_heads=8,
        qkv_bias=False,
        qk_scale=None,
        attn_drop=0.0,
        proj_drop=0.0,
        cosine_attention=False,
    ):
```

#### Parameters
- **dim** (`int`) - Required - Model dimension.
- **num_heads** (`int`) - Optional, default `8` - Attention heads.
- **qkv_bias** (`bool`) - Optional, default `False` - Bias on the `Linear(dim, dim * 3)` qkv projection.
- **qk_scale** (`float` or `None`) - Optional, default `None` - Score scale; defaults to `head_dim ** -0.5`.
- **attn_drop** (`float`) - Optional, default `0.0` - Dropout on attention probabilities.
- **proj_drop** (`float`) - Optional, default `0.0` - Dropout on the output projection.
- **cosine_attention** (`bool`) - Optional, default `False` - Cosine attention with learned `logit_scale` parameter (clamped to `log(1/0.01)`).

### Forward Method

```python
def forward(self, x, padding_mask=None, alibi_bias=None):
```

#### Parameters
- **x** (`torch.Tensor`) - Required - Input, `(B, N, C)`.
- **padding_mask** (`torch.Tensor` or `None`) - Optional, default `None` - Boolean `(B, N)` mask; masked positions become `-inf` before softmax.
- **alibi_bias** (`torch.Tensor` or `None`) - Optional, default `None` - ALiBi bias added to logits; first `alibi_bias.size(1)` head rows receive it.

#### Returns
Attention output of shape `(B, N, C)`. Softmax is always computed in `float32` regardless of input dtype.
```

--------------------------------

### main

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/feature-extraction-scripts.md

Main function that orchestrates feature extraction and saving to disk.

```APIDOC
## `main`

### Description
Main entry point that parses CLI arguments, creates output files, and iterates over features to write them to disk. It writes `{split}.npy` via `npy_append_array.NpyAppendArray` (a flat concatenated array) and `{split}.lengths` with one frame count per line. Existing `.npy` is removed first.

### Method
Function call

### Endpoint
`main()`

### Parameters
None (uses CLI arguments via `get_parser()`)

### Return
`None` (exit code 0)

### Example
```python
def main():
    parser = get_parser()
    args = parser.parse_args()
    os.makedirs(args.save_dir, exist_ok=True)
    ...
    save_path = osp.join(args.save_dir, args.split)
    npaa = create_files(save_path)
    generator, num = get_iterator(args)
    iterator = generator()
    with open(save_path + ".lengths", "w") as l_f:
        for d2v_feats in tqdm.tqdm(iterator, total=num):
            print(len(d2v_feats), file=l_f)
            if len(d2v_feats) > 0:
                npaa.append(d2v_feats.numpy())
```
```

--------------------------------

### load_dataset

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/iemocap-downstream.md

Loads preprocessed feature arrays, per-sample lengths, and optional labels from disk. Returns the flat feature array, sample sizes, offsets, and label strings.

```APIDOC
## load_dataset

### Description
Loads preprocessed feature arrays, per-sample lengths, and optional labels from disk. Returns the flat feature array, sample sizes, offsets, and label strings.

### Function Signature
```python
def load_dataset(data_path, labels=None, min_length=3, max_length=None):
```

### Parameters
- **data_path** (str) - Required - Base path; reads `{data_path}.npy`, `{data_path}.lengths`, and (if it exists) `{data_path}.{labels}`.
- **labels** (str or None) - Optional - Label file extension (e.g. `"emo"`); each line is `wav_name<TAB>label`. Only the label token is taken. Falls back to `None` when the file is missing.
- **min_length** (int) - Optional - Skip samples shorter than this (frames). Default: `3`.
- **max_length** (int or None) - Optional - Skip samples longer than this. Default: `None`.

### Return Value
Returns a tuple `(npy_data, sizes, offsets, emo_labels)` where:
- `npy_data` (np.ndarray) - The flat feature array.
- `sizes` (np.ndarray) - Per-sample frame counts.
- `offsets` (np.ndarray) - Per-sample start offsets into `npy_data`.
- `emo_labels` (list) - Label strings (empty list when `labels` is `None`).
```

--------------------------------

### post_save method

Source: https://github.com/ddlbojack/emotion2vec/blob/main/_autodocs/api-reference/emotion2vec-pretraining-task.md

Called by fairseq after each checkpoint save. If cfg.post_save_script is set, copies the checkpoint to checkpoint_eval_{num_updates}.pt and runs the script with the copied path. Throws AssertionError if the copy fails.

```python
    def post_save(self, cp_path, num_updates):
        if self.cfg.post_save_script is not None:
            logger.info(f"launching {self.cfg.post_save_script}")
            import os.path as osp
            from fairseq.file_io import PathManager

            eval_cp_path = osp.join(
                osp.dirname(cp_path), f"checkpoint_eval_{num_updates}.pt"
            )

            print(cp_path, eval_cp_path, osp.dirname(cp_path))

            assert PathManager.copy(
                cp_path, eval_cp_path, overwrite=True
            ), f"Failed to copy {cp_path} to {eval_cp_path}"

            import subprocess
            import shlex

            subprocess.call(shlex.split(f"{self.cfg.post_save_script} {eval_cp_path}"))
```

--------------------------------

### Load model and perform inference with PyTorch

Source: https://github.com/ddlbojack/emotion2vec/blob/main/iemocap_downstream/inference.ipynb

Loads a pre-trained BaseModel with input dimension 768 and 4 output classes, loads a checkpoint from a specific path, and performs a forward pass on a random input with a padding mask. Prints the predicted label index mapped to its string label.

```python
import torch
from model import BaseModel

label_dict={'ang': 0, 'hap': 1, 'neu': 2, 'sad': 3}
idx2label = {v: k for k, v in label_dict.items()}
model = BaseModel(input_dim=768, output_dim=len(label_dict))

ckpt = torch.load('outputs/2024-01-14/22-57-42/model_1.pth')
model.load_state_dict(ckpt)

feat = torch.randn(1, 100, 768)
padding_mask = torch.zeros(1, 100).bool()
outputs = model(feat, padding_mask)

_, predict = torch.max(outputs.data, dim=1)
print(idx2label[predict.item()])
```

```text
Output:
ang
```