### Install OpenSpeech

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_sources/notes/intro.md.txt

Installation methods using PyPI or source code.

```bash
pip install openspeech-core
```

```bash
$ ./install.sh
```

--------------------------------

### Install OpenSpeech from Source

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/notes/intro.html

Install OpenSpeech from its source code using the provided installation script. This is useful for development or if you need the latest changes.

```bash
./install.sh
```

--------------------------------

### Install OpenSpeech from Source

Source: https://github.com/openspeech-team/openspeech/wiki/Installation

Install OpenSpeech by cloning the source code and running the provided installation script. This method is for developers or users who need the latest code.

```bash
$ ./install.sh
```

--------------------------------

### LightningAIShellDataModule Setup

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/aishell/lit_data_module.html

The setup method initializes and splits the AISHELL-1 dataset into training, validation, and test sets. It also prepares the dataset instances with specified configurations and a tokenizer.

```APIDOC
## LightningAIShellDataModule.setup

### Description
Splits the AISHELL-1 dataset into \`train\`, \`valid\`, and \`test\` sets and prepares them for use.

### Method
`setup(stage: Optional[str] = None, tokenizer: Tokenizer = None)`

### Parameters
#### Arguments
- **stage** (str) - Optional - The current stage of training, e.g., \`train\` or \`valid\`.
- **tokenizer** (Tokenizer) - Optional - The tokenizer responsible for preparing model inputs.

### Returns
None
```

--------------------------------

### Data Preparation and Setup

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Details the recommended pattern for data processing using `prepare_data()` and `setup()` methods within a PyTorch Lightning DataModule. These steps are crucial for distributed processing.

```APIDOC
## Data Preparation and Setup

### Description

This section describes the recommended pattern for data processing within a PyTorch Lightning DataModule. The `prepare_data()` method is used for downloading data, and the `setup()` method is used for processing and splitting the data. These steps are particularly important when dealing with distributed processing.

### Methods

- `prepare_data()`: Download data. **Warning**: Do not assign state in this method.
- `setup()`: Process and split data. This method is called on every process in distributed training.

### Usage within `fit()`

The typical lifecycle within the `fit()` method includes:

1. `prepare_data()`
2. `setup()`
3. `train_dataloader()`
4. `val_dataloader()`
5. `test_dataloader()` (if implemented)

### Note on Samplers

PyTorch Lightning automatically adds the correct sampler for distributed and arbitrary hardware, so there is no need to set it manually.
```

--------------------------------

### Clone and Install NVIDIA Apex Library

Source: https://github.com/openspeech-team/openspeech/wiki/Installation

Clone the NVIDIA Apex repository and install it with C++ and CUDA extensions enabled for faster training. Ensure the correct CUDA and GCC modules are loaded before installation.

```bash
$ git clone https://github.com/NVIDIA/apex
$ cd apex

# ------------------------
# OPTIONAL: on your cluster you might need to load CUDA 10 or 9
# depending on how you installed PyTorch

# see available modules
module avail

# load correct CUDA before install
module load cuda-10.0
# ------------------------

# make sure you've loaded a cuda version > 4.0 and < 7.0
module load gcc-6.1.0

$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

--------------------------------

### Install OpenSpeech Dependencies

Source: https://github.com/openspeech-team/openspeech/blob/main/README.md

Commands to install required libraries for the OpenSpeech framework.

```bash
pip install numpy
```

```bash
conda install -c conda-forge librosa
```

```bash
pip install torchaudio==0.6.0
```

```bash
pip install sentencepiece
```

```bash
pip install pytorch-lightning
```

```bash
pip install hydra-core --upgrade
```

--------------------------------

### Clone and Install NVIDIA Apex

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/notes/intro.html

Clone the NVIDIA Apex repository and install it. This library is optional and used for faster 16-bit training. Ensure correct CUDA and GCC modules are loaded before installation.

```bash
git clone https://github.com/NVIDIA/apex
cd apex

# OPTIONAL: on your cluster you might need to load CUDA 10 or 9
# depending on how you installed PyTorch

# see available modules
module avail

# load correct CUDA before install
module load cuda-10.0
# ------------------------

# make sure you've loaded a cuda version > 4.0 and < 7.0
module load gcc-6.1.0

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

--------------------------------

### Install OpenSpeech Core from PyPI

Source: https://github.com/openspeech-team/openspeech/wiki/Installation

Install the core OpenSpeech package using pip. This is the recommended method for most users.

```bash
pip install openspeech-core
```

--------------------------------

### setup

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/ksponspeech/lit_data_module.html

Splits the dataset into train, valid, and test sets and initializes the SpeechToTextDataset objects.

```APIDOC
## setup

### Description
Splits the dataset into train and valid sets for training and initializes the internal dataset dictionary.

### Parameters
- **stage** (str) - Optional - The stage of training, either 'train' or 'valid'.
- **tokenizer** (Tokenizer) - Optional - The tokenizer used for preparing inputs.
```

--------------------------------

### Cross Entropy Loss Example

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/criterion/cross_entropy/cross_entropy.html

Example demonstrating how to instantiate and use the CrossEntropyLoss module. It shows the creation of dummy inputs and targets, calculating the loss, and performing backpropagation.

```python
>>> B, T1, C, T2 = 3, 128, 4, 10
>>> loss = CrossEntropyLoss()
>>> inputs = torch.randn(B, T1, C, requires_grad=True)
>>> targets = torch.empty(B, T2, dtype=torch.long).random_(T2)
>>> outputs = loss(inputs, targets)
>>> outputs.backward()
```

--------------------------------

### LightningLibriSpeechDataModule setup

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/librispeech/lit_data_module.html

Initializes the dataset by splitting audio paths and transcripts into train, validation, and test sets based on predefined counts.

```APIDOC
## setup

### Description
Splits the LibriSpeech dataset into train, valid, and test subsets and initializes the SpeechToTextDataset for each stage.

### Parameters
#### Path Parameters
- **stage** (str) - Optional - The current stage of the data module.
- **tokenizer** (Tokenizer) - Optional - The tokenizer instance used for SOS/EOS ID configuration.
```

--------------------------------

### Initialize MelSpectrogramFeatureTransform

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/melspectrogram/melspectrogram.html

Initializes the MelSpectrogramFeatureTransform with audio configurations. Requires librosa to be installed.

```python
import numpy as np
from omegaconf import DictConfig

from ... import register_audio_feature_transform
from ...audio.melspectrogram.configuration import MelSpectrogramConfigs
from ....utils import LIBROSA_IMPORT_ERROR

@register_audio_feature_transform("melspectrogram", dataclass=MelSpectrogramConfigs)
class MelSpectrogramFeatureTransform(object):
    r"""
    Create MelSpectrogram for a raw audio signal. This is a composition of Spectrogram and MelScale.

    Args:
        configs (DictConfig): configuraion set

    Returns:
        Tensor: A mel-spectrogram feature. The shape is `(seq_length, num_mels)`
    """
    def __init__(self, configs: DictConfig) -> None:
        super(MelSpectrogramFeatureTransform, self).__init__()
        try:
            import librosa
        except ImportError:
            raise ImportError(LIBROSA_IMPORT_ERROR)
        self.sample_rate = configs.audio.sample_rate
        self.num_mels = configs.audio.num_mels
        self.n_fft = int(round(configs.audio.sample_rate * 0.001 * configs.audio.frame_length))
        self.hop_length = int(round(configs.audio.sample_rate * 0.001 * configs.audio.frame_shift))
        self.function = librosa.feature.melspectrogram
        self.power_to_db = librosa.power_to_db
```

--------------------------------

### Train Quartznet with AISHELL-1 on GPU with FP16

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/notes/intro.html

Example of training a quartznet15x5 model using the AISHELL-1 dataset with MFCC features on a GPU with FP16 precision. Requires dataset download and path configuration.

```bash
python ./openspeech_cli/hydra_train.py \
    dataset=aishell \
    dataset.dataset_path=$DATASET_PATH \
    dataset.dataset_download=True \
    dataset.manifest_file_path=$MANIFEST_FILE_PATH \
    tokenizer=aishell_character \
    model=quartznet15x5 \
    audio=mfcc \
    lr_scheduler=warmup_reduce_lr_on_plateau \
    trainer=gpu-fp16 \
    criterion=ctc
```

--------------------------------

### Setup KsponSpeech Datasets

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/ksponspeech/lit_data_module.html

Splits the KsponSpeech dataset into train, valid, and test sets. Initializes SpeechToTextDataset for each stage with appropriate configurations.

```python
def setup(self, stage: Optional[str] = None, tokenizer: Tokenizer = None):
        r"""
        Split `train` and `valid` dataset for training.

        Args:
            stage (str): stage of training. `train` or `valid`
            tokenizer (Tokenizer): tokenizer is in charge of preparing the inputs for a model.

        Returns:
            None
        """
        valid_end_idx = self.KSPONSPEECH_TRAIN_NUM + self.KSPONSPEECH_VALID_NUM

        audio_paths, transcripts = self._parse_manifest_file()
        audio_paths = {
            "train": audio_paths[:self.KSPONSPEECH_TRAIN_NUM],
            "valid": audio_paths[self.KSPONSPEECH_TRAIN_NUM:valid_end_idx],
            "test": audio_paths[valid_end_idx:],
        }
        transcripts = {
            "train": transcripts[:self.KSPONSPEECH_TRAIN_NUM],
            "valid": transcripts[self.KSPONSPEECH_TRAIN_NUM:valid_end_idx],
            "test": transcripts[valid_end_idx:],
        }

        for stage in audio_paths.keys():
            if stage == 'test':
                dataset_path = self.configs.dataset.test_dataset_path
            else:
                dataset_path = self.configs.dataset.dataset_path

            self.dataset[stage] = SpeechToTextDataset(
                configs=self.configs,
                dataset_path=dataset_path,
                audio_paths=audio_paths[stage],
                transcripts=transcripts[stage],
                sos_id=tokenizer.sos_id,
                eos_id=tokenizer.eos_id,
                apply_spec_augment=self.configs.audio.apply_spec_augment if stage == 'train' else False,
                del_silence=self.configs.audio.del_silence if stage == 'train' else False,
            )
```

--------------------------------

### Setup AISHELL-1 Dataset

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/aishell/lit_data_module.html

Splits audio paths and transcripts into train, valid, and test sets and initializes the SpeechToTextDataset.

```python
def setup(self, stage: Optional[str] = None, tokenizer: Tokenizer = None):
    r"""
    Split `train` and `valid` dataset for training.

    Args:
        stage (str): stage of training. `train` or `valid`
        tokenizer (Tokenizer): tokenizer is in charge of preparing the inputs for a model.

    Returns:
        None
    """
    valid_end_idx = self.AISHELL_TRAIN_NUM + self.AISHELL_VALID_NUM
    audio_paths, transcripts = self._parse_manifest_file(self.configs.dataset.manifest_file_path)

    audio_paths = {
        "train": audio_paths[:self.AISHELL_TRAIN_NUM],
        "valid": audio_paths[self.AISHELL_TRAIN_NUM:valid_end_idx],
        "test": audio_paths[valid_end_idx:],
    }
    transcripts = {
        "train": transcripts[:self.AISHELL_TRAIN_NUM],
        "valid": transcripts[self.AISHELL_TRAIN_NUM:valid_end_idx],
        "test": transcripts[valid_end_idx:],
    }

    for stage in audio_paths.keys():
        self.dataset[stage] = SpeechToTextDataset(
            configs=self.configs,
            dataset_path=self.configs.dataset.dataset_path,
            audio_paths=audio_paths[stage],
            transcripts=transcripts[stage],
            sos_id=tokenizer.sos_id,
            eos_id=tokenizer.eos_id,
            apply_spec_augment=self.configs.audio.apply_spec_augment if stage == 'train' else False,
            del_silence=self.configs.audio.del_silence if stage == 'train' else False,
        )
```

--------------------------------

### Acoustic Model Manifest File Format

Source: https://github.com/openspeech-team/openspeech/wiki/Get-Started

Example of the manifest file structure used for acoustic model training, containing audio paths, text transcriptions, and tokenized sequences.

```text
LibriSpeech/test-other/8188/269288/8188-269288-0052.flac        ▁ANNIE ' S ▁MANNER ▁WAS ▁VERY ▁MYSTERIOUS       4039 20 5 531 17 84 2352
LibriSpeech/test-other/8188/269288/8188-269288-0053.flac        ▁ANNIE ▁DID ▁NOT ▁MEAN ▁TO ▁CONFIDE ▁IN ▁ANYONE ▁THAT ▁NIGHT ▁AND ▁THE ▁KIND EST ▁THING ▁WAS ▁TO ▁LEAVE ▁HER ▁A LONE    4039 99 35 251 9 4758 11 2454 16 199 6 4 323 200 255 17 9 370 30 10 492
LibriSpeech/test-other/8188/269288/8188-269288-0054.flac        ▁TIRED ▁OUT ▁LESLIE ▁HER SELF ▁DROPP ED ▁A SLEEP        1493 70 4708 30 115 1231 7 10 1706
LibriSpeech/test-other/8188/269288/8188-269288-0055.flac        ▁ANNIE ▁IS ▁THAT ▁YOU ▁SHE ▁CALL ED ▁OUT        4039 34 16 25 37 208 7 70
LibriSpeech/test-other/8188/269288/8188-269288-0056.flac        ▁THERE ▁WAS ▁NO ▁REPLY ▁BUT ▁THE ▁SOUND ▁OF ▁HURRY ING ▁STEPS ▁CAME ▁QUICK ER ▁AND ▁QUICK ER ▁NOW ▁AND ▁THEN ▁THEY ▁WERE ▁INTERRUPTED ▁BY ▁A ▁GROAN     57 17 56 1368 33 4 489 8 1783 14 1381 133 571 49 6 571 49 82 6 76 45 54 2351 44 10 3154
LibriSpeech/test-other/8188/269288/8188-269288-0057.flac        ▁OH ▁THIS ▁WILL ▁KILL ▁ME ▁MY ▁HEART ▁WILL ▁BREAK ▁THIS ▁WILL ▁KILL ▁ME 299 46 71 669 50 41 235 71 977 46 71 669 50
...
...
```

--------------------------------

### Initialize Tokenizer and Generate Manifests

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/datasets/librispeech/lit_data_module.html

Prepares the tokenizer based on configuration and generates manifest files if they do not exist. Requires appropriate configuration settings for the tokenizer unit and dataset paths.

```python
        tokenizer (Tokenizer): tokenizer is in charge of preparing the inputs for a model.
        """
        if self.configs.tokenizer.unit == 'libri_subword':
            from openspeech.datasets.librispeech.preprocess.subword import generate_manifest_files
        elif self.configs.tokenizer.unit == 'libri_character':
            from openspeech.datasets.librispeech.preprocess.character import generate_manifest_files
        else:
            raise ValueError(f"Unsupported vocabulary unit: {self.configs.tokenizer.unit}")

        if self.configs.dataset.dataset_download:
            self._download_dataset()

        if not os.path.exists(self.configs.dataset.manifest_file_path):
            self.logger.info("Manifest file is not exists !!\n"
                             "Generate manifest files..")

            if hasattr(self.configs.tokenizer, "vocab_size"):
                generate_manifest_files(
                    dataset_path=self.configs.dataset.dataset_path,
                    manifest_file_path=self.configs.dataset.manifest_file_path,
                    vocab_path=self.configs.tokenizer.vocab_path,
                    vocab_size=self.configs.tokenizer.vocab_size,
                )
            else:
                generate_manifest_files(
                    dataset_path=self.configs.dataset.dataset_path,
                    manifest_file_path=self.configs.dataset.manifest_file_path,
                    vocab_path=self.configs.tokenizer.vocab_path,
                )

        return TOKENIZER_REGISTRY[self.configs.tokenizer.unit](self.configs)
```

--------------------------------

### OpenspeechEncoderDecoderModel Initialization

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/models/openspeech_encoder_decoder_model.html

Initializes the OpenspeechEncoderDecoderModel with configuration, tokenizer, and sets up the criterion.

```APIDOC
## OpenspeechEncoderDecoderModel

### Description
Base class for OpenSpeech's encoder-decoder models.

### Method
__init__

### Parameters
#### Arguments
- **configs** (DictConfig) - configuration set.
- **tokenizer** (Tokenizer) - tokenizer is in charge of preparing the inputs for a model.

### Returns
None
```

--------------------------------

### Initialize and Build ContextNetTransducerModel

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/models/contextnet/model.html

Initializes the model with configuration and tokenizer, then constructs the encoder and decoder components using the provided configuration parameters.

```python
    def __init__(self, configs: DictConfig, tokenizer: Tokenizer, ) -> None:
        super(ContextNetTransducerModel, self).__init__(configs, tokenizer)

    def build_model(self):
        self.encoder = ContextNetEncoder(
            num_classes=self.num_classes,
            model_size=self.configs.model.model_size,
            input_dim=self.configs.audio.num_mels,
            num_layers=self.configs.model.num_encoder_layers,
            kernel_size=self.configs.model.kernel_size,
            num_channels=self.configs.model.num_channels,
            output_dim=self.configs.model.encoder_dim,
            joint_ctc_attention=False,
        )
        self.decoder = RNNTransducerDecoder(
            num_classes=self.num_classes,
            hidden_state_dim=self.configs.model.decoder_hidden_state_dim,
            output_dim=self.configs.model.decoder_output_dim,
            num_layers=self.configs.model.num_decoder_layers,
            rnn_type=self.configs.model.rnn_type,
            pad_id=self.tokenizer.pad_id,
            sos_id=self.tokenizer.sos_id,
            eos_id=self.tokenizer.eos_id,
            dropout_p=self.configs.model.decoder_dropout_p,
        )
```

--------------------------------

### LibriSpeech Manifest File Format Example

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/notes/intro.html

This is an example of the manifest file format for the LibriSpeech dataset. Each line contains a file path, followed by transcribed text and numerical features.

```text
LibriSpeech/test-other/8188/269288/8188-269288-0052.flac        ▁ANNIE ' S ▁MANNER ▁WAS ▁VERY ▁MYSTERIOUS       4039 20 5 531 17 84 2352
LibriSpeech/test-other/8188/269288/8188-269288-0053.flac        ▁ANNIE ▁DID ▁NOT ▁MEAN ▁TO ▁CONFIDE ▁IN ▁ANYONE ▁THAT ▁NIGHT ▁AND ▁THE ▁KIND EST ▁THING ▁WAS ▁TO ▁LEAVE ▁HER ▁A LONE    4039 99 35 251 9 4758 11 2454 16 199 6 4 323 200 255 17 9 370 30 10 492
LibriSpeech/test-other/8188/269288/8188-269288-0054.flac        ▁TIRED ▁OUT ▁LESLIE ▁HER SELF ▁DROPP ED ▁A SLEEP        1493 70 4708 30 115 1231 7 10 1706
LibriSpeech/test-other/8188/269288/8188-269288-0055.flac        ▁ANNIE ▁IS ▁THAT ▁YOU ▁SHE ▁CALL ED ▁OUT        4039 34 16 25 37 208 7 70
LibriSpeech/test-other/8188/269288/8188-269288-0056.flac        ▁THERE ▁WAS ▁NO ▁REPLY ▁BUT ▁THE ▁SOUND ▁OF ▁HURRY ING ▁STEPS ▁CAME ▁QUICK ER ▁AND ▁QUICK ER ▁NOW ▁AND ▁THEN ▁THEY ▁WERE ▁INTERRUPTED ▁BY ▁A ▁GROAN     57 17 56 1368 33 4 489 8 1783 14 1381 133 571 49 6 571 49 82 6 76 45 54 2351 44 10 3154
LibriSpeech/test-other/8188/269288/8188-269288-0057.flac        ▁OH ▁THIS ▁WILL ▁KILL ▁ME ▁MY ▁HEART ▁WILL ▁BREAK ▁THIS ▁WILL ▁KILL ▁ME 299 46 71 669 50 41 235 71 977 46 71 669 50
```

--------------------------------

### val_dataloader() - Multiple DataLoaders

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Example of returning multiple validation dataloaders.

```APIDOC
## POST /val_dataloader_list

### Description
Provides multiple PyTorch DataLoaders for validation.

### Method
POST

### Endpoint
/val_dataloader_list

### Parameters
#### Request Body
- **batch_size** (int) - Required - The batch size for the DataLoaders.

### Request Example
```json
{
  "batch_size": 32
}
```

### Response
#### Success Response (200)
- **loaders** (list) - A list containing multiple validation DataLoaders.

#### Response Example
```json
{
  "loaders": [
    "<DataLoader object A>",
    "<DataLoader object B>",
    "<DataLoader object N>"
  ]
}
```
```

--------------------------------

### Initialize Dataset with Augmentations

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/dataset.html

Initializes the dataset by duplicating audio paths, transcripts, and applying specified augmentations like time stretching and audio joining. Shuffles the dataset after augmentation.

```python
for idx in range(self.dataset_size):
                self.audio_paths.append(self.audio_paths[idx])
                self.transcripts.append(self.transcripts[idx])
                self.augments.append(self.TIME_STRETCH)

        if self.apply_joining_augment:
            self._joining_augment = JoiningAugment()
            for idx in range(self.dataset_size):
                self.audio_paths.append(self.audio_paths[idx])
                self.transcripts.append(self.transcripts[idx])
                self.augments.append(self.AUDIO_JOINING)

        self.total_size = len(self.audio_paths)

        tmp = list(zip(self.audio_paths, self.transcripts, self.augments))
        random.shuffle(tmp)
        self.audio_paths, self.transcripts, self.augments = zip(*tmp)
```

--------------------------------

### train_dataloader() - Dictionary of DataLoaders

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Example of returning a dictionary of training dataloaders (MNIST and CIFAR).

```APIDOC
## POST /train_dataloader_dict

### Description
Provides a dictionary of PyTorch DataLoaders for training, mapping dataset names to DataLoaders.

### Method
POST

### Endpoint
/train_dataloader_dict

### Parameters
#### Request Body
- **batch_size** (int) - Required - The batch size for the DataLoaders.

### Request Example
```json
{
  "batch_size": 32
}
```

### Response
#### Success Response (200)
- **loaders** (dict) - A dictionary mapping dataset names to their respective training DataLoaders (e.g., {'mnist': batch_mnist, 'cifar': batch_cifar}).

#### Response Example
```json
{
  "loaders": {
    "mnist": "<DataLoader object MNIST>",
    "cifar": "<DataLoader object CIFAR>"
  }
}
```
```

--------------------------------

### ContextNetTransducerModel Initialization

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/architectures/ContextNet.html

Initializes the ContextNetTransducerModel with configuration and tokenizer.

```APIDOC
## POST /openspeech/models/contextnet/model/ContextNetTransducerModel

### Description
Initializes the ContextNetTransducerModel with configuration and tokenizer.

### Method
POST

### Endpoint
/openspeech/models/contextnet/model/ContextNetTransducerModel

### Parameters
#### Request Body
- **configs** (omegaconf.dictconfig.DictConfig) - Required - Configuration set for the model.
- **tokenizer** (openspeech.tokenizers.tokenizer.Tokenizer) - Required - The tokenizer to be used by the model.

### Response
#### Success Response (200)
- **ContextNetTransducerModel** - An instance of the ContextNetTransducerModel.

#### Response Example
```json
{
  "ContextNetTransducerModel": "Instance of ContextNetTransducerModel"
}
```
```

--------------------------------

### train_dataloader() - List of DataLoaders

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Example of returning a list of training dataloaders (MNIST and CIFAR).

```APIDOC
## POST /train_dataloader_list

### Description
Provides a list of PyTorch DataLoaders for training, combining multiple datasets.

### Method
POST

### Endpoint
/train_dataloader_list

### Parameters
#### Request Body
- **batch_size** (int) - Required - The batch size for the DataLoaders.

### Request Example
```json
{
  "batch_size": 32
}
```

### Response
#### Success Response (200)
- **loaders** (list) - A list containing multiple training DataLoaders (e.g., [batch_mnist, batch_cifar]).

#### Response Example
```json
{
  "loaders": [
    "<DataLoader object MNIST>",
    "<DataLoader object CIFAR>"
  ]
}
```
```

--------------------------------

### QuartzNet Model Initialization and Inference

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/architectures/QuartzNet.html

Documentation for initializing and running inference on QuartzNet model variants.

```APIDOC
## QuartzNet Model Classes

### Description
Initializes the QuartzNet10x5Model or QuartzNet15x5Model for speech recognition tasks using 1D time-channel separable convolutions.

### Parameters
- **configs** (DictConfig) - Required - The configuration set for the model.
- **tokenizer** (Tokenizer) - Required - The tokenizer responsible for preparing model inputs.

### Inputs
- **inputs** (torch.FloatTensor) - Required - Padded input sequence of size (batch, seq_length, dimension).
- **input_lengths** (torch.LongTensor) - Required - The length of the input tensor (batch).

### Response
- **outputs** (dict) - Contains model predictions including y_hats, logits, and output_lengths.
```

--------------------------------

### val_dataloader() - Single DataLoader

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Example of implementing a single validation dataloader using MNIST dataset.

```APIDOC
## POST /val_dataloader

### Description
Provides a single PyTorch DataLoader for validation.

### Method
POST

### Endpoint
/val_dataloader

### Parameters
#### Request Body
- **batch_size** (int) - Required - The batch size for the DataLoader.

### Request Example
```json
{
  "batch_size": 32
}
```

### Response
#### Success Response (200)
- **loader** (torch.utils.data.DataLoader) - The configured validation DataLoader.

#### Response Example
```json
{
  "loader": "<DataLoader object>"
}
```
```

--------------------------------

### train_dataloader() - Single DataLoader

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/corpus/AISHELL-1.html

Example of implementing a single training dataloader using MNIST dataset.

```APIDOC
## POST /train_dataloader

### Description
Provides a single PyTorch DataLoader for training.

### Method
POST

### Endpoint
/train_dataloader

### Parameters
#### Request Body
- **batch_size** (int) - Required - The batch size for the DataLoader.

### Request Example
```json
{
  "batch_size": 32
}
```

### Response
#### Success Response (200)
- **loader** (torch.utils.data.DataLoader) - The configured training DataLoader.

#### Response Example
```json
{
  "loader": "<DataLoader object>"
}
```
```

--------------------------------

### Initialize and Calculate CTCLoss

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/criterion/ctc/ctc.html

Demonstrates the initialization of the CTCLoss module and the calculation of loss using padded target sequences.

```python
>>> # Target are to be padded
>>> T = 50      # Input sequence length
>>> C = 20      # Number of classes (including blank)
>>> N = 16      # Batch size
>>> S = 30      # Target sequence length of longest target in batch (padding length)
>>> S_min = 10  # Minimum target length, for demonstration purposes
>>>
>>> # Initialize random batch of input vectors, for *size = (T,N,C)
>>> input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
>>>
>>> # Initialize random batch of targets (0 = blank, 1:C = classes)
>>> target = torch.randint(low=1, high=C, size=(N, S), dtype=torch.long)
>>>
>>> input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
>>> target_lengths = torch.randint(low=S_min, high=S, size=(N,), dtype=torch.long)
>>> ctc_loss = nn.CTCLoss()
>>> loss = ctc_loss(input, target, input_lengths, target_lengths)
>>> loss.backward()
>>>
>>>
>>> # Target are to be un-padded
>>> T = 50      # Input sequence length
```

--------------------------------

### Get Item

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/dataset.html

Retrieves a pair of audio features and their corresponding transcript tokens for a given index.

```APIDOC
## Get Item

### Description
Provides a pair of processed audio features and their corresponding transcript tokens for a given index in the dataset. Handles audio joining augmentation if specified for the index.

### Method
`__getitem__(self, idx) -> tuple`

### Parameters
- `idx` (int): The index of the item to retrieve.

### Returns
- `tuple`: A tuple containing:
    - `feature` (Tensor): The processed audio features.
    - `transcript` (list): The list of token IDs for the transcript.
```

--------------------------------

### POST /encoders/conformer/initialize

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/modules/Encoders.html

Initializes a new ConformerEncoder instance with specified architecture parameters.

```APIDOC
## POST /encoders/conformer/initialize

### Description
Initializes the ConformerEncoder, which combines convolution neural networks and transformers to model local and global dependencies of an audio sequence.

### Method
POST

### Parameters
#### Request Body
- **num_classes** (int) - Required - Number of classification
- **input_dim** (int) - Optional - Dimension of input vector
- **encoder_dim** (int) - Optional - Dimension of conformer encoders
- **num_layers** (int) - Optional - Number of conformer blocks
- **num_attention_heads** (int) - Optional - Number of attention heads
- **feed_forward_expansion_factor** (int) - Optional - Expansion factor of feed forward module
- **conv_expansion_factor** (int) - Optional - Expansion factor of conformer convolution module
- **feed_forward_dropout_p** (float) - Optional - Probability of feed forward module dropout
- **attention_dropout_p** (float) - Optional - Probability of attention module dropout
- **conv_dropout_p** (float) - Optional - Probability of conformer convolution module dropout
- **conv_kernel_size** (int/tuple) - Optional - Size of the convolving kernel
- **half_step_residual** (bool) - Required - Flag indicating whether to use half step residual
- **joint_ctc_attention** (bool) - Optional - Flag indicating joint ctc attention
```

--------------------------------

### Language Model Data Format

Source: https://github.com/openspeech-team/openspeech/wiki/Get-Started

Example of the required text data format for language model training.

```text
openspeech is a framework for making end-to-end speech recognizers.
end to end automatic speech recognition is an emerging paradigm in the field of neural network-based speech recognition that offers multiple benefits.
because of these advantages, many end-to-end speech recognition related open sources have emerged.
...
...
```

--------------------------------

### Initialize OpenspeechTransducerModel

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/models/openspeech_transducer_model.html

Initializes the base transducer model, setting up the encoder, decoder, and a final linear layer for classification. Requires configuration and a tokenizer.

```python
def __init__(self, configs: DictConfig, tokenizer: Tokenizer) -> None:
    super(OpenspeechTransducerModel, self).__init__(configs, tokenizer)
    self.encoder = None
    self.decoder = None
    self.decode = self.greedy_decode

    if hasattr(self.configs.model, "encoder_dim"):
        in_features = self.configs.model.encoder_dim + self.configs.model.decoder_output_dim
    elif hasattr(self.configs.model, "output_dim"):
        in_features = self.configs.model.output_dim << 1
    else:
        raise ValueError("Transducer model must be contain `encoder_dim` or `encoder_hidden_state_dim` config.")

    self.fc = nn.Sequential(
        Linear(in_features=in_features, out_features=in_features),
        nn.Tanh(),
        Linear(in_features=in_features, out_features=self.num_classes),
    )
```

--------------------------------

### Get Hypothesis from Beam Search

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/search/beam_search_base.html

Retrieves the final predicted sequences (hypotheses) from the beam search process.

```python
def _get_hypothesis(self):
        predictions = list()

        for batch_idx, batch in enumerate(self.finished):
            # if there is no terminated sentences, bring ongoing sentence which has the highest probability instead
            if len(batch) == 0:
                prob_batch = self.cumulative_ps[batch_idx]
                top_beam_idx = int(prob_batch.topk(1)[1])
                predictions.append(self.ongoing_beams[batch_idx, top_beam_idx])

            # bring highest probability sentence
            else:
                top_beam_idx = int(torch.FloatTensor(self.finished_ps[batch_idx]).topk(1)[1])
                predictions.append(self.finished[batch_idx][top_beam_idx])

        predictions = self._fill_sequence(predictions)
        return predictions
```

--------------------------------

### Get Current Learning Rate

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/models/openspeech_model.html

Retrieves the current learning rate from the optimizer's parameter groups.

```python
def get_lr(self):
        for g in self.optimizer.param_groups:
            return g['lr']
```

--------------------------------

### OpenspeechCTCModel Initialization

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/models/openspeech_ctc_model.html

Initializes the OpenspeechCTCModel with configurations and a tokenizer.

```APIDOC
## OpenspeechCTCModel Initialization

### Description
Initializes the base class for OpenSpeech's encoder-only CTC models.

### Method
__init__

### Parameters
- **configs** (DictConfig) - Required - Configuration set for the model.
- **tokenizer** (Tokenizer) - Required - Tokenizer used for preparing model inputs.
```

--------------------------------

### Initialize and use ConformerEncoder

Source: https://context7.com/openspeech-team/openspeech/llms.txt

Configure the Conformer encoder with specific hyperparameters and perform a forward pass with audio features.

```python
import torch
from openspeech.encoders import ConformerEncoder

# Initialize Conformer encoder
encoder = ConformerEncoder(
    num_classes=5000,           # Vocabulary size
    input_dim=80,               # Number of mel filterbanks
    encoder_dim=512,            # Encoder hidden dimension
    num_layers=17,              # Number of conformer blocks
    num_attention_heads=8,      # Multi-head attention heads
    feed_forward_expansion_factor=4,
    conv_expansion_factor=2,
    input_dropout_p=0.1,
    feed_forward_dropout_p=0.1,
    attention_dropout_p=0.1,
    conv_dropout_p=0.1,
    conv_kernel_size=31,
    half_step_residual=True,
    joint_ctc_attention=True,
)

# Forward pass with audio features
# inputs: (batch, time, features) - e.g., mel spectrogram
# input_lengths: (batch,) - actual lengths before padding
inputs = torch.randn(4, 500, 80)  # Batch of 4, 500 frames, 80 mel bins
input_lengths = torch.tensor([500, 450, 400, 350])

# Returns: (encoder_outputs, encoder_logits, output_lengths)
outputs, logits, output_lengths = encoder(inputs, input_lengths)
# outputs: (batch, time/4, encoder_dim) - subsampled sequence
# logits: (batch, time/4, num_classes) - CTC logits if joint_ctc_attention=True
# output_lengths: (batch,) - output sequence lengths
print(f"Output shape: {outputs.shape}")  # torch.Size([4, 125, 512])
```

--------------------------------

### Model Prediction API

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/modules/Search.html

This endpoint is used to get predictions from ensemble models. It requires input lengths to be provided.

```APIDOC
## POST /predict

### Description
This endpoint takes input lengths and returns predictions from ensemble models.

### Method
POST

### Endpoint
/predict

### Parameters
#### Request Body
- **input_lengths** (torch.LongTensor) - Required - The length of the input tensor.

### Request Example
```json
{
  "input_lengths": [100, 150, 120]
}
```

### Response
#### Success Response (200)
- **predictions** (torch.LongTensor) - The prediction of ensemble models.

#### Response Example
```json
{
  "predictions": [1, 0, 1]
}
```
```

--------------------------------

### Get Successor in Beam Search

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/search/beam_search_base.html

Calculates and updates successor states for beam search, handling end-of-sentence tokens.

```python
def _get_successor(
            self,
            current_ps: torch.Tensor,
            current_vs: torch.Tensor,
            finished_ids: tuple,
            num_successor: int,
            eos_count: int,
            k: int
    ) -> int:
        finished_batch_idx, finished_idx = finished_ids

        successor_ids = current_ps.topk(k + num_successor)[1]
        successor_idx = successor_ids[finished_batch_idx, -1]

        successor_p = current_ps[finished_batch_idx, successor_idx]
        successor_v = current_vs[finished_batch_idx, successor_idx]

        prev_status_idx = (successor_idx // k)
        prev_status = self.ongoing_beams[finished_batch_idx, prev_status_idx]
        prev_status = prev_status.view(-1)[:-1]

        successor = torch.cat([prev_status, successor_v.view(1)])

        if int(successor_v) == self.eos_id:
            self.finished[finished_batch_idx].append(successor)
            self.finished_ps[finished_batch_idx].append(successor_p)
            eos_count = self._get_successor(
                current_ps=current_ps,
                current_vs=current_vs,
                finished_ids=finished_ids,
                num_successor=num_successor + eos_count,
                eos_count=eos_count + 1,
                k=k,
            )

        else:
            self.ongoing_beams[finished_batch_idx, finished_idx] = successor
            self.cumulative_ps[finished_batch_idx, finished_idx] = successor_p

        return eos_count
```

--------------------------------

### Dataset Initialization and Augmentation

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/dataset.html

Details the initialization of the dataset, including applying time stretch and audio joining augmentations, and shuffling the data.

```APIDOC
## Dataset Initialization and Augmentation

### Description
Initializes the dataset by loading audio paths and transcripts, applying specified augmentations (time stretch, audio joining), and shuffling the dataset.

### Method
`__init__` (constructor)

### Parameters
- `dataset_path` (str): Path to the dataset directory.
- `sample_rate` (int): The sample rate for audio processing.
- `del_silence` (bool): Whether to remove silence from audio.
- `apply_joining_augment` (bool): Whether to apply audio joining augmentation.
- `sos_id` (int): ID for the start-of-sequence token.
- `eos_id` (int): ID for the end-of-sequence token.
- `num_mels` (int): Number of mel filterbanks for feature extraction.

### Augmentations Defined
- `TIME_STRETCH`: Constant for time stretch augmentation.
- `AUDIO_JOINING`: Constant for audio joining augmentation.
- `NOISE_AUGMENT`: Constant for noise augmentation.
- `SPEC_AUGMENT`: Constant for spectral augmentation.

### Internal Methods
- `_load_audio`: Loads an audio file.
- `_joining_augment`: Augmentation for joining audio segments.
- `_time_stretch_augment`: Augmentation for time stretching audio.
- `_noise_injector`: Augmentation for adding noise.
- `_spec_augment`: Augmentation for spectral modification.
- `transforms`: Callable for audio signal transformations.
```

--------------------------------

### Get Dataset Length

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/dataset.html

Returns the total number of audio paths in the dataset, representing the dataset's size.

```python
def __len__(self):
        return len(self.audio_paths)
```

--------------------------------

### Transcript Parsing

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/data/audio/dataset.html

Processes raw transcript strings into a list of token IDs, including start and end tokens.

```APIDOC
## Parse Transcript

### Description
Parses a transcript string into a list of integer token IDs, adding start-of-sequence (SOS) and end-of-sequence (EOS) tokens.

### Method
`_parse_transcript(transcript: str) -> list`

### Parameters
- `transcript` (str): The raw transcript string, with words separated by spaces.

### Returns
- `transcript` (list): A list of integer token IDs, including `sos_id` and `eos_id`.
```

--------------------------------

### ConformerTransducerModel Initialization

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/architectures/Conformer.html

Initializes the ConformerTransducer model.

```APIDOC
## POST /conformer_transducer/initialize

### Description
Initializes the ConformerTransducer model with given configurations and tokenizer.

### Method
POST

### Endpoint
/conformer_transducer/initialize

### Parameters
#### Request Body
- **configs** (omegaconf.dictconfig.DictConfig) - Required - Configuration set for the model.
- **tokenizer** (openspeech.tokenizers.tokenizer.Tokenizer) - Required - Tokenizer to be used with the model.

### Response
#### Success Response (200)
- **message** (string) - Initialization successful message.

#### Response Example
```json
{
  "message": "ConformerTransducerModel initialized successfully."
}
```
```

--------------------------------

### Initialize ConformerEncoder

Source: https://github.com/openspeech-team/openspeech/blob/main/docs/_modules/openspeech/encoders/conformer_encoder.html

Initializes the ConformerEncoder with specified parameters for audio sequence modeling. Use this to set up the encoder for tasks like speech recognition.

```python
import torch
import torch.nn as nn
from typing import Tuple

from openspeech.encoders.openspeech_encoder import OpenspeechEncoder
from openspeech.modules import Conv2dSubsampling, Linear, ConformerBlock, Transpose

class ConformerEncoder(OpenspeechEncoder):
    r"""
    Transformer models are good at capturing content-based global interactions, while CNNs exploit local features
    effectively.  Conformer achieves the best of both worlds by studying how to combine convolution neural
    networks and transformers to model both local and global dependencies of an audio sequence
    in a parameter-efficient way.

    Args:
        num_classes (int): Number of classification
        input_dim (int, optional): Dimension of input vector
        encoder_dim (int, optional): Dimension of conformer encoders
        num_layers (int, optional): Number of conformer blocks
        num_attention_heads (int, optional): Number of attention heads
        feed_forward_expansion_factor (int, optional): Expansion factor of feed forward module
        conv_expansion_factor (int, optional): Expansion factor of conformer convolution module
        feed_forward_dropout_p (float, optional): Probability of feed forward module dropout
        attention_dropout_p (float, optional): Probability of attention module dropout
        conv_dropout_p (float, optional): Probability of conformer convolution module dropout
        conv_kernel_size (int or tuple, optional): Size of the convolving kernel
        half_step_residual (bool): Flag indication whether to use half step residual or not
            joint_ctc_attention (bool, optional): flag indication joint ctc attention or not

    Inputs: inputs, input_lengths
        - **inputs** (batch, time, dim): Tensor containing input vector
        - **input_lengths** (batch): list of sequence input lengths

    Returns: outputs, output_lengths
        - **outputs** (batch, out_channels, time): Tensor produces by conformer encoders.
        - **output_lengths** (batch): list of sequence output lengths

    Reference:
        Anmol Gulati et al: Conformer: Convolution-augmented Transformer for Speech Recognition
        https://arxiv.org/abs/2005.08100
    """
    def __init__(
            self,
            num_classes: int,
            input_dim: int = 80,
            encoder_dim: int = 512,
            num_layers: int = 17,
            num_attention_heads: int = 8,
            feed_forward_expansion_factor: int = 4,
            conv_expansion_factor: int = 2,
            input_dropout_p: float = 0.1,
            feed_forward_dropout_p: float = 0.1,
            attention_dropout_p: float = 0.1,
            conv_dropout_p: float = 0.1,
            conv_kernel_size: int = 31,
            half_step_residual: bool = True,
            joint_ctc_attention: bool = True,
    ) -> None:
        super(ConformerEncoder, self).__init__()
        self.joint_ctc_attention = joint_ctc_attention
        self.conv_subsample = Conv2dSubsampling(input_dim, in_channels=1, out_channels=encoder_dim)
        self.input_projection = nn.Sequential(
            Linear(self.conv_subsample.get_output_dim(), encoder_dim),
            nn.Dropout(p=input_dropout_p),
        )
        self.layers = nn.ModuleList([
            ConformerBlock(
                encoder_dim=encoder_dim,
                num_attention_heads=num_attention_heads,
                feed_forward_expansion_factor=feed_forward_expansion_factor,
                conv_expansion_factor=conv_expansion_factor,
                feed_forward_dropout_p=feed_forward_dropout_p,
                attention_dropout_p=attention_dropout_p,
                conv_dropout_p=conv_dropout_p,
                conv_kernel_size=conv_kernel_size,
                half_step_residual=half_step_residual,
            ) for _ in range(num_layers)
        ])
        if self.joint_ctc_attention:
            self.fc = nn.Sequential(
                Transpose(shape=(1, 2)),
                nn.Dropout(feed_forward_dropout_p),
                Linear(encoder_dim, num_classes, bias=False),
            )
```