RhythmMamba

https://github.com/zizheng-guo/rhythmmamba
Admin
RhythmMamba is a fast, lightweight, and accurate deep learning framework for remote physiological...
Tokens:2,657
Snippets:31
Trust Score:5.8
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Setup Environment

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/README.md

Commands to initialize the environment and install dependencies.

```bash
bash setup.sh
```

```bash
conda activate RhythmMamba
```

```bash
pip install -r requirements.txt
```

--------------------------------

### Initialize and run Mamba block

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Example of initializing the Mamba module and performing a forward pass with random input data.

```python
import torch
from mamba_ssm import Mamba

batch, length, dim = 2, 64, 16
x = torch.randn(batch, length, dim).to("cuda")
model = Mamba(
    # This module uses roughly 3 * expand * d_model^2 parameters
    d_model=dim, # Model dimension d_model
    d_state=16,  # SSM state expansion factor
    d_conv=4,    # Local convolution width
    expand=2,    # Block expansion factor
).to("cuda")
y = model(x)
assert y.shape == x.shape
```

--------------------------------

### Install Mamba dependencies

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Commands to install the required causal convolution layer and the core Mamba package.

```bash
pip install causal-conv1d>=1.2.0
```

```bash
pip install mamba-ssm
```

--------------------------------

### Train Models

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/README.md

Commands to initiate training for intra-dataset and cross-dataset configurations.

```bash
python main.py --config_file ./configs/train_configs/intra/0MMPD_RHYTHMMAMBA.yaml
```

```bash
python main.py --config_file ./configs/train_configs/cross/PURE_UBFC-rPPG_RHYTHMMAMBA.yaml
```

--------------------------------

### Run Inference with Pre-trained Models

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/README.md

Execute inference using a specific configuration file.

```bash
python main.py --config_file ./configs/infer_configs/MMPD_RHYTHMMAMBA.yaml
```

--------------------------------

### Run zero-shot evaluations

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Commands to execute zero-shot evaluations for Mamba and Hugging Face models using the lm-evaluation-harness framework.

```bash
python evals/lm_harness_eval.py --model mamba --model_args pretrained=state-spaces/mamba-130m --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande --device cuda --batch_size 64
python evals/lm_harness_eval.py --model hf --model_args pretrained=EleutherAI/pythia-160m --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande --device cuda --batch_size 64
```

```bash
python evals/lm_harness_eval.py --model mamba --model_args pretrained=state-spaces/mamba-2.8b-slimpj --tasks boolq,piqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa,race,truthfulqa_mc2 --device cuda --batch_size 64
python evals/lm_harness_eval.py --model mamba --model_args pretrained=state-spaces/mamba-2.8b-slimpj --tasks mmlu --num_fewshot 5 --device cuda --batch_size 64
```

--------------------------------

### Benchmark generation throughput

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Test model throughput using random prompts with larger batch sizes.

```bash
python benchmarks/benchmark_generation_mamba_simple.py --model-name "state-spaces/mamba-2.8b" --batch 128
python benchmarks/benchmark_generation_mamba_simple.py --model-name "EleutherAI/pythia-2.8b" --batch 128
```

--------------------------------

### Benchmark generation latency

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Run inference benchmarks with specific sampling strategies like top-p, temperature, and repetition penalty.

```bash
python benchmarks/benchmark_generation_mamba_simple.py --model-name "state-spaces/mamba-2.8b" --prompt "My cat wrote all this CUDA code for a new language model and" --topp 0.9 --temperature 0.7 --repetition-penalty 1.2
python benchmarks/benchmark_generation_mamba_simple.py --model-name "EleutherAI/pythia-2.8b" --prompt "My cat wrote all this CUDA code for a new language model and" --topp 0.9 --temperature 0.7 --repetition-penalty 1.2
python benchmarks/benchmark_generation_mamba_simple.py --model-name "state-spaces/mamba-2.8b" --prompt "My cat wrote all this CUDA code for a new language model and" --minp 0.05 --topk 0 --temperature 0.7 --repetition-penalty 1.2
```

--------------------------------

### Import Causal Conv1d Function

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/causal-conv1d-main/README.md

Import the causal_conv1d_fn function to use the CUDA-accelerated convolution.

```python
from causal_conv1d import causal_conv1d_fn
```

--------------------------------

### Citation BibTeX

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/README.md

BibTeX entries for citing the rPPG-Toolbox and RhythmMamba.

```bibtex
@article{liu2024rppg,
  title={rppg-toolbox: Deep remote ppg toolbox},
  author={Liu, Xin and Narayanswamy, Girish and Paruchuri, Akshay and Zhang, Xiaoyu and Tang, Jiankai and Zhang, Yuzhe and Sengupta, Roni and Patel, Shwetak and Wang, Yuntao and McDuff, Daniel},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}
```

```bibtex
@article{zou2024rhythmmamba,
  title={Rhythmmamba: Fast remote physiological measurement with arbitrary length videos},
  author={Zou, Bochao and Guo, Zizheng and Hu, Xiaocheng and Ma, Huimin},
  journal={arXiv preprint arXiv:2404.06483},
  year={2024}
}
```

--------------------------------

### Model Architecture Cost Calculation Variants

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/evaluation/cost_cal.md

Comparison between the original implementation and the optimized approach for calculating computational costs in models like DeepPhys, TSCAN, and EfficientPhys.

```python
elif img_size == 128: 
    self.final_dense_1 = nn.Linear(57600, self.nb_dense, bias=True)
```

```python
elif img_size == 128: 
    self.avg_pooling_3 = nn.AvgPool2d((4,4))
    self.final_dense_1 = nn.Linear(14400, self.nb_dense, bias=True)
```

--------------------------------

### Equivalent PyTorch Functional Implementation

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/causal-conv1d-main/README.md

Shows the standard PyTorch functional equivalent for the causal convolution operation.

```python
import torch.nn.functional as F

F.conv1d(x, weight.unsqueeze(1), bias, padding=width - 1, groups=dim)[..., :seqlen]
```

--------------------------------

### Causal Conv1d Function Signature

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/causal-conv1d-main/README.md

Defines the input parameters and expected shapes for the causal convolution function.

```python
def causal_conv1d_fn(x, weight, bias=None, activation=None):
    """
    x: (batch, dim, seqlen)
    weight: (dim, width)
    bias: (dim,)
    activation: either None or "silu" or "swish"

    out: (batch, dim, seqlen)
    """
```

--------------------------------

### Cite Mamba BibTeX

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/mamba/README.md

Use this BibTeX entry to cite the Mamba paper in academic publications.

```bibtex
@article{mamba,
  title={Mamba: Linear-Time Sequence Modeling with Selective State Spaces},
  author={Gu, Albert and Dao, Tri},
  journal={arXiv preprint arXiv:2312.00752},
  year={2023}
}
```

--------------------------------

### causal_conv1d_fn

Source: https://github.com/zizheng-guo/rhythmmamba/blob/main/setup/causal-conv1d-main/README.md

Performs a causal depthwise convolution on the input tensor using CUDA acceleration.

```APIDOC
## causal_conv1d_fn

### Description
Performs a causal depthwise convolution operation. This function is optimized for CUDA and supports fp32, fp16, and bf16 data types with kernel sizes of 2, 3, or 4.

### Signature
`causal_conv1d_fn(x, weight, bias=None, activation=None)`

### Parameters
- **x** (Tensor) - Input tensor of shape (batch, dim, seqlen).
- **weight** (Tensor) - Weight tensor of shape (dim, width).
- **bias** (Tensor, optional) - Bias tensor of shape (dim,).
- **activation** (str, optional) - Activation function to apply. Supported values: "silu", "swish", or None.

### Returns
- **out** (Tensor) - The output tensor of shape (batch, dim, seqlen).
```
Tokens:
Raw