TabDDPM

https://github.com/yandex-research/tab-ddpm
Admin
TabDDPM is a diffusion-based deep learning framework designed to generate realistic synthetic...
Tokens:4,387
Snippets:64
Trust Score:9.2
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Setup Environment

Source: https://github.com/yandex-research/tab-ddpm/blob/main/README.md

Commands to create and configure the Conda environment for TabDDPM.

```bash
export REPO_DIR=/path/to/the/code
cd $REPO_DIR

conda create -n tddpm python=3.9.7
conda activate tddpm

pip install torch==1.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

# if the following commands do not succeed, update conda
conda env config vars set PYTHONPATH=${PYTHONPATH}:${REPO_DIR}
conda env config vars set PROJECT_DIR=${REPO_DIR}

conda deactivate
conda activate tddpm
```

--------------------------------

### Set up local development environment

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Commands to create a virtual environment and install the package for development.

```shell
$ mkvirtualenv CTGAN
$ cd CTGAN/
$ make install-develop
```

--------------------------------

### Run TabDDPM Tuning

Source: https://github.com/yandex-research/tab-ddpm/blob/main/README.md

Template and example for tuning TabDDPM hyperparameters.

```bash
python scripts/tune_ddpm.py [ds_name] [train_size] synthetic [catboost|mlp] [exp_name] --eval_seeds
python scripts/tune_ddpm.py churn2 6500 synthetic catboost ddpm_tune --eval_seeds
```

--------------------------------

### Run TabDDPM Pipeline

Source: https://github.com/yandex-research/tab-ddpm/blob/main/README.md

Template and example for executing the training, sampling, and evaluation pipeline.

```bash
python scripts/pipeline.py --config [path_to_your_config] --train --sample --eval
python scripts/pipeline.py --config exp/churn2/ddpm_cb_best/config.toml --train --sample
```

--------------------------------

### Install CTGAN via pip

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/README.md

Use this command to install the standalone CTGAN library using the pip package manager.

```bash
pip install ctgan
```

--------------------------------

### TabDDPM Configuration Example

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CONFIG_DESCRIPTION.md

A complete TOML configuration file defining model parameters, diffusion settings, training hyperparameters, and preprocessing policies.

```toml
seed = 0
parent_dir = "exp/abalone/check"
real_data_path = "data/abalone/"
model_type = "mlp"
num_numerical_features = 7
device = "cuda:0"

[model_params]
is_y_cond = false
d_in = 11
num_classes = 0

[model_params.rtdl_params]
d_layers = [
    256,
    256,
]
dropout = 0.0

[diffusion_params]
num_timesteps = 1000
gaussian_loss_type = "mse"
scheduler = "cosine"

[train.main]
steps = 1000
lr = 0.001
weight_decay = 1e-05
batch_size = 4096

[train.T]
seed = 0
normalization = "quantile"
num_nan_policy = "__none__"
cat_nan_policy = "__none__"
cat_min_frequency = "__none__"
cat_encoding = "__none__"
y_policy = "default"

[sample]
num_samples = 20800
batch_size = 10000
seed = 0

[eval.type]
eval_model = "catboost"
eval_type = "synthetic"

[eval.T]
seed = 0
normalization = "__none__"
num_nan_policy = "__none__"
cat_nan_policy = "__none__"
cat_min_frequency = "__none__"
cat_encoding = "__none__"
y_policy = "default"
```

--------------------------------

### Specify development dependencies

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Examples of how to include development versions in project dependencies.

```python
install_requires = [
    ...
    'ctgan>=X.Y.Z.dev',
    ...
]
```

```bash
pip install 'ctgan>=X.Y.Z.dev'
```

--------------------------------

### Install CTGAN via conda

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/README.md

Use this command to install the standalone CTGAN library using the conda package manager with the required channels.

```bash
conda install -c pytorch -c conda-forge ctgan
```

--------------------------------

### Run Evaluation Over Seeds

Source: https://github.com/yandex-research/tab-ddpm/blob/main/README.md

Template and example for evaluating models using multiple sampling and evaluation seeds.

```bash
python scripts/eval_seeds.py --config [path_to_your_config] [n_eval_seeds] [ddpm|smote|ctabgan|ctabgan-plus|tvae] synthetic [catboost|mlp] [n_sample_seeds]
python scripts/eval_seeds.py --config exp/churn2/ddpm_cb_best/config.toml 10 ddpm synthetic catboost 5
```

--------------------------------

### Generate and view documentation

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Command to build and preview the project documentation locally.

```shell
$ make view-docs
```

--------------------------------

### Train and Sample with CTGANSynthesizer

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/README.md

Demonstrates loading a demo dataset, configuring discrete columns, fitting the model, and generating synthetic data.

```python3
from ctgan import CTGANSynthesizer
from ctgan import load_demo

data = load_demo()

# Names of the columns that are discrete
discrete_columns = [
    'workclass',
    'education',
    'marital-status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'native-country',
    'income'
]

ctgan = CTGANSynthesizer(epochs=10)
ctgan.fit(data, discrete_columns)

# Synthetic copy
samples = ctgan.sample(1000)
```

--------------------------------

### Run full test suite

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Command to verify changes across all supported Python versions.

```shell
$ make test-all
```

--------------------------------

### Upload release candidate

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Command to build and upload a pre-release version to PyPi.

```makefile
make release-candidate
```

--------------------------------

### Execute release commands

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Makefile commands to trigger different types of project releases.

```makefile
make release
```

```makefile
make release-minor
```

```makefile
make release-major
```

--------------------------------

### Download Datasets

Source: https://github.com/yandex-research/tab-ddpm/blob/main/README.md

Commands to download and extract the datasets used in the paper.

```bash
conda activate tddpm
cd $PROJECT_DIR
wget "https://www.dropbox.com/s/rpckvcs3vx7j605/data.tar?dl=0" -O data.tar
tar -xvf data.tar
```

--------------------------------

### Run quality assurance and testing commands

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Commands to lint code, run tests, and generate coverage reports.

```shell
$ make lint       # Check code styling
$ make test       # Run the tests
$ make coverage   # Get the coverage report
```

--------------------------------

### Build the Conda Package

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/conda/README.md

Builds the package using the specified channels after checking out the stable branch.

```bash
git checkout stable
cd conda
conda build -c sdv-dev -c pytorch -c conda-forge .
```

--------------------------------

### Upload Package to Anaconda

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/conda/README.md

Authenticates with the Anaconda server and uploads the built package to the sdv-dev channel.

```bash
anaconda login
anaconda upload -u sdv-dev <PATH_TO_PACKAGE>
```

--------------------------------

### Configure Problem Type

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTAB-GAN-Plus/README.md

Configuration syntax for specifying the dataset problem type, or setting it to None.

```python
problem_type= {None: None}
```

--------------------------------

### Create a development branch

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Command to create a new branch for your specific bugfix or feature.

```shell
$ git checkout -b name-of-your-bugfix-or-feature
```

--------------------------------

### Commit and push changes

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Standard git commands to stage, commit, and push your local changes to the remote repository.

```shell
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
```

--------------------------------

### Clone the CTGAN repository

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Use this command to clone your fork of the CTGAN repository locally.

```shell
$ git clone git@github.com:your_name_here/CTGAN.git
```

--------------------------------

### Required Package Versions

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTAB-GAN-Plus/README.md

List of specific package versions required to ensure compatibility with the model's implementation.

```text
numpy==1.21.0
torch==1.9.1
pandas==1.2.4
sklearn==0.24.1
dython==0.6.4.post1
scipy==1.4.1
```

--------------------------------

### Adjust Image Side Size for Large Datasets

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTAB-GAN-Plus/README.md

Configuration list defining the side sizes of the image-like format used for encoding data. Enlarge this list to accommodate datasets with a larger number of columns.

```python
sides = [4, 8, 16, 24, 32]
```

--------------------------------

### Aggregate experiment results into a DataFrame

Source: https://github.com/yandex-research/tab-ddpm/blob/main/agg_results.ipynb

Iterates through defined datasets and algorithm paths to load JSON metrics and populate a pandas DataFrame. Requires the lib module and specific directory structures for experiment outputs.

```python
import os
import lib
import numpy as np
import pandas as pd

DATASETS = [
    "abalone",
    "adult",
    "buddy",
    "california",
    "cardio",
    "churn2",
    "default",
    "diabetes",
    "fb-comments",
    "gesture",
    "higgs-small",
    "house",
    "insurance",
    "king",
    "miniboone",
    "wilt"
]

_REGRESSION = [
    "abalone",
    "california",
    "fb-comments",
    "house",
    "insurance",
    "king",
]


method2exp = {
    "real": "exp/{}/ddpm_cb_best/",
    "tab-ddpm": "exp/{}/ddpm_cb_best/",
    "smote": "exp/{}/smote/",
    "ctabgan+": "exp/{}/ctabgan-plus/",
    "ctabgan": "exp/{}/ctabgan/",
    "tvae": "exp/{}/tvae/"
}

eval_file = "eval_catboost.json"
show_std = False
df = pd.DataFrame(columns=["method"] + [_[:3].upper() for _ in DATASETS])

for algo in method2exp: 
    algo_res = []
    for ds in DATASETS:
        if not os.path.exists(os.path.join(method2exp[algo].format(ds), eval_file)):
            algo_res.append("--")
            continue
        metric = "r2" if ds in _REGRESSION else "f1"
        res_dict = lib.load_json(os.path.join(method2exp[algo].format(ds), eval_file))

        if algo == "real":
            res = f'{res_dict["real"]["test"][metric + "-mean"]:.4f}' 
            if show_std: res += f'+-{res_dict["real"]["test"][metric + "-std"]:.4f}'
        else:
            res = f'{res_dict["synthetic"]["test"][metric + "-mean"]:.4f}'
            if show_std: res += f'+-{res_dict["synthetic"]["test"][metric + "-std"]:.4f}'

        algo_res.append(res)
    df.loc[len(df)] = [algo] + algo_res
```

```python
df
```

--------------------------------

### Run test subsets with pytest

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/CONTRIBUTING.rst

Commands to execute specific test modules or filter tests by keyword.

```bash
$ python -m pytest tests.test_ctgan
$ python -m pytest -k 'foo'
```

--------------------------------

### Bibtex citation

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTAB-GAN/README.md

Bibtex entry for citing the CTAB-GAN paper.

```bibtex
@InProceedings{zhao21,
  title = 	 {CTAB-GAN: Effective Table Data Synthesizing},
  author =       {Zhao, Zilong and Kunar, Aditya and Birke, Robert and Chen, Lydia Y.},
  booktitle = 	 {Proceedings of The 13th Asian Conference on Machine Learning},
  pages = 	 {97--112},
  year = 	 {2021},
  editor = 	 {Balasubramanian, Vineeth N. and Tsang, Ivor},
  volume = 	 {157},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--19 Nov},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v157/zhao21a/zhao21a.pdf},
  url = 	 {https://proceedings.mlr.press/v157/zhao21a.html}
}
```

--------------------------------

### Bibtex Citation

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTAB-GAN-Plus/README.md

Bibtex entry for citing the CTAB-GAN+ paper.

```bibtex
@article{zhao2022ctab,
  title={CTAB-GAN+: Enhancing Tabular Data Synthesis},
  author={Zhao, Zilong and Kunar, Aditya and Birke, Robert and Chen, Lydia Y},
  journal={arXiv preprint arXiv:2204.00401},
  year={2022}
}
```

--------------------------------

### Cite CTGAN in LaTeX

Source: https://github.com/yandex-research/tab-ddpm/blob/main/CTGAN/CTGAN/README.md

Use this BibTeX entry to cite the CTGAN research paper in academic publications.

```LaTeX
@inproceedings{xu2019modeling,
  title={Modeling Tabular data using Conditional GAN},
  author={Xu, Lei and Skoularidou, Maria and Cuesta-Infante, Alfredo and Veeramachaneni, Kalyan},
  booktitle={Advances in Neural Information Processing Systems},
  year={2019}
}
```
Tokens:
Raw