WebDataset

https://rom1504.github.io/webdataset

Admin

WebDataset is a PyTorch IterableDataset implementation that provides efficient, sequential access to...

Tokens:266

Snippets:3

Trust Score:8.3

Update:1 year ago

Show doc for...

Tokens:

Raw

### Install WebDataset via pip

Source: https://rom1504.github.io/webdataset/index

This command installs the stable release of the WebDataset library from PyPI using pip, Python's package installer. It's the standard and recommended method for most users to get WebDataset.

```bash
$ pip install webdataset
```

--------------------------------

### Install WebDataset from GitHub

Source: https://rom1504.github.io/webdataset/index

This command allows users to install the latest development version of WebDataset directly from its GitHub repository. This is useful for accessing new features, bug fixes, or contributing to the project before a new stable release is available.

```bash
$ pip install git+https://github.com/tmbdev/webdataset.git
```

--------------------------------

### Load and Process Data with WebDataset in PyTorch

Source: https://rom1504.github.io/webdataset/index

This snippet demonstrates the fundamental usage of WebDataset within a PyTorch workflow. It shows how to initialize a WebDataset instance, apply common data transformations like shuffling and decoding, and then integrate it with PyTorch's DataLoader for batch processing and iteration.

```python
import webdataset as wds

dataset = wds.WebDataset(url).shuffle(1000).decode("torchrgb").to_tuple("jpg;png", "json")
dataloader = torch.utils.data.DataLoader(dataset, num_workers=4, batch_size=16)

for inputs, outputs in dataloader:
    ...
```

Tokens:

Raw