### Setup Development Environment

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Set up a Python virtual environment, activate it, and install img2dataset in editable mode.

```bash
python3 -m venv .env
source .env/bin/activate
pip install -e .
```

--------------------------------

### Environment Setup for img2dataset

Source: https://github.com/rom1504/img2dataset/blob/main/llm_doc/CLAUDE.md

Commands to set up a Python virtual environment, install the project in editable mode, and install test dependencies.

```bash
cd /path/to/img2dataset
python3 -m venv .env
source .env/bin/activate
pip install -e .
pip install -r requirements-test.txt
```

--------------------------------

### Install Ray

Source: https://github.com/rom1504/img2dataset/blob/main/examples/ray_example/README.md

Install the Ray library using pip. This is the first step before setting up a Ray cluster.

```bash
pip install ray
```

--------------------------------

### Install img2dataset

Source: https://github.com/rom1504/img2dataset/blob/main/notebook/img2dataset_getting_started.ipynb

Install the library using pip.

```bash
!pip install img2dataset
```

--------------------------------

### Install and Configure Bind9 Resolver

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Install bind9, configure its options for performance, and restart the service. Then, update resolv.conf to use the local bind9 resolver.

```bash
sudo apt install bind9
sudo vim /etc/bind/named.conf.options
```

```bash
sudo systemctl restart bind9
echo nameserver 127.0.0.1 | sudo tee -a /etc/resolv.conf
```

--------------------------------

### Install Testing Dependencies

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Install the necessary Python packages for running tests.

```bash
pip install -r requirements-test.txt
```

--------------------------------

### Start Spark master node

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Initialize the Spark master node on the designated host.

```bash
./spark-3.2.0-bin-hadoop3.2/sbin/start-master.sh -h master_node -p 7077
```

--------------------------------

### Start Spark Master

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Initializes the Spark master node on a specific port.

```bash
./spark-3.2.0-bin-hadoop3.2/sbin/start-master.sh -p 7077
```

--------------------------------

### Display Download Statistics

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Example output showing the total objects and size processed.

```text
Total Objects: 694047
   Total Size: 84.8 TiB
```

--------------------------------

### Configure and Start Knot Resolver

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Stop the systemd-resolved service, start Knot Resolver instances on each worker node, and configure /etc/resolv.conf to use the local resolver. Finally, test DNS resolution.

```bash
parallel-ssh -l $USER -i -h  ips.txt "sudo systemctl stop systemd-resolved"
parallel-ssh -l $USER -i -h  ips.txt "sudo systemctl start kresd@{1..4}.service"
parallel-ssh -l $USER -i -h  ips.txt 'sudo sh -c "echo nameserver 127.0.0.1 > /etc/resolv.conf"'
parallel-ssh -l $USER -i -h  ips.txt 'dig @localhost google.com'
```

--------------------------------

### Install and Configure Knot Resolver

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Commands to install and configure the Knot DNS resolver on Ubuntu for improved network performance.

```bash
wget https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb
sudo dpkg -i knot-resolver-release.deb
sudo apt update
sudo apt install -y knot-resolver
sudo sh -c 'echo `hostname -I` `hostname` >> /etc/hosts'
sudo sh -c 'echo nameserver 127.0.0.1 > /etc/resolv.conf'
sudo systemctl stop systemd-resolved
```

```bash
sudo systemctl start kresd@1.service
sudo systemctl start kresd@2.service
sudo systemctl start kresd@3.service
sudo systemctl start kresd@4.service
```

```bash
dig @localhost google.com
```

--------------------------------

### Basic CLI Download

Source: https://context7.com/rom1504/img2dataset/llms.txt

Install img2dataset using pip. This command downloads images from a list of URLs in a text file.

```bash
pip install img2dataset
```

```bash
img2dataset --url_list=urls.txt --output_folder=images --image_size=256
```

--------------------------------

### Install OpenJDK 17 and Configure Java

Source: https://github.com/rom1504/img2dataset/blob/main/llm_doc/CLAUDE.md

Use these commands to install OpenJDK 17 and set it as the default Java version. This resolves PySpark Java version incompatibility issues.

```bash
sudo apt install openjdk-17-jdk
sudo update-alternatives --config java  # Select Java 17
```

--------------------------------

### Start Spark worker nodes

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Launch worker nodes to connect to the Spark master.

```bash
parallel-ssh -l $USER -i -h  ips.txt  "./spark-3.2.0-bin-hadoop3.2/sbin/start-worker.sh -c 16 -m 16G spark://master_node:7077"
```

--------------------------------

### Download and Convert LAION-Face Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-face.md

This command sequence downloads the necessary metadata files and a conversion script. It then executes the script to prepare the metadata for image downloading. Ensure you have wget and python installed.

```bash
wget -l1 -r --no-parent https://the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/
mv the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ .
wget https://huggingface.co/datasets/FacePerceiver/laion-face/resolve/main/laion_face_ids.pth
wget https://raw.githubusercontent.com/FacePerceiver/LAION-Face/master/convert_parquet.py
python convert_parquet.py ./laion_face_ids.pth ./laion400m-meta ./laion_face_meta
```

--------------------------------

### Check Parallel SSH Setup

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Verify that parallel SSH connections to all worker nodes are successful by running a simple command like 'uname -a'.

```bash
MASTER_USER=rom1504
USER=rom1504
parallel-ssh -l $USER -i -h  ips.txt uname -a
```

--------------------------------

### Download Laion-high-resolution Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-high-resolution.md

Use this script to download the metadata files for the Laion-high-resolution dataset from Hugging Face. Ensure you have wget installed.

```bash
mkdir -p laion-high-resolution && cd laion-high-resolution

for i in {0..127}; do 
    wget "https://huggingface.co/datasets/laion/laion-high-resolution/resolve/main/part-$(printf "%05d" $i)-5d6701c4-b238-4c0a-84e4-fe8e9daea963-c000.snappy.parquet"
done

cd ..
```

--------------------------------

### Install Packages on Workers

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Install necessary packages on all worker nodes using parallel SSH. This includes Java, libraries for image processing, and monitoring tools.

```bash
sudo apt update
sudo apt install openjdk-11-jre-headless libgl1 htop tmux bwm-ng sshfs -y
parallel-ssh -l $USER -i -h  ips.txt "sudo apt update"
parallel-ssh -l $USER -i -h  ips.txt "sudo apt install openjdk-11-jre-headless libgl1 htop tmux bwm-ng sshfs -y"
```

--------------------------------

### Install Knot Resolver on Workers

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Install Knot Resolver on all worker nodes. This involves updating package lists, installing dependencies, and configuring the resolver service.

```bash
parallel-ssh -l $USER -i -h  ips.txt "sudo apt update && sudo apt install libgl1 htop tmux bwm-ng python3.8-venv awscli -y"
parallel-ssh -l $USER -i -h  ips.txt "wget https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb && sudo dpkg -i knot-resolver-release.deb && sudo apt update && sudo apt install -y knot-resolver"
```

--------------------------------

### Download and extract Spark on workers

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Install Apache Spark on worker nodes using parallel-ssh.

```bash
parallel-ssh -l $USER -i -h  ips.txt  "wget https://archive.apache.org/dist/spark/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz"
parallel-ssh -l $USER -i -h  ips.txt  "tar xf spark-3.2.0-bin-hadoop3.2.tgz"
```

--------------------------------

### Start Spark Worker

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Connects a worker node to the Spark master using parallel-ssh.

```bash
parallel-ssh -l $USER -i -h  ips.txt  './spark-3.2.0-bin-hadoop3.2/sbin/start-worker.sh -c 16 -m 24G "spark://172.31.46.59:7077"'
```

--------------------------------

### Download with WebP Encoding

Source: https://context7.com/rom1504/img2dataset/llms.txt

Use this snippet to download images and encode them into WebP format with specified quality. Ensure the 'webp' library is installed.

```python
download(
    url_list="urls.txt",
    output_folder="webp_output",
    encode_format="webp",
    encode_quality=95,                # 0-100, >100 = lossless
    image_size=256,
)
```

--------------------------------

### Run Linting and Tests

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Execute the linting and testing commands using make.

```bash
make lint
make test
```

--------------------------------

### Configure PySpark Session and Run Download

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Sets up a Spark session with specific environment variables and configurations, then executes the image download process.

```python
from img2dataset import download
import shutil
import os
from pyspark.sql import SparkSession  # pylint: disable=import-outside-toplevel

from pyspark import SparkConf, SparkContext

def create_spark_session():
    # this must be a path that is available on all worker nodes
    pex_file = "/home/ubuntu/img2dataset.pex"
    
    os.environ['PYSPARK_PYTHON'] = pex_file
    spark = (
        SparkSession.builder
        .config("spark.submit.deployMode", "client") \
        #.config("spark.files", pex_file) \ # you may choose to uncomment this option if you want spark to automatically download the pex file, but it may be slow
        .config("spark.executorEnv.PEX_ROOT", "./.pex")
        #.config("spark.executor.cores", "16")
        #.config("spark.cores.max", "48") # you can reduce this number if you want to use only some cores ; if you're using yarn the option name is different, check spark doc
        .config("spark.driver.port", "5678")
        .config("spark.driver.blockManager.port", "6678")
        .config("spark.driver.host", "172.31.44.42")
        .config("spark.driver.bindAddress", "172.31.44.42")
        .config("spark.executor.memory", "16G") # make sure to increase this if you're using more cores per executor
        .config("spark.executor.memoryOverhead", "8G")
        .config("spark.task.maxFailures", "100")
        .master("spark://172.31.44.42:7077") # this should point to your master node, if using the tunnelling version, keep this to localhost
        .appName("spark-stats")
        .getOrCreate()
    )
    return spark

spark = create_spark_session()

url_list = "s3://laion-us-east-1/laion-metadata/laion2B-en/"
output_dir = "s3://laion-us-east-1/laion-data/laion2B-data"

download(
    processes_count=1,
    thread_count=64,
    url_list = url_list,
    image_size=384,
    resize_only_if_bigger=True,
    resize_mode="keep_ratio",
    skip_reencode=True,
    output_folder=output_dir,
    output_format="webdataset",
    input_format="parquet",
    url_col="URL",
    caption_col="TEXT",
    enable_wandb=True,
    number_sample_per_shard=10000,
    distributor="pyspark",
    save_additional_columns=["NSFW","similarity","LICENSE"],
    oom_shard_count=6,
)
```

--------------------------------

### Select output format options

Source: https://context7.com/rom1504/img2dataset/llms.txt

Choose between different storage formats based on the target ML framework or use case.

```python
from img2dataset import download

# Files format - individual files in folders (good for <1M images)
download(
    url_list="small_dataset.txt",
    output_folder="files_output",
    output_format="files",
    image_size=256,
)

# Parquet format - images stored as bytes in columnar format
download(
    url_list="medium_dataset.parquet",
    input_format="parquet",
    url_col="url",
    caption_col="text",
    output_folder="parquet_output",
    output_format="parquet",          # Good for Spark/PyArrow ecosystems
    image_size=256,
)

# TFRecord format - for TensorFlow pipelines
download(
    url_list="tf_dataset.parquet",
    input_format="parquet",
    url_col="url",
    caption_col="text",
    output_folder="tfrecord_output",
    output_format="tfrecord",         # Requires tensorflow and tensorflow_io
    image_size=256,
)

# Dummy format - for benchmarking (no actual writes)
download(
    url_list="benchmark.txt",
    output_folder="dummy_output",
    output_format="dummy",            # Benchmarking only
    image_size=256,
)
```

--------------------------------

### Run Image Download

Source: https://github.com/rom1504/img2dataset/blob/main/notebook/img2dataset_getting_started.ipynb

Execute the download process using the specified URL list and output folder.

```bash
!img2dataset --url_list=myimglist.txt --output_folder=output_folder --thread_count=64 --image_size=256
```

--------------------------------

### Run 10000 Image Benchmark

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Navigate to the test files directory and execute the benchmark script for 10,000 images.

```bash
cd tests/test_files
bash benchmark.sh
```

--------------------------------

### Cloud Storage Support - GCS

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download images to Google Cloud Storage (GCS). Ensure fsspec and gcsfs are installed.

```python
# Download to Google Cloud Storage
download(
    url_list="gcs://my-bucket/urls.parquet",
    input_format="parquet",
    url_col="url",
    output_folder="gcs://my-bucket/output/",
    output_format="webdataset",
    image_size=256,
)
```

--------------------------------

### List Output Directory

Source: https://github.com/rom1504/img2dataset/blob/main/notebook/img2dataset_getting_started.ipynb

Verify the downloaded files in the output directory.

```bash
!ls -R output_folder
```

--------------------------------

### Run 18M Image Benchmark

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Navigate to the tests directory and execute the large benchmark script after downloading the first part of the crawling dataset.

```bash
cd tests
bash large_bench.sh
```

--------------------------------

### Download Normal LAION-5B Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Commands to create directories and download standard parquet metadata files for LAION-5B subsets.

```bash
mkdir laion2B-en && cd laion2B-en
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en/resolve/main/part-$i-5114fd87-297e-42b0-9d11-50f1df323dfa-c000.snappy.parquet; done
cd ..
```

```bash
mkdir laion2B-multi && cd laion2B-multi
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-multi/resolve/main/part-$i-fc82da14-99c9-4ff6-ab6a-ac853ac82819-c000.snappy.parquet; done
cd ..
```

```bash
mkdir laion1B-nolang && cd laion1B-nolang
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion1B-nolang/resolve/main/part-$i-d6a94da9-d368-4d5b-9ab7-3f6d3c7abdb3-c000.snappy.parquet; done
```

--------------------------------

### Basic Image Download and Resize

Source: https://context7.com/rom1504/img2dataset/llms.txt

Use this snippet for basic image downloading from a text file. It handles resizing and saving images as individual files with metadata. Ensure the output directory is clean before running.

```python
from img2dataset import download
import shutil
import os

# Clean output directory
output_dir = "my_dataset"
if os.path.exists(output_dir):
    shutil.rmtree(output_dir)

# Basic download with text file input
download(
    url_list="urls.txt",              # File with one URL per line
    output_folder=output_dir,
    image_size=256,                   # Resize to 256x256
    resize_mode="border",             # Add white border to maintain aspect ratio
    output_format="files",            # Save as individual files
    processes_count=16,               # Number of parallel processes
    thread_count=64,                  # Threads per process for downloading
    encode_format="jpg",              # Output image format
    encode_quality=95,                # JPEG quality
)
# Output structure:
# my_dataset/
#   00000/
#     000000000.jpg
#     000000000.json  (metadata)
#     000000001.jpg
#     ...
#   00000.parquet  (shard metadata)
```

--------------------------------

### Submit img2dataset Job to Ray Cluster

Source: https://github.com/rom1504/img2dataset/blob/main/examples/ray_example/README.md

Submit a Python script (`ray_example.py`) to your Ray cluster. This command requires specifying the cluster configuration file, the script to run, and arguments for the script such as `--url_list` and `--out_folder`.

```bash
ray submit cluster_minimal.yaml ray_example.py -- --url_list <url_list> --out_folder <out_folder>
```

--------------------------------

### Download LAION-400M Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion400m.md

Use wget to download the metadata files for the LAION-400M dataset. Ensure you have sufficient disk space.

```bash
wget -l1 -r --no-parent https://the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/
mv the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ .

```

--------------------------------

### Create Image URL List

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Use this bash command to create a text file containing image URLs, one URL per line.

```bash
echo 'https://picsum.photos/200/305' >> myimglist.txt
echo 'https://picsum.photos/200/304' >> myimglist.txt
echo 'https://picsum.photos/200/303' >> myimglist.txt
```

--------------------------------

### Download LAION-Aesthetic Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-aesthetic.md

Downloads parquet metadata files for the laion2B-en-aesthetic subset using wget.

```bash
mkdir laion2B-en-aesthetic && cd laion2B-en-aesthetic
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en-aesthetic/resolve/main/part-$i-9230b837-b1e0-4254-8b88-ed2976e9cee9-c000.snappy.parquet; done
cd ..
```

--------------------------------

### download - Main Entry Point

Source: https://context7.com/rom1504/img2dataset/llms.txt

The primary function for downloading and processing image datasets from a URL list file.

```APIDOC
## download(url_list, output_folder, ...)

### Description
The primary function for downloading and processing image datasets. It accepts a URL list file and handles all aspects of downloading, resizing, and storing images with their metadata.

### Parameters
- **url_list** (str) - Required - Path to the file containing URLs.
- **output_folder** (str) - Required - Path to the output directory.
- **image_size** (int) - Optional - Target size for resizing.
- **resize_mode** (str) - Optional - Resizing mode (e.g., 'border', 'keep_ratio', 'center_crop').
- **output_format** (str) - Optional - Output format (e.g., 'files', 'webdataset', 'parquet', 'tfrecord').
- **processes_count** (int) - Optional - Number of parallel processes.
- **thread_count** (int) - Optional - Threads per process.
- **encode_format** (str) - Optional - Output image format (e.g., 'jpg').
- **encode_quality** (int) - Optional - JPEG quality.

### Request Example
```python
download(url_list="urls.txt", output_folder="my_dataset", image_size=256, resize_mode="border", output_format="files")
```
```

--------------------------------

### Configure and run distributed img2dataset

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Python script to initialize a Spark session and execute the download function in distributed mode.

```python
from img2dataset import download
import shutil
import os
from pyspark.sql import SparkSession  # pylint: disable=import-outside-toplevel

from pyspark import SparkConf, SparkContext

def create_spark_session():
    # this must be a path that is available on all worker nodes
    pex_file = "/home/rom1504/img2dataset.pex"
    
    os.environ['PYSPARK_PYTHON'] = pex_file
    spark = (
        SparkSession.builder
        .config("spark.submit.deployMode", "client") \
        #.config("spark.files", pex_file) \ # you may choose to uncomment this option if you want spark to automatically download the pex file, but it may be slow
        .config("spark.executorEnv.PEX_ROOT", "./.pex")
        #.config("spark.executor.cores", "2") # this can be set to the number of cores of the machine
        #.config("spark.cores.max", "200") # total number of cores to use over the whole spark cluster
        .config("spark.driver.port", "5678")
        .config("spark.driver.blockManager.port", "6678")
        .config("spark.driver.host", "master_node")
        .config("spark.driver.bindAddress", "master_node")
        .config("spark.executor.memory", "16GB") # make sure to increase this if you're using more cores per executor
        .config("spark.executor.memoryOverhead", "8GB")
        .config("spark.task.maxFailures", "100")
        .master("spark://master_node:7077") # this should point to your master node, if using the tunnelling version, keep this to localhost
        .appName("spark-stats")
        .getOrCreate()
    )
    return spark

output_dir = "/tmp/bench"


spark = create_spark_session()

url_list = "some_file.parquet"

download(
    processes_count=1, # this is not used with spark, instead one task for each core will be started (nb executor * nb core per executor)
    thread_count=32,
    retries=0,
    url_list = url_list,
    image_size=384,
    resize_only_if_bigger=True,
    resize_mode="keep_ratio",
    skip_reencode=True,
    output_folder=output_dir,
    output_format="webdataset",
    input_format="parquet",
    url_col="URL",
    caption_col="TEXT",
    enable_wandb=False,
    number_sample_per_shard=10000,
    distributor="pyspark",
    save_additional_columns=["NSFW","similarity","LICENSE"]
)
```

--------------------------------

### Run img2dataset Command

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Execute the img2dataset tool with specified URL list, output folder, thread count, and image size.

```bash
img2dataset --url_list=myimglist.txt --output_folder=output_folder --thread_count=64 --image_size=256
```

--------------------------------

### CLI Download with Filtering

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download images from a parquet file with advanced filtering options, including image size constraints and aspect ratio.

```bash
img2dataset --url_list=large_dataset.parquet \
    --input_format=parquet \
    --url_col=url \
    --caption_col=caption \
    --output_folder=filtered \
    --output_format=webdataset \
    --image_size=512 \
    --resize_mode=keep_ratio \
    --resize_only_if_bigger=True \
    --min_image_size=200 \
    --max_aspect_ratio=3.0
```

--------------------------------

### CLI Download from Parquet with Captions

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download images from a parquet file, specifying URL and caption columns. Outputs in webdataset format.

```bash
img2dataset --url_list=dataset.parquet \
    --input_format=parquet \
    --url_col=URL \
    --caption_col=TEXT \
    --output_folder=webdataset_output \
    --output_format=webdataset \
    --processes_count=16 \
    --thread_count=64 \
    --image_size=384
```

--------------------------------

### Ray Distribution for Downloads

Source: https://context7.com/rom1504/img2dataset/llms.txt

Utilize Ray for distributed image processing across a Ray cluster. Initialize Ray and set `distributor='ray'`.

```python
import ray
from img2dataset import download

# Initialize Ray cluster
ray.init(address="localhost:6379")    # Or "auto" for existing cluster

download(
    url_list="large_dataset.parquet",
    input_format="parquet",
    url_col="url",
    caption_col="alt",
    output_folder="ray_output",
    output_format="webdataset",
    image_size=512,
    resize_mode="keep_ratio_largest",
    resize_only_if_bigger=True,
    skip_reencode=True,
    processes_count=1,
    thread_count=32,
    distributor="ray",                # Use Ray for distribution
    subjob_size=11520,                # Shards per Ray task
    number_sample_per_shard=10000,
    oom_shard_count=8,
    compute_hash="sha256",
    save_additional_columns=["uid"],
    enable_wandb=True,
)
```

--------------------------------

### Download LAION-art metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-art.md

Retrieves the parquet metadata file for the LAION-art dataset.

```bash
wget https://huggingface.co/datasets/laion/laion-art/resolve/main/laion-art.parquet
```

--------------------------------

### Download LAION-Face Images with img2dataset

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-face.md

This script downloads the LAION-Face image data using the prepared metadata. The process can take a significant amount of time and disk space. It downloads 50 million image-text pairs in 32 parts.

```bash
wget https://raw.githubusercontent.com/FacePerceiver/LAION-Face/master/download.sh
bash download.sh ./laion_face_meta ./laion_face_data
```

--------------------------------

### Dummy Output Format

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Use the dummy output format to avoid saving any files, which helps in removing storage bottlenecks during testing. This is useful for benchmarking.

```bash
--output_format dummy
```

--------------------------------

### Download img2dataset on all nodes

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Use parallel-ssh to download and prepare the img2dataset executable across multiple nodes.

```bash
parallel-ssh -i -h ips.txt  "wget -c https://github.com/rom1504/img2dataset/releases/latest/download/img2dataset.pex -O img2dataset_new.pex"
```

```bash
parallel-ssh -l $USER -i -h  ips.txt  "mv img2dataset_new.pex img2dataset.pex"
parallel-ssh -l $USER -i -h  ips.txt  "chmod +x img2dataset.pex"
```

--------------------------------

### Download img2dataset PEX

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Download the img2dataset PEX executable. This file should be accessible to all nodes in the cluster.

```bash
wget https://github.com/rom1504/img2dataset/releases/latest/download/img2dataset.pex -O img2dataset.pex
```

--------------------------------

### Spin up Ray Cluster on AWS

Source: https://github.com/rom1504/img2dataset/blob/main/examples/ray_example/README.md

Use the `ray up` command with your cluster configuration file to launch a Ray cluster on AWS. Ensure `cluster_minimal.yaml` is correctly configured for your AWS environment.

```bash
ray up cluster_minimal.yaml
```

--------------------------------

### Cloud Storage Support - Hugging Face Hub

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download images from the Hugging Face Hub. Specify the dataset and file paths.

```python
# Download to Hugging Face Hub
download(
    url_list="hf://datasets/my-dataset/urls.parquet",
    input_format="parquet",
    url_col="url",
    output_folder="hf://datasets/my-dataset/images/",
    output_format="webdataset",
    image_size=256,
)
```

--------------------------------

### Download and Extract Spark

Source: https://github.com/rom1504/img2dataset/blob/main/examples/distributed_img2dataset_tutorial.md

Download the Spark archive and extract it on the master node. Ensure you use the correct version compatible with your Hadoop distribution.

```bash
wget https://archive.apache.org/^[email protected]/spark-3.2.0/spark-3.2.0-bin-hadoop3.2.tgz
tar xf spark-3.2.0-bin-hadoop3.2.tgz
```

--------------------------------

### Execute img2dataset download

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/cc12m.md

Runs the img2dataset command to download and process images into the webdataset format.

```bash
img2dataset --url_list cc12m.tsv --input_format "tsv"\
         --url_col "url" --caption_col "caption" --output_format webdataset\
           --output_folder cc12m --processes_count 16 --thread_count 64 --image_size 256\
             --enable_wandb True
```

--------------------------------

### Copy LAION-Aesthetic Metadata to S3

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-aesthetic.md

Streams parquet metadata files directly to an S3 bucket using wget and the AWS CLI.

```bash
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en-aesthetic/resolve/main/part-$i-9230b837-b1e0-4254-8b88-ed2976e9cee9-c000.snappy.parquet -O - | aws s3 cp - s3://s-laion/laion-aesthetic/metadata/laion2B-en-aesthetic/part-$i-9230b837-b1e0-4254-8b88-ed2976e9cee9-c000.snappy.parquet; done
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-multi-aesthetic/resolve/main/part-$i-41ee6475-31c6-4d39-960e-7dbbe96bc95b-c000.snappy.parquet -O - | aws s3 cp - s3://s-laion/laion-aesthetic/metadata/laion2B-multi-aesthetic/part-$i-41ee6475-31c6-4d39-960e-7dbbe96bc95b-c000.snappy.parquet; done
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion1B-nolang-aesthetic/resolve/main/part-$i-604e83c4-a4f2-460a-8aae-1c0fa1d4f6d5-c000.snappy.parquet -O - | aws s3 cp - s3://s-laion/laion-aesthetic/metadata/laion1B-nolang-aesthetic/part-$i-604e83c4-a4f2-460a-8aae-1c0fa1d4f6d5-c000.snappy.parquet; done
```

--------------------------------

### Create Image URL List

Source: https://github.com/rom1504/img2dataset/blob/main/notebook/img2dataset_getting_started.ipynb

Append image URLs to a text file for processing.

```bash
!echo 'https://placekitten.com/200/305' >> myimglist.txt
!echo 'https://placekitten.com/200/304' >> myimglist.txt
!echo 'https://placekitten.com/200/303' >> myimglist.txt
```

--------------------------------

### Clean S3 and Execute Download

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Removes existing data from an S3 bucket and triggers the download process via PEX.

```bash
aws s3 rm --recursive s3://laion-us-east-1/test_output/
./img2dataset.pex download.py
```

--------------------------------

### Download COYO-700M Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/coyo-700m.md

Downloads the dataset parquet files from Hugging Face into a local directory.

```bash
mkdir coyo-700m && cd coyo-700m
for i in {00000..00127}; do wget https://huggingface.co/datasets/kakaobrain/coyo-700m/resolve/main/data/part-$i-17da4908-939c-46e5-91d0-15f256041956-c000.snappy.parquet; done
cd ..
```

--------------------------------

### Stream Metadata to AWS S3

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Commands to pipe metadata downloads directly to AWS S3 storage using the AWS CLI.

```bash
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en-joined/resolve/main/part-$i-4cfd6e30-f032-46ee-9105-8696034a8373-c000.snappy.parquet -O - | aws s3 cp - s3://laion5b/metadata/laion2B-en-joined/part-$i-4cfd6e30-f032-46ee-9105-8696034a8373-c000.snappy.parquet; done
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-multi-joined/resolve/main/part-$i-fcd86c9b-36f4-49ff-bea1-8c9a0e029fb7-c000.snappy.parquet -O - | aws s3 cp - s3://laion5b/metadata/laion2B-multi-joined/art-$i-fcd86c9b-36f4-49ff-bea1-8c9a0e029fb7-c000.snappy.parquet; done
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion1B-nolang-joined/resolve/main/part-$i-4852663c-9585-44b0-9a45-f95c2b89c792-c000.snappy.parquet -O - | aws s3 cp - s3://laion5b/metadata/laion1B-nolang-joined/part-$i-4852663c-9585-44b0-9a45-f95c2b89c792-c000.snappy.parquet; done
```

--------------------------------

### Incremental Mode

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Explanation of how to use incremental mode to resume interrupted downloads.

```APIDOC
## Incremental Mode

### Description

Incremental mode allows you to resume interrupted downloads. If a download process is stopped for any reason, you can restart img2dataset with the same output folder, number of samples per shard, and input URLs, and it will automatically continue downloading the remaining shards.

### Method

This functionality is typically invoked by setting the `incremental_mode` parameter.

### Endpoint

N/A (This is a configuration option, not a specific API endpoint.)

### Usage

To enable incremental mode, use the `--incremental` flag or set the `incremental_mode` parameter to `"incremental"` (which is the default behavior).

```bash
img2dataset --url_list urls.txt --output_folder dataset --incremental_mode incremental
```

Or simply:

```bash
img2dataset --url_list urls.txt --output_folder dataset
```

### Notes

- Ensure that the `output_folder`, `number_sample_per_shard`, and input URLs remain consistent between the interrupted run and the resumed run for incremental mode to work correctly.
```

--------------------------------

### Manage incremental and extend download modes

Source: https://context7.com/rom1504/img2dataset/llms.txt

Control how the downloader handles existing data, allowing for resuming interrupted tasks, extending datasets, or overwriting existing files.

```python
from img2dataset import download

# Resume interrupted download (default behavior)
download(
    url_list="large_dataset.parquet",
    input_format="parquet",
    url_col="url",
    output_folder="resumable_output",
    output_format="webdataset",
    image_size=256,
    incremental_mode="incremental",   # Skip already completed shards
    processes_count=16,
    thread_count=64,
)

# Extend dataset with new URLs (continues from last shard number)
download(
    url_list="additional_urls.parquet",
    input_format="parquet",
    url_col="url",
    output_folder="resumable_output",  # Same output folder
    output_format="webdataset",
    image_size=256,
    incremental_mode="extend",        # Start from next shard ID
    processes_count=16,
    thread_count=64,
)

# Overwrite existing data
download(
    url_list="fresh_dataset.parquet",
    input_format="parquet",
    url_col="url",
    output_folder="fresh_output",
    output_format="webdataset",
    image_size=256,
    incremental_mode="overwrite",     # Delete and restart
    processes_count=16,
    thread_count=64,
)
```

--------------------------------

### CLI Download CC3M Dataset

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download the CC3M dataset from a TSV file, specifying URL and caption columns. Enables Weights & Biases logging.

```bash
img2dataset --url_list=cc3m.tsv \
    --input_format=tsv \
    --url_col=url \
    --caption_col=caption \
    --output_format=webdataset \
    --output_folder=cc3m \
    --processes_count=16 \
    --thread_count=64 \
    --image_size=256 \
    --enable_wandb=True
```

--------------------------------

### Display Image

Source: https://github.com/rom1504/img2dataset/blob/main/notebook/img2dataset_getting_started.ipynb

Display a downloaded image using IPython.

```python
from IPython.display import Image
Image(filename='output_folder/00000/000000000.jpg')
```

--------------------------------

### Download MSCOCO Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/mscoco.md

Use wget to download the mscoco.parquet metadata file. This file contains URLs and captions for the dataset.

```bash
wget https://huggingface.co/datasets/ChristophSchuhmann/MS_COCO_2017_URL_TEXT/resolve/main/mscoco.parquet
```

--------------------------------

### img2dataset Resize Modes

Source: https://context7.com/rom1504/img2dataset/llms.txt

Demonstrates various resize modes for image preprocessing with img2dataset. Use these options to control how images are resized to fit the specified image_size.

```python
from img2dataset import download

# No resize - keep original dimensions
download(url_list="urls.txt", output_folder="no_resize", resize_mode="no", image_size=256)
```

```python
# Border mode - resize to fit in square, add white borders
download(url_list="urls.txt", output_folder="border", resize_mode="border", image_size=256)
# 800x600 image -> 256x192 scaled, then 256x256 with white borders
```

```python
# Keep ratio - resize smallest side to image_size
download(url_list="urls.txt", output_folder="keep_ratio", resize_mode="keep_ratio", image_size=256)
# 800x600 image -> 341x256 (smallest side = 256)
```

```python
# Keep ratio largest - resize largest side to image_size
download(url_list="urls.txt", output_folder="keep_ratio_largest", resize_mode="keep_ratio_largest", image_size=256)
# 800x600 image -> 256x192 (largest side = 256)
```

```python
# Center crop - resize then crop to square
download(url_list="urls.txt", output_folder="center_crop", resize_mode="center_crop", image_size=256)
# 800x600 image -> 341x256 scaled, then center cropped to 256x256
```

```python
# Only resize if bigger
download(
    url_list="urls.txt",
    output_folder="conditional_resize",
    resize_mode="keep_ratio",
    resize_only_if_bigger=True,       # Small images stay original size
    image_size=256,
)
```

--------------------------------

### FSSpec Configuration for S3

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Configure fsspec for S3-compatible file systems like Minio by creating a .config/fsspec/s3.json file with endpoint and credentials.

```json
{
  "s3": {
    "client_kwargs": {
            "endpoint_url": "https://some_endpoint",
            "aws_access_key_id": "your_user",
           "aws_secret_access_key": "your_password"
    }
  }
}
```

--------------------------------

### Download Images with img2dataset

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/SBUcaptions.md

Execute the img2dataset command to download images from the SBU Captions JSON metadata file.

```bash
img2dataset --url_list sbu-captions-all.json --input_format "json" --url_col "image_urls" --caption_col "captions" --output_format webdataset --output_folder sbucaptions --processes_count 16 --thread_count 64 --image_size 256
```

--------------------------------

### Running img2dataset Tests

Source: https://github.com/rom1504/img2dataset/blob/main/llm_doc/CLAUDE.md

Commands to run specific tests for blurrer and center_crop functionality, or to execute all tests using pytest.

```bash
# Test our specific fixes
python -m pytest -v tests/test_blurrer.py::test_blurrer
python -m pytest -v tests/test_main.py -k "center_crop"

# Run all tests
make test
# OR
python -m pytest -v tests
```

--------------------------------

### Download Joined LAION-5B Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md

Commands to download metadata collections that include additional punsafe and pwatermark fields.

```bash
mkdir laion2B-en && cd laion2B-en
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-en-joined/resolve/main/part-$i-4cfd6e30-f032-46ee-9105-8696034a8373-c000.snappy.parquet; done
cd ..
```

```bash
mkdir laion2B-multi && cd laion2B-multi
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion2B-multi-joined/resolve/main/part-$i-fcd86c9b-36f4-49ff-bea1-8c9a0e029fb7-c000.snappy.parquet; done
cd ..
```

```bash
mkdir laion1B-nolang && cd laion1B-nolang
for i in {00000..00127}; do wget https://huggingface.co/datasets/laion/laion1B-nolang-joined/resolve/main/part-$i-4852663c-9585-44b0-9a45-f95c2b89c792-c000.snappy.parquet; done
```

--------------------------------

### Download SBU Captions Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/SBUcaptions.md

Use wget and tar to retrieve and extract the SBU Captions dataset metadata archive.

```bash
wget https://www.cs.rice.edu/~vo9/sbucaptions/sbu-captions-all.tar.gz
tar -xvzf sbu-captions-all.tar.gz
```

--------------------------------

### Configure opt-out directive handling

Source: https://context7.com/rom1504/img2dataset/llms.txt

Control whether the downloader respects X-Robots-Tag headers like noai or noindex. Use with caution when disabling.

```python
from img2dataset import download

# Default behavior respects noai, noimageai, noindex, noimageindex headers
download(
    url_list="urls.txt",
    output_folder="respectful_output",
    image_size=256,
    # Default: disallowed_header_directives=["noai", "noimageai", "noindex", "noimageindex"]
)

# Disable opt-out checking (use responsibly)
download(
    url_list="urls.txt",
    output_folder="all_images_output",
    image_size=256,
    disallowed_header_directives=[],  # Download all images regardless of headers
)
```

--------------------------------

### Reformat Code with Black

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

Use the 'make black' command to reformat the code according to Black's style.

```bash
make black
```

--------------------------------

### Download CC12M metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/cc12m.md

Downloads the CC12M TSV file using wget.

```bash
wget https://storage.googleapis.com/conceptual_12m/cc12m.tsv
```

--------------------------------

### Download LAION-COCO Metadata

Source: https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-coco.md

Downloads the parquet metadata files from the Hugging Face dataset repository using wget.

```bash
mkdir -p laion-coco && cd laion-coco/

for i in {0..127}; do 
    wget "https://huggingface.co/datasets/laion/laion-coco/resolve/main/part-$(printf "%05d" $i)-2256f782-126f-4dc6-b9c6-e6757637749d-c000.snappy.parquet"
done

cd ..
```

--------------------------------

### Bind9 Resolver Options Configuration

Source: https://github.com/rom1504/img2dataset/blob/main/README.md

These options enhance the performance of the bind9 DNS resolver for high query volumes.

```bind
recursive-clients 10000;
resolver-query-timeout 30000;
max-clients-per-query 10000;
max-cache-size 2000m;
```

--------------------------------

### Verify image integrity with hashes

Source: https://context7.com/rom1504/img2dataset/llms.txt

Use hash comparison to ensure downloaded images match expected values. Mismatched images are marked with a hash mismatch error.

```python
from img2dataset import download

download(
    url_list="dataset_with_hashes.parquet",
    input_format="parquet",
    url_col="url",
    caption_col="caption",
    output_folder="verified_dataset",
    output_format="webdataset",
    image_size=256,
    compute_hash="md5",               # Compute MD5 hash of downloaded images
    verify_hash=["md5", "md5"],       # [column_name, hash_type] to verify against
    processes_count=16,
    thread_count=64,
)
# Images with mismatched hashes are marked as "failed_to_download" with "hash mismatch" error
```

--------------------------------

### Set image encoding formats

Source: https://context7.com/rom1504/img2dataset/llms.txt

Specify the output image format and quality settings for compression.

```python
from img2dataset import download

# JPEG encoding (default, good compression)
download(
    url_list="urls.txt",
    output_folder="jpg_output",
    encode_format="jpg",
    encode_quality=95,                # 0-100, higher = better quality
    image_size=256,
)

# PNG encoding (lossless)
download(
    url_list="urls.txt",
    output_folder="png_output",
    encode_format="png",
    encode_quality=6,                 # 0-9, compression level (not quality)
    image_size=256,
)
```

--------------------------------

### Parquet Input with Captions to WebDataset

Source: https://context7.com/rom1504/img2dataset/llms.txt

Download images from a parquet file, using specified URL and caption columns. Saves the dataset in webdataset format, suitable for ML training. Enables Weights & Biases logging for monitoring.

```python
from img2dataset import download

download(
    url_list="dataset.parquet",
    input_format="parquet",
    url_col="URL",                    # Column name for image URLs
    caption_col="TEXT",               # Column name for captions
    output_folder="webdataset_output",
    output_format="webdataset",       # Save as tar files
    image_size=384,
    resize_mode="keep_ratio",         # Keep aspect ratio, resize smallest side
    resize_only_if_bigger=True,       # Don't upscale small images
    processes_count=16,
    thread_count=64,
    number_sample_per_shard=10000,    # Images per tar file
    compute_hash="sha256",            # Compute and store image hash
    enable_wandb=True,                # Enable W&B logging
)
# Output structure:
# webdataset_output/
#   00000.tar  (containing 000000000.jpg, 000000000.txt, 000000000.json, ...)
#   00000.parquet
#   00001.tar
#   ...
```