### Install TensorFlow Machine Learning Framework

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Installs the TensorFlow library, another widely used open-source machine learning framework. This is an alternative to PyTorch for users who prefer TensorFlow for their deep learning workflows.

```bash
pip install tensorflow
```

--------------------------------

### Install PyTorch Machine Learning Framework

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Installs the PyTorch library, a popular open-source machine learning framework. This is often a prerequisite for using 🤗 Datasets in conjunction with deep learning models.

```bash
pip install torch
```

--------------------------------

### Verify Hugging Face Datasets Installation (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Verifies the installation of the Hugging Face Datasets library by attempting to load and print the first example from the SQuAD training dataset. This command serves as a basic check for a successful setup.

```python
from datasets import load_dataset
print(load_dataset('rajpurkar/squad', split='train')[0])
```

--------------------------------

### Apply Torchvision Data Augmentations

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Defines a data augmentation pipeline using `torchvision.transforms`. This example includes `ColorJitter` for random color property changes and `ToTensor` to convert images to PyTorch tensors. It's applied to the dataset using a custom `transforms` function.

```python
from torchvision.transforms import Compose, ColorJitter, ToTensor

jitter = Compose([
    ColorJitter(brightness=0.5, hue=0.5), ToTensor()
])

def transforms(examples):
    examples["pixel_values"] = [jitter(image.convert("RGB")) for image in examples["image"]]
    return examples

dataset = dataset.with_transform(transforms)
```

--------------------------------

### Install Datasets with Audio Support (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Installs the Hugging Face Datasets library with the 'audio' extra dependency. This enables support for handling audio datasets.

```bash
pip install datasets[audio]
```

--------------------------------

### Apply Albumentations Data Augmentations for TensorFlow

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Applies data augmentations using the `albumentations` library, specifically for use with TensorFlow. This involves installing `albumentations` and `opencv-python`, defining an augmentation pipeline, and applying it using `dataset.set_transform`. The processed dataset is then prepared for TensorFlow using `model.prepare_tf_dataset`.

```bash
pip install -U albumentations opencv-python
```

```python
import albumentations
import numpy as np

transform = albumentations.Compose([
    albumentations.RandomCrop(width=256, height=256),
    albumentations.HorizontalFlip(p=0.5),
    albumentations.RandomBrightnessContrast(p=0.2),
])

def transforms(examples):
    examples["pixel_values"] = [
        transform(image=np.array(image))["image"] for image in examples["image"]
    ]
    return examples

dataset.set_transform(transforms)
tf_dataset = model.prepare_tf_dataset(
    dataset,
    batch_size=4,
    shuffle=True,
)
```

--------------------------------

### Convert Audio Features to PyTorch Tensors

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_pytorch

Explains the conversion of `Audio` features to PyTorch tensors, requiring the `audio` extra (`pip install datasets[audio]`). The example shows how audio data is accessed as a tensor array and its corresponding sampling rate.

```python
>>> from datasets import Dataset, Features, Audio, Image
>>> audio = ["path/to/audio.wav"] * 10
>>> features = Features({"audio": Audio()})
>>> ds = Dataset.from_dict({"audio": audio}, features=features) 
>>> ds = ds.with_format("torch")  
>>> ds[0]["audio"]["array"]
tensor([ 6.1035e-05,  1.5259e-05,  1.6785e-04,  ..., -1.5259e-05,
        -1.5259e-05,  1.5259e-05])
>>> ds[0]["audio"]["sampling_rate"]
tensor(44100)
```

--------------------------------

### Install Datasets with Vision Support (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Installs the Hugging Face Datasets library with the 'vision' extra dependency. This enables support for handling image datasets.

```bash
pip install datasets[vision]
```

--------------------------------

### Convert Image Features to PyTorch Tensors

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_pytorch

Demonstrates converting `Image` features to PyTorch tensors. This requires installing the `vision` extra (`pip install datasets[vision]`). The example shows how image data is represented as tensors with dimensions corresponding to height, width, and channels.

```python
>>> from datasets import Dataset, Features, Audio, Image
>>> images = ["path/to/image.png"] * 10
>>> features = Features({"image": Image()})
>>> ds = Dataset.from_dict({"image": images}, features=features) 
>>> ds = ds.with_format("torch")
>>> ds[0]["image"].shape
torch.Size([512, 512, 4])
>>> ds[0]
{'image': tensor([[[255, 215, 106, 255],
         [255, 215, 106, 255],
         ...,
         [255, 255, 255, 255],
         [255, 255, 255, 255]]], dtype=torch.uint8)}
>>> ds[:2]["image"].shape
torch.Size([2, 512, 512, 4])
>>> ds[:2]
{'image': tensor([[[[255, 215, 106, 255],
          [255, 215, 106, 255],
          ...,
          [255, 255, 255, 255],
          [255, 255, 255, 255]]]], dtype=torch.uint8)}
```

--------------------------------

### Create and Navigate Project Directory (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Creates a new directory for your project and navigates into it. This is a standard command-line operation for organizing projects.

```bash
mkdir ~/my-project
cd ~/my-project
```

--------------------------------

### Install Transformers Library

Source: https://huggingface.co/docs/datasets/v4.4.2/use_dataset

Installs the Hugging Face Transformers library, which is necessary for using tokenizers and data collators in dataset preprocessing.

```bash
pip install transformers
```

--------------------------------

### Install huggingface_hub Library

Source: https://huggingface.co/docs/datasets/v4.4.2/upload_dataset

Installs the `huggingface_hub` library, which is necessary for interacting with the Hugging Face Hub programmatically from Python. This is a prerequisite for uploading datasets using Python.

```bash
pip install huggingface_hub
```

--------------------------------

### Install Hugging Face Datasets from Source (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Installs the Hugging Face Datasets library in editable mode after cloning the repository. This allows for direct modification and testing of the library's code.

```bash
pip install -e .
```

--------------------------------

### Install Hugging Face Datasets with Pip (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Installs the Hugging Face Datasets library using pip. This command fetches and installs the latest stable version from the Python Package Index.

```bash
pip install datasets
```

--------------------------------

### Install Albumentations

Source: https://huggingface.co/docs/datasets/v4.4.2/depth_estimation

Installs or upgrades the Albumentations library. This is a prerequisite for using data augmentation for computer vision tasks.

```bash
pip install -U albumentations
```

--------------------------------

### Take First N Examples from IterableDataset (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/stream

Illustrates how to use the `take()` method to extract a specified number of initial examples from an IterableDataset. This is useful for quickly inspecting or using a subset of the data. Requires the 'datasets' library.

```python
>>> dataset = load_dataset('HuggingFaceFW/fineweb', split='train', streaming=True)
>>> dataset_head = dataset.take(2)
>>> list(dataset_head)
[{'text': "How AP reported in all formats from tor...}, {'text': 'Did you know you have two little yellow...'}]
```

--------------------------------

### Format Dataset for PyTorch DataLoader

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Selects relevant columns and formats the dataset into PyTorch tensors using `with_format`. The formatted dataset is then wrapped in a `torch.utils.data.DataLoader` for batch processing during training.

```python
import torch

dataset = dataset.select_columns(["input_ids", "token_type_ids", "attention_mask", "labels"])
dataset = dataset.with_format(type="torch")
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
```

--------------------------------

### Install Hugging Face Datasets with Conda (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Installs the Hugging Face Datasets library using the conda package manager from the huggingface and conda-forge channels. This is an alternative installation method for users who prefer conda.

```bash
conda install -c huggingface -c conda-forge datasets
```

--------------------------------

### Create Virtual Environment (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Creates a new Python virtual environment named '.env' within the current directory. This isolates project dependencies.

```bash
python -m venv .env
```

--------------------------------

### Load Audio Dataset using `load_dataset`

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Loads the MInDS-14 audio dataset from Hugging Face. Requires the dataset name, configuration, and split. It uses the `load_dataset` function and returns a `Dataset` object.

```python
from datasets import load_dataset, Audio

dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
```

--------------------------------

### Install Libraries

Source: https://huggingface.co/docs/datasets/v4.4.2/image_classification

Installs the required albumentations and opencv-python libraries for image processing. Ensure you have up-to-date versions.

```bash
pip install -U albumentations opencv-python
```

--------------------------------

### Clone Hugging Face Datasets Repository (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Clones the Hugging Face Datasets GitHub repository to your local machine. This is necessary for building the library from source.

```bash
git clone https://github.com/huggingface/datasets.git
cd datasets
```

--------------------------------

### Split Long Examples into Chunks using Batch Processing

Source: https://huggingface.co/docs/datasets/v4.4.2/process

This example demonstrates batch processing with `map()` to split long text entries into smaller, manageable chunks. The `chunk_examples` function iterates through sentences in a batch, creating substrings of a specified length (50 characters in this case), and returns them as a new list of chunks.

```python
>>> def chunk_examples(examples):
...     chunks = []
...     for sentence in examples["sentence1"]:
...         chunks += [sentence[i:i + 50] for i in range(0, len(sentence), 50)]
...     return {"chunks": chunks}
```

--------------------------------

### Load and Inspect Object Detection Dataset

Source: https://huggingface.co/docs/datasets/v4.4.2/object_detection

Loads the 'cppe-5' object detection dataset from Hugging Face and displays an example entry. This example shows the structure of image data, including bounding box metadata and category information.

```python
from datasets import load_dataset

ds = load_dataset("rishitdagli/cppe-5")
example = ds['train'][0]
print(example)
```

--------------------------------

### Select a percentage of a split using string slicing and ReadInstruction

Source: https://huggingface.co/docs/datasets/v4.4.2/loading

Illustrates how to load a specific percentage of data from a split. This includes examples for selecting the first 10% and using ReadInstruction for percentage-based slicing.

```python
>>> train_10pct_ds = datasets.load_dataset("ajibawa-2023/General-Stories-Collection", split="train[:10%]")
```

```python
>>> train_10_20_ds = datasets.load_dataset("ajibawa-2023/General-Stories-Collection", split=datasets.ReadInstruction("train", to=10, unit="%"))
```

--------------------------------

### Load Dataset, Audio Feature, and Feature Extractor (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/use_dataset

Loads the MInDS-14 dataset, the Audio feature, and a Wav2Vec2 feature extractor. This is the initial setup required before resampling.

```python
from transformers import AutoFeatureExtractor
from datasets import load_dataset, Audio

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")
dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
```

--------------------------------

### Load Image Data as TensorFlow Tensors

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_tensorflow

Demonstrates loading image data from file paths into a Hugging Face Dataset and formatting it for TensorFlow. Requires the 'datasets[vision]' extra to be installed.

```python
from datasets import Dataset, Features, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features)
ds = ds.with_format("tf")
print(ds[0])
print(ds[:2])
```

--------------------------------

### Prepare TensorFlow Dataset using `prepare_tf_dataset`

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Prepares the Hugging Face dataset to be compatible with TensorFlow by converting it into a `tf.data.Dataset`. This method handles collation and batching, allowing direct use with Keras `fit()` methods.

```python
import tensorflow as tf

tf_dataset = model.prepare_tf_dataset(
    dataset,
    batch_size=4,
    shuffle=True,
)
```

--------------------------------

### Load and Format Audio Features as NumPy Arrays

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_numpy

Demonstrates how `Audio` features are loaded and represented when the dataset is formatted for NumPy. Requires the `audio` extra to be installed. The output includes the audio array and its sampling rate.

```python
from datasets import Dataset, Features, Audio
audio = ["path/to/audio.wav"] * 10
features = Features({"audio": Audio()})
ds = Dataset.from_dict({"audio": audio}, features=features)
ds = ds.with_format("numpy")
print(ds[0]["audio"]["array"])
print(ds[0]["audio"]["sampling_rate"])
```

--------------------------------

### Tokenize a Single Text Example (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/use_dataset

Demonstrates how to tokenize a single text entry from the dataset using the loaded tokenizer. The output includes `input_ids`, `token_type_ids`, and `attention_mask`, which are the required inputs for machine learning models.

```python
tokenizer(dataset[0]["text"])
```

--------------------------------

### Upload Dataset to Hugging Face Hub

Source: https://huggingface.co/docs/datasets/v4.4.2/image_dataset

Uploads a processed dataset to the Hugging Face Hub using the `push_to_hub()` method. Requires the 'huggingface_hub' library to be installed and authentication with a Hugging Face account.

```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("imagefolder", data_dir="/path/to/folder", split="train")
>>> dataset.push_to_hub("stevhliu/my-image-captioning-dataset")
```

--------------------------------

### Load MRPC Dataset with Datasets Library

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Loads the Microsoft Research Paraphrase Corpus (MRPC) training dataset using the `load_dataset` function from the `datasets` library. This function requires the dataset name and configuration.

```python
from datasets import load_dataset

dataset = load_dataset("nyu-mll/glue", "mrpc", split="train")
```

--------------------------------

### Load Dataset with Hugging Face Datasets

Source: https://huggingface.co/docs/datasets/v4.4.2/process

Loads the MRPC dataset from Hugging Face's model hub for processing. This is a common starting point for dataset manipulation tasks. It requires the 'datasets' library to be installed.

```python
from datasets import load_dataset
dataset = load_dataset("nyu-mll/glue", "mrpc", split="train")
```

--------------------------------

### Load Wav2Vec2 Model and Feature Extractor

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Loads a pre-trained Wav2Vec2 model for audio classification and its corresponding feature extractor from the Transformers library. This is necessary for processing audio data for the model. Expect warnings about uninitialized weights, which are normal for training on a new task.

```python
from transformers import AutoModelForAudioClassification, AutoFeatureExtractor

model = AutoModelForAudioClassification.from_pretrained("facebook/wav2vec2-base")
feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

--------------------------------

### Create Audio Dataset from Local Files and Upload

Source: https://huggingface.co/docs/datasets/v4.4.2/audio_dataset

Demonstrates creating an audio dataset from local files and uploading it to the Hugging Face Hub. It involves casting a column of audio file paths to the `Audio` feature and then using `push_to_hub()` to upload the dataset.

```python
from datasets import Audio, Dataset

audio_dataset = Dataset.from_dict({"audio": ["path/to/audio_1", "path/to/audio_2", ..., "path/to/audio_n"]}).cast_column("audio", Audio())
audio_dataset.push_to_hub("/my_dataset")
```

--------------------------------

### Filter Dataset Rows by Text Content using Hugging Face Datasets

Source: https://huggingface.co/docs/datasets/v4.4.2/stream

This example shows how to filter rows from a dataset based on a condition applied to the 'text' column. It uses the `Dataset.filter()` method with a lambda function that checks if the text starts with a specific string. This is useful for selecting subsets of data that meet certain criteria.

```python
from datasets import load_dataset
dataset = load_dataset('HuggingFaceFW/fineweb', streaming=True, split='train')
start_with_ar = dataset.filter(lambda example: example['text'].startswith('San Francisco'))
print(next(iter(start_with_ar)))
```

--------------------------------

### Preprocess Audio Data with Feature Extractor using `map`

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Defines a function to preprocess audio arrays using a feature extractor, including padding and truncation. The `map` function applies this preprocessing to the dataset efficiently in batches. It takes audio arrays, sampling rate, and padding/truncation parameters as input and returns processed inputs.

```python
def preprocess_function(examples):
    audio_arrays = [x.get_all_samples().data for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=16000,
        padding=True,
        max_length=100000,
        truncation=True,
    )
    return inputs

dataset = dataset.map(preprocess_function, batched=True)
```

--------------------------------

### Load Image Dataset with Hugging Face Datasets

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Loads an image dataset using the `load_dataset` function from the `datasets` library. It takes the dataset name and a split as arguments. The `Image` type hint is used to specify image data.

```python
from datasets import load_dataset, Image

dataset = load_dataset("AI-Lab-Makerere/beans", split="train")
```

--------------------------------

### Query Elasticsearch Index for Nearest Examples (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/faiss_es

Retrieves the top 'k' nearest examples from a specified Elasticsearch index within a Hugging Face dataset based on a given query. Returns scores and the retrieved examples.

```python
query = "machine"
scores, retrieved_examples = squad.get_nearest_examples("context", query, k=10)
print(retrieved_examples["title"][0])
```

--------------------------------

### Set PyTorch Dataset Format and Create DataLoader

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Sets the dataset format to PyTorch tensors and specifies the columns to be formatted ('input_values' and 'labels'). It then wraps the formatted dataset in a PyTorch DataLoader for batch processing during training.

```python
from torch.utils.data import DataLoader

dataset.set_format(type="torch", columns=["input_values", "labels"])
dataloader = DataLoader(dataset, batch_size=4)
```

--------------------------------

### Create Audio Dataset from Dictionary (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/create_dataset

Demonstrates how to create an audio dataset by first creating a dataset from a dictionary of audio file paths and then casting the 'audio' column to the Audio feature type.

```python
from datasets import Dataset, Audio
audio_dataset = Dataset.from_dict({"audio": ["path/to/audio_1", ..., "path/to/audio_n"]}).cast_column("audio", Audio())

```

--------------------------------

### Eager Data Processing with Dataset.map()

Source: https://huggingface.co/docs/datasets/v4.4.2/about_mapstyle_vs_iterable

Applies a processing function ('process_fn') to all examples in an Arrow Dataset immediately. This is similar to how pandas operates. The output shows the first processed example.

```python
# Assuming my_dataset is already loaded
# def process_fn(example):
#     # Your processing logic here
#     return example

my_dataset = my_dataset.map(process_fn)
print(my_dataset[0])
```

--------------------------------

### Map Function to Process IterableDataset Examples

Source: https://huggingface.co/docs/datasets/v4.4.2/stream

Applies a processing function to each example in an IterableDataset on-the-fly. The function must accept and return a dictionary. This can be used to modify existing data or create new columns.

```python
from datasets import load_dataset

def add_prefix(example):
    example['text'] = 'My text: ' + example['text']
    return example

dataset = load_dataset('allenai/c4', 'en', streaming=True, split='train')
updated_dataset = dataset.map(add_prefix)

# To view results:
# print(list(updated_dataset.take(3)))
```

--------------------------------

### Load BERT Model and Tokenizer with Transformers (PyTorch)

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Loads a pre-trained BERT base uncased model for sequence classification and its corresponding tokenizer using `AutoModelForSequenceClassification` and `AutoTokenizer` from the `transformers` library. This is suitable for PyTorch integration.

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
```

--------------------------------

### Add Index to Dataset Column using map() with Indices

Source: https://huggingface.co/docs/datasets/v4.4.2/process

This snippet illustrates using map() with the with_indices=True argument to access the index of each example. A lambda function is used to prepend the index to the 'sentence2' column. The function signature now includes both the example and its index.

```python
>>> updated_dataset = dataset.map(lambda example, idx: {"sentence2": f"{idx}: " + example["sentence2"]}, with_indices=True)
>>> updated_dataset["sentence2"][:5]
['0: Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .',
 "1: Yucaipa bought Dominick 's in 1995 for $ 693 million and sold it to Safeway for $ 1.8 billion in 1998 .",
 "2: On June 10 , the ship 's owners had published an advertisement on the Internet , offering the explosives for sale .",
 '3: Tab shares jumped 20 cents , or 4.6 % , to set a record closing high at A $ 4.57 .',
 '4: PG & E Corp. shares jumped $ 1.63 or 8 percent to $ 21.03 on the New York Stock Exchange on Friday .'
]
```

--------------------------------

### Skip First N Examples from IterableDataset (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/stream

Demonstrates how to use the `skip()` method to omit a specified number of initial examples from an IterableDataset, returning the rest. This is often used for creating training or validation splits. Requires the 'datasets' library.

```python
>>> train_dataset = shuffled_dataset.skip(1000)
```

--------------------------------

### Load Specific Files with Glob Patterns (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/nlp_load

Loads specific files from a dataset using glob patterns. This allows you to select a subset of files, for example, all files matching a certain naming convention within a larger dataset. Uses the 'allenai/c4' dataset as an example.

```python
from datasets import load_dataset
c4_subset = load_dataset("allenai/c4", data_files="en/c4-train.0000*-of-01024.json.gz")
```

--------------------------------

### Load Video Dataset with Metadata (JSONL)

Source: https://huggingface.co/docs/datasets/v4.4.2/video_dataset

Loads a video dataset using a `metadata.jsonl` file. This JSONL file maps video files (using `file_name` or `*_file_name`) to associated metadata. It supports single video files, multiple input/output videos, or lists of video files.

```python
from datasets import load_dataset

# Example with single video file and additional feature
dataset_single = load_dataset("videofolder", data_dir="/path/to/folder_single", split="train")

# Example with multiple input and output video files
dataset_multi_io = load_dataset("videofolder", data_dir="/path/to/folder_multi_io", split="train")

# Example with a list of video files and a label
dataset_list = load_dataset("videofolder", data_dir="/path/to/folder_list", split="train")
```

--------------------------------

### Apply Image and Depth Map Augmentations to Dataset Examples in Python

Source: https://huggingface.co/docs/datasets/v4.4.2/depth_estimation

This Python function iterates through images and their corresponding depth maps within a dataset batch. It applies the defined augmentation pipeline (`aug`) to each pair and updates the dataset examples with the transformed 'pixel_values' (images) and 'labels' (depth maps).

```python
>>> def apply_transforms(examples):
...     transformed_images, transformed_maps = [], []
...     for image, depth_map in zip(examples["image"], examples["depth_map"]):
...         image, depth_map = np.array(image), np.array(depth_map)
...         transformed = aug(image=image, depth=depth_map)
...         transformed_images.append(transformed["image"])
...         transformed_maps.append(transformed["depth"])
...
...     examples["pixel_values"] = transformed_images
...     examples["labels"] = transformed_maps
...     return examples
```

--------------------------------

### Prepare Dataset for TensorFlow Training

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Prepares the dataset for TensorFlow compatibility using the `prepare_tf_dataset` method from the `transformers` library. This method wraps a Hugging Face Dataset as a `tf.data.Dataset`, automatically handling collation and batching for direct use with Keras `fit()` methods.

```python
import tensorflow as tf

tf_dataset = model.prepare_tf_dataset(
    dataset,
    batch_size=4,
)
```

--------------------------------

### Visualize Transformed Example for Object Detection (Python)

Source: https://huggingface.co/docs/datasets/v4.4.2/object_detection

Visualizes the 10th example of a dataset after transformations have been applied, specifically for object detection tasks. This code snippet demonstrates how to draw bounding boxes on an image using the transformed bounding box and category information. It relies on helper functions like `to_pil_image`, `draw_bounding_boxes`, and `box_convert`.

```python
>>> example = ds['train'][10]
>>> to_pil_image(
...     draw_bounding_boxes(
...         example['image'],
...         box_convert(example['bbox'], 'xywh', 'xyxy'),
...         colors='red',
...         labels=[categories.int2str(x) for x in example['category']]
...     )
... )
```

--------------------------------

### Load Audio Data as TensorFlow Tensors

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_tensorflow

Shows how to load audio data from file paths into a Hugging Face Dataset and format it for TensorFlow. Requires the 'datasets[audio]' extra to be installed. The output includes the audio array and sampling rate.

```python
from datasets import Dataset, Features, Audio, Image
audio = ["path/to/audio.wav"] * 10
features = Features({"audio": Audio()})
ds = Dataset.from_dict({"audio": audio}, features=features)
ds = ds.with_format("tf")
print(ds[0]["audio"]["array"])
print(ds[0]["audio"]["sampling_rate"])
```

--------------------------------

### Load Object Detection Dataset with ImageFolder

Source: https://huggingface.co/docs/datasets/v4.4.2/image_dataset

Loads an object detection dataset using the 'imagefolder' handler. It expects a JSONL file with 'file_name' and 'objects' (containing 'bbox' and 'categories') fields. The loaded dataset will have an 'objects' column with bounding box and category information.

```python
>>> dataset = load_dataset("imagefolder", data_dir="/path/to/folder", split="train")
>>> dataset[0]["objects"]
{"bbox": [[302.0, 109.0, 73.0, 52.0]], "categories": [0]}
```

--------------------------------

### Verify Albumentations Transformation

Source: https://huggingface.co/docs/datasets/v4.4.2/semantic_segmentation

Retrieves a transformed example from the dataset after `set_transform` has been applied and visualizes the augmented image and mask using the `visualize_seg_mask` function.

```python
# Assume dataset is loaded, transforms are set, and visualize_seg_mask is defined
index = 10
image = np.array(dataset[index]["pixel_values"])
mask = np.array(dataset[index]["label"])

visualize_seg_mask(image, mask)
```

--------------------------------

### Show Datasets CLI Help

Source: https://huggingface.co/docs/datasets/v4.4.2/cli

Displays the available commands and their usage for the Datasets CLI. This is a fundamental command to understand the CLI's capabilities.

```bash
datasets-cli --help
```

--------------------------------

### Load Dataset with Metadata from CSV

Source: https://huggingface.co/docs/datasets/v4.4.2/audio_dataset

Demonstrates loading an audio dataset where metadata is provided in a CSV file. The CSV file must contain a 'file_name' or similar field linking to the audio files, along with additional features. This allows for richer dataset creation beyond just audio and inferred labels.

```csv
file_name,additional_feature
0001.mp3,This is a first value of a text feature you added to your audio files
0002.mp3,This is a second value of a text feature you added to your audio files
0003.mp3,This is a third value of a text feature you added to your audio files
```

```python
from datasets import load_dataset

dataset = load_dataset("audiofolder", data_dir="/path/to/folder", data_files="metadata.csv")
```

--------------------------------

### Upload to an Organization Repository

Source: https://huggingface.co/docs/datasets/v4.4.2/share

Upload content to a repository owned by an organization by specifying the organization name in the `repo_id`. The example shows uploading to `MyCoolOrganization/my-cool-dataset`.

```bash
>>> huggingface-cli upload MyCoolOrganization/my-cool-dataset . . --repo-type dataset
https://huggingface.co/datasetsMyCoolOrganization/my-cool-dataset/tree/main/
```

--------------------------------

### Load and Inspect Beans Dataset

Source: https://huggingface.co/docs/datasets/v4.4.2/image_classification

Loads the 'AI-Lab-Makerere/beans' dataset using the datasets library and displays an example entry, showing the image, file path, and label.

```python
from datasets import load_dataset

dataset = load_dataset("AI-Lab-Makerere/beans")
print(dataset["train"Показать]10))

```

--------------------------------

### Format Dataset to JAX Arrays

Source: https://huggingface.co/docs/datasets/v4.4.2/use_with_jax

Converts a `datasets.Dataset` object to use JAX arrays as its format. This allows for direct integration with JAX operations. Requires `jax` and `jaxlib` to be installed.

```python
from datasets import Dataset
data = [[1, 2], [3, 4]]
ds = Dataset.from_dict({"data": data})
ds = ds.with_format("jax")
print(ds[0])
print(ds[:2])
```

--------------------------------

### Deactivate Virtual Environment (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Deactivates the currently active Python virtual environment. This returns your shell session to using the system's default Python interpreter.

```bash
source .env/bin/deactivate
```

--------------------------------

### Select combined percentages from a split using string slicing and ReadInstruction

Source: https://huggingface.co/docs/datasets/v4.4.2/loading

Demonstrates how to select a combination of percentage slices from a single split, such as the first 10% and the last 80%. Both string slicing and ReadInstruction methods are shown.

```python
>>> train_10_80pct_ds = datasets.load_dataset("ajibawa-2023/General-Stories-Collection", split="train[:10%]+train[-80%:]")
```

```python
>>> ri = (datasets.ReadInstruction("train", to=10, unit="%") + datasets.ReadInstruction("train", from_=-80, unit="%"))
>>> train_10_80pct_ds = datasets.load_dataset("ajibawa-2023/General-Stories-Collection", split=ri)
```

--------------------------------

### Query Dataset with FAISS Index

Source: https://huggingface.co/docs/datasets/v4.4.2/faiss_es

Loads the DPR question encoder, generates an embedding for a query question, and then uses the FAISS index to retrieve the nearest examples from the dataset.

```python
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
q_encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
q_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base")

question = "Is it serious ?"
question_embedding = q_encoder(**q_tokenizer(question, return_tensors="pt"))[0][0].numpy()
scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', question_embedding, k=10)
retrieved_examples["line"][0]
```

--------------------------------

### Loading Specific Dataset Configurations in Python

Source: https://huggingface.co/docs/datasets/v4.4.2/repository_structure

Python code example showing how to load specific dataset configurations ('main_data' or 'additional_data') from a Hugging Face Hub repository using the `load_dataset` function.

```python
from datasets import load_dataset

main_data = load_dataset("my_dataset_repository", "main_data")
additional_data = load_dataset("my_dataset_repository", "additional_data")
```

--------------------------------

### Activate Virtual Environment (Bash)

Source: https://huggingface.co/docs/datasets/v4.4.2/installation

Activates the Python virtual environment named '.env'. This command modifies your shell session to use the environment's Python interpreter and packages.

```bash
source .env/bin/activate
```

--------------------------------

### Prepare PyTorch DataLoader with Custom Collate Function

Source: https://huggingface.co/docs/datasets/v4.4.2/quickstart

Sets up a PyTorch `DataLoader` for training. It requires a custom `collate_fn` to batch the preprocessed images (`pixel_values`) and labels. The dataset is wrapped to facilitate batching for PyTorch models.

```python
from torch.utils.data import DataLoader
import torch

def collate_fn(examples):
    images = []
    labels = []
    for example in examples:
        images.append((example["pixel_values"]))
        labels.append(example["labels"])

    pixel_values = torch.stack(images)
    labels = torch.tensor(labels)
    return {"pixel_values": pixel_values, "labels": labels}

dataloader = DataLoader(dataset, collate_fn=collate_fn, batch_size=4)
```

--------------------------------

### Compute Dataset Embeddings with DPR

Source: https://huggingface.co/docs/datasets/v4.4.2/faiss_es

Loads a dataset and computes vector embeddings for each example using the downloaded DPR model and tokenizer. The embeddings are stored in a new 'embeddings' column.

```python
from datasets import load_dataset
ds = load_dataset('community-datasets/crime_and_punish', split='train[:100]')
ds_with_embeddings = ds.map(lambda example: {'embeddings': ctx_encoder(**ctx_tokenizer(example["line"], return_tensors="pt"))[0][0].numpy()})
```

--------------------------------

### Create Audio Dataset with AudioFolder Builder

Source: https://huggingface.co/docs/datasets/v4.4.2/create_dataset

Creates an audio dataset using the `audiofolder` builder. Similar to `ImageFolder`, this builder simplifies the creation of audio datasets from a folder structure, automatically handling features, splits, and labels. It supports various audio formats decoded via ffmpeg.

```python
from datasets import load_dataset

dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```