### Clone Repository and Install Project Dependencies

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Clones the PFL research repository, navigates into the benchmarks directory, sets the Python environment for poetry, and installs project dependencies including PyTorch and TensorFlow extras. Finally, it activates the poetry shell.

```shell
git clone git@github.com:apple/pfl-research.git
cd pfl-research/benchmarks/
poetry env use `which python`
poetry install -E pytorch -E tf
poetry shell
```

--------------------------------

### Install Subset of Poetry Dependencies

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Installs specific subsets of project dependencies using poetry, controlled by extras. This is useful for installing only the dependencies needed for TensorFlow or PyTorch examples, or both, without development dependencies.

```shell
# Install to run tf examples
poetry install -E tf --no-dev
# Install to run pytorch examples
poetry install -E pytorch --no-dev
# Install to run tf, pytorch and tests
poetry install -E pytorch -E tf
```

--------------------------------

### Install Poetry

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Installs the latest version of Poetry, a dependency management tool for Python, using a curl script. This is a prerequisite for setting up the project environment.

```shell
curl -sSL https://install.python-poetry.org | python3 -
```

--------------------------------

### Clone pfl-research repository and navigate to benchmarks

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/installation.rst

Clones the 'pfl-research' repository from GitHub and changes the current directory to the 'benchmarks' folder. This is the starting point for accessing official benchmarks and for conducting custom research.

```bash
git clone https://github.com/apple/pfl-research.git
cd pfl-research/benchmarks
```

--------------------------------

### Install HuggingFace Modules

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Creating Federated Dataset for PFL Experiment.ipynb

Installs the necessary HuggingFace `datasets` and `transformers` libraries using pip. This is a prerequisite for the subsequent code examples.

```bash
!pip3 -q install datasets transformers
```

--------------------------------

### PFL Distributed Simulation Setup (Command Line)

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/guides/simulation_distributed.rst

This example shows how to set up PFL for distributed simulation on multiple GPUs on the same machine using environment variables and background processes. No changes to the Python code are required.

```bash
export PFL_WORKER_ADDRESSES=localhost:8000,localhost:8001
PFL_WORKER_RANK=0 python train.py &
PFL_WORKER_RANK=1 python train.py &

```

--------------------------------

### Install FedML CIFAR10 Benchmark

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/fedml/cifar10/README.md

Installs the necessary dependencies and sets up the FedML environment for the CIFAR10 benchmark. This script should be executed in the project root directory.

```bash
./setup.sh
```

--------------------------------

### Load Model Weights and Set Up Simulated Backend

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Loads a pre-trained model and initializes a simulated backend for federated learning. The backend is configured with training data and an aggregator. This setup is essential before starting the model training process.

```python
model.load('tutorial_model')

simulated_backend = SimulatedBackend(
    training_data=train_federated_dataset,
    val_data=None,
    aggregator=MaxMagAggregator())
```

--------------------------------

### Federated Learning Model Initialization and Training Setup

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This Python code snippet demonstrates the initialization of a federated learning model and the setup of a simulated backend for training. It includes loading a pre-trained model, configuring a simulated backend with training and validation data, and defining sparsity postprocessors. It also sets up hyperparameters for local model training.

```python
model.load('tutorial_model')

simulated_backend = SimulatedBackend(
    training_data=train_federated_dataset,
    val_data=None,
    postprocessors=[
        SparsifyTopK(0.5),
    ])

model_train_params = NNTrainHyperParams(
    local_learning_rate=0.5,
    local_num_epochs=2,
    local_batch_size=5)
```

--------------------------------

### Installing Documentation Dependencies

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/support/contributing.rst

A command to install only the documentation-related dependencies for the project using Poetry.

```bash
poetry install --only docs
```

--------------------------------

### Install Jupyter and IPython Kernel for MLX

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Federated Learning with CIFAR10 and MLX.ipynb

Installs Jupyter and adds a custom IPython kernel named 'notebook-tutorial' to the environment. This is a prerequisite for running the MLX-based federated learning notebook.

```bash
pip install jupyter
python -m ipykernel install --name "notebook-tutorial"
```

--------------------------------

### Install Environment and `pfl` for Local Jupyter Notebook

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Differential Privacy with Federated Learning.ipynb

Installs necessary packages and sets up the environment for running the tutorial locally with Jupyter. It ensures `pfl` and other required libraries like `matplotlib`, `nest_asyncio`, `torch`, and `tqdm` are installed.

```python
import sys
# Additional packages needed to run notebook
!{sys.executable} -m pip install matplotlib nest_asyncio torch tqdm
```

--------------------------------

### Install Environment and PFL for Google Colab

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Sets up the PFL research environment in Google Colab by cloning the repository, changing the working directory, and installing the PFL package along with other required libraries. It assumes PyTorch is pre-installed in the Colab environment.

```python
!git clone https://github.com/apple/pfl-research.git
import os
# Set the working directory to same as if running locally.
os.chdir('pfl-research/tutorials')
```

```python
import sys
# PyTorch should already be installed in colab
!{sys.executable} -m pip install pfl
# Additional packages to run notebook
!{sys.executable} -m pip install h5py matplotlib nest_asyncio pandas torchvision
```

--------------------------------

### Run Distributed CIFAR10 Training with Horovod

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Initiates a distributed training job for the CIFAR10 CNN using Horovod. This example runs the training with 2 processes on the same machine ('localhost') using the Gloo backend.

```shell
horovodrun --gloo -np 2 -H localhost:2 python image_classification/pytorch/train.py --args_config image_classification/configs/baseline.yaml
```

--------------------------------

### Configure PyTorch Model with Mixed Precision and Torch Compile

Source: https://context7.com/apple/pfl-research/llms.txt

Demonstrates how to initialize a PyTorch model for federated learning with options for mixed precision (AMP) using `torch.float16` and faster training via `torch.compile`. Includes examples for setting parameters, getting model differences, and evaluating the model.

```python
pfl_model_amp = PyTorchModel(
    model=pytorch_model,
    local_optimizer_create=lambda params: torch.optim.Adam(params, lr=0.001),
    central_optimizer=torch.optim.SGD(pytorch_model.parameters(), lr=1.0),
    amp_dtype=torch.float16
)

pfl_model_compiled = PyTorchModel(
    model=pytorch_model,
    local_optimizer_create=torch.optim.Adam,
    central_optimizer=torch.optim.SGD(pytorch_model.parameters(), lr=1.0),
    use_torch_compile=True
)

# Model operations
params = pfl_model.get_parameters()  # Get current weights as MappedVectorStatistics
pfl_model.set_parameters(params)     # Restore weights
diff = pfl_model.get_model_difference(params)  # Compute update

# Evaluate model
from pfl.data.dataset import Dataset
eval_data = Dataset(raw_data=(torch.randn(100, 20).long(), torch.randint(0, 10000, (100,)))
metrics = pfl_model.evaluate(
    dataset=eval_data,
    name_formatting_fn=lambda n: f"eval_{n}"
)
print(metrics)
# Output: {'eval_loss': Weighted(...), 'eval_accuracy': Weighted(...), 'eval_perplexity': Weighted(...)}
```

--------------------------------

### Install Environment and PFL for Local Jupyter Notebooks

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Installs necessary packages and configures the Jupyter environment for local PFL research. It uses pip to install packages like h5py, matplotlib, pandas, and torchvision, and requires restarting the kernel after installation.

```python
import sys
# Additional packages to run notebook
!{sys.executable} -m pip install h5py matplotlib nest_asyncio pandas torchvision
```

--------------------------------

### TensorFlow Multi-GPU Training Setup (Environment Variables)

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/guides/simulation_distributed.rst

This example illustrates setting up TensorFlow for distributed training across multiple GPUs on a single machine. It involves defining worker addresses via an environment variable and launching multiple Python processes with unique ranks.

```bash
export PFL_WORKER_ADDRESSES=localhost:8000,localhost:8001,localhost:8002,localhost:8003,localhost:8004,localhost:8005,localhost:8006,localhost:8007
for i in {0..7}; do PFL_WORKER_RANK=$i python train.py & done

```

--------------------------------

### Install pfl with Deep Learning Frameworks using pip

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/installation.rst

Installs the 'pfl' library from PyPI with optional extras for deep learning frameworks. Supported extras include 'tf' for TensorFlow and Keras, and 'pytorch' for PyTorch. This command assumes pip is installed and configured.

```bash
pip install 'pfl[tf]'

pip install 'pfl[pytorch]'

pip install 'pfl[trees]'
```

--------------------------------

### Install Hugging Face Modules for LLM Benchmarks

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/llm/README.md

Installs essential Hugging Face libraries required for running federated LLM benchmarks. This command uses pip to install the transformers, peft, and datasets modules.

```bash
pip install transformers peft datasets
```

--------------------------------

### Run All Tests with tox

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/support/contributing.rst

Tests code compatibility across different environments. Install tox and run in the root directory.

```bash
tox
```

--------------------------------

### Installing Project Dependencies with Poetry

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/support/contributing.rst

Commands to configure Poetry to use the current Python environment and install project dependencies with specific extras and development groups.

```bash
poetry env use `which python`
poetry install -E pytorch -E tf -E trees --with dev,docs
poetry shell
```

--------------------------------

### Install MLX and Supporting Packages

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Federated Learning with CIFAR10 and MLX.ipynb

Installs necessary Python packages including matplotlib, nest_asyncio, and pfl with the MLX extras. nest_asyncio is required to run asynchronous operations within the Jupyter notebook environment.

```python
import sys
# Additional packages needed to run notebook
!{sys.executable} -m pip install matplotlib nest_asyncio
```

--------------------------------

### Install PFL with PyTorch Support for Colab

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Creating Federated Dataset for PFL Experiment.ipynb

Installs the 'pfl' package with PyTorch support on Google Colab. This is a prerequisite for using PyTorch-based federated learning functionalities.

```python
import sys
# PyTorch should already be installed in colab
!{sys.executable} -m pip install pfl
```

--------------------------------

### Install Environment and `pfl` for Google Colab

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Differential Privacy with Federated Learning.ipynb

Installs `pfl` and other prerequisite packages for running the tutorial in Google Colab. It clones the `pfl-research` repository and adjusts the working directory.

```python
!git clone https://github.com/apple/pfl-research.git
import os
# Set the working directory to same as if running locally.
os.chdir('pfl-research/tutorials')
```

```python
import sys
# PyTorch should already be installed in colab
!{sys.executable} -m pip install pfl
# Additional packages to run notebook
!{sys.executable} -m pip install matplotlib nest_asyncio torch tqdm
```

--------------------------------

### Download and Preprocess CIFAR10 Data

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/flute/cifar10/README.md

This script downloads and preprocesses the CIFAR10 dataset using the `pfl-research` library. It requires Python and the `pfl-research` package to be installed. The output is directed to a specified directory.

```shell
( cd ../../../../benchmarks/ 
  python -m dataset.cifar10.download_preprocess --output_dir ../publications/pfl/flute/cifar10/data/cifar10 
)
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Creates a new conda environment named 'py310' with Python 3.10 and then activates it. This ensures a compatible Python version for the project.

```shell
conda create -n py310 python=3.10
conda activate py310
```

--------------------------------

### Add Project Root to PYTHONPATH

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Exports the current working directory (benchmarks directory) to the PYTHONPATH environment variable. This allows the utility modules within the project to be imported correctly.

```shell
export PYTHONPATH=`pwd`:$PYTHONPATH
```

--------------------------------

### Run FLUTE CIFAR10 Benchmark Training

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/flute/cifar10/README.md

This command executes the end-to-end trainer for the CIFAR10 benchmark using FLUTE. It requires the FLUTE repository to be cloned and its dependencies installed. The command sets up the Python path, specifies the number of processes per node, and points to configuration and data directories.

```shell
cd msrflute
PYTHONPATH=./core/:.:../:../../../ python -m torch.distributed.run  --nproc_per_node=1  e2e_trainer.py -dataPath ./data -outputPath ./outputTest  -config ./experiments/cifar10/config.yaml -task cifar10 -backend nccl
```

--------------------------------

### Install `pfl` and Dependencies (Colab)

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Federated Learning with CIFAR10 and TensorFlow.ipynb

Installs the `pfl` library with TensorFlow support and other required packages like `matplotlib` and `nest_asyncio` for use in a Google Colab environment. It also clones the `pfl-research` repository and adjusts the working directory.

```python
!git clone https://github.com/apple/pfl-research.git
import os
# Set the working directory to same as if running locally.
os.chdir('pfl-research/tutorials')
```

```python
import sys
!{sys.executable} -m pip install pfl[tf]
# Additional packages needed to run notebook
!{sys.executable} -m pip install matplotlib nest_asyncio
```

--------------------------------

### Import Libraries and Setup PFL

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Imports essential Python libraries including matplotlib, h5py, numpy, pandas, sys, and torch for data manipulation and machine learning. It also applies `nest_asyncio` to allow PFL to run within the notebook's event loop and sets up manual seeds for reproducibility.

```python
import matplotlib.pyplot as plt
import h5py
import numpy as np
import pandas as pd
import sys
import torch

# Both Jupyter and `pfl` use async. `nest_asyncio` allows `pfl` to run inside the notebook 
import nest_asyncio
nest_asyncio.apply()

# append the root directory to your paths to be able to reach the examples.  
sys.path.append('../benchmarks')
torch.random.manual_seed(1)
np.random.seed(1)

# Always import the `pfl` model first before initializing any `pfl` components to let `pfl` know which Deep Learning framework you will use.
from pfl.model.pytorch import PyTorchModel
```

--------------------------------

### Train CNN on CIFAR10

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/README.md

Trains a small Convolutional Neural Network (CNN) on the CIFAR10 dataset using PyTorch. Configuration is loaded from a specified YAML file.

```python
python image_classification/pytorch/train.py --args_config image_classification/configs/baseline.yaml
```

--------------------------------

### Create FederatedDataset with torch.utils.data.Dataset for Large Datasets

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Creating Federated Dataset for PFL Experiment.ipynb

This example demonstrates creating a `FederatedDataset` that supports `torch.utils.data.Dataset` for handling user datasets that may be too large to fit into memory. It defines a `make_dataset_fn` to create individual user datasets as `torch.utils.data.TensorDataset` and then initializes `FederatedDataset` with this function and a `user_sampler`.

```python
def make_dataset_fn(user_id):
    # Create a `torch.utils.data.Dataset` for a single user
    user_data = [torch.as_tensor(data) for data in user_id_to_data[user_id]]
    pytorch_data = torch.utils.data.TensorDataset(*user_data)
    return PyTorchDataDataset(raw_data=pytorch_data, user_id=user_id)

federated_dataset = FederatedDataset(make_dataset_fn=make_dataset_fn, user_sampler=user_sampler)
print("FederatedDataset with torch.utils.data.Dataset and real user partition: ")
for i in range(5):
    user_dataset, seed = next(federated_dataset)
    print(f"\tReal user {i} has size of {len(user_dataset)}.")
```

--------------------------------

### Run FedScale CIFAR10 Benchmark

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/fedscale/cifar10/README.md

This snippet shows how to run the FedScale CIFAR10 benchmark. It involves setting environment variables for log directory and FedScale home, then executing the driver script with a configuration file. Ensure FedScale is installed and configured correctly before running.

```bash
export LOG_DIR=./logs
export FEDSCALE_HOME=.
python FedScale/docker/driver.py start fedscale_config.yaml
```

--------------------------------

### Federated Learning Model Training Setup (Python)

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This Python snippet initializes and configures a federated learning model for training. It involves loading pre-trained weights, setting up a local learning rate scheduler, defining a simulated backend with training and validation data, and specifying hyperparameters for local model training. The code then executes the federated learning algorithm using these components.

```python
model.load('tutorial_model')

local_lr = LocalLRWarmup(0.0001, 0.1)

simulated_backend = SimulatedBackend(
    training_data=train_federated_dataset,
    val_data=None)

model_train_params = NNTrainHyperParams(
    local_learning_rate=local_lr,
    local_num_epochs=2,
    local_batch_size=5)

model = MyAlgorithm().run(
    algorithm_params=algorithm_params,
    backend=simulated_backend,
    model=model,
    model_train_params=model_train_params,
    model_eval_params=model_eval_params,
    callbacks=callbacks + [local_lr])
```

--------------------------------

### Initialize MLX and PFL Environment

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Federated Learning with CIFAR10 and MLX.ipynb

Initializes the MLX framework, sets random seeds for reproducibility, and imports necessary modules including `pfl.model.mlx.MLXModel`. It also applies `nest_asyncio` to enable `pfl`'s asynchronous operations within the notebook.

```python
import matplotlib.pyplot as plt
import numpy as np
import sys
import mlx
import mlx.core as mx
import mlx.nn as nn

# Both Jupyter and `pfl` uses async, nest_asyncio allows `pfl` to run inside the notebook 
import nest_asyncio
nest_asyncio.apply()

# append the root directory to your paths to be able to reach the examples.  
sys.path.append('../benchmarks')
mx.random.seed(7)
np.random.seed(7)

# Always import the `pfl` model first to let `pfl` know which Deep Learning framework you will use.
from pfl.model.mlx import MLXModel
```

--------------------------------

### Python: Load and Run Model Training Algorithm

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This Python code snippet demonstrates how to load a pre-trained model and then run a custom training algorithm. It requires specifying algorithm parameters, a backend, the model itself, and parameters for model training and evaluation, along with any necessary callbacks. The output is an updated model ready for further steps.

```Python
model.load('tutorial_model')

model = MyAlgorithm().run(
    algorithm_params=algorithm_params,
    backend=simulated_backend,
    model=model,
    model_train_params=model_train_params,
    model_eval_params=model_eval_params,
    callbacks=callbacks)

```

--------------------------------

### Download and Preprocess CIFAR10 Dataset

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Federated Learning with CIFAR10 and MLX.ipynb

Executes a script to download and preprocess the CIFAR10 dataset, saving the processed data into pickle files in the specified output directory. This step prepares the dataset for use in federated learning simulations.

```bash
!(cd ../benchmarks; {sys.executable} -m dataset.cifar10.download_preprocess --output_dir ./data/cifar10)
```

--------------------------------

### Download Small FLAIR Dataset

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Downloads a small, preprocessed version of the FLAIR dataset using wget. The dataset is saved to the '../benchmarks/data/' directory and is intended for easier testing and running of the notebook.

```bash
!wget -P ../benchmarks/data/ https://pfl-data.s3.us-east-2.amazonaws.com/flair/flair_federated_small.hdf5
```

--------------------------------

### Install gRPC for macOS M1

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/flower/cifar10/README.md

Installs gRPC and gRPC-tools with specific linker flags required for compatibility on macOS M1 machines. This is a prerequisite for running Flower benchmarks locally on M1.

```bash
GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation" pip install grpcio --no-binary :all:
GRPC_PYTHON_LDFLAGS=" -framework CoreFoundation" pip install grpcio-tools --no-binary :all:
```

--------------------------------

### Create PFL PyTorchModel Adapter

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This Python code demonstrates how to create a `PyTorchModel` adapter for use with the 'pfl' library. It configures the model with a local optimizer and a central optimizer, and includes saving the initial model state.

```python
params = [p for p in pytorch_model.parameters() if p.requires_grad]

model = PyTorchModel(pytorch_model, 
                     local_optimizer_create=torch.optim.SGD,
                     central_optimizer=torch.optim.SGD(params, 1.0))

# Save initial model
model.save('tutorial_model')
```

--------------------------------

### Install TensorFlow Federated and Dependencies

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/tff/cifar10/README.md

Installs TensorFlow, TensorFlow Federated, and related libraries necessary for the CIFAR10 benchmark. This command uses pip for package management and specifies exact versions for TensorFlow and TensorFlow Federated.

```bash
git clone git@github.com:google-research/federated.git google-research-federated 
pip install tensorflow==2.11 tensorflow_federated==0.48.0 \
    tensorflow_datasets \
    tensorflow_addons \
    pandas -f https://storage.googleapis.com/jax-releases/jax_releases.html
```

--------------------------------

### Initialize SimulatedBackend for PFL

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This Python code snippet shows the initialization of `SimulatedBackend` from the 'pfl' library. This backend is used for simulating the aggregation of statistics from user devices in a federated learning context.

```python
from pfl.aggregate.simulate import SimulatedBackend

cohort_size = 10
central_num_iterations = 5
```

--------------------------------

### Install pfl framework with optional dependencies

Source: https://github.com/apple/pfl-research/blob/develop/README.md

Installs the 'pfl' Python framework using pip. It supports optional dependencies for TensorFlow ('tf'), PyTorch ('pytorch'), and tree-based models ('trees').

```bash
pip install 'pfl[tf,pytorch,trees]'
```

--------------------------------

### Run Flower CIFAR10 Benchmark

Source: https://github.com/apple/pfl-research/blob/develop/publications/pfl/flower/cifar10/README.md

Launches the CIFAR10 benchmark using Python and a specified configuration. This command sets the PYTHONPATH and executes the main script with the provided config-path.

```bash
PYTHONPATH=../../:. python -m main --config-path conf/cifar10
```

--------------------------------

### Instantiate Simulated Backend for Federated Learning

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Initializes a `SimulatedBackend` for federated learning simulations. This backend requires training data and optionally validation data. It simulates the behavior of distributed data and devices for the federated learning process.

```python
simulated_backend = SimulatedBackend(
    training_data=train_federated_dataset,
    val_data=None)

```

--------------------------------

### Iterate and Visualize Federated Dataset Sample

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Demonstrates how to iterate through a `FederatedDataset`, sample a user dataset, and visualize the first few images along with their corresponding labels. It utilizes `itertools.islice` for efficient iteration and `matplotlib` for plotting.

```python
import itertools
import matplotlib.pyplot as plt
import torch

user, seed = next(train_federated_dataset)
print('User: {}\nunique user seed: {}\ndataset length: {}\nfirst 10 images:'.format(user.user_id, seed, len(user)))
fig, axes = plt.subplots(1,min(len(user),10),figsize=(20,12))
for ax, image, label in itertools.islice(zip(axes, *user.raw_data),10):
    ax.set_title('labels={}'.format(torch.nonzero(label).squeeze().tolist()))
    ax.imshow((image.cpu().numpy()*255+128).astype(np.uint8))
```

--------------------------------

### Initialize FederatedDataset

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

Demonstrates the initialization of a FederatedDataset using a user sampling function and a dataset creation function. This is a key step for setting up federated learning experiments.

```python
from pfl.data.dataset import Dataset
from pfl.data.federated_dataset import FederatedDataset

# Assuming user_sampler and make_dataset_fn are defined elsewhere
# federated_dataset = FederatedDataset(user_sampler, make_dataset_fn)
```

--------------------------------

### Setup Simulated Backend for FL Training (Python)

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to Differential Privacy with Federated Learning.ipynb

Configures a SimulatedBackend for federated learning training. This setup explicitly excludes Differential Privacy (DP) by providing an empty list for postprocessors. It takes federated datasets for training and validation as input.

```python
# FL training, without using DP

# remove central privacy from postprocessors
simulated_backend = SimulatedBackend(
    training_data=train_federated_dataset,
    val_data=val_federated_dataset,
    postprocessors=[])

```

--------------------------------

### Import Libraries for CIFAR10 User Simulation

Source: https://github.com/apple/pfl-research/blob/develop/publications/mdm/mdm_paper/notebooks/cifar10_visualisations.ipynb

Imports necessary libraries for data manipulation, machine learning, and visualization, including specific modules from Ramsay and Polya Mixture for data sampling and preprocessing.

```python
import os

import matplotlib.pyplot as plt
import numpy as np
from sklearn.manifold import TSNE
import joblib

from ramsay.data.sampling import get_data_sampler
from polya_mixture.datasets.cifar10_dataset import load_and_preprocess
from polya_mixture.datasets.sampler import DirichletDataSampler
```

--------------------------------

### Run Unit Tests with pytest

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/support/contributing.rst

Executes unit tests to ensure code functionality. Install pytest and run using make or poetry commands.

```bash
make test
```

```bash
poetry run pytest -svx
```

--------------------------------

### Get User Sampler (Python)

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/guides/fl_introduction.rst

Demonstrates how to obtain a user sampler from the PFL library. It shows the usage of 'minimize_reuse' sampling strategy and iterates through sampled user IDs.

```python
from pfl.data.sampling import get_user_sampler
user_ids = ['user1', 'user2', 'user3']
sampler = get_user_sampler('minimize_reuse', user_ids)
for _ in range(5):
   print('sampled ', sampler())
```

--------------------------------

### Download and Preprocess CIFAR10 Dataset

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/image_classification/README.md

This command downloads the CIFAR10 dataset from its original source and preprocesses it into a pickle format. It requires Python to be installed and specifies an output directory for the processed data.

```shell
python -m dataset.cifar10.download_preprocess --output_dir data/cifar10
```

--------------------------------

### Python: Get Default Framework Module for Operations

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/support/contributing.rst

Demonstrates how to import and utilize the default framework module selector in PFL to access framework-specific operations. This is a common pattern for abstracting deep learning framework dependencies.

```python
from pfl.internal.ops.selector import get_default_framework_module as ops
ops().get_shape(tensor)
```

--------------------------------

### Download and Preprocess StackOverflow Dataset

Source: https://github.com/apple/pfl-research/blob/develop/benchmarks/lm/README.md

This command downloads and preprocesses the StackOverflow dataset. It requires TensorFlow to be installed. Arguments for vocabulary size and maximum sequence length are available, with defaults set to 10000 and 20 respectively.

```Python
python -m dataset.stackoverflow.download_preprocess --output_dir data/stackoverflow
```

--------------------------------

### Get User Image Counts and IDs

Source: https://github.com/apple/pfl-research/blob/develop/tutorials/Introduction to PFL research with FLAIR.ipynb

This snippet retrieves the number of images per user from an HDF5 file and sorts the user IDs. It is used to understand dataset distribution and prepare for federated learning.

```python
user_num_images = get_user_num_images(hdf5_path, 'train')
user_ids = sorted(list(user_num_images.keys()))
```

--------------------------------

### Create Federated Dataset Instance (Python)

Source: https://github.com/apple/pfl-research/blob/develop/docs/source/guides/fl_introduction.rst

Illustrates how to instantiate a FederatedDataset using a provided dataset creation function and a user sampler. This is a key step before iterating through the federated dataset.

```python
dataset = FederatedDataset(make_dataset_fn, sampler)
```