FedLUAR

https://github.com/swblaster/fedluar
Admin
FedLUAR is a federated learning framework implementing layer-wise update recycling for...
Tokens:4,127
Snippets:41
Trust Score:5.1
Update:6 months ago
Show doc for...
Tokens:
Raw
### Model Summary and Forward Pass Example

Source: https://context7.com/swblaster/fedluar/llms.txt

Generates a summary of a built model and demonstrates a forward pass with dummy input data. Useful for verifying model structure and output shapes.

```python
# Model summary
model_cifar10.summary()
# Output: resnet20 with ~270K parameters

# Example forward pass
import numpy as np
dummy_input = np.random.randn(32, 32, 32, 3).astype('float32')
predictions = model_cifar10(dummy_input)
# Output shape: (32, 10)
```

--------------------------------

### Initialize and Run the Federated Training Framework

Source: https://context7.com/swblaster/fedluar/llms.txt

Orchestrates the federated learning process, including MPI communication, local training, model averaging, and validation. Requires MPI and TensorFlow setup for distributed training.

```python
from train import framework
from mpi4py import MPI
import tensorflow as tf

# Initialize MPI environment
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

# Configure GPU visibility for distributed training
gpus = tf.config.experimental.list_physical_devices('GPU')
local_rank = rank % len(gpus)
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)
if gpus:
    tf.config.experimental.set_visible_devices(gpus[local_rank], 'GPU')

# Initialize the training framework
trainer = framework(
    models=models,              # List of local model instances
    dataset=dataset,            # Dataset feeder object
    solver=solver,              # FedAvg or FedLUAR solver
    num_epochs=200,             # Total training epochs
    min_lr=0.2,                 # Minimum learning rate
    max_lr=0.2,                 # Maximum learning rate
    decay_epochs=[100, 150],    # Epochs at which to decay LR
    num_classes=10,             # Number of output classes
    num_workers=32,             # Total number of workers
    num_clients=128,            # Total number of clients
    num_candidates=48,          # Candidates for client sampling
    average_interval=20,        # Local steps before averaging
    phi=1,                      # Client reshuffling interval
    do_checkpoint=0             # Enable/disable checkpointing
)

# Start training
trainer.train()

# Output files generated:
# - loss.txt: Training loss per epoch
# - acc.txt: Validation accuracy per epoch
# - ./checkpoint/: Model checkpoints (if enabled)

```

--------------------------------

### Get CIFAR Training and Validation Iterators

Source: https://context7.com/swblaster/fedluar/llms.txt

Retrieves iterators for training and validation datasets from the CIFAR feeder. Demonstrates fetching a batch of images and labels.

```python
# Get training dataset iterator for a specific client
train_iter = dataset_cifar.train_dataset(client_id=0)
images, labels = train_iter.next()
# images shape: (32, 32, 32, 3), labels shape: (32, 10)

# Get validation dataset iterator
valid_iter = dataset_cifar.valid_dataset()
images, labels = valid_iter.next()
```

--------------------------------

### Run Training with MPI

Source: https://github.com/swblaster/fedluar/blob/master/README.md

Execute the training process using mpirun with 8 processes.

```bash
mpirun -n 8 python main.py
```

--------------------------------

### Initialize FedAvg Solver

Source: https://context7.com/swblaster/fedluar/llms.txt

Sets up the standard Federated Averaging solver. Configure with the number of output classes, total workers, and the interval for local training steps before averaging.

```python
from solvers.fedavg import FedAvg
from mpi4py import MPI

# Initialize FedAvg solver
solver = FedAvg(
    num_classes=10,        # Number of output classes
    num_workers=32,        # Total number of workers
    average_interval=20    # Local training steps per round
)

# Solver methods:

# Execute one training round on a local worker

```

--------------------------------

### Initialize Client Sampler

Source: https://context7.com/swblaster/fedluar/llms.txt

Initializes the client sampling module with the total number of clients, active workers per round, and candidate pool size.

```python
from client_sampling import sampling

# Initialize sampler
sampler = sampling(
    num_clients=128,       # Total number of clients
    num_workers=32,        # Active workers per round
    num_candidates=48      # Candidate pool size
)
```

--------------------------------

### Launch Distributed Training with MPI

Source: https://context7.com/swblaster/fedluar/llms.txt

Command to execute distributed training using MPI after configuring the settings in config.py.

```bash
# Configure config.py with desired settings
# dataset = "cifar10"
# optimizer = 1  # FedLUAR
# num_workers = 32
# active_ratio = 0.25
# alpha = 0.1

# Launch distributed training with 8 MPI processes
mpirun -n 8 python main.py

# Output during training:
# ---------------------------------------------------
# dataset: cifar10
# number of workers: 32
# average interval: 20
# batch_size: 32
# training epochs: 200
# ---------------------------------------------------
# FedLUAR solver!
# Epoch 0 lr: 0.2 validation acc = 0.15 training loss = 2.3
# comm: 75.0 percent
# ...

# Generated output files:
# - loss.txt: One loss value per line, one per epoch
# - acc.txt: One accuracy value per line, one per epoch
# - weight.txt: Layer recycling weights (FedLUAR only)
```

--------------------------------

### Initialize and Execute FedLUAR Solver

Source: https://context7.com/swblaster/fedluar/llms.txt

Initializes the FedLUAR solver and executes a single training round. Requires model, dataset, and worker information.

```python
from solvers.fedluar import FedLUAR
from mpi4py import MPI
import config as cfg

# Initialize FedLUAR solver
solver = FedLUAR(
    model=models[0],       # Reference model for parameter shapes
    num_classes=10,        # Number of output classes
    num_workers=32,        # Total number of workers
    average_interval=20    # Local training steps per round
)

# Execute one training round (same interface as FedAvg)
loss_mean = solver.round(
    round_id=0,
    models=models,
    datasets=datasets,
    local_id=0
)
```

--------------------------------

### Initialize FEMNIST Dataset Feeder

Source: https://context7.com/swblaster/fedluar/llms.txt

Initializes a dataset feeder for the FEMNIST dataset. Configures batch size, workers, clients, classes, and active client ratio.

```python
# FEMNIST federated dataset (from LEAF benchmark)
dataset_femnist = federated_emnist(
    batch_size=20,
    num_workers=32,
    num_clients=128,
    num_classes=62,
    active_ratio=0.25
)
# Data path: leaf/data/femnist/data/train/
```

--------------------------------

### Initialize AG News Dataset Feeder

Source: https://context7.com/swblaster/fedluar/llms.txt

Initializes a dataset feeder for the AG News text classification dataset. Configures batch size, classes, clients, and partitioning alpha. Uses DistilBERT tokenizer.

```python
# AG News text classification dataset
dataset_agnews = agnews(
    batch_size=128,
    num_classes=4,
    num_clients=128,
    alpha=0.1
)
# Uses DistilBERT tokenizer for text preprocessing
# sample_length: 128 tokens max
```

--------------------------------

### Compute Similarity and Sample Clients

Source: https://context7.com/swblaster/fedluar/llms.txt

Utility functions for calculating cosine similarity between gradient tensors and performing power-of-choice client sampling based on Dirichlet weights.

```python
from util import cos_sim, power_d
import tensorflow as tf

# Compute cosine similarity between two lists of tensors
# Useful for comparing gradient directions across clients
gradients_a = [tf.random.normal((10, 10)) for _ in range(5)]
gradients_b = [tf.random.normal((10, 10)) for _ in range(5)]
similarity = cos_sim(gradients_a, gradients_b)
# Returns: scalar cosine similarity value

# Power-of-choice client sampling with custom weights
import numpy as np
num_clients = 128
num_workers = 32
weights = np.random.dirichlet(np.ones(num_clients))  # Probability weights
selected_clients = power_d(num_clients, num_workers, weights)
# Returns: array of 32 client indices sampled by weights
```

--------------------------------

### Initialize CIFAR Dataset Feeder

Source: https://context7.com/swblaster/fedluar/llms.txt

Initializes a dataset feeder for CIFAR-10/100. Configures batch size, number of workers, clients, classes, and partitioning alpha for non-IID data.

```python
from feeders.feeder_cifar import cifar
from feeders.feeder_femnist import federated_emnist
from feeders.feeder_agnews import agnews

# CIFAR-10/100 dataset feeder
dataset_cifar = cifar(
    batch_size=32,
    num_workers=32,
    num_clients=128,       # Total number of FL clients
    num_classes=10,
    alpha=0.1              # Dirichlet parameter for non-IID partitioning
)
```

--------------------------------

### Configure FedLUAR Hyperparameters and Settings

Source: https://context7.com/swblaster/fedluar/llms.txt

Defines dataset-specific configurations, FedLUAR settings, distributed training parameters, optimizer choice, and federated learning hyperparameters like active ratio and data heterogeneity.

```python
# config.py - Main configuration settings

# Dataset-specific configurations
cifar10_config = {
    "batch_size": 32,
    "min_lr": 0.2,
    "max_lr": 0.2,
    "num_classes": 10,
    "epochs": 200,
    "decay": {100, 150},
    "weight_decay": 0.0001,
}

cifar100_config = {
    "batch_size": 32,
    "min_lr": 0.1,
    "max_lr": 0.4,
    "num_classes": 100,
    "epochs": 300,
    "decay": {150, 200},
    "weight_decay": 0.0005,
}

femnist_config = {
    "batch_size": 20,
    "min_lr": 0.01,
    "max_lr": 0.01,
    "num_classes": 62,
    "epochs": 200,
    "decay": {100, 150},
    "weight_decay": 0.0001,
}

agnews_config = {
    "batch_size": 128,
    "min_lr": 0.00001,
    "max_lr": 0.00001,
    "num_classes": 4,
    "epochs": 100,
    "decay": {60, 80},
    "weight_decay": 0.0001,
}

# FedLUAR-specific settings
recycling_layers = 12  # Number of layers to recycle updates

# Distributed training settings
num_processes_per_node = 8
dataset = "cifar10"  # Options: "cifar10", "cifar100", "femnist", "agnews"
average_interval = 20  # Local steps before averaging
phi = 1
num_workers = 32
num_candidates = 48
checkpoint = 0  # Set to 1 to enable checkpointing

# Optimizer selection
# 0: FedAvg
# 1: FedLUAR
optimizer = 1

# Federated Learning settings
active_ratio = 0.25  # Device activation ratio (0.25, 0.5, 1)
alpha = 0.1  # Dirichlet's concentration parameter (0.1, 0.5, 1)

```

--------------------------------

### Perform Random Client Selection and Reset Sampler

Source: https://context7.com/swblaster/fedluar/llms.txt

Performs random client selection without replacement and resets the sampler state. Useful for managing client selection across training epochs or LR decay.

```python
# Random client selection (uniform sampling without replacement)
active_devices = sampler.random()
# Returns: numpy array of 32 selected client indices

# Reset sampler state (called after LR decay)
sampler.reset()
```

--------------------------------

### Define ResNet-20 Model for CIFAR-10

Source: https://context7.com/swblaster/fedluar/llms.txt

Builds a ResNet-20 model with specified weight decay, number of classes, and width multiplier. Use for CIFAR-10 datasets.

```python
from model import resnet20, wideresnet28, cnn, distilBert

# ResNet-20 for CIFAR-10
model_cifar10 = resnet20(
    weight_decay=0.0001,
    num_classes=10,
    ratio=1.0              # Width multiplier
).build_model()
```

--------------------------------

### Define WideResNet-28 Model for CIFAR-100

Source: https://context7.com/swblaster/fedluar/llms.txt

Builds a WideResNet-28-10 model with specified weight decay and number of classes. Optimized for CIFAR-100 datasets.

```python
# WideResNet-28-10 for CIFAR-100
model_cifar100 = wideresnet28(
    weight_decay=0.0005,
    num_classes=100
).build_model()
```

--------------------------------

### Define CNN Model for FEMNIST

Source: https://context7.com/swblaster/fedluar/llms.txt

Builds a simple CNN model with specified weight decay and number of classes. Suitable for FEMNIST handwritten character recognition.

```python
# Simple CNN for FEMNIST (handwritten characters)
model_femnist = cnn(
    weight_decay=0.0001,
    num_classes=62          # 62 character classes
).build_model()
```

--------------------------------

### Define DistilBERT Model for AG News

Source: https://context7.com/swblaster/fedluar/llms.txt

Builds a DistilBERT model for text classification with specified weight decay, sample length, and number of classes. Used for AG News dataset.

```python
# DistilBERT for AG News (text classification)
model_agnews = distilBert(
    weight_decay=0.0001,
    sample_length=128,      # Maximum sequence length
    num_classes=4           # 4 news categories
).build_model()
```

--------------------------------

### Average Model with Layer-wise Update Recycling

Source: https://context7.com/swblaster/fedluar/llms.txt

Averages model parameters across workers, selectively recycling layer updates to reduce communication. Active layers are communicated, while recycled layers reuse previous updates.

```python
# Average model with layer-wise update recycling
# - Active layers: Communicate and compute new updates
# - Recycled layers: Reuse previous updates without communication
solver.average_model(checkpoint, epoch_id=0)
```
Tokens:
Raw