### Basic FedML Client Initialization and Run

Source: https://docs.tensoropera.ai/federate/cross-silo/example/mqtt_s3_fedavg_attack_mnist_lr_example

Initializes the FedML framework, sets up the device, loads data and model, and starts the client training process. This is a standard setup for a FedML client.

```python
import fedml
from fedml.cross_silo import Client

if __name__ == "__main__":
    args = fedml.init()

    # init device
    device = fedml.device.get_device(args)

    # load data
    dataset, output_dim = fedml.data.load(args)

    # load model
    model = fedml.model.create(args, output_dim)

    # start training
    client = Client(args, device, dataset, model)
    client.run()
```

--------------------------------

### Setup and Bind Node

Source: https://docs.tensoropera.ai/render

Run this command to verify the installation and bind the node to the TensorOpera platform using your Render auth token.

```bash
sudo curl -sSf https://doc.fedml.ai/render/shell/setup.sh | bash  
```

--------------------------------

### Install FedML from source

Source: https://docs.tensoropera.ai/open-source/installation

Clones the repository and installs the package using setup.py.

```bash
git clone https://github.com/FedML-AI/FedML.git && \  
cd ./FedML/python && \  
python setup.py install  
```

```bash
git clone https://github.com/FedML-AI/FedML.git && \  
cd ./FedML/python && \  
python install -e '.[tensorflow]'  
python install -e '.[jax]'  
python install -e '.[mxnet]'  
```

```bash
git clone https://github.com/FedML-AI/FedML.git && \  
cd ./FedML && git checkout COMMIT_ID && \  
cd python && \  
python setup.py install  
```

--------------------------------

### Confirm Job Launch and Get Run URL

Source: https://docs.tensoropera.ai/launch/on-cloud/cloud-cluster

Once resources are confirmed, the cluster is created, and the job runs. You will receive a URL to track your run details. This example shows the confirmation prompt and the subsequent run URL.

```bash
Do you want to launch the job with the above matched GPU resource? [y/N]: y  
  
You can track your run details at this URL:  
https://tensoropera.ai/run?projectId=1717276102352834560&runId=1717307383354626048  
  
For querying the realtime status of your run, please run the following command.  
fedml run logs -rid 1717307383354626048  
```

--------------------------------

### Run Base Framework Example

Source: https://docs.tensoropera.ai/federate/simulation/examples/mpi_torch_fedavg_mnist_lr_example

Executes the base framework example for developing new federated learning algorithms. This script initializes FedML and starts the MPI simulator, providing a foundation for custom algorithm implementation.

```python
import fedml
from fedml import SimulatorMPI

if __name__ == "__main__":
    # init FedML framework
    args = fedml.init()

    # start training
    simulator = SimulatorMPI(args, None, None, None)
    simulator.run() 
```

--------------------------------

### Install FedML

Source: https://docs.tensoropera.ai/federate/getting_started

Install the FedML library using pip. This is the first step to get started with the FedML open-source library.

```bash
pip install fedml  

```

--------------------------------

### Jobs Start

Source: https://docs.tensoropera.ai/federate/cli

Instructions and examples for starting a job on the MLOps platform using the `fedml jobs start` command.

```APIDOC
## Jobs Start

Start a job at the MLOps platform using the `fedml jobs start` command.

### Usage

```bash
fedml jobs start [OPTIONS]
```

### Options

- **-pf, --platform** (TEXT) - The platform name at the MLOps platform (options: octopus, parrot, spider, beehive).
- **-prj, --project_name** (TEXT) - The project name at the MLOps platform.
- **-app, --application_name** (TEXT) - Application name in the My Application list at the MLOps platform.
- **-d, --devices** (TEXT) - The devices with the format: `[{"serverId": 727, "edgeIds": ["693"], "account": 105}]`.
- **-u, --user** (TEXT) - User ID or API key.
- **-k, --api_key** (TEXT) - User API key.
- **-v, --version** (TEXT) - Start job at which version of MLOps platform (options: dev, test, release).
- **--help** - Show this message and exit.

### Example

```bash
fedml jobs start -pf octopus -prj test-fedml -app test-alex-app -d '[{"serverId":706,"edgeIds":["705"],"account":214}]' -u 214 -k c9356b9c4ce44363bb66366d21030
```
```

--------------------------------

### Download Docker Installation Script

Source: https://docs.tensoropera.ai/open-source/installation/rpi

Fetches the official Docker installation script from the web.

```bash
curl -fsSL https://get.docker.com -o get-docker.sh  
```

--------------------------------

### Build Federated Learning Setup with FedML

Source: https://docs.tensoropera.ai/category/command-line-interfaces-clis

Configures and builds the setup for federated learning.

```bash
fedml federate build [OPTIONS]
```

--------------------------------

### View directory structure

Source: https://docs.tensoropera.ai/launch/on-cloud/install

Display the file structure of the hello_world example project.

```bash
tree -l  
  
/example  
├── hello_world  
|  ├── hello_world.py  
├── job.yaml
```

--------------------------------

### Run Step-by-Step FL Example

Source: https://docs.tensoropera.ai/federate/simulation/examples/mpi_torch_fedavg_mnist_lr_example

Execute the federated learning training script with a specified number of workers. This command initiates the step-by-step example for MPI-based FL simulation.

```shell
sh run_step_by_step_example.sh 4  
```

--------------------------------

### Example Job Launch Command and Output

Source: https://docs.tensoropera.ai/launch/on-prem/install

Example of executing the `fedml launch` command with a specific cluster name and its expected output, indicating the job has been submitted.

```text
fedml launch job.yaml -c hello-world  

```

--------------------------------

### Install fedml library

Source: https://docs.tensoropera.ai/deploy/quickstart

Install the required serving library provided by TensorOpera AI.

```bash
pip install fedml  
```

--------------------------------

### Install NVIDIA Drivers and CUDA Toolkit

Source: https://docs.tensoropera.ai/render/prerequisite

If NVIDIA drivers are not installed, use these commands to automatically install them, followed by the CUDA toolkit. A reboot is required after installation. Finally, verify the installation with `nvidia-smi`.

```bash
sudo ubuntu-drivers autoinstall
sudo apt install nvidia-cuda-toolkit
sudo reboot
nvidia-smi
```

--------------------------------

### Full Model Configuration YAML Example

Source: https://docs.tensoropera.ai/deploy/yaml_ref

This example demonstrates all supported configuration options for creating a model card, covering common required fields, fedml-based Docker images, customized Docker images, and optional fields for GPU cloud and on-premise modes.

```yaml
# Common required fields  
workspace: "./src"  
  
# Required fields for fedml based docker image, i.e. starts with fedml/ in the Dockerhub  
entry_point: "main_entry.py"  
  
# Required fields for customized docker image  
container_run_command: "tritonserver --model-repository=/repo_inside_container"  
  
# Common optional fields  
inference_image_name: "fedml/fedml-default-inference-backend"  
image_pull_policy: "IfNotPresent"  
registry_specs:  
    registry_user_name: "fedml"  
    registry_user_password: "passwd"  
    registry_provider: "DockerHub"  
bootstrap: |
  echo "Bootstrap start..."  
  sh ./config/bootstrap.sh  
  echo "Bootstrap finished"  
port: 2345  
volume_mounts:   
  - workspace_path: "./path/to/workspace"  
    mount_path: "/path/to/container"  
environment_variables:  
  PROMPT_STYLE: "llama_orca"  
readiness_probe:  
  httpGet:  
    path: "/ready"  
service:  
  httpPost:  
    path: "/predict"  
expose_subdomains: true  
deploy_timeout: 600  
request_input_example: '{"text": "Hello"}'  
  
# GPU Cloud mode optional fields  
endpoint_api_type: "text2text_llm_openai_chat_completions"  
  
# On-premise mode optional fields  
use_gpu: true  
num_gpus: 1  
authentication_token: "myPrivateToken"  

```

--------------------------------

### Install FedML Library

Source: https://docs.tensoropera.ai/deploy/create_model

Install the TensorOpera serving library required for model card creation.

```bash
pip install fedml
```

--------------------------------

### Run Execution Commands

Source: https://docs.tensoropera.ai/federate/federated-analytics/Examples/heavy_hitter_twitter_data_example

Commands to start the server and individual client nodes.

```bash
bash run_server.sh a  
```

```bash
bash run_client.sh 1 a  
```

```bash
bash run_client.sh 2 a  
```

--------------------------------

### FedML Simulation Entry Point

Source: https://docs.tensoropera.ai/federate/simulation/examples/sp_fedavg_mnist_lr_example

This Python script serves as the entry point for running a federated learning simulation using the FedML library. It requires the fedml library to be installed and typically calls fedml.run_simulation() to start the process.

```python
import fedml  

if __name__ == "__main__":  
    fedml.run_simulation()  
```

--------------------------------

### Install FedML Library

Source: https://docs.tensoropera.ai/launch/share-and-earn

Execute the binding script to install the FedML library and required dependencies on the compute node.

```bash
sudo curl -sSf https://doc.fedml.ai/shell/bind_gpu.sh | bash && exec $SHELL  
```

--------------------------------

### Start Server Execution

Source: https://docs.tensoropera.ai/federate/federated-analytics/Examples/frequency_estimation_fake_data_example

Command to initiate the server process. Replace 'a' with your desired run ID.

```bash
bash run_server.sh a  
```

--------------------------------

### Start Client 1 Execution

Source: https://docs.tensoropera.ai/federate/federated-analytics/Examples/frequency_estimation_fake_data_example

Command to start the first client. Ensure the rank is set to 1 and the run ID matches the server's.

```bash
bash run_client.sh 1 a  
```

--------------------------------

### Execution Commands

Source: https://docs.tensoropera.ai/federate/cross-silo/example/mqtt_s3_fedavg_hierarchical_mnist_lr_example

Commands to start the server and individual client nodes.

```bash
bash run_server.sh  
```

```bash
bash run_client.sh 1  
```

```bash
bash run_client.sh 2  
```

--------------------------------

### Install FedML and Libraries

Source: https://docs.tensoropera.ai/render

Execute this command on the compute node to install the necessary FedML libraries and dependencies.

```bash
sudo curl -sSf https://doc.fedml.ai/render/shell/install.sh | bash && exec $SHELL  
```

--------------------------------

### Install NVIDIA Drivers and CUDA Toolkit

Source: https://docs.tensoropera.ai/launch/share-and-earn

Install necessary drivers and the CUDA toolkit if they are not present on the system.

```bash
sudo ubuntu-drivers autoinstall  
sudo apt install nvidia-cuda-toolkit  
sudo reboot  
nvidia-smi  
```

--------------------------------

### Start Client 2 Execution

Source: https://docs.tensoropera.ai/federate/federated-analytics/Examples/frequency_estimation_fake_data_example

Command to start the second client. Ensure the rank is set to 2 and the run ID matches the server's.

```bash
bash run_client.sh 2 a  
```

--------------------------------

### Project Directory Structure Example

Source: https://docs.tensoropera.ai/launch/on-prem/install

Example directory structure for a FedML job, showing the location of source code (`hello_world.py`, `requirements.txt`) and the `job.yaml` configuration file.

```text
❯ tree -l 2  
▁  
/Users/alay/Desktop/FedML/example  
├── hello_world  
|  ├── hello_world.py  
|  └── requirements.txt  
└── job.yaml  
  
directory: 1 file: 3  


```

--------------------------------

### Execute Docker Installation Script

Source: https://docs.tensoropera.ai/open-source/installation/rpi

Runs the downloaded script to install Docker on your Raspberry Pi.

```bash
sudo sh get-docker.sh  
```

--------------------------------

### FedML Login Command Output Example

Source: https://docs.tensoropera.ai/launch/on-prem/install

Example output after running the `fedml login` command, indicating successful connection and device binding to the TensorOpera AI platform.

```text
(fedml) alay@a6000:~$ fedml login 3b24dd2f****************206e8669 -v dev  

 Welcome to FedML.ai!  
 Start to login the current device to the MLOps (https://tensoropera.ai)...  
  
(fedml) alay@a6000:~$ Found existing installation: fedml 0.8.7  
Uninstalling fedml-0.8.7:  
  Successfully uninstalled fedml-0.8.7  
  Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple  
Collecting fedml==0.8.8a156  
  Obtaining dependency information for fedml==0.8.8a156 from https://test-files.pythonhosted.org/packages/e8/44/06b4773fe095760c8dd4933c2f75ee7ea9594938038fb8293afa22028906/fedml-0.8.8a156-py2.py3-none-any.whl.metadata  
  Downloading https://test-files.pythonhosted.org/packages/e8/44/06b4773fe095760c8dd4933c2f75ee7ea9594938038fb8293afa22028906/fedml-0.8.8a156-py2.py3-none-any.whl.metadata (4.8 kB)  
Requirement already satisfied: numpy>=1.21 in ./.pyenv/versions/fedml/lib/python3.10/site-packages (from fedml==0.8.8a156  
.  
.  
.  
.  
  
Congratulations, your device is connected to the TensorOpera AI platform successfully!  
Your FedML Edge ID is 201610, unique device ID is 0xffdc89fad658@Linux.Edge.Device  


```

--------------------------------

### Confirm job launch

Source: https://docs.tensoropera.ai/launch/on-cloud/install

Example output after confirming the job launch and tracking the run status.

```bash
Do you want to launch the job with the above matched GPU resource? [y/N]: y  
  
Launching the job with the above matched GPU resource.  
Failed to list run with response.status_code = 200, response.content: b'{"message":"Succeeded to process request","code":"SUCCESS","data":null}'  
  
You can track your run details at this URL:  
https://tensoropera.ai/run?projectId=1717259066058870784&runId=1717260771043446784  
  
For querying the realtime status of your run, please run the following command.  
fedml run logs -rid 1717260771043446784
```

--------------------------------

### Start all clusters

Source: https://docs.tensoropera.ai/open-source/cli/fedml-cluster

Initiates the startup process for all stopped clusters found in the environment.

```bash
fedml cluster startall
```

--------------------------------

### Install FedML on Ubuntu

Source: https://docs.tensoropera.ai/open-source/installation/linux

Installs pip3 and FedML on Ubuntu. Ensure to export the PATH to use fedml commands.

```bash
sudo apt install python3-pip
pip3 install fedml
export PATH=$HOME/.local/bin:$PATH
fedml env
```

--------------------------------

### Install FedML with Pip

Source: https://docs.tensoropera.ai/open-source/installation/rpi

Installs the FedML package using pip. If '/home/user/.local/bin' is not in your PATH, run 'source ~/.profile' afterwards.

```bash
pip install fedml  
source ~/.profile  ## run this command if '/home/user/.local/bin' is not on PATH' after installation  
```

--------------------------------

### Building a Training Package

Source: https://docs.tensoropera.ai/open-source/cli/fedml-train

Example command to build a package from a YAML file and the resulting output path.

```bash
fedml train build train_job.yaml  
  
Your train package file is located at: /home/fedml/launch/fedml-train-packages/client-package.zip  
```

--------------------------------

### Python MPI Decentralized FL Example

Source: https://docs.tensoropera.ai/federate/simulation/examples/mpi_torch_fedavg_mnist_lr_example

The main Python script for the decentralized federated learning simulation using the FedML framework with MPI. It initializes the framework and starts the simulation.

```python
import fedml
from fedml import SimulatorMPI

if __name__ == "__main__":
    # init FedML framework
    args = fedml.init()

    # start training
    simulator = SimulatorMPI(args, None, None, None)
    simulator.run()
```

--------------------------------

### Initialize UnitedLLM Environment

Source: https://docs.tensoropera.ai/train/advanced/cross_cloud

Commands to clone the repository, navigate to the project directory, install dependencies, and prepare datasets.

```bash
# clone the repo and the submodules  
git clone https://github.com/FedML-AI/FedML.git -b unitedllm  
  
# go to the project directory  
cd python/spotlight_prj/unitedllm  
  
# Install dependencies  
pip install -r requirements.txt  
  
# Update the repo  
git pull  
  
# Download the example datasets.  
bash scripts/setup.sh  
```

--------------------------------

### Install Conda Packages

Source: https://docs.tensoropera.ai/open-source/installation/macos

If you encounter C/C++ compiler issues during installation, try installing necessary packages using Conda.

```bash
conda install … 

```

--------------------------------

### Install MPI using Conda on MacOS

Source: https://docs.tensoropera.ai/open-source/installation/macos

Use this command to install MPI and related libraries via Conda. Ensure Python 3.8+ is installed.

```bash
conda install mpi4py openmpi  

```

--------------------------------

### Initialize and Run FedML Client

Source: https://docs.tensoropera.ai/federate/cross-silo/example/mqtt_s3_fedavg_mnist_lr_example

Python script to initialize FedML, set up the device, load data and model, and start the client training process. Requires FedML library and appropriate arguments.

```python
import fedml  
from fedml.cross_silo import Client  
  
if __name__ == "__main__":  
    args = fedml.init()  
  
    # init device  
    device = fedml.device.get_device(args)  
  
    # load data  
    dataset, output_dim = fedml.data.load(args)  
  
    # load model  
    model = fedml.model.create(args, output_dim)  
  
    # start training  
    client = Client(args, device, dataset, model)  
    client.run()  
```

--------------------------------

### Install mpi4py with Conda

Source: https://docs.tensoropera.ai/federate/simulation/examples/mpi_torch_fedavg_mnist_lr_example

Install the mpi4py package using conda to ensure proper MPI implementation in Python. Pip installation may lead to errors.

```bash
conda install mpi4py  
```

--------------------------------

### Launch a job

Source: https://docs.tensoropera.ai/launch/on-cloud/install

Command to initiate a job on the GPU cloud using a configuration file.

```bash
fedml launch /path/to/job.yaml
```

--------------------------------

### Install NVIDIA Drivers and CUDA Toolkit

Source: https://docs.tensoropera.ai/share-and-earn/prerequisites

Use these commands to automatically install NVIDIA drivers and the CUDA toolkit if they are not already present. A reboot is required after installation.

```bash
sudo ubuntu-drivers autoinstall
sudo apt install nvidia-cuda-toolkit
sudo reboot
```

--------------------------------

### FedML Launch CLI Help

Source: https://docs.tensoropera.ai/launch/cli

Displays help information for the `fedml launch` command, outlining its usage, options for specifying cluster, API key, and platform version.

```bash
❯ fedml launch -h
fedml launch [OPTIONS] YAML_FILE
  Launch job at the TensorOpera AI platform

Options:
  -h, --help          Show this message and exit.
  -c, --cluster TEXT  If a cluster name is specified, you labelled the
                      searched resource by launch with the cluster name. So
                      later you can reuse the same cluster resource without
                      warmup after the first launch. The cluster can be
                      stopped by CLI: fedml cluster stop, or it would be
                      automatically stopped after 15-minute idle time.
  -k, --api_key TEXT  user api key.
  -v, --version TEXT  version of TensorOpera AI Platform. It should be dev,
                      test or release. Default is release.

```

--------------------------------

### Resolve sklearn Installation Errors on Jetson

Source: https://docs.tensoropera.ai/open-source/installation/jetson

Install essential build tools and libraries required for installing Python packages like 'sklearn' on Jetson devices.

```bash
sudo apt-get install build-essential libatlas-base-dev gfortran  
```

--------------------------------

### Start Cluster

Source: https://docs.tensoropera.ai/open-source/cli/fedml-cluster

Start clusters on TensorOpera AI Platform.

```APIDOC
## POST /api/cluster/start

### Description
Start clusters from TensorOpera AI Platform.

### Method
POST

### Endpoint
`/api/cluster/start`

### Parameters
#### Query Parameters
- `api_key` (TEXT) - Required - The user API key.
- `version` (TEXT) - Optional - Version of the TensorOpera AI Platform. It should be dev, test or release.

#### Request Body
- `cluster_names` (array of strings) - Required - Names of the clusters to start.

### Request Example
```json
{
  "cluster_names": ["my_cluster"]
}
```

### Response
#### Success Response (200)
- `message` (string) - Confirmation message.

#### Response Example
```json
{
  "message": "Cluster my_cluster have been started."
}
```
```

--------------------------------

### Build Client and Server Packages

Source: https://docs.tensoropera.ai/federate/cli

Instructions and examples for building client and server packages using the `fedml build` command.

```APIDOC
## Build the client and server package in the TensorOpera AI platform (fedml.ai)

Use the `fedml build` command to create client or server packages.

### Usage

```bash
fedml build -t <client or server> -sf <source_folder> -ep <entry_point_file> -cf <config_folder> -df <destination_package_folder> [--ignore <ignore_file_and_directory>]
```

### Parameters

- **-t, --type** (string) - Required - Specifies whether to build a 'client' or 'server' package.
- **-sf, --source_folder** (string) - Required - Path to the source code folder.
- **-ep, --entry_point** (string) - Required - The entry point file for the package.
- **-cf, --config_folder** (string) - Required - Path to the configuration folder.
- **-df, --destination_package_folder** (string) - Required - Folder where the built package will be saved.
- **--ignore** (string) - Optional - Comma-separated list of files and directories to ignore during the build process.

### Examples

**Build Client Package:**
```bash
SOURCE=./../cross_silo/client/
ENTRY=torch_client.py
CONFIG=./../cross_silo/config
DEST=./
IGNORE=__pycache__,*.git

fedml build -t client \
-sf $SOURCE \
-ep $ENTRY \
-cf $CONFIG \
-df $DEST \
--ignore $IGNORE
```

**Build Server Package:**
```bash
SOURCE=./../cross_silo/server/
ENTRY=torch_server.py
CONFIG=./../cross_silo/config
DEST=./
IGNORE=__pycache__,*.git

fedml build -t server \
-sf $SOURCE \
-ep $ENTRY \
-cf $CONFIG \
-df $DEST \
--ignore $IGNORE
```

*Refer to the FedML GitHub repository for more examples: [https://github.com/FedML-AI/FedML/blob/master/python/tests/smoke_test/cli/build.sh](https://github.com/FedML-AI/FedML/blob/master/python/tests/smoke_test/cli/build.sh)*
```

--------------------------------

### Install FedML Dependencies

Source: https://docs.tensoropera.ai/federate/simulation/examples/sp_fedavg_mnist_lr_example

Upgrade pip and install the FedML package.

```bash
python -m pip install --upgrade pip  
python -m pip install fedml  
```

--------------------------------

### Initialize and Run FedML ServerMNN

Source: https://docs.tensoropera.ai/federate/cross-device/api

This snippet demonstrates the complete process of initializing the FedML framework, configuring W&B for logging, setting up the device, loading datasets and dataloaders, creating a model, and finally initializing and running the ServerMNN for federated learning. Ensure MNN and FedML are installed and the necessary dataset and model files are available.

```python
import MNN
import fedml
import wandb
from fedml.cross_device import ServerMNN
from fedml.model import create_mnn_lenet5_model

from my_dataset import MnistDataset

if __name__ == "__main__":
    # init FedML framework
    args = fedml.init()

    if args.enable_wandb:
        args.wandb_obj = wandb.init(
            entity="fedml", project="mobile", name="MNN-Mobile", config=args
        )

    # init device
    device = fedml.device.get_device(args)

    # load data
    train_dataset = MnistDataset(True)
    test_dataset = MnistDataset(False)
    train_dataloader = MNN.data.DataLoader(train_dataset, batch_size=64, shuffle=True)
    test_dataloader = MNN.data.DataLoader(
        test_dataset, batch_size=args.batch_size, shuffle=False
    )
    class_num = 10

    # load model
    create_mnn_lenet5_model(args.global_model_file_path)

    # start training
    server = ServerMNN(
        args, device, test_dataloader, None
    )  # for MNN, the model is loaded using MNN file
    server.run()
```

--------------------------------

### Resolve h5py Installation Errors on Jetson

Source: https://docs.tensoropera.ai/open-source/installation/jetson

These commands help resolve common issues when installing the 'h5py' library on Jetson by installing necessary system packages and creating a symbolic link.

```bash
sudo apt-get install subversion  
ln -s /usr/include/locale.h /usr/include/xlocale.h  
sudo apt-get install libhdf5-serial-dev  
```

--------------------------------

### Install FedML with Pip (No Dependencies)

Source: https://docs.tensoropera.ai/open-source/installation/jetson

Install the FedML library using pip without its dependencies. This is recommended when PyTorch is not available via pip on Jetson, and you plan to install dependencies manually.

```bash
pip3 install fedml --no-dependencies
```

--------------------------------

### Build Client Package

Source: https://docs.tensoropera.ai/federate/cross-silo/user_guide

Commands to clone the repository and build the client-side deployable package.

```bash
git clone https://github.com/FedML-AI/FedML  
cd ./FedML/python/examples/cross_silo/mqtt_s3_fedavg_mnist_lr_example/one_line  
  
SOURCE=client  
ENTRY=torch_client.py  
CONFIG=config  
DEST=./mlops  
fedml build -t client -sf $SOURCE -ep $ENTRY -cf $CONFIG -df $DEST
```

--------------------------------

### Cluster Start API

Source: https://docs.tensoropera.ai/open-source/api/api-launch

Start selected clusters on the TensorOpera AI platform.

```APIDOC
## POST /api/cluster_start

### Description
Start selected clusters on TensorOpera AI platform.

### Method
POST

### Endpoint
/api/cluster_start

### Parameters
#### Request Body
- **cluster_names** (Tuple[str]) - Required - Tuple of cluster names to start.
- **api_key** (str) - Optional - Your API key from TensorOpera AI platform (if not configured already).

### Response
#### Success Response (200)
- **success** (Boolean) - True if the clusters were successfully started, False otherwise.
```

--------------------------------

### Build Server Package

Source: https://docs.tensoropera.ai/federate/cross-silo/user_guide

Commands to build the server-side deployable package.

```bash
SOURCE=server  
ENTRY=torch_server.py  
CONFIG=config  
DEST=./mlops  
fedml build -t server -sf $SOURCE -ep $ENTRY -cf $CONFIG -df $DEST
```

--------------------------------

### Check NVIDIA Driver Installation

Source: https://docs.tensoropera.ai/share-and-earn

Verify if NVIDIA drivers are currently installed on the system.

```bash
nvidia-smi  

```

--------------------------------

### Cluster Start All API

Source: https://docs.tensoropera.ai/open-source/api/api-launch

Start all existing clusters on your account on the TensorOpera AI platform.

```APIDOC
## POST /api/cluster_startall

### Description
Start all existing clusters on your account on TensorOpera AI platform.

### Method
POST

### Endpoint
/api/cluster_startall

### Parameters
#### Request Body
- **api_key** (str) - Optional - Your API key from TensorOpera AI platform (if not configured already).

### Response
#### Success Response (200)
- **success** (Boolean) - True if all clusters were successfully started, False otherwise.
```

--------------------------------

### Verify FedML Installation

Source: https://docs.tensoropera.ai/share-and-earn

Run the verification script to ensure the FedML library is correctly installed.

```bash
sudo curl -sSf https://doc.fedml.ai/shell/verify_installation_driver.sh | bash  

```

--------------------------------

### FedML Configuration File Example

Source: https://docs.tensoropera.ai/federate/getting_started

A sample `fedml_config.yaml` file demonstrating how to configure various aspects of a FedML cross-silo training run, including data, model, training, and communication arguments.

```yaml
common_args:
  training_type: 'cross_silo'
  scenario: 'horizontal'
  using_mlops: false
  random_seed: 0

environment_args:
  bootstrap: config/bootstrap.sh

data_args:
  dataset: 'mnist'
  data_cache_dir: ~/fedml_data
  partition_method: 'hetero'
  partition_alpha: 0.5

model_args:
  model: 'lr'
  model_file_cache_folder: './model_file_cache' # will be filled by the server automatically
  global_model_file_path: './model_file_cache/global_model.pt'

train_args:
  federated_optimizer: 'FedAvg'
  client_id_list: 
  client_num_in_total: 1
  client_num_per_round: 2
  comm_round: 10
  epochs: 1
  batch_size: 10
  client_optimizer: sgd
  learning_rate: 0.03
  weight_decay: 0.001

validation_args:
  frequency_of_the_test: 1

device_args:
  worker_num: 2
  using_gpu: false
  gpu_mapping_file: config/gpu_mapping.yaml
  gpu_mapping_key: mapping_default

comm_args:
  backend: 'MQTT_S3'
  mqtt_config_path: config/mqtt_config.yaml
  s3_config_path: config/s3_config.yaml
  # If you want to use your customized MQTT or s3 server as training backends, you should uncomment and set the following lines. 
  #customized_training_mqtt_config: {'BROKER_HOST': 'your mqtt server address or domain name', 'MQTT_PWD': 'your mqtt password', 'BROKER_PORT': 1883, 'MQTT_KEEPALIVE': 180, 'MQTT_USER': 'your mqtt user'}
  #customized_training_s3_config: {'CN_S3_SAK': 'your s3 aws_secret_access_key', 'CN_REGION_NAME': 'your s3 region name', 'CN_S3_AKI': 'your s3 aws_access_key_id', 'BUCKET_NAME': 'your s3 bucket name'}

tracking_args:
  # When running on MLOps platform(fedml.ai), the default log path is at ~/fedml-client/fedml/logs/ and ~/fedml-server/fedml/logs/
  enable_wandb: false
```

--------------------------------

### Build Packages

Source: https://docs.tensoropera.ai/federate/cli

Commands to package client or server code for deployment on the platform.

```bash
fedml build -t client(or server) -sf source_folder -ep entry_point_file -cf config_folder -df destination_package_folder --ignore ignore_file_and_directory(concat with ,)  
```

```bash
# build client package  
SOURCE=./../cross_silo/client/  
ENTRY=torch_client.py  
CONFIG=./../cross_silo/config  
DEST=./  
IGNORE=__pycache__,*.git  
  
fedml build -t client \
-sf $SOURCE \
-ep $ENTRY \
-cf $CONFIG \
-df $DEST  \
--ignore $IGNORE  
```

```bash
# build server package  
SOURCE=./../cross_silo/server/  
ENTRY=torch_server.py  
CONFIG=./../cross_silo/config  
DEST=./  
IGNORE=__pycache__,*.git  
  
fedml build -t server \
-sf $SOURCE \
-ep $ENTRY \
-cf $CONFIG \
-df $DEST  \
--ignore $IGNORE  
```

--------------------------------

### Verify FedML installation

Source: https://docs.tensoropera.ai/open-source/installation

Checks if the library is correctly installed by attempting an import in the Python shell.

```python
Python 3.9.6 (default, Aug 11 2023, 19:44:49)   
[Clang 15.0.0 (clang-1500.0.40.1)] on darwin  
Type "help", "copyright", "credits" or "license" for more information.  
>>> import fedml  
>>>   
```

--------------------------------

### Install FedML via conda

Source: https://docs.tensoropera.ai/open-source/installation

Creates a dedicated conda environment and installs FedML within it.

```bash
conda create --name fedml-pip python=3.8  
conda activate fedml-pip  
conda install --name fedml-pip pip  
pip install fedml  
```

--------------------------------

### Launch job with verbose output

Source: https://docs.tensoropera.ai/launch/on-cloud/install

Example output showing resource matching and confirmation prompt after running the launch command with the -v flag.

```bash
❯ fedml launch job.yaml -v  
Submitting your job to TensorOpera AI Platform: 100%|█████████████████████████████| 2.92k/2.92k [00:00<00:00, 16.7kB/s]  
  
Searched and matched the following GPU resource for your job:  
+-----------+-------------------+---------+------------+-------------------------+---------+------+----------+  
|  Provider |      Instance     | vCPU(s) | Memory(GB) |          GPU(s)         |  Region | Cost | Selected |  
+-----------+-------------------+---------+------------+-------------------------+---------+------+----------+  
| FedML Inc | FEDML_A100_NODE_2 |   256   |  2003.85   | NVIDIA A100-SXM4-80GB:8 | DEFAULT | 1.09 |    √     |  
+-----------+-------------------+---------+------------+-------------------------+---------+------+----------+  
  
You can also view the matched GPU resource with Web UI at:  
https://tensoropera.ai/launch/confirm-start-job?projectId=1717259066058870784&projectName=my-project&jobId=1717260771043446784  
Do you want to launch the job with the above matched GPU resource? [y/N]:
```

--------------------------------

### Server and Client Execution Scripts

Source: https://docs.tensoropera.ai/federate/security/Examples/mqtt_s3_fedavg_attack_mnist_lr_example

Bash scripts to initialize the server and client processes with specific configuration paths.

```bash
#!/usr/bin/env bash  
  
RUN_ID=$1  
python3 torch_server.py --cf config/byzantine/fedml_config.yaml --rank 0 --role server --run_id $RUN_ID  
```

```bash
#!/usr/bin/env bash  
RANK=$1  
RUN_ID=$2  
python3 torch_client.py --cf config/byzantine/fedml_config.yaml --rank $RANK --role client --run_id $RUN_ID  
```

--------------------------------

### Install FedML with MPI support

Source: https://docs.tensoropera.ai/open-source/installation

Installs the MPI message exchanging mechanism for local distributed simulations.

```bash
pip install "fedml[MPI]"  
```

--------------------------------

### Install additional ML engines

Source: https://docs.tensoropera.ai/open-source/installation

Commands to install optional machine learning framework support for FedML.

```bash
pip install "fedml[tensorflow]"  
pip install "fedml[jax]"  
pip install "fedml[mxnet]"  
```

```bash
pip install "fedml[tensorflow,jax,mxnet]"  
```

--------------------------------

### Install FedML with Conda

Source: https://docs.tensoropera.ai/open-source/installation/windows

Creates a dedicated conda environment and installs the FedML package via pip.

```bash
conda create --name fedml python  
conda activate fedml  
conda install --name fedml pip  
pip install fedml  

```

--------------------------------

### Verify FedML Installation

Source: https://docs.tensoropera.ai/launch/share-and-earn

Run the verification script to ensure the FedML driver environment is correctly installed.

```bash
sudo curl -sSf https://doc.fedml.ai/shell/verify_installation_driver.sh | bash  
```

--------------------------------

### Initialize Server and Client Logic

Source: https://docs.tensoropera.ai/federate/federated-analytics/Examples/heavy_hitter_twitter_data_example

Python entry points for the server and client nodes using the FedML framework.

```python
from fedml.fa import init, FARunner  
from fedml.fa.data import fa_load_data  
  
  
if __name__ == "__main__":  
    args = init()  
    dataset = fa_load_data(args)  
    fa_runner = FARunner(args, dataset)  
    fa_runner.run()  
```

```python
from fedml.fa import init, FARunner  
from fedml.fa.data import fa_load_data  
  
if __name__ == "__main__":  
    args = init()  
    dataset = fa_load_data(args)  
    fa_runner = FARunner(args, dataset)  
    fa_runner.run()  
```

--------------------------------

### Inference Server Response Example

Source: https://docs.tensoropera.ai/deploy/deploy_local

This is an example of the expected output from the inference server when a request is successfully processed.

```json
"{'generated_text': '...'}"  

```

--------------------------------

### Install FedML on CentOS

Source: https://docs.tensoropera.ai/open-source/installation/linux

Installs pip3 and FedML on CentOS. Ensure to export the PATH to use fedml commands.

```bash
yum –y install python3-pip
pip3 install fedml
export PATH=$HOME/.local/bin:$PATH
fedml env
```

--------------------------------

### Start All Clusters

Source: https://docs.tensoropera.ai/open-source/cli/fedml-cluster

Starts all available clusters on the TensorOpera AI Platform. This is useful for bringing all resources online simultaneously.

```bash
fedml cluster startall

```

--------------------------------

### Run FedML MNIST LR Step-by-Step Example

Source: https://docs.tensoropera.ai/federate/simulation/examples/sp_fedavg_mnist_lr_example

Execute a stand-alone simulated FedAvg algorithm on the MNIST dataset using the step-by-step API. This requires a configuration file.

```bash
python torch_fedavg_mnist_lr_step_by_step_example.py --cf fedml_config.yaml  
```

--------------------------------

### Initialize and Run FedML Client

Source: https://docs.tensoropera.ai/federate/cross-silo/example/mqtt_s3_fedavg_hierarchical_mnist_lr_example

Initializes the FedML framework, sets up the device, loads data, defines the model, and starts the federated learning client. Assumes MNIST dataset (28x28 images).

```python
if __name__ == "__main__":
    # init FedML framework
    args = fedml.init()

    device = fedml.device.get_device(args)

    # load data
    dataset, output_dim = load_data(args)

    # load model (the size of MNIST image is 28 x 28)
    model = LogisticRegression(28 * 28, output_dim)

    # start training
    client = Client(args, device, dataset, model)
    client.run()
```