### Install Dependencies

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Run this command once to install all necessary project dependencies before starting development.

```bash
npm install
```

--------------------------------

### Run MNIST Training Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Execute the provided Python script to train the MNIST model. Ensure necessary Python libraries are installed.

```bash
python examples/MNIST_RAW_format/mnist_dataset_training_example.py
```

--------------------------------

### Install Dependencies for Local Development

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_model_training/tensorflow/README.md

Run this command to install all necessary dependencies for local development of the federated training module.

```bash
python -m pip install -r requirements.txt
```

--------------------------------

### Install Recommended GPU Driver on Ubuntu

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Steps to install the recommended NVIDIA GPU driver on an Ubuntu machine. This involves SSHing into the worker node and using apt commands.

```bash
# SSH into the worker machine with GPU
$ ssh USERNAME@EXTERNAL_IP

# Verify ubuntu driver
$ sudo apt install ubuntu-drivers-common
$ ubuntu-drivers devices

# Install the recommended driver
$ sudo ubuntu-drivers autoinstall

# Reboot the machine
$ sudo reboot

# After the reboot, test if the driver is installed correctly
$ nvidia-smi
```

--------------------------------

### Run Development Server

Source: https://github.com/ertis-research/kafka-ml/blob/master/mlcode_executor/tfexecutor/README.md

Starts the development server using gunicorn. The IP and port can be customized.

```bash
gunicorn app:app --bind 0.0.0.0:8001 --timeout 0
```

--------------------------------

### Install Nvidia Docker on Ubuntu

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Install Nvidia Docker 2 on Ubuntu by adding the necessary repositories and then updating the package list and installing the package. Restart Docker afterwards.

```bash
# SSH into the worker machine with GPU
$ ssh USERNAME@EXTERNAL_IP

# Add the package repositories
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
```

--------------------------------

### Run Federated MNIST Training Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Execute the federated MNIST training script. Ensure the _deployment_id_ is correctly configured and necessary Python libraries are installed.

```bash
python examples/FEDERATED_MNIST_RAW_format/mnist_dataset_federated_training_example.py
```

--------------------------------

### Start Development Server

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Executes the development server and enables live reloading. Access the application at http://localhost:4200/.

```bash
ng serve
```

--------------------------------

### Build Frontend and Push Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Installs frontend dependencies, builds the production version of the frontend, and then builds and pushes the frontend Docker image to the local registry. Node version management might be required.

```bash
cd frontend
npm install # nvm install 10 & nvm use 10.24.1
npm i -g @angular/cli@9.1.15
ng build -c production
docker build --tag localhost:5000/frontend .
docker push localhost:5000/frontend
```

--------------------------------

### Install Kubernetes GPU Sharing Extender and Device Plugin

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Install the necessary Kubernetes components for GPU sharing, including the extender and device plugin. This involves downloading YAML configuration files and applying them using kubectl.

```bash
# From your local machine that has access to the Kubernetes API
$ curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml
$ kubectl create -f gpushare-schd-extender.yaml

$ wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
$ kubectl create -f device-plugin-rbac.yaml

$ wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml
# update the local file so the first line is 'apiVersion: apps/v1'
$ kubectl create -f device-plugin-ds.yaml

# From your local machine that has access to the Kubernetes API
$ kubectl label node worker-gpu-0 gpushare=true
```

--------------------------------

### Run Development Server

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_backend/README.md

Starts the Django development server. Access the admin UI at http://localhost:8085/admin.

```bash
python manage.py runserver 0.0.0.0:8085
```

--------------------------------

### Execute the Training Task

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_model_training/tensorflow/README.md

Once all environment variables are configured, run this command to start the federated training task.

```bash
python training.py
```

--------------------------------

### Execute Incremental MNIST Training Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Run the provided Python script to perform incremental training on the MNIST dataset. Ensure all required Python libraries are installed from datasources/requirements.txt.

```bash
python examples/MNIST_RAW_format/mnist_dataset_online_training_example.py
```

--------------------------------

### Kustomize Installation Configuration

Source: https://github.com/ertis-research/kafka-ml/blob/master/kustomize/README.md

Defines the Kustomize resources and namespace for Kafka-ML deployment. Customize the 'resources' field to select a specific version.

```yaml
resources:
  # Choose your kustomize version
  - github.com/ertis-research/kafka-ml/kustomize/master

# Namespace where Kafka-ML will be deployed
namespace: kafkaml

configMapGenerator:
  - name: kafkaml-configmap
    behavior: merge
    literals:
      # Comma separated list of Kafka brokers
      - brokers=kafka1,kafka2,kafka3

```

--------------------------------

### Deploy Kafka-ML Locally

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Command to apply Kafka-ML local installation using kubectl. Uncomment the export line if running on Apple Silicon.

```sh
# Uncomment only if you are running Kafka-ML on Apple Silicon
# export DOCKER_DEFAULT_PLATFORM=linux/amd64
kubectl apply -k "github.com/ertis-research/kafka-ml/kustomize/local?ref=v1.2"
```

--------------------------------

### Run Development Server

Source: https://github.com/ertis-research/kafka-ml/blob/master/backend/README.md

Starts the Django development server. Access the admin UI at http://localhost:8000/admin. The IP and port can be customized.

```bash
python manage.py runserver 0.0.0.0:8000
```

--------------------------------

### Run MNIST Unsupervised Training Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Execute the Python script for incremental unsupervised training of the MNIST model. You may need to install Python libraries from datasources/requirements.txt.

```python
python examples/MNIST_RAW_format/mnist_dataset_unsupervised_training_example.py
```

--------------------------------

### Execute Kafka Control Logger

Source: https://github.com/ertis-research/kafka-ml/blob/master/kafka_control_logger/README.md

Run this command to start the Kafka control logger task after setting up environment variables.

```bash
python logger.py
```

--------------------------------

### Export Environment Variable Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/model_inference/tensorflow/README.md

Sets the INPUT_TOPIC environment variable for configuring the Kafka input topic.

```bash
export INPUT_TOPIC=ertis-input
```

--------------------------------

### Export Environment Variable

Source: https://github.com/ertis-research/kafka-ml/blob/master/kafka_control_logger/README.md

Example of exporting the CONTROL_TOPIC environment variable. Ensure all required environment variables are set before execution.

```bash
export CONTROL_TOPIC=control
```

--------------------------------

### Deploy Kafka-ML Federated Module Locally

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Use this command to apply the necessary Kubernetes resources for a local installation of the Kafka-ML Federated Module. Uncomment the export line if running on Apple Silicon.

```sh
# Uncomment only if you are running Kafka-ML Federated Module on Apple Silicon
# export DOCKER_DEFAULT_PLATFORM=linux/amd64
kubectl apply -k "github.com/ertis-research/kafka-ml/federated-module/kustomize/local?ref=v1.1"
```

--------------------------------

### Run MNIST Inference Example

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Execute the provided Python script to test the deployed MNIST model for inference. This streams data to the configured topics.

```bash
python examples/MNIST_RAW_format/mnist_dataset_inference_example.py
```

--------------------------------

### Define PyTorch Neural Network Model

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Example of defining a PyTorch neural network with required methods for loss, optimizer, and metrics. Ensure PyTorch and related modules are imported.

```python
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
            nn.Softmax()
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

    def loss_fn(self):
        return nn.CrossEntropyLoss()

    def optimizer(self):
        return torch.optim.Adam(model.parameters(), lr=0.001)

    def metrics(self):
        val_metrics = {
            "accuracy": Accuracy(),
            "loss": Loss(self.loss_fn())
         }
        return val_metrics

model = NeuralNetwork()
```

--------------------------------

### Define TensorFlow Keras Model

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Example of defining a simple Sequential model using TensorFlow Keras for the MNIST dataset. Ensure TensorFlow is imported.

```python
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(
    optimizer=tf.keras.optimizers.Adam(0.001),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
)
```

--------------------------------

### Build and Push Backend Docker Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Build the backend Docker image and push it to the local registry. Ensure you are in the 'backend' directory.

```bash
cd backend
docker build --tag localhost:5000/federated_backend .
docker push localhost:5000/federated_backend
```

--------------------------------

### Angular CLI Help

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Access comprehensive help and documentation for the Angular CLI.

```bash
ng help
```

--------------------------------

### Deploy Kafka-ML using Kustomize

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/kustomize/README.md

Apply the Kustomize configuration to your Kubernetes cluster. Ensure the target namespace exists before applying.

```sh
# Create the namespace first if it doesn't exists
kubectl create namespace kafkaml
kubectl apply -k .
```

--------------------------------

### Run Local Docker Registry

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Deploy a local Docker registry to upload your Docker images. This is the first step in building the module locally.

```bash
docker run -d -p 5000:5000 --restart=always --name registry registry:2
```

--------------------------------

### Build and Push TensorFlow Model Training Docker Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Build the TensorFlow model training component's Docker image and push it to the local registry. Navigate to the correct directory first.

```bash
cd federated_model_training/tensorflow
docker build --tag localhost:5000/federated_tensorflow_model_training .
docker push localhost:5000/federated_tensorflow_model_training
```

--------------------------------

### Build for Kubernetes Deployment

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Builds the project for deployment. Use the --prod flag for a production build, which is necessary for generating the front-end image for Kubernetes.

```bash
ng build --prod
```

--------------------------------

### Build and Push PyTorch Model Training Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the PyTorch model training Docker image and pushes it to the local registry. Ensure you are in the 'pytorch' directory within 'model_training'.

```bash
cd ../pytorch
docker build --tag localhost:5000/pytorch_model_training .
docker push localhost:5000/pytorch_model_training
```

--------------------------------

### Deploy Kafka-ML Components to Kubernetes

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Applies Kubernetes configuration files to deploy Kafka-ML components in order. This includes Zookeeper, Kafka, backend, frontend, ML executors, and the control logger.

```bash
kubectl apply -f zookeeper-pod.yaml
kubectl apply -f zookeeper-service.yaml

kubectl apply -f kafka-pod.yaml
kubectl apply -f kafka-service.yaml

kubectl apply -f backend-deployment.yaml
kubectl apply -f backend-service.yaml

kubectl apply -f frontend-deployment.yaml
kubectl apply -f frontend-service.yaml

kubectl apply -f tf-executor-deployment.yaml
kubectl apply -f tf-executor-service.yaml

kubectl apply -f pth-executor-deployment.yaml
kubectl apply -f pth-executor-service.yaml

kubectl apply -f kafka-control-logger-deployment.yaml
```

--------------------------------

### Build and Push Kafka Control Logger Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the Kafka control logger Docker image and pushes it to the local registry. Ensure you are in the 'kafka_control_logger' directory.

```bash
cd kafka_control_logger
docker build --tag localhost:5000/kafka_control_logger .
docker push localhost:5000/kafka_control_logger
```

--------------------------------

### Build and Push Docker Images for Backend

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds and pushes the backend Docker image to a specified registry. Ensure the image tag includes the registry address.

```bash
cd backend
docker build --tag x.x.x.x:5000/backend .
docker push x.x.x.x:5000/backend
```

--------------------------------

### Build and Push PyTorch Code Executor

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the PyTorch Code Executor Docker image and pushes it to the local registry. Navigate to the correct directory before execution.

```bash
cd mlcode_executor/pthexecutor
docker build --tag localhost:5000/pthexecutor .
docker push localhost:5000/pthexecutor
```

--------------------------------

### Kubernetes Deployment Commands

Source: https://github.com/ertis-research/kafka-ml/blob/master/kustomize/README.md

Commands to apply the Kustomize configuration to a Kubernetes cluster. Ensure the namespace exists before applying.

```sh
# Create the namespace first if it doesn't exists
kubectl create namespace kafkaml
kubectl apply -k .

```

--------------------------------

### Run Unit Tests

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_backend/README.md

Executes all unit tests for the project.

```bash
python manage.py test
```

--------------------------------

### Kustomize Configuration for Kafka-ML Deployment

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/kustomize/README.md

Define the resources and namespace for Kafka-ML deployment. Choose your desired Kustomize version from the available options. Configure Kafka brokers using literals in the configMapGenerator.

```yaml
resources:
  # Choose your kustomize version
  - github.com/ertis-research/kafka-ml/federated-module/kustomize/master

# Namespace where Kafka-ML will be deployed
namespace: kafkaml

configMapGenerator:
  - name: federated-kafkaml-configmap
    behavior: merge
    literals:
      # Comma separated list of Kafka brokers
      - brokers=kafka1,kafka2,kafka3
```

--------------------------------

### Build and Push Model Control Logger Docker Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Build the federated_model_control_logger component's Docker image and push it to the local registry. Ensure you are in the correct directory.

```bash
cd federated_model_control_logger
docker build --tag localhost:5000/federated_model_control_logger .
docker push localhost:5000/federated_model_control_logger
```

--------------------------------

### Build and Push Data Control Logger Docker Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Build the federated_data_control_logger component's Docker image and push it to the local registry. Navigate to the correct directory before building.

```bash
cd federated_data_control_logger
docker build --tag localhost:5000/federated_data_control_logger .
docker push localhost:5000/federated_data_control_logger
```

--------------------------------

### Build and Push PyTorch Model Inference Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the PyTorch model inference Docker image and pushes it to the local registry. Ensure you are in the 'pytorch' directory within 'model_inference'.

```bash
cd ../pytorch
docker build --tag localhost:5000/pytorch_model_inference .
docker push localhost:5000/pytorch_model_inference
```

--------------------------------

### Run Unit Tests

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Executes the unit tests for the front-end application using Karma.

```bash
ng test
```

--------------------------------

### Create Superuser

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_backend/README.md

Creates a superuser account for accessing the Django admin interface.

```bash
python manage.py createsuperuser
```

--------------------------------

### Synchronize Database

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_backend/README.md

Synchronizes and creates the database schema. Run again after changes in models.py.

```bash
python manage.py makemigrations --noinput
python manage.py migrate --run-syncdb
```

--------------------------------

### Configure Kubernetes Master URL and Token in Backend Deployment

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Updates the `backend-deployment.yaml` file with the Kubernetes master URL and the retrieved service account token for API access.

```yaml
    - name: KUBE_TOKEN
      value: # include token here (and remove #)
    - name: KUBE_HOST
      value: # include kubernetes master URL here
```

--------------------------------

### Update Frontend Deployment Image Configuration

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Modifies the `frontend-deployment.yaml` file to use the correct backend image and URL.

```yaml
 containers:
 -   - image: localhost:5000/backend
 +   - image: x.x.x.x:5000/backend

    - name: BACKEND_URL
-     value: http://localhost:8000
+     value: http://x.x.x.x:8000
```

--------------------------------

### Create Kubernetes Service Account and Cluster Role Binding

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Creates a service account named 'k8sadmin' in the 'kube-system' namespace and binds it to the 'cluster-admin' role for administrative access.

```bash
$ sudo kubectl create serviceaccount k8sadmin -n kube-system

$ sudo kubectl create clusterrolebinding k8sadmin --clusterrole=cluster-admin --serviceaccount=kube-system:k8sadmin
```

--------------------------------

### Execute Inference Task

Source: https://github.com/ertis-research/kafka-ml/blob/master/model_inference/tensorflow/README.md

Runs the main inference script after setting up all required environment variables.

```bash
python inference.py
```

--------------------------------

### PyTorch Model for Parking Occupance

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/MLGPARK_STREAM_RAW_format/README.md

Implements a PyTorch neural network for parking occupancy prediction. Includes L1Loss and MeanSquaredError metrics. Uses SGD optimizer with a learning rate of 1e-3. Batch size is 16, with max 25 epochs and shuffling enabled.

```python
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(6, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 1),
        )

    def forward(self, x):
        return self.linear_relu_stack(x)

    def loss_fn(self):
        return nn.L1Loss()

    def optimizer(self):
        return torch.optim.SGD(model.parameters(), lr=1e-3)

    def metrics(self):
        val_metrics = {
            "mse": MeanSquaredError(),
            "loss": Loss(self.loss_fn())
         }
        return val_metrics

model = NeuralNetwork()
```

--------------------------------

### Build and Push TensorFlow Code Executor

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the TensorFlow Code Executor Docker image and pushes it to the local registry. Ensure you are in the correct directory.

```bash
cd mlcode_executor/tfexecutor
docker build --tag localhost:5000/tfexecutor .
docker push localhost:5000/tfexecutor
```

--------------------------------

### Kubernetes Configuration for External Deployment

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_backend/README.md

Configures the backend to load Kubernetes configuration externally, useful when running outside a Docker container.

```python
#config.load_incluster_config() # To run inside the container
config.load_kube_config() # To run externally
```

--------------------------------

### Build and Push TensorFlow Model Inference Image

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Builds the TensorFlow model inference Docker image and pushes it to the local registry. Ensure you are in the 'tensorflow' directory within 'model_inference'.

```bash
cd model_inference/tensorflow
docker build --tag localhost:5000/tensorflow_model_inference .
docker push localhost:5000/tensorflow_model_inference
```

--------------------------------

### Configure Docker Daemon for Nvidia Runtime

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/README.md

Configure the Docker daemon to use the NVIDIA container runtime by modifying the daemon.json file. This requires restarting the Docker daemon.

```bash
# SSH into the worker machine with GPU
$ ssh USERNAME@EXTERNAL_IP
$ sudo tee /etc/docker/daemon.json <<EOF
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
EOF
$ sudo pkill -SIGHUP docker
$ sudo reboot
```

--------------------------------

### Update Backend Deployment Image Configuration

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Modifies the `backend-deployment.yaml` file to use the newly pushed Docker image and configure environment variables for Kubernetes integration.

```yaml
 containers:
 -   - image: localhost:5000/backend
 +   - image: x.x.x.x:5000/backend

    - name: BOOTSTRAP_SERVERS
      value: kafka-cluster:9092 # You can specify all the Kafka Bootstrap Servers that you have. e.g.: kafka-cluster-2:9092,kafka-cluster-3:9092,kafka-cluster-4:9092,kafka-cluster-5:9092,kafka-cluster-6:9092,kafka-cluster-7:9092

    - name: TRAINING_MODEL_IMAGE
-     value: localhost:5000/model_training
+     value: x.x.x.x:5000/model_training
    - name: INFERENCE_MODEL_IMAGE
-     value: localhost:5000/model_inference
+     value: x.x.x.x:5000/model_inference
    - name: FRONTEND_URL
-     value: http://localhost
+     value: http://x.x.x.x
```

--------------------------------

### Add External IP to Backend Allowed Hosts

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Updates the `backend-deployment.yaml` file to include the assigned external IP address in the `ALLOWED_HOSTS` environment variable, enabling external access.

```yaml
    - name: ALLOWED_HOSTS
      value: y.y.y.y, localhost
```

--------------------------------

### Export Environment Variable for Kafka Bootstrap Servers

Source: https://github.com/ertis-research/kafka-ml/blob/master/federated-module/federated_model_training/tensorflow/README.md

Set the BOOTSTRAP_SERVERS environment variable to specify the Kafka bootstrap server address for local testing.

```bash
export BOOTSTRAP_SERVERS=localhost:9094
```

--------------------------------

### PyTorch VGG16 Model for SO2SAT

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/SO2SAT_RAW_format/README.md

Implements a VGG16 model in PyTorch for the SO2SAT dataset. This class includes methods for defining the loss function, optimizer, and metrics.

```python
class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        self.pretrained = models.vgg16(pretrained=True)
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(1000, 512),
            nn.ReLU(),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 17),
            nn.Softmax()
        )
        
    def forward(self, x):
        x = self.pretrained(x)
        x = self.flatten(x)
        output = self.linear_relu_stack(x)
        return output

    def loss_fn(self):
        return nn.CrossEntropyLoss()

    def optimizer(self):
        return torch.optim.SGD(model.parameters(), lr=1e-3)

    def metrics(self):
        val_metrics = {
            "accuracy": Accuracy(),
            "loss": Loss(self.loss_fn())
         }
        return val_metrics

model = VGG16()
```

--------------------------------

### Kafka-ML BibTeX Reference

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

BibTeX reference for citing the Kafka-ML article.

```bibtex
@article{martin2022kafka,
  title={Kafka-ML: connecting the data stream with ML/AI frameworks},
  author={Mart{\'i}n, Cristian and Langendoerfer, Peter and Zarrin, Pouya Soltani and D{\'i}az, Manuel and Rubio, Bartolom{\'e}},
  journal={Future Generation Computer Systems},
  volume={126},
  pages={15--33},
  year={2022},
  publisher={Elsevier}
}
```

--------------------------------

### Build TCN Model for Disaggregation

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/TCN_NILM_RAW_format/README.md

Defines a TCN block and a function to build the disaggregation model. The model is compiled with Adam optimizer and Mean Squared Error loss.

```python
def tcn_block(x, filters, kernel_size=3, dilation_rate=1):
    conv = Conv1D(filters, kernel_size, padding='causal',
                    dilation_rate=dilation_rate, activation='relu')(x)
    conv = SpatialDropout1D(0.2)(conv)
    res  = Conv1D(filters, 1, padding='same')(x)
    return Add()([res, conv])

def build_disaggregation_model(input_length=120, num_features=2, num_outputs=3):
    inp = layers.Input(shape=(input_length, num_features))
    x   = tcn_block(inp, 16, 3, 1)
    x   = tcn_block(x  ,  8, 3, 2)
    out = Conv1D(num_outputs, 1, padding='same', activation='linear')(x)
    model = models.Model(inp, out)
    model.compile("adam", loss='mse', metrics=['mae', 'mse', 'mape'])
    return model

model = build_disaggregation_model(120, 2, 3)
```

--------------------------------

### TensorFlow Model for Parking Occupance

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/MLGPARK_STREAM_RAW_format/README.md

Defines a sequential Keras model for parking occupancy prediction. Uses SGD optimizer with Mean Absolute Error loss. Batch size is 16, with 25 epochs and shuffling enabled.

```python
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(256, activation='relu', input_shape=(6,)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(1)
  ])
model.compile(optimizer='sgd',
    loss='mae',
    metrics=['mae', 'mse'])
```

--------------------------------

### Generate Angular Component

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Use this command to generate new components, directives, pipes, services, classes, guards, interfaces, enums, or modules within the Angular project.

```bash
ng generate component component-name
```

--------------------------------

### MNIST Prediction Visualization Configuration (Classification)

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

JSON configuration for visualizing MNIST model predictions in classification mode. Specifies labels, colors, and averaging parameters.

```json
{
  "average_updated": false,
  "average_window": 10000,
  "type": "classification",
  "labels": [
    {
      "id": 0,
      "color": "#fff100",
      "label": "Zero"
    },
    {
      "id": 1,
      "color": "#ff8c00",
      "label": "One"
    },
    {
      "id": 2,
      "color": "#e81123",
      "label": "Two"
    },
    {
      "id": 3,
      "color": "#ec008c",
      "label": "Three"
    },
    {
      "id": 4,
      "color": "#68217a",
      "label": "Four"
    },
    {
      "id": 5,
      "color": "#00188f",
      "label": "Five"
    },
    {
      "id": 6,
      "color": "#00bcf2",
      "label": "Six"
    },
    {
      "id": 7,
      "color": "#00b294",
      "label": "Seven"
    },
    {
      "id": 8,
      "color": "#009e49",
      "label": "Eight"
    },
    {
      "id": 9,
      "color": "#bad80a",
      "label": "Nine"
    }
  ]
}
```

--------------------------------

### Run End-to-End Tests

Source: https://github.com/ertis-research/kafka-ml/blob/master/frontend/README.md

Executes the end-to-end tests for the front-end application using Protractor.

```bash
ng e2e
```

--------------------------------

### PyTorch VGG16 Model for EUROSAT

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/EUROSAT_RAW_format/README.md

Implements a VGG16 model in PyTorch for the EUROSAT dataset. This class defines the model structure, including pre-trained weights, flattening, and fully connected layers with ReLU activation and Softmax output. It also specifies the loss function, optimizer (SGD), and metrics (accuracy, loss).

```python
class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        self.pretrained = models.vgg16(pretrained=True)
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(1000, 512),
            nn.ReLU(),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
            nn.Softmax()
        )
        
    def forward(self, x):
        x = self.pretrained(x)
        x = self.flatten(x)
        output = self.linear_relu_stack(x)
        return output

    def loss_fn(self):
        return nn.CrossEntropyLoss()

    def optimizer(self):
        return torch.optim.SGD(model.parameters(), lr=1e-3)

    def metrics(self):
        val_metrics = {
            "accuracy": Accuracy(),
            "loss": Loss(self.loss_fn())
         }
        return val_metrics

model = VGG16()
```

--------------------------------

### Retrieve Kubernetes Service Account Token

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Retrieves the authentication token for the 'k8sadmin' service account. This token is required to authenticate with the Kubernetes API.

```bash
$ sudo kubectl -n kube-system describe secret $(sudo kubectl -n kube-system get secret | (grep k8sadmin || echo "$_") | awk '{print $1}') | grep token: | awk '{print $2}'
```

--------------------------------

### Configure External IP for Backend Service

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Assigns a specific external IP address to the backend service in Kubernetes, allowing access from outside the cluster. This IP must be available on a cluster node.

```yaml
  type: LoadBalancer
+ externalIPs:
+ - y.y.y.y.y.y
```

--------------------------------

### Define Distributed ML Models in Python

Source: https://github.com/ertis-research/kafka-ml/blob/master/README.md

Define the architecture for distributed ML models using TensorFlow/Keras. This includes creating separate models for edge, fog, and cloud layers, specifying their inputs and outputs.

```python
edge_input = keras.Input(shape=(28,28,1), name='input_img')
x = tf.keras.layers.Conv2D(28, kernel_size=(3,3), name='conv2d')(edge_input)
x = tf.keras.layers.MaxPooling2D(pool_size=(2,2), name='maxpooling')(x)
x = tf.keras.layers.Flatten(name='flatten')(x)
output_to_fog = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='output_to_fog')(x)
edge_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='edge_output')(output_to_fog)
edge_model = keras.Model(inputs=[edge_input], outputs=[output_to_fog, edge_output], name='edge_model')

fog_input = keras.Input(shape=64, name='fog_input')
output_to_cloud = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='output_to_cloud')(fog_input)
fog_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='fog_output')(output_to_cloud)
fog_model = keras.Model(inputs=[fog_input], outputs=[output_to_cloud, fog_output], name='fog_model')

cloud_input = keras.Input(shape=64, name='cloud_input')
x = tf.keras.layers.Dense(64, activation=tf.nn.relu, name='relu1')(cloud_input)
x = tf.keras.layers.Dense(128, activation=tf.nn.relu, name='relu2')(x)
x = tf.keras.layers.Dropout(0.2)(x)
cloud_output = tf.keras.layers.Dense(10, activation=tf.nn.softmax, name='cloud_output')(x)
cloud_model = keras.Model(inputs=cloud_input, outputs=[cloud_output], name='cloud_model')
```

--------------------------------

### PyTorch VGG16 Model Definition

Source: https://github.com/ertis-research/kafka-ml/blob/master/examples/VGG16_CIFAR10_RAW_format/README.md

Implements a VGG16 model using PyTorch, leveraging pre-trained weights and custom linear layers for CIFAR10 classification. This is suitable for PyTorch-based pipelines.

```python
class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        self.pretrained = models.vgg16(pretrained=True)
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
			nn.Linear(1000, 512),
			nn.ReLU(),
			nn.Linear(512, 256),
			nn.ReLU(),
			nn.Linear(256, 128),
			nn.ReLU(),
			nn.Linear(128, 10),
			nn.Softmax()
		)
		
    def forward(self, x):
        x = self.pretrained(x)
        x = self.flatten(x)
        output = self.linear_relu_stack(x)
        return output

    def loss_fn(self):
        return nn.CrossEntropyLoss()

    def optimizer(self):
        return torch.optim.SGD(model.parameters(), lr=1e-3)

    def metrics(self):
        val_metrics = {
            "accuracy": Accuracy(),
            "loss": Loss(self.loss_fn())
         }
        return val_metrics

model = VGG16()
```