### Install KFP v2

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/getting-started.md

Install the KFP SDK v2 using pip. This is the first step to start using KFP.

```sh
pip install kfp
```

--------------------------------

### Example Apt-get Install in Dockerfile

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/container-images.md

Install Linux packages using apt-get within a Dockerfile. Remember to switch to root user before running apt-get and switch back afterwards.

```dockerfile
USER root
RUN apt-get update && apt-get install -y --no-install-recommends \
    package1 \
    package2 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
USER $NB_USER
```

--------------------------------

### Verify GPU Setup for Containers

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

Check NVIDIA driver installation on the host, verify the NVIDIA Container Toolkit by running a CUDA container, and ensure GPUs are requested in your trainer configuration.

```bash
# 1. Verify NVIDIA drivers on host
nvidia-smi

# 2. Verify NVIDIA Container Toolkit
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi

# 3. Request GPU in your trainer
trainer = CustomTrainer(
    func=train_model,
    resources_per_node={"gpu": "1"}
)
```

--------------------------------

### Install Katib Python SDK (Specific Commit)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/installation.md

Installs the Katib Python SDK from a specific GitHub commit hash. Useful for reproducible installations.

```shell
pip install git+https://github.com/kubeflow/katib.git@ea46a7f2b73b2d316b6b7619f99eb440ede1909b#subdirectory=sdk/python/v1beta1
```

--------------------------------

### Example s6-overlay Cont-init Script

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/container-images.md

A startup script for s6-overlay placed in `/etc/cont-init.d/`. This example uses `with-contenv` to access environment variables within the script.

```bash
#!/bin/sh
# shellcheck disable=SC2039

. /usr/lib/s6/functions/with-contenv

# Snapshot Kubernetes environment variables for kubectl
with-contenv

# Example: snapshotting KUBERNETES_* variables
for var in $(env | grep '^KUBERNETES_' | cut -d= -f1);
do
  echo "export $var=${!var}" >> /home/jovyan/.Renviron.site
done

exit 0
```

--------------------------------

### List Pipeline Runs Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/reference/api/kubeflow-pipeline-api-spec.md

Send a GET request to this URL to list pipeline runs in a specific namespace. Ensure you replace `https://kubeflow.example.com` with your Kubeflow deployment URL and `team-1` with your target namespace.

```bash
https://kubeflow.example.com/pipeline/apis/v2beta1/runs?namespace=team-1
```

--------------------------------

### Install Git Pre-Commit Hooks

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/developer-guide.md

Install the pre-commit package manager and set up the Git pre-commit hooks for the repository.

```bash
# Using pip
pip install pre-commit

# Using conda
conda install -c conda-forge pre-commit

# Using Homebrew
brew install pre-commit
```

```bash
pre-commit install

pre-commit install-hooks
```

--------------------------------

### Example s6-overlay Service Run Script

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/container-images.md

The `run` script for an s6-overlay service, typically placed in `/etc/services.d/<service_name>/run`. This example starts JupyterLab.

```bash
#!/bin/sh

exec /usr/local/bin/start-notebook.sh
```

--------------------------------

### Example Pip Install in Dockerfile

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/container-images.md

Install Python packages using pip within a Dockerfile. Avoid using `pip install --user` to prevent conflicts with pre-installed packages.

```dockerfile
RUN pip install --upgrade pip
RUN pip install --no-cache-dir \
    package1 \
    package2
```

--------------------------------

### Install Kubeflow SDK with Docker Support

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

Install the Kubeflow SDK, ensuring Docker support is included for local execution.

```bash
pip install "kubeflow[docker]"
```

--------------------------------

### Install Kubeflow SDK with Podman Support

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/podman.md

Install the Kubeflow SDK, ensuring Podman support is included for container operations.

```bash
pip install "kubeflow[podman]"
```

--------------------------------

### Install kfp-kubernetes Library

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/core-functions/platform-specific-features.md

Install the `kfp-kubernetes` library to enable Kubernetes-specific features. This command includes the base `kfp` package along with the necessary Kubernetes extras.

```sh
pip install kfp[kubernetes]
```

--------------------------------

### Install Kubeflow SDK

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/local_process.md

Install the base Kubeflow SDK package using pip.

```bash
pip install kubeflow
```

--------------------------------

### Install All Default Runtimes with Helm

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/operator-guides/installation.md

Install the Kubeflow Trainer Helm chart and enable all default training runtimes (torch, deepspeed, mlx, jax, torchtune).

```bash
helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version ${VERSION#v} \
    --set runtimes.defaultEnabled=true
```

--------------------------------

### Install Katib Python SDK (Latest Changes)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/installation.md

Installs the latest changes of the Katib Python SDK directly from the master branch of the GitHub repository.

```shell
pip install git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1
```

--------------------------------

### Install Kubeflow Python SDK (From Source)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/getting-started.md

Install the latest changes of the Kubeflow Python SDK directly from its source repository using pip.

```bash
pip install git+https://github.com/kubeflow/sdk.git@main
```

--------------------------------

### Configure TLS During Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/hub/installation.md

Set the INSECURE_SKIP_VERIFY environment variable for the Model Registry UI deployment during local installation. This is for development only.

```bash
kubectl set env deployment/model-registry-ui INSECURE_SKIP_VERIFY=true -n $PROFILE_NAME
```

--------------------------------

### Start Docker Daemon

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

Commands to start the Docker daemon on different operating systems and verify its status. This is necessary to resolve 'Connection refused' errors.

```bash
# macOS/Windows: Start Docker Desktop
# Linux: Start Docker daemon
sudo systemctl start docker

# Verify Docker is running
docker ps
```

--------------------------------

### Start Local Hugo Server

Source: https://github.com/kubeflow/website/blob/master/README.md

Starts a local Hugo development server with draft content enabled (`-D`). Ensure you are in the root directory of the repository before running.

```bash
# NOTE: You should ensure that you are in the root directory of the repository.
hugo server -D
```

--------------------------------

### Install Model Registry Python Client

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/hub/getting-started.md

Install the Model Registry Python client and KServe. Note potential dependency version conflicts with packages like pydantic.

```bash
!pip install model-registry=="{{% hub/latest-version %}}"
!pip install kserve=="0.13"
```

--------------------------------

### Example Conda Install in Dockerfile

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/container-images.md

Install Python packages using conda within a Dockerfile. Ensure the correct environment is activated before installing.

```dockerfile
RUN conda install --yes \
    package1 \
    package2 && \
    conda clean --all -y
```

--------------------------------

### KatibConfig Initialization Parameters

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/user-guides/katib-config.md

Example of KatibConfig initialization parameters, including settings for certGenerator and controller.

```yaml
apiVersion: config.kubeflow.org/v1beta1
kind: KatibConfig
init:
  certGenerator:
    enable: true
    ...
  controller:
    trialResources:
      - Job.v1.batch
      - TFJob.v1.kubeflow.org
    ...

```

--------------------------------

### Complete ClusterTrainingRuntime Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/operator-guides/job-template.md

An example demonstrating the configuration of a `ClusterTrainingRuntime` with multiple `replicatedJobs`, including dataset initializer, model initializer, launcher, and the main training node, along with their dependencies and ancestor labels.

```yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: example-runtime
  labels:
    trainer.kubeflow.org/framework: mlx
spec:
  template:
    spec:
      replicatedJobs:
        - name: dataset-initializer
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
            spec:
              template:
                spec:
                  containers:
                    - name: dataset-initializer
                      image: ghcr.io/kubeflow/trainer/dataset-initializer
        - name: model-initializer
          dependsOn:
            - name: dataset-initializer
              status: Complete
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
            spec:
              template:
                spec:
                  containers:
                    - name: model-initializer
                      image: ghcr.io/kubeflow/trainer/model-initializer
        - name: launcher
          dependsOn:
            - name: model-initializer
              status: Complete
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: trainer
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: ghcr.io/kubeflow/trainer/mlx-runtime
                      securityContext:
                        runAsUser: 1000
        - name: node
          template:
            spec:
              template:
                spec:
                  containers:
                    - name: node
                      image: ghcr.io/kubeflow/trainer/mlx-runtime
                      securityContext:
                        runAsUser: 1000
                      command:
                        - /usr/sbin/sshd
                      args:
                        - -De
                        - -f
                        - /home/mpiuser/.sshd_config
                      readinessProbe:
                        tcpSocket:
                          port: 2222
                        initialDelaySeconds: 5
```

--------------------------------

### Example: Compile Pipeline with Parameters

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/core-functions/cli.md

Demonstrates how to compile a pipeline and provide specific parameters using a JSON string. This example sets 'param1' to 2.0 and 'param2' to 'my_val'.

```shell
kfp dsl compile --py path/to/pipeline.py --output path/to/output.yaml --pipeline-parameters '{"param1": 2.0, "param2": "my_val"}'
```

--------------------------------

### Retrieve TorchTune Runtime

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/builtin-trainer/torchtune.md

Fetches a specific runtime configuration for TorchTune from the available manifests. Use this to get the correct setup for your desired model.

```python
runtime=client.get_runtime("torchtune-llama3.2-1b")
```

--------------------------------

### Define a Hello World Component

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/concepts/component.md

This example demonstrates how to define a basic 'hello world' component using the `@dsl.component` decorator. It takes an optional name parameter and prints a greeting.

```python
from kfp.dsl import component, Output, Dataset

# hello world component
@component()
def hello_world(name: str = "World") -> str:
    print(f"Hello {name}!")
    return name

```

--------------------------------

### KatibConfig Runtime Parameters

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/user-guides/katib-config.md

Example of KatibConfig runtime parameters, showing configurations for metricsCollectors, suggestions, and earlyStoppings.

```yaml
apiVersion: config.kubeflow.org/v1beta1
kind: KatibConfig
runtime:
  metricsCollectors:
    - kind: StdOut
      image: docker.io/kubeflowkatib/file-metrics-collector:latest
    ...
  suggestions:
    - algorithmName: random
      image: docker.io/kubeflowkatib/suggestion-hyperopt:latest
    ...
  earlyStoppings:
    - algorithmName: medianstop
      image: docker.io/kubeflowkatib/earlystopping-medianstop:latest
    ...

```

--------------------------------

### Deploy Katib with PostgreSQL Database

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/user-guides/installation-options.md

Run this command to install Katib using PostgreSQL as the database backend instead of MySQL.

```shell
kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone-postgres?ref=master"
```

--------------------------------

### SparkApplication Init-Containers Configuration

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/user-guide/writing-sparkapplication.md

Specify optional init-containers for driver or executor pods to perform setup tasks before the main application starts. Specification follows the Kubernetes Container API. Requires the mutating admission webhook to be enabled.

```yaml
spec:
  driver:
    initContainers:
    - name: "init-container1"
      image: "init-container1:latest"
      ...
  executor:
    initContainers:
    - name: "init-container1"
      image: "init-container1:latest"
      ...
```

--------------------------------

### Install Kubeflow Training Runtimes (Released Version)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/operator-guides/installation.md

Deploy a released version of the Kubeflow Training Runtimes using kubectl. Ensure the VERSION environment variable is set.

```bash
kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=${VERSION}"
```

--------------------------------

### Install Specific Hugo Version with Homebrew

Source: https://github.com/kubeflow/website/blob/master/README.md

Installs Hugo version 0.124.1 using a specific Homebrew commit to ensure compatibility. This is recommended over `brew install hugo` which installs the latest version.

```bash
# WARNING: using `brew install hugo` will install the latest version of hugo
#          which may not be compatible with the website

# TIP: to install hugo run the following commands
HOMEBREW_COMMIT="9d025105a8be086b2eeb3b1b2697974f848dbaac" # 0.124.1
curl -fL -o "hugo.rb" "https://raw.githubusercontent.com/Homebrew/homebrew-core/${HOMEBREW_COMMIT}/Formula/h/hugo.rb"
HOMEBREW_DEVELOPER="true" brew install ./hugo.rb
brew pin hugo
```

--------------------------------

### Describe Notebook Resource

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/notebooks/troubleshooting.md

Check the events of a Notebook resource to identify startup errors. Replace placeholders with your notebook and namespace names.

```shell
kubectl describe notebooks "${MY_NOTEBOOK_NAME}" --namespace "${MY_PROFILE_NAMESPACE}"
```

--------------------------------

### Check KFP CLI Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/core-functions/cli.md

Verify if the KFP CLI is installed in your environment. This command should be run after installing the KFP SDK.

```shell
kfp --version
```

--------------------------------

### Install Custom Packages from Private Index

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/components/notebook-component.md

Install packages from a private repository using `packages_to_install` and `pip_index_urls`. The component will install these packages before execution.

```python
@dsl.notebook_component(
    notebook_path="train.ipynb",
    packages_to_install=['custom-ml-package==0.0.1'],
    pip_index_urls=['http://myprivaterepo.com/simple'])
def evaluate_model(dataset_uri: str):
    from custom_ml_package import model_trainer
    dsl.run_notebook(
        dataset_uri=dataset_uri,
        output_model_path=model_trainer.path,
    )
```

--------------------------------

### Setting up Python Virtual Environment

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/local_process.md

This command sequence sets up a Python virtual environment, installs pip, and upgrades it. It's used for isolating dependencies during local execution.

```bash
python -m venv --without-pip /tmp/a1b2c3d4e5f_xyz/
source /tmp/a1b2c3d4e5f_xyz/bin/activate
python -m ensurepip --upgrade --default-pip
```

--------------------------------

### Basic Docker Backend Configuration

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

Configure the container backend to use Docker. This is the most basic setup.

```python
backend_config = ContainerBackendConfig(
    container_runtime="docker",
)
```

--------------------------------

### Install Spark Operator Instance 1

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/user-guide/running-multiple-instances-of-the-operator.md

Installs the first Spark operator instance, specifying the namespace it should watch. Ensure the target namespace exists before installation.

```bash
# Create the spark-1 namespace if it does not exist
kubectl create ns spark-1

# Install the Spark operator with release name spark-operator-1
helm install spark-operator-1 spark-operator/spark-operator \
    --namespace spark-operator \
    --create-namespace \
    --set 'spark.jobNamespaces={spark-1}'
```

--------------------------------

### Install Spark Operator with Volcano Enabled

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/user-guide/volcano-integration.md

Install the Spark Operator using Helm, enabling the webhook and batch scheduler for Volcano integration. Ensure Volcano is installed separately.

```shell
helm repo add spark-operator https://kubeflow.github.io/spark-operator

helm install my-release spark-operator/spark-operator \
    --namespace spark-operator \
    --set webhook.enable=true \
    --set controller.batchScheduler.enable=true
```

--------------------------------

### Basic Local Process Backend Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/local_process.md

Demonstrates running a simple PyTorch training function locally using the Local Process Backend. It configures the backend to automatically clean up virtual environments after completion and waits for the job to finish.

```python
from kubeflow.trainer import CustomTrainer, TrainerClient, LocalProcessBackendConfig

# Define your training function
def train_model():
    import torch
    import time

    print("Starting training...")

    # Your training code here
    model = torch.nn.Linear(10, 1)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    for epoch in range(5):
        # Training loop
        loss = torch.nn.functional.mse_loss(
            model(torch.randn(32, 10)),
            torch.randn(32, 1)
        )
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print(f"Epoch {epoch + 1}/5, Loss: {loss.item():.4f}")

    print("Training completed!")

# Configure the backend
backend_config = LocalProcessBackendConfig(
    cleanup_venv=True  # Automatically clean up virtual environments after completion
)

# Create the client
client = TrainerClient(backend_config=backend_config)

# Create the trainer
trainer = CustomTrainer(
    func=train_model,
    num_nodes=1,  # Local process backend ignores this parameter
)

# Start the TrainJob
job_name = client.train(trainer=trainer)
print(f"TrainJob started: {job_name}")

# Wait for completion
job = client.wait_for_job_status(
    job_name,
)

print(f"Job completed with status: {job.status}")
```

--------------------------------

### Verify Clustertrainingruntime Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/data-cache.md

Checks if the 'torch-distributed-with-cache' runtime is installed and available in the cluster.

```bash
$ kubectl get clustertrainingruntime

NAME                           AGE
torch-distributed-with-cache   14h
```

--------------------------------

### Install Spark Operator Chart

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/getting-started.md

Installs the Spark Operator using the Helm chart. Replace [RELEASE_NAME] with your desired release name. The operator will be installed in the 'spark-operator' namespace, which will be created if it doesn't exist.

```shell
helm install [RELEASE_NAME] spark-operator/spark-operator
```

--------------------------------

### Control KFP Package Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/components/lightweight-python-components.md

Use `install_kfp_package=False` to prevent automatic installation of the `kfp` package. This allows installing a different version via `packages_to_install` and `pip_index_urls`. This is rarely needed and generally discouraged.

```python
import kfp
from kfp import dsl

# Example demonstrating the concept, actual usage might involve specific versions
# @dsl.component(install_kfp_package=False, packages_to_install=['kfp==2.0.0'], pip_index_urls=['http://myprivaterepo.com/simple'])
# def my_component():
#     pass
```

--------------------------------

### Verify Docker Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

Check if Docker is installed and running, and test its daemon connectivity.

```bash
# Check Docker is running
docker version

# Test Docker daemon connectivity
docker ps

```

--------------------------------

### Katib Experiment with FromVolume Resume Policy

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/user-guides/resume-experiment.md

Example of a Katib Experiment configuration using the `FromVolume` resume policy. This policy attaches a volume to the suggestion's deployment, allowing suggestion data to be retained and recovered upon restarting the experiment.

```yaml
apiVersion: "kubeflow.org/v1beta1"
kind: Experiment
metadata:
  name: from-volume-resume
  namespace: kubeflow
spec:
  parallelTrialCount: 3
  maxTrialCount: 6
  maxFailedTrialCount: 3
  resumePolicy: FromVolume
  objective:
    type: minimize
    goal: 0.001
    objectiveMetricName: accuracy
  algorithm:
    algorithmName: random
    parameters:
    - name: "--lr"
      type: "double"
      feasibleSpace:
        min: "0.0001"
        max: "0.1"
    - name: "--num-layers"
      type: "int"
      feasibleSpace:
        min: "1"
        max: "5"
  trialTemplate:
    primaryContainerName: "training-container"
    trialParameters:
    - name: "lr"
      reference: "--lr"
    - name: "num-layers"
      reference: "--num-layers"
    trialSpec:
      environmentVariables:
      - name: "TF_CONFIG"
        value: ""
      restartPolicy:
        name: Never
      containers:
      - name: "training-container"
        image: "docker.io/kubeflow/tf-operator:v0.7.0-rc.0"
        command:
        - "python"
        - "/app/run.py"
        - "--lr={{.Trial."lr"}}-{{.Trial."num-layers"}} "

```

--------------------------------

### Verify Podman Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/podman.md

Check if Podman is installed and running correctly by verifying its version and listing active containers.

```bash
podman version
podman ps
```

--------------------------------

### Initialize KFPClientManager and Create KFP Client

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/core-functions/connect-api.md

Initialize a KFPClientManager with API URL and authentication details, then create an authenticated KFP client. Useful for authenticated environments.

```python
kfp_client_manager = KFPClientManager(
    api_url="http://localhost:8080/pipeline",
    skip_tls_verify=True,

    dex_username="user@example.com",
    dex_password="12341234",

    # can be 'ldap' or 'local' depending on your Dex configuration
    dex_auth_type="local",
)

# get a newly authenticated KFP client
# TIP: long-lived sessions might need to get a new client when their session expires
kfp_client = kfp_client_manager.create_kfp_client()

# test the client by listing experiments
experiments = kfp_client.list_experiments(namespace="my-profile")
print(experiments)
```

--------------------------------

### Spark Operator Events Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/getting-started.md

Example of events generated by the Spark Operator for a SparkApplication, indicating its lifecycle stages.

```text
Events:
  Type    Reason                      Age   From            Message
  ----    ------                      ----  ----            -------
  Normal  SparkApplicationAdded       5m    spark-operator  SparkApplication spark-pi was added, enqueued it for submission
  Normal  SparkApplicationTerminated  4m    spark-operator  SparkApplication spark-pi terminated with state: COMPLETED

```

--------------------------------

### Install Kubeflow Training Runtimes (Latest Changes)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/operator-guides/installation.md

Deploy the latest development version of the Kubeflow Training Runtimes from the master branch. Use this for testing unreleased features.

```bash
kubectl apply --server-side -k "https://github.com/kubeflow/trainer.git/manifests/overlays/runtimes?ref=master"
```

--------------------------------

### SparkApplication YAML Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/getting-started.md

An example of a SparkApplication custom resource definition, showing driver, executor, and application configurations.

```yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  ...
spec:
  deps: {}
  driver:
    coreLimit: 1200m
    cores: 1
    labels:
      version: 2.3.0
    memory: 512m
    serviceAccount: spark
  executor:
    cores: 1
    instances: 1
    labels:
      version: 2.3.0
    memory: 512m
  image: gcr.io/ynli-k8s/spark:v3.1.1
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar
  mainClass: org.apache.spark.examples.SparkPi
  mode: cluster
  restartPolicy:
      type: OnFailure
      onFailureRetries: 3
      onFailureRetryInterval: 10
      onSubmissionFailureRetries: 5
      onSubmissionFailureRetryInterval: 20
  type: Scala
status:
  sparkApplicationId: spark-5f4ba921c85ff3f1cb04bef324f9154c9
  applicationState:
    state: COMPLETED
  completionTime: 2018-02-20T23:33:55Z
  driverInfo:
    podName: spark-pi-83ba921c85ff3f1cb04bef324f9154c9-driver
    webUIAddress: 35.192.234.248:31064
    webUIPort: 31064
    webUIServiceName: spark-pi-2402118027-ui-svc
    webUIIngressName: spark-pi-ui-ingress
    webUIIngressAddress: spark-pi.ingress.cluster.com
  executorState:
    spark-pi-83ba921c85ff3f1cb04bef324f9154c9-exec-1: COMPLETED
  LastSubmissionAttemptTime: 2018-02-20T23:32:27Z

```

--------------------------------

### Basic Docker Container Backend Training Example

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/docker.md

A Python script demonstrating a simple training function executed using the Docker Container Backend. It configures the backend for Docker, sets up a custom trainer for multi-node distributed training, and initiates a TrainJob.

```python
from kubeflow.trainer import CustomTrainer, TrainerClient, ContainerBackendConfig

def train_model():
    """Simple training function."""
    import torch
    import os

    rank = int(os.environ.get('RANK', '0'))
    world_size = int(os.environ.get('WORLD_SIZE', '1'))

    print(f"Training on rank {rank}/{world_size}")

    # Your training code
    model = torch.nn.Linear(10, 1)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    for epoch in range(5):
        loss = torch.nn.functional.mse_loss(
            model(torch.randn(32, 10)),
            torch.randn(32, 1)
        )
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print(f"[Rank {rank}] Epoch {epoch + 1}/5, Loss: {loss.item():.4f}")

    print(f"[Rank {rank}] Training completed!")

# Configure the Docker backend
backend_config = ContainerBackendConfig(
    container_runtime="docker",  # Explicitly use Docker
    pull_policy="IfNotPresent",  # Pull image if not cached locally
    auto_remove=True             # Clean up containers after completion
)

# Create the client
client = TrainerClient(backend_config=backend_config)

# Create a trainer with multi-node support
trainer = CustomTrainer(
    func=train_model,
    num_nodes=2  # Run distributed training across 2 containers
)

# Start the TrainJob
job_name = client.train(trainer=trainer)
print(f"TrainJob started: {job_name}")

# Wait for completion
job = client.wait_for_job_status(
    job_name,
)

print(f"Job completed with status: {job.status}")

```

--------------------------------

### Verify Namespace RBAC Installation

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/data-cache.md

Verifies that the ServiceAccount and RoleBinding for the cache initializer are installed in the specified namespace (e.g., 'default').

```bash
$ kubectl get sa,rolebinding -n default | grep cache-initializer

serviceaccount/kubeflow-trainer-cache-initializer
rolebinding.rbac.authorization.k8s.io/kubeflow-trainer-cache-initializer
```

--------------------------------

### TFJob Example with Summaries

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/katib/user-guides/metrics-collector.md

This YAML configuration demonstrates how to integrate Katib with a TensorFlow Job (TFJob) that uses summaries for metrics collection.

```yaml
apiVersion: "kubeflow.org/v1beta1"
kind: "Katib"
metadata:
  name: "tfjob-mnist-with-summaries"
spec:
  trials:
    maxTrialCount: 1
    trialTemplate:
      template:
        spec:
          containers:
            - name: "training-container"
              image: "tensorflow/tensorflow:latest-gpu"
              command:
                - "python"
                - "mnist_with_summaries.py"
              volumeMounts:
                - name: "tensorboard-log"
                  mountPath: "/mnt/tensorboard"
  collector:
    kind: "TensorFlowEvent"
    source:
      fileSystemPath:
        path: "/mnt/tensorboard"
  volumes:
    - name: "tensorboard-log"
      emptyDir: {}
```

--------------------------------

### Install Kubeflow Python SDK (Stable)

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/getting-started.md

Install the latest stable version of the Kubeflow Python SDK using pip.

```bash
pip install -U kubeflow
```

--------------------------------

### General KFP CLI Syntax

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/user-guides/core-functions/cli.md

Illustrates the basic structure for all KFP CLI commands. Use this to understand how to invoke commands and their arguments.

```shell
kfp [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### Install Locust and Dependencies

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/spark-operator/performance/benchmarking.md

Installs Locust and its required dependencies within a virtual environment. Ensure Python 3.12 is available.

```sh
python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

--------------------------------

### Run Distributed Training Locally with Podman

Source: https://github.com/kubeflow/website/blob/master/content/en/docs/components/trainer/user-guides/local-execution-mode/podman.md

This example demonstrates how to configure the TrainerClient to use Podman as the container runtime for local distributed training. It includes a sample training function that utilizes PyTorch for a simple distributed task.

```python
from kubeflow.trainer import CustomTrainer, TrainerClient, ContainerBackendConfig

def train_model():
    """Simple training function."""
    import torch
    import os

    # Environment variables set by torchrun
    rank = int(os.environ.get('RANK', '0'))
    world_size = int(os.environ.get('WORLD_SIZE', '1'))

    print(f"Training on rank {rank}/{world_size}")

    # Your training code
    model = torch.nn.Linear(10, 1)
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    for epoch in range(5):
        loss = torch.nn.functional.mse_loss(
            model(torch.randn(32, 10)),
            torch.randn(32, 1)
        )
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print(f"[Rank {rank}] Epoch {epoch + 1}/5, Loss: {loss.item():.4f}")

    print(f"[Rank {rank}] Training completed!")

# Configure the Podman backend
backend_config = ContainerBackendConfig(
    container_runtime="podman",   # Explicitly use Podman
    pull_policy="IfNotPresent",   # Pull image if not cached locally
    auto_remove=True              # Clean up containers after completion
)

# Create the client
client = TrainerClient(backend_config=backend_config)

# Create a trainer with multi-node support
trainer = CustomTrainer(
    func=train_model,
    num_nodes=2  # Run distributed training across 2 containers
)

# Start the TrainJob
job_name = client.train(trainer=trainer)
print(f"TrainJob started: {job_name}")

# Wait for completion
job = client.wait_for_job_status(
    job_name,
)

print(f"Job completed with status: {job.status}")

```