### Install Katib SDK via Setuptools Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md Install the Katib Python SDK using Setuptools. You can install it for the current user or for all users with sudo. ```sh python setup.py install --user ``` ```sh sudo python setup.py install ``` -------------------------------- ### Install Katib SDK Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb Install the Katib SDK from a GitHub repository. This command installs the latest version with new APIs. ```python # TODO (andreyvelich): Change to release version when SDK with the new APIs is published. !pip install git+https://github.com/kubeflow/katib.git#subdirectory=sdk/python/v1beta1 ``` -------------------------------- ### Deploy Katib Components Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/kind-cluster/README.md Run this script to install Katib components on your Kind cluster. Ensure you have Docker and Kind installed. ```bash ./deploy.sh ``` -------------------------------- ### Import Katib Package Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md Import the Katib package after installation to start using the SDK. ```python from kubeflow import katib ``` -------------------------------- ### Install Katib SDK Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/nas-with-darts.ipynb Install the required Katib SDK package. Ensure you are using a compatible version as specified. ```python # Install required package (Katib SDK). # TODO (andreyvelich): Update this example with the latest SDK. !pip install kubeflow-katib==0.13.0 ``` -------------------------------- ### Install build for Publishing Katib Models Source: https://github.com/kubeflow/katib/blob/master/docs/release/README.md Install the 'build' package, which is required for publishing Katib models. This is part of the release prerequisites. ```bash pip install build ``` -------------------------------- ### Install Kubeflow Pipelines and Katib SDK Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/kubeflow-pipelines/early-stopping.ipynb Use pip to install the specified versions of kfp and kubeflow-katib. Ensure these versions are compatible with your environment. ```python !pip install kfp==1.8.12 !pip install kubeflow-katib==0.13.0 ``` -------------------------------- ### Apply XGBoost Parameter Tuning Example Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/fpga/README.md Applies a Kubernetes configuration file to run an example XGBoost parameter tuning experiment. This assumes the 'xgboost-example.yaml' file is present and configured. ```sh kubectl apply -f xgboost-example.yaml ``` -------------------------------- ### Install Twine for Publishing SDK Package Source: https://github.com/kubeflow/katib/blob/master/docs/release/README.md Install the 'twine' package with a specific version for publishing the Katib SDK package to PyPI. This is a prerequisite for releasing the SDK. ```bash pip install twine==3.4.1 ``` -------------------------------- ### Install Kubeflow Python SDKs Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/mnist-with-push-metrics-collection.ipynb Install the Kubeflow SDKs to enable interaction with Kubeflow services. This command installs a specific version from a Git repository, which might be a development or release candidate version. ```python # TODO (Electronic-Waste): Change to release version when SDK with the updated `tune()` is published. %pip install git+https://github.com/kubeflow/katib.git#subdirectory=sdk/python/v1beta1 ``` -------------------------------- ### Install Kubeflow Python SDKs Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/tune-train-from-func.ipynb Installs the necessary TensorFlow package and Kubeflow SDKs for v1beta1 Katib and the Training Operator. Ensure to use release versions when available. ```python !pip install tensorflow==2.9.1 # TODO (andreyvelich): Change to release version when SDK with the new APIs is published. !pip install git+https://github.com/kubeflow/katib.git#subdirectory=sdk/python/v1beta1 !pip install git+https://github.com/kubeflow/training-operator.git#subdirectory=sdk/python ``` -------------------------------- ### Install Katib Python SDK (Latest Stable) Source: https://github.com/kubeflow/katib/blob/master/README.md Installs the latest stable release of the Katib Python SDK using pip. This SDK simplifies the creation of hyperparameter tuning jobs. ```sh pip install -U kubeflow-katib ``` -------------------------------- ### Resume Experiment from Volume Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment configured to resume from a volume. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: resume-from-volume-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random resumePolicy: fromVolume trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" ``` -------------------------------- ### Install PyGithub for Changelog Generation Source: https://github.com/kubeflow/katib/blob/master/docs/release/README.md Install the PyGithub library with a specific version to generate the changelog. This is a prerequisite for the release process. ```bash pip install PyGithub==1.55 ``` -------------------------------- ### Start Angular Development Server Source: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/frontend/README.md Run this command to start the Angular development server. The app will automatically reload when source files are changed. Access the app at http://localhost:4200/. ```bash npm run start ``` -------------------------------- ### Install Kubeflow Katib SDK via pip Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/README.md Use this command to install the Katib Python SDK using pip. Ensure you have Python 3.8 or higher. ```sh pip install kubeflow-katib ``` -------------------------------- ### Install Katib Standalone Control Plane (Latest Changes) Source: https://github.com/kubeflow/katib/blob/master/README.md Installs the latest changes from the master branch of the Katib control plane using kubectl. This is useful for testing development versions. ```bash kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=master" ``` -------------------------------- ### Example Usage of Push-based Metrics Collection Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2340-push-based-metrics-collector/README.md A simple example demonstrating how to use the `report_metrics` function within an objective function for push-based metrics collection. ```APIDOC ## Example: Push-based Metrics Collection ### Description This example shows a basic implementation of a push-based metrics collection strategy using Katib's Python SDK. ### Method (Conceptual - This is a Python SDK example) ### Endpoint (Conceptual - This is a Python SDK example) ### Parameters (Conceptual - Refers to parameters used in the `tune` function and `report_metrics` function) ### Request Example ```python import kubeflow.katib as katib # Step 1. Define an objective function that reports metrics using report_metrics. def objective(parameters): # Import required packages within the objective function if needed. import kubeflow.katib as katib # Calculate the objective function value. # Example calculation: result = 4 * a - b^2 result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2 # Push the calculated metric to Katib DB. # Ensure the trial name is passed as an environment variable 'KATIB_TRIAL_NAME'. katib.report_metrics({"result": result}) # Step 2. Define the hyperparameter search space. parameters = { "a": katib.search.int(min=10, max=20), "b": katib.search.double(min=0.1, max=0.2) } # Step 3. (Conceptual) Call the tune function with metrics_collector_config set to "Push". # This part is conceptual as the actual call to tune would be outside this snippet. # Example: # katib_client = katib.KatibClient() # katib_client.tune( # name="my-experiment", # objective=objective, # parameters=parameters, # metrics_collector_config={"kind": "Push"} # ) ``` ### Response #### Success Response (200) (Conceptual - Success is indicated by the experiment running without errors and metrics being reported.) #### Response Example None (The output is the successful execution of the training job and reporting of metrics.) ``` -------------------------------- ### Trial with ConfigMap Source Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment using a ConfigMap as the source for the Trial template. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: trial-configmap-source-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" volumeMounts: - name: nas-storage mountPath: /mnt/nas volumes: - name: nas-storage persistentVolumeClaim: claimName: nas-pvc trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" configMap: name: my-trial-configmap ``` -------------------------------- ### Install Katib Standalone Control Plane (Stable) Source: https://github.com/kubeflow/katib/blob/master/README.md Installs the latest stable release of the Katib control plane using kubectl. Ensure you have a compatible Kubernetes cluster and kubectl configured. ```bash kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=v0.17.0" ``` -------------------------------- ### Example Log Entry for Argo Workflow Integration Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/argo/README.md An example log entry from the Katib controller confirming that the 'Workflow' CRD kind has been successfully added to its watch list, indicating successful integration with Argo Workflows. ```json {"level":"info","ts":"2024-07-13T10:02:10Z","logger":"trial-controller","msg":"Job watch added successfully","CRD Group":"argoproj.io","CRD Version":"v1alpha1","CRD Kind":"Workflow"} ``` -------------------------------- ### Metrics Collection Strategy Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment demonstrating a Metrics Collection Strategy. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: metrics-collection-strategy-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" volumeMounts: - name: nas-storage mountPath: /mnt/nas volumes: - name: nas-storage persistentVolumeClaim: claimName: nas-pvc metricsCollectorSpec: source: fileSystemPath: path: "/mnt/nas/metrics" kind: File collectorTemplate: name: metrics-collection-strategy source: fileSystemPath: path: "/mnt/nas/metrics" kind: File ``` -------------------------------- ### Trial with Metadata Substitution Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment using metadata substitution in the Trial template. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: trial-metadata-substitution-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" volumeMounts: - name: nas-storage mountPath: /mnt/nas volumes: - name: nas-storage persistentVolumeClaim: claimName: nas-pvc trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" env: - name: TRIAL_NAME value: "${trial.name}" - name: TRIAL_HYPERMETERS value: "${trialParameters.learningRate},${trialParameters.batchSize}" ``` -------------------------------- ### PBT Hyperparameter Tuning Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment using PBT for hyperparameter tuning. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: pbt-example spec: parallelTrialCount: 3 maxTrialCount: 15 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: pbt trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" ``` -------------------------------- ### File Metrics Collector Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment using the File Metrics Collector. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: file-metrics-collector-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" volumeMounts: - name: nas-storage mountPath: /mnt/nas volumes: - name: nas-storage persistentVolumeClaim: claimName: nas-pvc metricsCollectorSpec: source: fileSystemPath: path: "/mnt/nas/metrics" kind: File collectorTemplate: name: file-metrics-collector source: fileSystemPath: path: "/mnt/nas/metrics" kind: File ``` -------------------------------- ### PBT Proof of Concept Trial Image Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Trial image for PBT proof of concept. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: simple-pbt-example spec: parallelTrialCount: 3 maxTrialCount: 15 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: pbt trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/simple-pbt:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" ``` -------------------------------- ### Grid Search Hyperparameter Tuning Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Katib Experiment using Grid Search for hyperparameter tuning. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: grid-search-example spec: parallelTrialCount: 2 maxTrialCount: 4 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: grid trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/tf-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" ``` -------------------------------- ### Get Angular CLI Help Source: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/frontend/README.md Display help information for the Angular CLI, providing details on available commands and options. ```bash ng help ``` -------------------------------- ### Serve Katib UI from Backend Source: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/README.md Runs the Katib UI by serving the frontend artifacts via the Go backend. This is the recommended method for end-to-end testing. It requires the `katib-db-manager` service to be accessible. ```bash export APP_DISABLE_AUTH=true go run main.go --build-dir=../../../pkg/ui/v1beta1/frontend/dist --port=8080 --db-manager-address=localhost:6789 ``` -------------------------------- ### Create and Run a Katib Tuning Experiment Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2339-hpo-for-llm-fine-tuning/README.md Instantiate the Katib client, define an experiment with model, dataset, and trainer parameters, and then run the tuning process. Ensure to specify objective metrics and resource allocation per trial. ```python cl = KatibClient(namespace="kubeflow") # Run the tuning experiment. exp_name = "llm-experiment1" cl.tune( name = exp_name, # BERT model URI and type of Transformer to train it. model_provider_parameters = HuggingFaceModelParams( model_uri = "hf://google-bert/bert-base-cased", transformer_type = transformers.AutoModelForSequenceClassification, ), # Use 3000 samples from Yelp dataset. dataset_provider_parameters = HuggingFaceDatasetParams( repo_id = "yelp_review_full", split = "train[:3000]", ), # Specify HuggingFace Trainer parameters. trainer_parameters = HuggingFaceTrainerParams( training_parameters = transformers.TrainingArguments( output_dir = "test_tune_api", save_strategy = "no", learning_rate = katib.search.double(min=1e-05, max=5e-05), num_train_epochs=3, ), # Set LoRA config to reduce number of trainable model parameters. lora_config = LoraConfig( r = katib.search.int(min=8, max=32), lora_alpha = 8, lora_dropout = 0.1, bias = "none", ), ), objective_metric_name = "train_loss", objective_type = "minimize", algorithm_name = "random", max_trial_count = 10, parallel_trial_count = 2, resources_per_trial={ "gpu": "2", "cpu": "4", "memory": "10G", }, # For distribued training, please specify `resources_per_trial` using `types.TrainerResources` (To be implemented). ) # Wait until Katib Experiment is complete cl.wait_for_experiment_condition(name=exp_name) # Get the best hyperparameters. print(cl.get_optimal_hyperparameters(exp_name)) ``` -------------------------------- ### Build Katib from Source Source: https://github.com/kubeflow/katib/blob/master/CONTRIBUTING.md Use this command to build Katib images from the source code. Replace placeholders with your registry and tag. ```bash make build REGISTRY= TAG= ``` -------------------------------- ### Create Experiments with Different Resume Policies Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb Create Katib experiments using the client, including those with 'Never' and 'FromVolume' resume policies. ```python # Create Experiment with never resume. kclient.create_experiment(experiment_never_resume,namespace=namespace) # Create Experiment with from volume resume. kclient.create_experiment(experiment_from_volume_resume,namespace=namespace) ``` -------------------------------- ### Install InAccel FPGA Operator Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/fpga/README.md Installs the InAccel FPGA Operator in the 'kube-system' namespace. This operator manages FPGA resources within the Kubernetes cluster. ```sh helm install -n kube-system inaccel inaccel/fpga-operator ``` -------------------------------- ### Create Katib Client and Experiment Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb Instantiate the Katib client and use it to create a new experiment in the specified namespace. ```python # Create Katib client. kclient = KatibClient() # Create your Experiment. kclient.create_experiment(experiment,namespace=namespace) ``` -------------------------------- ### KatibClient Initialization Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Initialize the KatibClient with optional configuration parameters. ```APIDOC ## KatibClient Constructor ### Description Initializes the KatibClient, allowing users to load authentication and cluster information from a kube-config file. ### Parameters #### Path Parameters - **config_file** (str) - Optional - Name of the kube-config file. Defaults to ~/.kube/config. - **context** (str) - Optional - Set the active context. Defaults to current_context from the kube-config. - **client_configuration** (kubernetes.client.Configuration) - Optional - The kubernetes.client.Configuration to set configs to. - **persist_config** (bool) - Optional - If True, config file will be updated when changed. Defaults to True. ``` -------------------------------- ### Initialize Katib Client Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/nas-with-darts.ipynb Instantiate the KatibClient to interact with the Katib service. This client is necessary for submitting and managing experiments. ```python # Create client. kclient = KatibClient() ``` -------------------------------- ### Install Argo Workflows v3.1.3 Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/argo/README.md Deploy Argo Workflows version 3.1.3 to your Kubernetes cluster. Ensure Argo Workflows is installed before proceeding with Katib integration. ```bash kubectl create namespace argo kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.1.3/install.yaml ``` -------------------------------- ### Example: Objective function with push-based metrics Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2340-push-based-metrics-collector/README.md Demonstrates how to define an objective function that calculates a result and pushes it to Katib DB using `report_metrics`. ```python import kubeflow.katib as katib # Step 1. Create an objective function with push-based metrics collection. def objective(parameters): # Import required packages. import kubeflow.katib as katib # Calculate objective function. result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2 # Push metrics to Katib DB. katib.report_metrics({"result": result}) # Step 2. Create HyperParameter search space. parameters = { "a": katib.search.int(min=10, max=20), "b": katib.search.double(min=0.1, max=0.2) } ``` -------------------------------- ### Start a Katib Experiment Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/mnist-with-push-metrics-collection.ipynb Initiate a Katib hyperparameter tuning experiment by specifying the experiment name, objective function, base Docker image, hyperparameter distributions, and tuning algorithm. Note the TODO comments for potential future updates. ```python # Start the Katib Experiment. # TODO (Electronic-Waste): # 1. Change `kubeflow-katib` to release version when `0.18.0` is ready. # 2. Change `base_image` to official image when `kubeflow-katib` release version `0.18.0` is ready. exp_name = "tune-mnist" katib_client = katib.KatibClient(namespace="kubeflow") katib_client.tune( name=exp_name, objective=train_mnist_model, # Objective function. base_image="docker.io/electronicwaste/pytorch:gitv1", parameters=parameters, # HyperParameters to tune. algorithm_name="random", # Alorithm to use. objective_metric_name="accuracy", # Katib is going to optimize "accuracy". additional_metric_names=["loss"], # Katib is going to collect these metrics in addition to the objective metric. max_trial_count=12, # Trial Threshold. parallel_trial_count=2, packages_to_install=["git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1"], metrics_collector_config={"kind": "Push"}, ) ``` -------------------------------- ### Build Katib UI Docker Image Source: https://github.com/kubeflow/katib/blob/master/pkg/ui/v1beta1/README.md Use this command to build a Docker image for the Katib UI. Replace `` with your desired image name. ```bash docker build . -f cmd/ui/v1beta1/Dockerfile -t ``` -------------------------------- ### Define Experiments with Resume Policies Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/cmaes-and-resume-policies.ipynb Define new Katib Experiments based on a previous configuration, specifying different resume policies. This example sets up experiments with 'Never' and 'FromVolume' resume policies. ```python experiment_never_resume_name = "never-resume-cmaes" experiment_from_volume_resume_name = "from-volume-resume-cmaes" # Create new Experiments from the previous Experiment info. ``` -------------------------------- ### GetSuggestionsRequest Source: https://github.com/kubeflow/katib/blob/master/pkg/apis/manager/v1beta1/gen-doc/index.html Request object for getting hyperparameter suggestions. ```APIDOC ## GetSuggestionsRequest ### Description Request object for getting hyperparameter suggestions. ### Fields - **experiment** ([Experiment](#api.v1.beta1.Experiment)) - The experiment configuration. - **trials** ([Trial](#api.v1.beta1.Trial)) - repeated - All completed trials owned by the experiment. - **current_request_number** (int32) - The number of Suggestions requested at one time. When you set 3 to current_request_number, you get three Suggestions at one time. - **total_request_number** (int32) - The number of Suggestions requested till now. ``` -------------------------------- ### Import Katib and Kubernetes Client Libraries Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/sdk/nas-with-darts.ipynb Import necessary classes from the Katib SDK and Kubernetes client library for defining a Katib Experiment. ```python from kubeflow.katib import KatibClient from kubernetes.client import V1ObjectMeta from kubeflow.katib import V1beta1Experiment from kubeflow.katib import V1beta1AlgorithmSpec from kubeflow.katib import V1beta1AlgorithmSetting from kubeflow.katib import V1beta1ObjectiveSpec from kubeflow.katib import V1beta1MetricsCollectorSpec from kubeflow.katib import V1beta1CollectorSpec from kubeflow.katib import V1beta1SourceSpec from kubeflow.katib import V1beta1FilterSpec from kubeflow.katib import V1beta1FeasibleSpace from kubeflow.katib import V1beta1ExperimentSpec from kubeflow.katib import V1beta1NasConfig from kubeflow.katib import V1beta1GraphConfig from kubeflow.katib import V1beta1Operation from kubeflow.katib import V1beta1ParameterSpec from kubeflow.katib import V1beta1TrialTemplate from kubeflow.katib import V1beta1TrialParameterSpec ``` -------------------------------- ### GET /is_experiment_succeeded Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Checks if a Katib Experiment has successfully completed. ```APIDOC ## GET /is_experiment_succeeded ### Description Check if Experiment has succeeded. ### Method GET ### Endpoint /is_experiment_succeeded ### Parameters #### Query Parameters - **name** (str) - Required - The name of the experiment. - **namespace** (str) - Optional - The namespace of the experiment. ### Response #### Success Response (200) - **bool** - True if the experiment has succeeded, False otherwise. ``` -------------------------------- ### Traverse and Set Up Hyperparameters Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2339-hpo-for-llm-fine-tuning/README.md This code snippet demonstrates how to traverse training parameters and LoRA configurations to identify tunable hyperparameters. It sets up these parameters for the Katib Experiment and Trials, preparing them to be passed as arguments to worker container specs. ```python # Traverse and set up hyperparameters input_params = {} experiment_params = [] trial_params = [] training_args = trainer_parameters.training_parameters for p_name, p_value in training_args.to_dict().items(): if not hasattr(training_args, p_name): logger.warning(f"Training parameter {p_name} is not supported by the current transformer.") continue if isinstance(p_value, models.V1beta1ParameterSpec): value = f"${{trialParameters.{p_name}}}" setattr(training_args, p_name, value) p_value.name = p_name experiment_params.append(p_value) trial_params.append(models.V1beta1TrialParameterSpec(name=p_name, reference=p_name)) elif p_value is not None: value = type(old_attr)(p_value) setattr(training_args, p_name, value) input_params['training_args'] = training_args # Note: Repeat similar logic for `lora_config` ``` -------------------------------- ### GET /get_experiment_status Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves the current status of a Katib Experiment. ```APIDOC ## GET /get_experiment_status ### Description Get the Experiment current status. ### Method GET ### Endpoint /get_experiment_status ### Parameters #### Query Parameters - **name** (str) - Required - The name of the experiment. - **namespace** (str) - Optional - The namespace of the experiment. ### Response #### Success Response (200) - **dict** - The status of the experiment. ``` -------------------------------- ### GET /list_experiments Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Lists all Katib Experiments in the specified namespace. ```APIDOC ## GET /list_experiments ### Description List all Katib Experiments. If the namespace is `None`, it takes "default" namespace. ### Method GET ### Endpoint /list_experiments ### Parameters #### Query Parameters - **namespace** (str) - Optional - The namespace to list experiments from. Defaults to "default" if not provided. ### Response #### Success Response (200) - **list** - A list of Experiment objects. ``` -------------------------------- ### Get Optimal Hyperparameters Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves the current optimal hyperparameters from an experiment. ```APIDOC ## GET /experiments/{name}/optimal-hyperparameters ### Description Get the current optimal Trial from the Experiment. If the namespace is `None`, it takes namespace from the Experiment or "default". Return current optimal Trial for the Experiment. ### Method GET ### Endpoint /experiments/{name}/optimal-hyperparameters ### Parameters #### Path Parameters - **name** (str) - Required - Experiment name. #### Query Parameters - **namespace** (str) - Optional - Experiment namespace. ### Response #### Success Response (200) - **optimal_hyperparameters** (dict) - A dictionary containing the optimal hyperparameters. #### Response Example ```json { "optimal_hyperparameters": { "lr": 0.001, "batch_size": 32 } } ``` ``` -------------------------------- ### Get Experiment Status Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves the current status of a specific experiment. ```APIDOC ## GET /experiments/{name}/status ### Description Get the Experiment current status. If the namespace is `None`, it takes "default" namespace. Return the current Experiment status. ### Method GET ### Endpoint /experiments/{name}/status ### Parameters #### Path Parameters - **name** (str) - Required - Experiment name. #### Query Parameters - **namespace** (str) - Optional - Experiment namespace. ### Response #### Success Response (200) - **status** (str) - The current status of the experiment. #### Response Example ```json { "status": "Running" } ``` ``` -------------------------------- ### Create Katib Experiment with 12 Trials and 2 GPUs Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2340-push-based-metrics-collector/README.md Use the Katib Python SDK to create an experiment with a specified number of trials and resources per trial. Ensure the `objective` and `parameters` are defined elsewhere. ```python katib_client = katib.KatibClient(namespace="kubeflow") name = "tune-experiment" katib_client.tune( name=name, objective=objective, parameters=parameters, objective_metric_name="result", max_trial_count=12, resources_per_trial={"gpu": "2"}, metrics_collector_config={"kind": "Push"}, ) ``` -------------------------------- ### Run Kubeflow Conformance Tests (Makefile) Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2044-conformance-program/README.md Use these Makefile targets to set up, run, and report on conformance tests for Katib and Training Operator. Ensure you have the necessary Kubernetes resources and profiles configured. ```makefile run: setup run-katib run-training-operator ``` ```makefile setup: kubectl apply -f ./setup.yaml mkdir -p /tmp/kf-conformance ``` ```makefile run-katib: kubectl apply -f ./katib-conformance.yaml ``` ```makefile run-training-operator: kubectl apply -f ./training-operator-conformance.yaml ``` ```makefile report: ./report-conformance.sh ``` ```makefile cleanup: kubectl delete -f ./setup.yaml kubectl delete -f ./katib-conformance.yaml kubectl delete -f ./training-operator-conformance.yaml rm -rf /tmp/kf-conformance ``` -------------------------------- ### Get Experiments Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves a list of experiments within a specified namespace. ```APIDOC ## GET /experiments ### Description Retrieves a list of experiments. If the namespace is `None`, it takes the 'default' namespace. ### Method GET ### Endpoint /experiments ### Parameters #### Query Parameters - **namespace** (str) - Optional - Experiment namespace. ### Response #### Success Response (200) - **experiments** (list[V1beta1Experiment]) - A list of V1beta1Experiment objects. #### Response Example ```json { "experiments": [ { "name": "example-experiment", "namespace": "default" } ] } ``` ``` -------------------------------- ### Deploy Katib Manifests Source: https://github.com/kubeflow/katib/blob/master/CONTRIBUTING.md Command to deploy Katib v1beta1 manifests into a Kubernetes cluster. ```bash make deploy ``` -------------------------------- ### GET /list_trials Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Lists all Trials associated with a specific Katib Experiment. ```APIDOC ## GET /list_trials ### Description List all Experiment's Trials. ### Method GET ### Endpoint /list_trials ### Parameters #### Query Parameters - **experiment_name** (str) - Required - The name of the experiment. - **namespace** (str) - Optional - The namespace of the experiment. ### Response #### Success Response (200) - **list** - A list of Trial objects. ``` -------------------------------- ### PyTorch MNIST Trial Image Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/README.md Example of a Trial image for PyTorch MNIST. ```yaml apiVersion: "kubeflow.org/v1beta1" kind: Experiment metadata: name: pytorch-mnist-example spec: parallelTrialCount: 6 maxTrialCount: 12 objective: type: minimize metricName: accuracy goal: 0.95 algorithm: name: random trialTemplate: primaryContainerName: training-container trialParameters: - name: learningRate infeasibleValue: ">1.0" - name: batchSize infeasibleValue: "<1" trialSpec: retries: 3 trialTimeout: "600s" podTemplate: spec: containers: - name: training-container image: docker.io/kubeflow/pytorch-operator:v0.7.0 command: - "python" - "/app/run.py" - "--learningRate" - "${trialParameters.learningRate}" - "--batchSize" - "${trialParameters.batchSize}" ``` -------------------------------- ### Configure Katib Controller Trial Resources Source: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/tekton/README.md Update Katib's controller configuration to include Tekton PipelineRun as a supported trial resource. ```yaml trialResources: - PipelineRun.v1beta1.tekton.dev ``` -------------------------------- ### Initiate Hyperparameter Tuning Experiment Source: https://github.com/kubeflow/katib/blob/master/docs/proposals/2339-hpo-for-llm-fine-tuning/README.md Use the `tune` method of `KatibClient` to start a hyperparameter tuning experiment. Configure model, dataset, trainer, and objective parameters. Supports various provider options like HuggingFace and S3, and custom objective functions. ```python import kubeflow.katib as katib from kubeflow.katib import KatibClient class KatibClient(object): def tune( self, name: str, namespace: Optional[str] = None, model_provider_parameters: Optional[HuggingFaceModelParams] = None, dataset_provider_parameters: Optional[Union[HuggingFaceDatasetParams, S3DatasetParams]] = None, trainer_parameters: Union[HuggingFaceTrainerParams, Dict[str, Any]] = None, storage_config: Dict[str, Optional[Union[str, List[str]]]] = { "size": constants.PVC_DEFAULT_SIZE, "storage_class": None, "access_modes": constants.PVC_DEFAULT_ACCESS_MODES, }, objective: Optional[Callable] = None, base_image: Optional[str] = None, algorithm_name: str = "random", algorithm_settings: Union[dict, List[models.V1beta1AlgorithmSetting], None] = None, objective_metric_name: str = "eval_accuracy", additional_metric_names: List[str] = [], objective_type: str = "maximize", objective_goal: float = None, max_trial_count: int = None, parallel_trial_count: int = None, max_failed_trial_count: int = None, resources_per_trial = Union[dict, client.V1ResourceRequirements, types.TrainerResources, None] = None, retain_trials: bool = False, env_per_trial: Optional[Union[Dict[str, str], List[Union[client.V1EnvVar, client.V1EnvFromSource]]]] = None, packages_to_install: List[str] = None, pip_index_url: str = "https://pypi.org/simple", ): """ Initiates a hyperparameter tuning experiment in Katib. Model, dataset and parameters can be configured using one of the following options: - Using the Storage Initializer: Specify `model_provider_parameters`, `dataset_provider_parameters`, and `trainer_parameters`. This option downloads models and datasets from external platforms like HuggingFace and S3, and utilizes `Trainer.train()` in HuggingFace to train the model. - Defining a custom objective function: Specify the `objective` parameter to define your own objective function, and use the `base_image` parameter to execute the objective function. Parameters: - name: Name for the experiment. - namespace: Namespace for the experiment. Defaults to the namespace of the 'KatibClient' object. - model_provider_parameters: Parameters for providing the model. Compatible with model providers like HuggingFace. - dataset_provider_parameters: Parameters for providing the dataset. Compatible with dataset providers like HuggingFace or S3. - trainer_parameters: Parameters for configuring the training process, including settings for hyperparameters search space. - storage_config: Configuration for Storage Initializer PVC to download pre-trained model and dataset. - objective: Objective function that Katib uses to train the model. - base_image: Image to use when executing the objective function. - algorithm_name: Tuning algorithm name (e.g., 'random', 'bayesian'). - algorithm_settings: Settings for the tuning algorithm. - objective_metric_name: Primary metric to optimize. - additional_metric_names: List of additional metrics to collect. - objective_type: Optimization direction for the objective metric, "minimize" or "maximize". - objective_goal: Desired value of the objective metric. - max_trial_count: Maximum number of trials to run. - parallel_trial_count: Number of trials to run in parallel. - max_failed_trial_count: Maximum number of allowed failed trials. - resources_per_trial: Resources assigned to per trial, which can be specified using one of the following options: - Non-distributed Training: Specify a kubernetes.client.V1ResourceRequirements object or a dicitionary that includes one or more of the following keys: `cpu`, `memory`, or `gpu` (other keys will be ignored). - Distributed Training in Pytorch: Specify a types.TrainerResources, which includes the following parameters: - num_workers: Number of PyTorchJob workers. - num_procs_per_worker: Number of processes per PyTorchJob worker. - resources_per_worker: Resources assigned to per PyTorchJob worker container, specified as either a kubernetes.client.V1ResourceRequirements object or a dicitionary that includes one or more of the following keys: `cpu`, `memory`, or `gpu` (other keys will be ignored). - retain_trials: Whether to retain trial resources after completion. - env_per_trial: Environment variables for worker containers. - packages_to_install: Additional Python packages to install. - pip_index_url: URL of the PyPI index for installing packages. """ pass # Implementation logic for initiating the experiment ``` -------------------------------- ### Get Success Trial Details Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves details of succeeded trials for an experiment. ```APIDOC ## GET /experiments/{name}/trials/succeeded ### Description Get the Trial details that have succeeded for an Experiment. If the namespace is `None`, it takes namespace from the Experiment or "default". Return Trial names with the hyperparameters and metrics. ### Method GET ### Endpoint /experiments/{name}/trials/succeeded ### Parameters #### Path Parameters - **name** (str) - Required - Experiment name. #### Query Parameters - **namespace** (str) - Optional - Experiment namespace. ### Response #### Success Response (200) - **trial_details** (list[dict]) - A list of dictionaries, each containing details of a succeeded trial. #### Response Example ```json { "trial_details": [ { "trial_name": "trial-1", "hyperparameters": {"lr": 0.01}, "metrics": {"accuracy": 0.95} } ] } ``` ``` -------------------------------- ### Build Katib Images Source: https://github.com/kubeflow/katib/blob/master/docs/release/README.md Build all Katib images using 'make build'. Ensure Docker Desktop has containerd image store enabled for multi-arch images. Replace 'private-registry' with your registry and 'latest' with the desired tag. ```bash make build REGISTRY=private-registry TAG=latest ``` -------------------------------- ### GET /get_optimal_hyperparameters Source: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/docs/KatibClient.md Retrieves the current optimal hyperparameters found for a Katib Experiment. ```APIDOC ## GET /get_optimal_hyperparameters ### Description Get the current optimal Trial from the Experiment. ### Method GET ### Endpoint /get_optimal_hyperparameters ### Parameters #### Query Parameters - **experiment_name** (str) - Required - The name of the experiment. - **namespace** (str) - Optional - The namespace of the experiment. ### Response #### Success Response (200) - **dict** - The optimal hyperparameters. ```