### Install Protocol Buffer Compiler Source: https://github.com/kubeflow/pipelines/blob/master/frontend/CONTRIBUTING.md Example command to install the Protocol Buffer compiler (protoc) on a Debian-based system. Ensure the correct version is installed. ```bash # Example: apt install -y protobuf-compiler=3.15.8 ``` -------------------------------- ### Install bmizerany/assert Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Install the bmizerany/assert library using the go get command. This is a prerequisite for using the assertion functions in Go tests. ```bash $ go get github.com/bmizerany/assert ``` -------------------------------- ### Install kfp-server-api via Setuptools Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/README.md Install the Python package using Setuptools. This method is an alternative to pip installation. ```sh python setup.py install --user (or `sudo python setup.py install` to install the package for all users) ``` -------------------------------- ### Local Development Environment Setup Source: https://github.com/kubeflow/pipelines/blob/master/CLAUDE.md Sets up a Python virtual environment and installs KFP packages in editable mode for local development. Ensure you have Make, pip, setuptools, and wheel installed. ```bash python3 -m venv .venv source .venv/bin/activate python -m pip install -U pip setuptools wheel make -C api python-dev make -C kubernetes_platform python-dev pip install -e api/v2alpha1/python --config-settings editable_mode=strict pip install -e sdk/python --config-settings editable_mode=strict pip install -e kubernetes_platform/python --config-settings editable_mode=strict ``` -------------------------------- ### Create Experiment Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v2beta1/python_http_client/docs/ExperimentServiceApi.md Creates a new experiment. This example shows the initial setup for API client configuration with bearer token authentication. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. ``` -------------------------------- ### First-Time Setup for UI Smoke Test Tool Source: https://github.com/kubeflow/pipelines/blob/master/frontend/scripts/ui-smoke-test/README.md Installs tool dependencies, Playwright, and captures screenshots for visual regression testing. Ensure Docker is running for the compare workflow. ```bash cd frontend/scripts/ui-smoke-test npm install npx playwright install chromium open -a Docker node smoke-test-runner.js --compare master ``` -------------------------------- ### Install Argo CLI Source: https://github.com/kubeflow/pipelines/wiki/Tests Installs the Argo command-line interface. This is a one-time setup step for GKE testing. ```bash argo install ``` -------------------------------- ### Install and Build Frontend Dependencies Source: https://github.com/kubeflow/pipelines/blob/master/frontend/README.md Navigate to the frontend directory, install NPM dependencies, and build the project for local development. ```bash cd ${WORKING_DIRECTORY}/frontend npm ci npm run build ``` -------------------------------- ### Example DEV_IMAGE_PREFIX Configuration Source: https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/README.md An example of how to configure the DEV_IMAGE_PREFIX in the .env file for pushing built development images. ```makefile export DEV_IMAGE_PREFIX=ghcr.io/kubeflow-test/kfp-/kfp- ``` -------------------------------- ### Go Prompt Example Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Demonstrates how to use the prompt.Choose function to ask the user a question and select from a list of predefined options. The selected option's index is returned. ```go package main import "github.com/segmentio/go-prompt" var langs = []string{ "c", "c++", "lua", "go", "js", "ruby", "python", } func main() { i := prompt.Choose("What's your favorite language?", langs) println("picked: " + langs[i]) } ``` -------------------------------- ### Install Ginkgo CLI with Go Install Source: https://github.com/kubeflow/pipelines/blob/master/AGENTS.md Installs the Ginkgo CLI tool directly into a project-local bin directory using 'go install'. Ensures the binary is added to the system's PATH. ```bash GOBIN=$PWD/bin go install github.com/onsi/ginkgo/v2/ginkgo@latest export PATH="$PWD/bin:$PATH" ``` -------------------------------- ### Install Kubeflow Pipelines SDK Source: https://github.com/kubeflow/pipelines/blob/master/samples/tutorials/Data passing in python components.ipynb Install the Kubeflow Pipelines SDK. Use the --user argument if you encounter permission errors during installation. ```python # Install Kubeflow Pipelines SDK. Add the --user argument if you get permission errors. !PIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install 'kfp>=1.4.0' --quiet --user ``` -------------------------------- ### Install Kubernetes Platform Package Source: https://github.com/kubeflow/pipelines/blob/master/sdk/CONTRIBUTING.md Install the kfp kubernetes_platform package. Use 'make python-dev' for development or omit '-dev' for a regular installation. ```bash pushd kubernetes_platform make python-dev # omit -dev for regular install popd ``` -------------------------------- ### List Objects Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Initializes an OSS client, gets a bucket handle, and lists all objects within the bucket. ```go client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret") if err != nil { // HandleError(err) } bucket, err := client.Bucket("my-bucket") if err != nil { // HandleError(err) } lsRes, err := bucket.ListObjects() if err != nil { // HandleError(err) } for _, object := range lsRes.Objects { fmt.Println("Objects:", object.Key) } ``` -------------------------------- ### Install Kubeflow Pipelines SDK Source: https://github.com/kubeflow/pipelines/blob/master/samples/core/kfp_env_validation/kfp_env_validation.ipynb Installs the KFP SDK using pip. This command should be run once in the environment. ```python # Install Pipeline SDK - This only needs to be run once in the environment. !python3 -m pip install 'kfp>=0.1.31' --user --quiet ``` -------------------------------- ### Install and Use Node.js Version Source: https://github.com/kubeflow/pipelines/blob/master/CLAUDE.md Installs and switches to the Node.js version specified in frontend/.nvmrc using fnm or nvm. ```bash # With fnm (faster) fnm install "$(cat frontend/.nvmrc)" && fnm use "$(cat frontend/.nvmrc)" # With nvm nvm install "$(cat frontend/.nvmrc)" && nvm use "$(cat frontend/.nvmrc)" ``` -------------------------------- ### Install Pre-commit Hooks Source: https://github.com/kubeflow/pipelines/blob/master/sdk/CONTRIBUTING.md Install the pre-commit framework to automatically check code quality before commits. This is recommended for all contributors. ```sh pre-commit install ``` -------------------------------- ### Get Object To File Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Initializes an OSS client, gets a bucket handle, and downloads an object from the bucket to a local file. ```go client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret") if err != nil { // HandleError(err) } bucket, err := client.Bucket("my-bucket") if err != nil { // HandleError(err) } err = bucket.GetObjectToFile("my-object", "LocalFile") if err != nil { // HandleError(err) } ``` -------------------------------- ### Put Object From File Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Initializes an OSS client, gets a bucket handle, and uploads a local file to a specified object key within the bucket. ```go client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret") if err != nil { // HandleError(err) } bucket, err := client.Bucket("my-bucket") if err != nil { // HandleError(err) } err = bucket.PutObjectFromFile("my-object", "LocalFile") if err != nil { // HandleError(err) } ``` -------------------------------- ### Delete Object Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Initializes an OSS client, gets a bucket handle, and deletes a specified object from the bucket. ```go client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret") if err != nil { // HandleError(err) } bucket, err := client.Bucket("my-bucket") if err != nil { // HandleError(err) } err = bucket.DeleteObject("my-object") if err != nil { // HandleError(err) } ``` -------------------------------- ### Start Proxy for Client UI Development Source: https://github.com/kubeflow/pipelines/blob/master/frontend/CONTRIBUTING.md Starts the development server with proxying enabled for client UI development against a single-user KFP instance. ```bash npm run start:proxy ``` -------------------------------- ### Install NPM Dependencies Source: https://github.com/kubeflow/pipelines/blob/master/frontend/CONTRIBUTING.md Installs exact NPM dependency versions based on package-lock.json. Recommended for initial setup, after package.json changes, and in CI/CD. ```bash npm ci ``` -------------------------------- ### Upload Pipeline Example Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/docs/PipelineUploadServiceApi.md Demonstrates how to upload a pipeline using the PipelineUploadServiceApi. It shows configuration for API key authentication and how to pass pipeline details. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure API key authorization: Bearer configuration = kfp_server_api.Configuration( host = "http://localhost", api_key = { 'authorization': 'YOUR_API_KEY' } ) # Uncomment below to setup prefix (e.g. Bearer) for API key, if needed # configuration.api_key_prefix['authorization'] = 'Bearer' # Enter a context with an instance of the API client with kfp_server_api.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = kfp_server_api.PipelineUploadServiceApi(api_client) uploadfile = '/path/to/file' # file | The pipeline to upload. Maximum size of 32MB is supported. name = 'name_example' # str | (optional) description = 'description_example' # str | (optional) namespace = 'namespace_example' # str | (optional) try: api_response = api_instance.upload_pipeline(uploadfile, name=name, description=description, namespace=namespace) pprint(api_response) except ApiException as e: print("Exception when calling PipelineUploadServiceApi->upload_pipeline: %s\n" % e) ``` -------------------------------- ### Install Google Cloud Pipeline Components Source: https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/README.md Install the latest release of Google Cloud Pipeline Components from PyPI. Use this command to get the most recent stable version. ```shell pip install -U google-cloud-pipeline-components ``` -------------------------------- ### Start Local Development with Proxy and Server Source: https://github.com/kubeflow/pipelines/blob/master/AGENTS.md Starts the local development server, proxying to the cluster and enabling hot reload for full integration testing. ```bash npm run start:proxy-and-server ``` -------------------------------- ### Set up a Virtual Environment Source: https://github.com/kubeflow/pipelines/blob/master/sdk/CONTRIBUTING.md Create and activate a Python virtual environment to isolate project dependencies. Replace 'env' with your preferred environment name. ```bash # optional, replace with your tool of choice python -m venv env && source ./env/bin/activate ``` -------------------------------- ### Setup and Check Python SDK Imports Source: https://github.com/kubeflow/pipelines/blob/master/CLAUDE.md Installs development dependencies for the Python SDK and checks for import order and unused imports using pycln and isort. Ensure you have the correct requirements installed before running. ```bash pip install -r sdk/python/requirements-dev.txt pycln --check sdk/python isort --check --profile google sdk/python ``` -------------------------------- ### Get Experiment Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/docs/ExperimentServiceApi.md Retrieve a specific experiment by its ID using this code example. Ensure the API client is properly configured. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost ``` -------------------------------- ### Example Usage of prsha Function Source: https://github.com/kubeflow/pipelines/blob/master/RELEASE.md Demonstrates how to use the `prsha` function to get the commit SHA for a specific pull request URL. ```bash > prsha https://github.com/kubeflow/pipelines/pull/13344 67a73cf809f7c76fee33c749f302f2bceedd8fab ``` -------------------------------- ### List Buckets Source: https://github.com/kubeflow/pipelines/blob/master/third_party/minio/license.txt Initializes an OSS client and lists all buckets associated with the provided credentials. ```go client, err := oss.New("Endpoint", "AccessKeyId", "AccessKeySecret") if err != nil { // HandleError(err) } lsRes, err := client.ListBuckets() if err != nil { // HandleError(err) } for _, bucket := range lsRes.Buckets { fmt.Println("Buckets:", bucket.Name) } ``` -------------------------------- ### List Pipeline Versions Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/docs/PipelineServiceApi.md Lists all available pipeline versions for a given pipeline. This example shows the setup for API key authentication. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. ``` -------------------------------- ### Install Protobuf Compiler and gRPC Go Plugins Source: https://github.com/kubeflow/pipelines/blob/master/third_party/ml-metadata/README.md Install specific versions of the protobuf-compiler, protoc-gen-go, and protoc-gen-go-grpc required for building the Go gRPC client. ```bash apt install -y protobuf-compiler=3.15.8 go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1 ``` -------------------------------- ### Kubeflow Pipelines Benchmark Setup Source: https://github.com/kubeflow/pipelines/blob/master/tools/benchmarks/run_service_api.ipynb Sets up the necessary imports and configurations for running Kubeflow Pipelines benchmarks. This includes defining the KFP host, the pipeline file URL, and the desired number of runs. ```python # This benchmark measures the performance of run related operations in Kubeflow pipelines, including run durations and latencies of creating/getting/deleting runs. import random import kfp import kfp_server_api import os import string import time from google.cloud import storage from kfp.components import create_component_from_func from datetime import datetime, timezone, timedelta import numpy as np import pandas as pd import seaborn as sns import matplotlib.pyplot as plt from scipy import stats # CHANGE necessary paramters here # host is your KFP endpoint host = 'http://127.0.0.1:3001' # Use the pipeline you prefer pipeline_file_url = 'https://storage.googleapis.com/jingzhangjz-project-pipelines/benchmarks/taxi.yaml' # number of runs you want to create num_runs = 5 ``` -------------------------------- ### Install Sample Test Dependencies Source: https://github.com/kubeflow/pipelines/blob/master/backend/src/v2/README.md Installs Python dependencies required for running sample tests. Navigate to the 'test' directory before running. ```bash cd test pip install -r requirements.txt ``` -------------------------------- ### Multi-user Setup Environment Variable Source: https://github.com/kubeflow/pipelines/blob/master/frontend/CONTRIBUTING.md Set the VITE_NAMESPACE environment variable for multi-user Kubeflow Pipelines installations. This is typically done before building the frontend. ```bash export VITE_NAMESPACE=kubeflow-user-example-com npm run build ``` -------------------------------- ### Install Documentation Generation Tools Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/README.md Install the necessary npm packages for generating API reference documentation. This includes 'bootprint' and 'bootprint-openapi' for processing OpenAPI specifications and 'html-inline' for creating self-contained HTML files. ```bash npm install -g bootprint npm install -g bootprint-openapi npm -g install html-inline ``` -------------------------------- ### Retry a Run Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v2beta1/python_http_client/docs/RunServiceApi.md Re-initiates a failed or terminated run using its ID. This example demonstrates the basic setup for authentication and API client configuration. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure API key authorization: Bearer configuration = kfp_server_api.Configuration( host = "http://localhost", api_key = { 'authorization': 'YOUR_API_KEY' } ) # Uncomment below to setup prefix (e.g. Bearer) for API key, if needed # configuration.api_key_prefix['authorization'] = 'Bearer' # Example for retry_run (assuming api_instance is already created and configured) # run_id = 'run_id_example' # str | The ID of the run to retry. # experiment_id = 'experiment_id_example' # str | The ID of the experiment that the run belongs to. (optional) # try: # api_response = api_instance.run_service_retry_run(run_id, experiment_id=experiment_id) # pprint(api_response) # except ApiException as e: # print("Exception when calling RunServiceApi->run_service_retry_run: %s\n" % e) ``` -------------------------------- ### Start Interactive Shell in a Docker Image Source: https://github.com/kubeflow/pipelines/blob/master/third_party/README.md Run an interactive shell within a Docker container created from a specified image. This requires the image to have a shell installed. ```bash docker run -it --entrypoint sh $IMAGE_NAME ``` -------------------------------- ### Get Pipeline Details using Python HTTP Client Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v2beta1/python_http_client/docs/PipelineServiceApi.md Retrieves details for a specific pipeline using its ID. This example uses API key authentication. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure API key authorization: Bearer configuration = kfp_server_api.Configuration( host = "http://localhost", api_key = { 'authorization': 'YOUR_API_KEY' } ) # Uncomment below to setup prefix (e.g. Bearer) for API key, if needed # configuration.api_key_prefix['authorization'] = 'Bearer' # Enter a context with an instance of the API client with kfp_server_api.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = kfp_server_api.PipelineServiceApi(api_client) pipeline_id = 'pipeline_id_example' # str | Required input. The ID of the pipeline to be retrieved. try: # Finds a specific pipeline by ID. api_response = api_instance.pipeline_service_get_pipeline(pipeline_id) pprint(api_response) except ApiException as e: print("Exception when calling PipelineServiceApi->pipeline_service_get_pipeline: %s\n" % e) ``` -------------------------------- ### Start Frontend Development Server Source: https://github.com/kubeflow/pipelines/blob/master/CLAUDE.md Starts the frontend development server using npm. This command is used for local frontend development and testing. ```bash cd frontend && npm start ``` -------------------------------- ### Get Pipeline Template by ID Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/docs/PipelineServiceApi.md Retrieves the YAML template for a pipeline, including its description, parameters, and metadata. This example demonstrates API key authentication and error handling. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure API key authorization: Bearer configuration = kfp_server_api.Configuration( host = "http://localhost", api_key = { 'authorization': 'YOUR_API_KEY' } ) # Uncomment below to setup prefix (e.g. Bearer) for API key, if needed # configuration.api_key_prefix['authorization'] = 'Bearer' # Enter a context with an instance of the API client with kfp_server_api.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = kfp_server_api.PipelineServiceApi(api_client) id = 'id_example' # str | The ID of the pipeline whose template is to be retrieved. try: # Returns a single YAML template that contains the description, parameters, and metadata associated with the pipeline provided. api_response = api_instance.pipeline_service_get_template(id) pprint(api_response) except ApiException as e: print("Exception when calling PipelineServiceApi->pipeline_service_get_template: %s\n" % e) ``` ``` -------------------------------- ### Install Development Requirements Source: https://github.com/kubeflow/pipelines/blob/master/sdk/CONTRIBUTING.md Install essential tools like pip, wheel, and setuptools. Ensure your package management tools are up-to-date before proceeding. ```bash python -m pip install -U pip wheel setuptools ``` -------------------------------- ### Build All Backend Images Source: https://github.com/kubeflow/pipelines/blob/master/third_party/argo/UPGRADE.md Run 'make image_all' from the 'backend' directory to verify that backend images still build after an upgrade. ```bash make image_all ``` -------------------------------- ### Accessing UI Variables in Kubeflow Pipelines Source: https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/visualization/README.md Shows how to retrieve custom variables passed from the Kubeflow Pipelines UI to a visualization script. Includes examples for getting values with and without default fallbacks, and checking for their existence. ```python # Get a value for a specified key key = variables.get("key") # Get a value for a specified key with a default key = variables.get("key", "default_value") # Check if a value for a specified key exists if variables.get("key", "default_value") is "default_value": # Value for a specified key does not exist pass else: # Value for a specified key does exist pass ``` -------------------------------- ### Report Run Metrics V1 Example Source: https://github.com/kubeflow/pipelines/blob/master/backend/api/v1beta1/python_http_client/docs/RunServiceApi.md Demonstrates how to report metrics for a specific run using the RunServiceApi. Includes setup for API key authentication and error handling. The API accepts partial failures and ignores duplicate reporting. ```python from __future__ import print_function import time import kfp_server_api from kfp_server_api.rest import ApiException from pprint import pprint # Defining the host is optional and defaults to http://localhost # See configuration.py for a list of all supported configuration parameters. configuration = kfp_server_api.Configuration( host = "http://localhost" ) # The client must configure the authentication and authorization parameters # in accordance with the API server security policy. # Examples for each auth method are provided below, use the example that # satisfies your auth use case. # Configure API key authorization: Bearer configuration = kfp_server_api.Configuration( host = "http://localhost", api_key = { 'authorization': 'YOUR_API_KEY' } ) # Uncomment below to setup prefix (e.g. Bearer) for API key, if needed # configuration.api_key_prefix['authorization'] = 'Bearer' # Enter a context with an instance of the API client with kfp_server_api.ApiClient(configuration) as api_client: # Create an instance of the API class api_instance = kfp_server_api.RunServiceApi(api_client) run_id = 'run_id_example' # str | Required. The parent run ID of the metric. body = kfp_server_api.RunServiceReportRunMetricsV1Body() # RunServiceReportRunMetricsV1Body | try: # ReportRunMetrics reports metrics of a run. Each metric is reported in its own transaction, so this API accepts partial failures. Metric can be uniquely identified by (run_id, node_id, name). Duplicate reporting will be ignored by the API. First reporting wins. api_response = api_instance.run_service_report_run_metrics_v1(run_id, body) pprint(api_response) except ApiException as e: print("Exception when calling RunServiceApi->run_service_report_run_metrics_v1: %s\n" % e) ```