Valohai

https://docs.valohai.com
Admin
Valohai is a modular MLOps platform that orchestrates ML workflows through configuration, enabling...
Tokens:348,706
Snippets:4,573
Trust Score:9.9
Update:5 months ago
Show doc for...
Tokens:
Raw
### Agent Query Example

Source: https://docs.valohai.com/valohai.yaml-overview/lint

This example demonstrates how to query the documentation dynamically using the `ask` query parameter in an HTTP GET request.

```http
GET https://docs.valohai.com/valohai.yaml-overview/lint.md?ask=<question>
```

--------------------------------

### Example GET Request for Dynamic Documentation Querying

Source: https://docs.valohai.com/reusable-step-libraries/ecosystem-libraries/bigquery

This example demonstrates how to dynamically query the documentation by making an HTTP GET request to the current page URL with an 'ask' query parameter. This is useful for retrieving specific information or clarifications not explicitly present on the page.

```http
GET https://docs.valohai.com/reusable-step-libraries/ecosystem-libraries/bigquery.md?ask=<question>
```

--------------------------------

### Query Documentation Example

Source: https://docs.valohai.com/notebook-executions/work-with-data

Illustrates how to query the documentation dynamically by making an HTTP GET request to the page URL with an `ask` query parameter.

```http
GET https://docs.valohai.com/notebook-executions/work-with-data.md?ask=<question>
```

--------------------------------

### Complete Workflow Example: Mount, Process, Save

Source: https://docs.valohai.com/data/data-nfs/azure-files

An example valohai.yaml configuration for processing data from Azure Files, including installing dependencies and setting environment variables. The Python script processes videos from the mounted file share.

```yaml
- step:
    name: extract-video-features
    image: python:3.9
    command:
      - pip install opencv-python numpy pandas
      - python extract_features.py
    mounts:
      - destination: /mnt/raw-videos
        source: ${FILE_SHARE}
        type: smb
        options:
          username: ${USERNAME}
          password: ${PASSWORD}
        readonly: true
    environment-variables:
      - name: SAMPLE_RATE
        default: "1"  # Process every Nth frame
```

```python
import os
import cv2
import numpy as np
import pandas as pd
import json

# Configuration
FILESHARE_PATH = '/mnt/raw-videos/'
OUTPUT_PATH = '/valohai/outputs/'
SAMPLE_RATE = int(os.getenv('SAMPLE_RATE', '1'))

# 1. Scan file share for videos (NOT versioned)
print(f"Scanning file share mount: {FILESHARE_PATH}")
video_files = [f for f in os.listdir(FILESHARE_PATH) if f.endswith(('.mp4', '.avi'))]
print(f"Found {len(video_files)} videos on file share")

```

--------------------------------

### Making an HTTP GET Request

Source: https://docs.valohai.com/data/data-databases/snowflake

Example of how to make an HTTP GET request to query documentation dynamically. Replace `<question>` with your specific query.

```http
GET https://docs.valohai.com/data/data-databases/snowflake.md?ask=<question>
```

--------------------------------

### Query Documentation API Example

Source: https://docs.valohai.com/automation-overview/triggers/notification-triggers/notification-payload-formats

Demonstrates how to query the documentation dynamically using an HTTP GET request with the 'ask' query parameter. This is useful for retrieving information not explicitly present on the page.

```http
GET https://docs.valohai.com/automation-overview/triggers/notification-triggers/notification-payload-formats.md?ask=<question>
```

--------------------------------

### Basic On-Premises NFS Server Setup (Ubuntu)

Source: https://docs.valohai.com/installation-and-setup/advanced-topics/shared-cache

Installs and configures an NFS server on Ubuntu. Ensure the export path and network range are appropriate for your environment.

```shell
sudo apt-get update
sudo apt-get install nfs-kernel-server

# Create shared directory
sudo mkdir -p /mnt/valohai-cache
sudo chown nobody:nogroup /mnt/valohai-cache
sudo chmod 777 /mnt/valohai-cache

# Configure exports
sudo nano /etc/exports
# Add line:
/mnt/valohai-cache 10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash)

# Apply configuration
sudo exportfs -ra
sudo systemctl restart nfs-kernel-server
```

--------------------------------

### Dynamic Documentation Query Mechanism

Source: https://docs.valohai.com/reusable-step-libraries/ecosystem-libraries/snowflake

This example demonstrates how to query the documentation dynamically using an HTTP GET request. Append the `ask` query parameter with your question to the documentation URL to retrieve specific information or clarifications.

```http
GET https://docs.valohai.com/reusable-step-libraries/ecosystem-libraries/snowflake.md?ask=<question>
```

--------------------------------

### Valohai Watcher Setup for YOLO

Source: https://docs.valohai.com/experiment-tracking/collect-metrics/yolo

Configure your `valohai.yaml` to install necessary dependencies and start the `valohai_watch.py` script in the background before your training process begins.

```yaml
command:
  - pip install watchdog valohai-utils
  - nohup python ./scripts/valohai_watch.py &  # Start watcher first
  - python train.py  # Then start training
```

--------------------------------

### Default Dockerfile Example

Source: https://docs.valohai.com/reusable-step-libraries/build-your-own-library/docker-image-builder

This is a default Dockerfile used if no specific Dockerfile is provided. It sets up a Python 3.9 environment with `valohai-utils` and `tensorflow` installed.

```dockerfile
FROM python:3.9
RUN pip install valohai-utils tensorflow==2.6.0
```

--------------------------------

### Dynamic Documentation Query Example

Source: https://docs.valohai.com/automation-overview/triggers/notification-triggers/dataset-version-trigger

Perform an HTTP GET request to query documentation dynamically. Use the `ask` query parameter with a specific, self-contained question in natural language.

```http
GET https://docs.valohai.com/automation-overview/triggers/notification-triggers/dataset-version-trigger.md?ask=<question>
```

--------------------------------

### Query Documentation via API

Source: https://docs.valohai.com/installation-and-setup/kubernetes/oracle

This example demonstrates how to query the documentation dynamically by making an HTTP GET request to the page URL with an 'ask' query parameter. This is useful for retrieving specific information not explicitly present on the page.

```http
GET https://docs.valohai.com/installation-and-setup/kubernetes/oracle.md?ask=<question>
```

--------------------------------

### Enable Peon Services to Start on Boot

Source: https://docs.valohai.com/installation-and-setup/on-premises/linux-workers

Enable the Peon, Peon-clean, and Peon-warden services to automatically start when the system boots up.

```shell
systemctl enable peon
systemctl enable peon-clean
systemctl enable peon-warden
```

--------------------------------

### Dynamic Documentation Query Example

Source: https://docs.valohai.com/installation-and-setup/aws/kubernetes-inference

Perform an HTTP GET request to the current page URL with the 'ask' query parameter to dynamically query the documentation. The question should be specific and in natural language.

```http
GET https://docs.valohai.com/installation-and-setup/aws/kubernetes-inference.md?ask=<question>
```

--------------------------------

### Correct Indentation Example

Source: https://docs.valohai.com/valohai.yaml-overview/lint

This example demonstrates the correct indentation for a step in a valohai.yaml file.

```yaml
# ✅ Correct indentation
- step:
    name: train
    image: python:3.9
```

--------------------------------

### Install Langfuse and Valohai LLM

Source: https://docs.valohai.com/genai/llm-evaluations-local-execution/showing-full-traces-of-llm-calls-with-langfuse

Install the necessary libraries using pip. Ensure you have Python and pip configured.

```bash
pip install langfuse valohai-llm
```

--------------------------------

### Install Python Packages On-the-Fly

Source: https://docs.valohai.com/notebook-executions/troubleshooting

Use this method to install specific library versions within a notebook session. Packages installed this way are temporary and will not persist after the session ends.

```shell
!pip install matplotlib==3.5.0 scikit-learn
```

--------------------------------

### Install valohai-utils

Source: https://docs.valohai.com/experiment-tracking/collect-metrics

Install the valohai-utils library using pip. This provides helper functions for logging.

```shell
pip install valohai-utils
```

--------------------------------

### Verify Python installation

Source: https://docs.valohai.com/automation-overview/rest-api/getting-started

Verify that your Python installation meets the minimum version requirement (3.8+) for the examples in this guide.

```shell
python --version
```

--------------------------------

### Initialize Valohai Project

Source: https://docs.valohai.com/cli-commands

Interactively sets up a new Valohai project in your current directory.

```bash
vh init [OPTIONS]
```

--------------------------------

### Complete Workflow: Mount, Preprocess, Save

Source: https://docs.valohai.com/data/data-nfs

A full example demonstrating mounting an NFS source, processing images using Python, saving results to Valohai outputs, and creating dataset versions.

```yaml
- step:
    name: preprocess-images
    image: python:3.9
    command:
      - pip install pillow pandas
      - python preprocess_images.py
    mounts:
      - destination: /mnt/raw-images
        source: <cloud-specific-nfs-source>
        type: nfs
        readonly: true
```

```python
import os
from PIL import Image
import pandas as pd
import json

# 1. Access raw data from network mount (NOT versioned)
mount_path = "/mnt/raw-images/"
image_files = [f for f in os.listdir(mount_path) if f.endswith(".jpg")]

print(f"Found {len(image_files)} images on network mount")

# 2. Process images
processed_data = []
output_dir = "/valohai/outputs/processed_images/"
os.makedirs(output_dir, exist_ok=True)

for filename in image_files:
    # Read from mount
    input_path = os.path.join(mount_path, filename)
    img = Image.open(input_path)

    # Process (resize, augment, etc.)
    img_resized = img.resize((224, 224))

    # Save processed image to Valohai outputs (VERSIONED ✅)
    output_path = os.path.join(output_dir, filename)
    img_resized.save(output_path)

    # Track metadata
    processed_data.append(
        {
            "filename": filename,
            "original_size": img.size,
            "processed_size": img_resized.size,
        },
    )

# 3. Save metadata
df = pd.DataFrame(processed_data)
df.to_csv("/valohai/outputs/processing_metadata.csv", index=False)

# 4. Create dataset version from processed images
metadata = {
    f"processed_images/{filename}": {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://preprocessed-images/v1",
            },
        ],
    }
    for filename in image_files
}

# Add metadata file itself
metadata["processing_metadata.csv"] = {
    "valohai.dataset-versions": [{"uri": "dataset://preprocessed-images/v1"}],
}

```

--------------------------------

### Complete Example: Time-Series Dataset Packaging

Source: https://docs.valohai.com/data/datasets/package-datasets

This example demonstrates processing daily sensor data, saving it as CSV files, and creating metadata with the packaging flag enabled for automatic packaging. It also includes the corresponding `valohai.yaml` configuration for preparing and training.

```python
import json
import pandas as pd
import os

# Step 1: Process data files
output_dir = "/valohai/outputs/"
dates = pd.date_range("2024-01-01", "2024-12-31", freq="D")

metadata = {}

for date in dates:
    # Process sensor data for this date
    sensor_data = load_sensor_data(date)

    # Save as CSV
    filename = f"sensor_{date.strftime('%Y%m%d')}.csv"
    filepath = os.path.join(output_dir, filename)
    sensor_data.to_csv(filepath, index=False)

    # Add to metadata with packaging enabled
    metadata[filename] = {
        "valohai.dataset-versions": [
            {
                "uri": "dataset://sensor-data/year-2024",
                "packaging": True,
            },
        ],
    }

# Step 2: Save metadata
metadata_path = os.path.join(output_dir, "valohai.metadata.jsonl")
with open(metadata_path, "w") as f:
    for filename, file_metadata in metadata.items():
        json.dump({"file": filename, "metadata": file_metadata}, f)
        f.write("\n")

print(f"Created dataset with {len(metadata)} files")
print("Packaging will happen automatically")
```

```yaml
- step:
    name: prepare-sensor-data
    image: python:3.9
    command: python prepare_data.py
    environment-variables:
      - name: VH_ENABLE_DATASET_VERSION_PACKAGING
        default: "1"

- step:
    name: train-forecasting-model
    image: python:3.9
    command: python train.py
    inputs:
      - name: sensor-data
        default: dataset://sensor-data/year-2024
```

--------------------------------

### Example GET Request for Dynamic Querying

Source: https://docs.valohai.com/docker-in-valohai/private-docker-registries/gcp

To get additional information not explicitly on the page, make an HTTP GET request to the page URL with an `ask` query parameter. The question should be self-contained and in natural language.

```http
GET https://docs.valohai.com/docker-in-valohai/private-docker-registries/gcp.md?ask=<question>
```

--------------------------------

### Network Mount Address Examples

Source: https://docs.valohai.com/installation-and-setup/advanced-topics/shared-cache

Examples of network mount addresses for different cloud providers and on-premises setups. Use these when providing details to Valohai support.

```text
AWS: fs-1234aa12.efs.eu-west-1.amazonaws.com:/
GCP: 10.123.12.123:/valohai_cache
Azure: //mystorageaccount.file.core.windows.net/valohai-cache
On-premises: nfs-server.example.com:/mnt/valohai-cache
```

--------------------------------

### Start an Execution with Basic Options

Source: https://docs.valohai.com/cli-commands

Use this command to start a new execution. Specify the step name and any required step options. You can also override default settings using various flags.

```bash
vh execution run [OPTIONS] STEP-NAME STEP-OPTIONS...
```

--------------------------------

### HTTP GET Request Example for Querying Documentation

Source: https://docs.valohai.com/automation-overview/triggers/webhooks/examples/v7-integration

To get additional information not directly on the page, you can query the documentation dynamically. Perform an HTTP GET request to the page URL with an `ask` query parameter containing your specific question.

```http
GET https://docs.valohai.com/automation-overview/triggers/webhooks/examples/v7-integration.md?ask=<question>
```

--------------------------------

### Workflow Summary with Example IDs

Source: https://docs.valohai.com/data/datasets/adding-readme-to-valohai-datasets

This outlines the steps for generating and associating a README.md file with Valohai datasets and dataset versions, including example IDs for each stage.

```text
Dataset ID: a3f7b2c1-4d5e-6f7a-8b9c-0d1e2f3a4b5c
↓
Run readme.py execution
↓
README.md Output Datum ID: 7c8d9e0f-1a2b-3c4d-5e6f-7a8b9c0d1e2f
↓
POST to /api/v0/datasets/a3f7b2c1-4d5e-6f7a-8b9c-0d1e2f3a4b5c/readme/
```

```text
Dataset Version ID: f1e2d3c4-b5a6-9876-5432-10fedcba9876
↓
Run readme.py execution with this dataset version
↓
README.md Output Datum ID: 2b4c6d8e-0f1a-2b3c-4d5e-6f7a8b9c0d1e
↓
POST to /api/v0/dataset-versions/f1e2d3c4-b5a6-9876-5432-10fedcba9876/readme/
```

--------------------------------

### Complete Image Comparison Example

Source: https://docs.valohai.com/experiment-tracking/compare-images

This example demonstrates generating prediction and ground truth images, saving them, and then saving prediction metadata including comparison properties.

```python
import json
import matplotlib.pyplot as plt
import numpy as np

# Generate prediction and ground truth
prediction = model.predict(input_image)
ground_truth = load_ground_truth()

# Save ground truth
plt.imsave("/valohai/outputs/ground_truth.png", ground_truth)

# Save prediction with metadata
plt.imsave("/valohai/outputs/prediction.png", prediction)

metadata = {
    "vhic_group": "segmentation/building-detection",
    "vhic_base": "base_image.png",
    "vhic_truth": "ground_truth.png",
    "vhic_name": "ResNet50 Prediction",
    "vhic_blend": "multiply",
    "vhic_color": "red",
}

with open("/valohai/outputs/prediction.png.metadata.json", "w") as f:
    json.dump(metadata, f)
```

--------------------------------

### Start Xorg Server

Source: https://docs.valohai.com/executions/system-environment-variables

Determine whether to start an Xorg server for the duration of the execution. Set to true to enable Xorg.

```shell
VH_XORG=1    # true values: 1, yes, true

```

--------------------------------

### Dockerfile for Julia Workloads

Source: https://docs.valohai.com/docker-in-valohai/building-images

Create a Julia environment using the official Julia image. This example installs the DataFrames, CSV, and Plots packages.

```dockerfile
FROM julia:1.9

RUN julia -e 'using Pkg; Pkg.add(["DataFrames", "CSV", "Plots"])

WORKDIR /workspace
```

--------------------------------

### Install Setuptools for Deprecated Distutils

Source: https://docs.valohai.com/installation-and-setup/on-premises/linux-workers

Install the `setuptools` package within the Peon virtual environment to address potential issues with the deprecated `distutils` module on Ubuntu 24.04 and Python 3.12.

```bash
/opt/valohai/peon-venv/bin/python -m pip install setuptools
```

--------------------------------

### SQL Query Example

Source: https://docs.valohai.com/data/data-databases

Example of a SQL query to select customer data based on signup date. The number of rows returned can change over time.

```sql
SELECT * FROM customer_data WHERE signup_date >= '2024-01-01'
-- Returns: 50,000 rows
```

```sql
SELECT * FROM customer_data WHERE signup_date >= '2024-01-01'
-- Returns: 65,000 rows (15,000 new customers!)
```

--------------------------------

### Query Valohai API with 'ask' Parameter

Source: https://docs.valohai.com/automation-overview/rest-api/examples/fetch-failed-executions

Example of how to query Valohai documentation dynamically using an HTTP GET request with the 'ask' query parameter.

```http
GET https://docs.valohai.com/automation-overview/rest-api/examples/fetch-failed-executions.md?ask=<question>
```

--------------------------------

### Example Input URLs for Web UI Import

Source: https://docs.valohai.com/data/data-versioning/add-existing-files

Provide these URLs, one per line, in the Valohai UI when adopting existing files.

```text
s3://my-bucket/datasets/training-data-v1.csv
s3://my-bucket/datasets/training-data-v2.csv
s3://my-bucket/models/baseline-model.pkl
```

--------------------------------

### Get Ingress Address

Source: https://docs.valohai.com/installation-and-setup/aws/self-hosted-eks

Retrieves the ingress address for your Valohai installation after the load balancer controller pods are running. This address is used to access the Valohai application.

```bash
kubectl get ingress
```

--------------------------------

### Dockerfile for R Workloads

Source: https://docs.valohai.com/docker-in-valohai/building-images

Set up an R environment using an R-base image. This example installs common R packages like tidyverse, caret, and randomForest.

```dockerfile
FROM r-base:4.3.0

RUN R -e "install.packages(c('tidyverse', 'caret', 'randomForest'), repos='https://cran.rstudio.com/')"

WORKDIR /workspace
```

--------------------------------

### Dockerfile for GPU Workloads with PyTorch

Source: https://docs.valohai.com/docker-in-valohai/building-images

Use NVIDIA's official CUDA images as a base for GPU-enabled workloads. This example installs Python, pip, and PyTorch with CUDA support.

```dockerfile
FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3 python3-pip
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
```

--------------------------------

### Example Metadata File Structure

Source: https://docs.valohai.com/serving-your-models/real-time-endpoints/metadata

An example of the `valohai-metadata.json` file structure, showing top-level keys like `deployment`, `version`, `endpoint`, `project`, and `meta`.

```json
{
  "deployment": {
    "name": "fraud-detection-api",
    "target": "production-cluster",
    "url": "https://app.valohai.com/api/v0/deployments/..."
  },
  "version": {
    "name": "20240315.2",
    "enabled": true,
    "ctime": "2024-03-15T14:23:11.431914Z"
  },
  "endpoint": {
    "name": "predict",
    "cpu_request": 0.1,
    "memory_limit": 512,
    "files": [
      {
        "name": "model",
        "datum": {
          "name": "fraud_model_v2.pkl",
          "size": 1048576
        }
      }
    ]
  },
  "project": {
    "name": "fraud-detection",
    "owner": {
      "username": "acme"
    }
  }
}
```

--------------------------------

### Download and Install frps on Jump Host

Source: https://docs.valohai.com/installation-and-setup/advanced-topics/workers-behind-nat

Install the frps server binary on the jump host by downloading it, making it executable, and placing it in the /opt/bin directory.

```shell
# Create installation directory
sudo mkdir -p /opt/bin
cd /opt/bin

# Download frps
sudo wget https://dist.valohai.com/frp/frp_0.61.0_linux_amd64/frps.gz
sudo gunzip frps.gz
sudo chmod a+x frps
```

--------------------------------

### Set Up Python Virtual Environment

Source: https://docs.valohai.com/installation-and-setup/index

Create a Python 3.9+ virtual environment accessible from SLURM compute nodes. Choose a path on shared network storage.

```shell
python3 -m venv /path/to/valohai-env
```

```shell
virtualenv /path/to/valohai-env
```

```shell
uv venv /path/to/valohai-env
```

--------------------------------

### Custom Values for Helm Installation

Source: https://docs.valohai.com/installation-and-setup/kubernetes/oracle

Example structure of a custom values file for Helm. These parameters, such as site name, image pull credentials, and Sentry DSN, are typically provided by Valohai.

```yaml
siteName: SITE_NAME
imagePullCredentials:
  email: EMAIL
  username: USERNAME
  password: PASSWORD
cleaner:
  sentryDsn: SENTRY_URL
```

--------------------------------

### Troubleshoot Service Start with Full Python Path

Source: https://docs.valohai.com/installation-and-setup/on-premises/linux-workers

If services fail to start, specify the full Python module path in the ExecStart directive within the service configuration file.

```ini
ExecStart=/usr/bin/env python3 -m peon.cli
```

--------------------------------

### Install Valohai CLI

Source: https://docs.valohai.com/getting-started/intro/jobs

Install the Valohai CLI and utilities for experiment tracking. Consider using `pipx` to avoid dependency conflicts.

```shell
pip install valohai-cli
```

--------------------------------

### Small Organization Quota Configuration

Source: https://docs.valohai.com/user-and-organization-management/environments-and-access-control/team-quotas

Example configuration for a small organization with two teams, distributing quotas across different environments. This setup prioritizes the development team for CPU resources.

```text
Environment: aws-cpu-medium (Max Scale: 10)
- Team dev-team: Quota = 6
- Team ml-team: Quota = 4

Environment: aws-gpu-v100 (Max Scale: 4)
- Team dev-team: Quota = 2
- Team ml-team: Quota = 2
```

--------------------------------

### List Available Execution Environments

Source: https://docs.valohai.com/cli-commands

Use this command to see all available execution environments. Options can filter the output to show GPU specifications, pricing, queue information, or descriptions.

```bash
vh environments [OPTIONS]
```

--------------------------------

### Agent Instructions: Querying Documentation

Source: https://docs.valohai.com/models/manage-models

Perform an HTTP GET request with an 'ask' query parameter to dynamically query documentation. The question should be specific and self-contained.

```http
GET https://docs.valohai.com/models/manage-models.md?ask=<question>
```

--------------------------------

### Verify frps Installation

Source: https://docs.valohai.com/installation-and-setup/advanced-topics/workers-behind-nat

Check the installed frps version to confirm successful installation.

```shell
/opt/bin/frps --version
```

--------------------------------

### Start Peon Service

Source: https://docs.valohai.com/installation-and-setup/on-premises/linux-workers

Start the Peon service after configuring it for dispatch mode. Ensure all prerequisites are met before starting.

```shell
sudo systemctl start peon
```

--------------------------------

### PyTorch Distributed Setup with Valohai

Source: https://docs.valohai.com/distributed-training/distributed-tasks

Initialize PyTorch's distributed backend using Valohai's connection information. This requires the `torch.distributed` module and Valohai's distributed utilities.

```python
import torch.distributed as dist
import valohai

if valohai.distributed.is_distributed_task():
    master = valohai.distributed.master()
    rank = valohai.distributed.rank
    world_size = valohai.distributed.required_count

    # Initialize PyTorch distributed
    dist.init_process_group(
        backend="nccl",
        init_method=f"tcp://{master.primary_local_ip}:29500",
        rank=rank,
        world_size=world_size,
    )
```

--------------------------------

### Install Snowflake Connector in valohai.yaml

Source: https://docs.valohai.com/data/data-databases/snowflake

Install the Snowflake Python connector and run your script by adding pip install command to your valohai.yaml configuration.

```yaml
- step:
    name: query-snowflake
    image: python:3.11
    command:
      # Install Snowflake connector
      - pip install snowflake-connector-python
      # Run your script
      - python query_snowflake.py
```

--------------------------------

### Use valohai-utils for Output Paths and Live Uploads

Source: https://docs.valohai.com/migration-strategy/migrate-job-outputs

Demonstrates using the `valohai-utils` library to generate output paths and enable live uploading of log files.

```python
import valohai

# Generate output path
output_path = valohai.outputs().path("model.h5")
model.save(output_path)

# Live logging
valohai.outputs().live_upload("training.log")
```

--------------------------------

### Install Multiple Private Packages

Source: https://docs.valohai.com/executions/advanced-features/private-pypi-jfrog

Install several private packages by listing them after the pip install command. Ensure the PRIVATE_PYPI_INDEX_URL environment variable is set.

```yaml
- step:
    name: install-multiple-private
    image: python:3.12
    command:
      - pip config set global.index-url $PRIVATE_PYPI_INDEX_URL
      - pip install my-ml-library my-data-utils my-viz-tools
      - python run_pipeline.py
```

--------------------------------

### Install All Valohai Agent Skills

Source: https://docs.valohai.com/migration-strategy/migrate-with-ai-skills

Installs all Valohai Agent Skills for all detected AI coding agents on your system. This command configures the skills to work with your installed agents.

```bash
npx skills add valohai/valohai-skills --all
```

--------------------------------

### Verify Python Installation

Source: https://docs.valohai.com/installation-and-setup/on-premises/linux-workers

Check if Python 3.10+ is installed on the system.

```shell
python3 --version
```

--------------------------------

### Create requirements-deployment.txt

Source: https://docs.valohai.com/serving-your-models/real-time-endpoints/install-packages

Add this file to your repository to specify packages for deployment. Valohai installs these when building your deployment.

```text
sqlalchemy==2.0.0
fastapi==0.104.1
pydantic==2.5.0
```

--------------------------------

### Example valohai.yaml Configuration

Source: https://docs.valohai.com/valohai.yaml-overview

Defines multiple steps (preprocess, train, batch-inference) and a pipeline for an ML workflow. Includes Docker images, commands, inputs with defaults, and parameters for the training step.

```yaml
- step:
    name: preprocess-dataset
    image: python:3.9
    command:
      - pip install numpy valohai-utils
      - python ./preprocess_dataset.py
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz

- step:
    name: train-model
    image: tensorflow/tensorflow:2.6.0
    command:
      - pip install valohai-utils
      - python ./train_model.py {parameters}
    parameters:
      - name: epochs
        default: 5
        type: integer
      - name: learning_rate
        default: 0.001
        type: float
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz

- step:
    name: batch-inference
    image: tensorflow/tensorflow:2.6.0
    command:
    - pip install pillow valohai-utils
    - python ./batch_inference.py
    inputs:
    - name: model
    - name: images
      default:
      - https://valohaidemo.blob.core.windows.net/mnist/four-inverted.png
      - https://valohaidemo.blob.core.windows.net/mnist/five-inverted.png
      - https://valohaidemo.blob.core.windows.net/mnist/five-normal.jpg

- pipeline:
    name: training-pipeline
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-dataset
      - name: train
        type: execution
        step: train-model
        override:
          inputs:
            - name: dataset
      - name: evaluate
        type: execution
        step: batch-inference
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
      - [train.output.model*, evaluate.input.model]
```

--------------------------------

### Python example with valohai-utils

Source: https://docs.valohai.com/pipelines/parallel-runs-in-a-pipeline/parallel-executions-in-pipeline

This is a placeholder for a Python code example using valohai-utils, intended to demonstrate best practices for output naming in task nodes.

```python
import valohai

```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.valohai.com/pipelines/configure-different-environments

Perform an HTTP GET request on the current page URL with the `ask` query parameter to get dynamic answers and documentation excerpts.

```http
GET https://docs.valohai.com/pipelines/configure-different-environments.md?ask=<question>
```

--------------------------------

### Query Documentation with 'ask' Parameter

Source: https://docs.valohai.com/installation-and-setup/advanced-topics/configure-ssh-access

This example demonstrates how to dynamically query the documentation by appending the 'ask' query parameter to the page URL. This is useful for retrieving specific information or clarification not explicitly present on the page.

```http
GET https://docs.valohai.com/installation-and-setup/advanced-topics/configure-ssh-access.md?ask=<question>
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.valohai.com/executions/system-configuration-files

Perform an HTTP GET request on the current page URL with the `ask` query parameter to get additional information not directly available on the page.

```http
GET https://docs.valohai.com/executions/system-configuration-files.md?ask=<question>
```

--------------------------------

### Querying Documentation via GET Request

Source: https://docs.valohai.com/automation-overview/triggers/webhooks/webhook-reference

To get additional information not directly present on a page, perform an HTTP GET request with the 'ask' query parameter. The question should be specific and self-contained.

```http
GET https://docs.valohai.com/automation-overview/triggers/webhooks/webhook-reference.md?ask=<question>
```

--------------------------------

### Create a New Valohai Project

Source: https://docs.valohai.com/getting-started/intro/jobs

Set up a new project directory and connect it to Valohai for experiment tracking. This links your local directory to Valohai.

```shell
mkdir my-ml-project
cd my-ml-project
vh project create --name my-ml-project
```

--------------------------------

### Query Documentation Dynamically via HTTP GET

Source: https://docs.valohai.com/automation-overview/rest-api/getting-started

Perform an HTTP GET request to a documentation URL with an 'ask' query parameter to get specific answers and relevant excerpts from the documentation.

```http
GET https://docs.valohai.com/automation-overview/rest-api/getting-started.md?ask=<question>
```

--------------------------------

### Initialize Valohai Admin User

Source: https://docs.valohai.com/installation-and-setup/openshift/self-hosted

Run the `roi_init` command within the Valohai pod to create an administrator account. Save the printed credentials securely.

```bash
python manage.py roi_init --mode dev
```

--------------------------------

### Example SLURM Configuration

Source: https://docs.valohai.com/installation-and-setup/index

An example JSON configuration file for Valohai's SLURM integration, demonstrating SSH connection details and job parameters.

```json
{
  "job": {
    "account": "ml_research",
    "partition": "gpu",
    "time_limit_minutes": 720
  },
  "worker": {
    "working_directory": "/scratch/valohai",
    "peon_command": "/shared/valohai-env/bin/valohai-peon"
  },
  "ssh": {
    "hostname": "slurm-head.example.com",
    "username": "valohai",
    "port": 22,
    "password": null,
    "allowed_host_keys": [
      "ssh-rsa AAAAB3NzaC1..."
    ]
  }
}
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://docs.valohai.com/installation-and-setup/kubernetes

To get information not directly on the page, perform an HTTP GET request to the current page URL with the 'ask' query parameter. The question should be specific and self-contained.

```http
GET https://docs.valohai.com/installation-and-setup/kubernetes.md?ask=<question>
```
Tokens:
Raw