### Run Dashboard UI Locally

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to install dependencies and start the UI development server.

```sh
cd ui
npm install
PUBLIC_API_BASE_URL=http://localhost:6060 npm run dev
```

--------------------------------

### Start Tilt

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to start the Tilt development environment.

```sh
tilt up --context kind-kvisor
```

--------------------------------

### Start Colima

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to start Colima as a Docker Desktop replacement.

```sh
colima start  --cpu 2 --memory 4 --disk 100 -t vz --mount-type virtiofs
```

--------------------------------

### Kvisor Sizing Report Example Output

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Example output from the sizing report script, showing per-node analysis, per-nodepool summary, and suggested values.yaml configurations for different sizing profiles.

```yaml
OBI Sizing Report
═════════════════════════════════════════════════════════════════

Cluster:          prod-cluster
Ports analyzed:   8080,8443

Nodes scanned:    188
Nodes with match: 80 (42%)

Per-Node Analysis (nodes with matching containers)
────────────────────────────────────────────────────────────────
NODE                                          POOL                  PROCS  STEADY      PEAK     RECOMMEND  PROFILE
gke-prod-cast-pool-521d416f                   c2d-highmem-32           22   634 Mi    684 Mi      664 Mi    Large
gke-prod-cast-pool-656bc8c9                   c2d-standard-56          19   553 Mi    603 Mi      583 Mi    Large
gke-prod-cast-pool-dc246266                   c2d-standard-32          17   499 Mi    549 Mi      529 Mi    Large
gke-prod-cast-pool-650882f0                   c2d-standard-112         12   364 Mi    414 Mi      394 Mi   Medium
gke-prod-cast-pool-48cb64eb                   n2d-highcpu-16            9   283 Mi    333 Mi      313 Mi   Medium
...

Per-Nodepool Summary
────────────────────────────────────────────────────────────────
POOL                 NODES  MIN  MAX  AVG  RECOMMENDED LIMIT  PROFILE
c2d-highmem-32          15    1   22    6          768Mi       Large
c2d-standard-32          1   17   17   17          768Mi       Large
c2d-standard-56          3    7   19   11          768Mi       Large
c2d-standard-112         2   12   12   12          512Mi      Medium
n2d-highcpu-16           2    9    9    9          384Mi      Medium
c2d-highmem-8           29    0    3    0          256Mi       Small
n2d-highcpu-8           26    0    0    0          256Mi       Small
...

Suggested values.yaml
────────────────────────────────────────────────────────────────
  # Recommended sizing profile based on max observed density
  # (22 containers on busiest node):
  agent:
    reliabilityMetrics:
      enabled: true
      obi:
        sizingProfile: "large"
        openPorts: "8080,8443"

  # Profile 'large' → requests: 384Mi, limits: 768Mi

  # ⚡ High variance detected (0–22 containers/node).
  # Consider dynamic sizing to auto-tune per node:
  agent:
    reliabilityMetrics:
      enabled: true
      obi:
        sizingProfile: "xlarge"
        dynamicSizing: true
        openPorts: "8080,8443"

  # Or use custom sizing for full control:
  agent:
    reliabilityMetrics:
      enabled: true
      obi:
        sizingProfile: "custom"
        openPorts: "8080,8443"
        resources:
          requests:
            memory: 384Mi
          limits:
            memory: 768Mi
```

--------------------------------

### Setup Local Kubernetes with Kind

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to create a Kind cluster and verify connectivity.

```sh
kind create cluster --name=kvisor --config ./e2e/kind-config.yaml
kubectl cluster-info --context kind-kvisor
```

--------------------------------

### Install Kind Cluster

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to deploy a local Kind cluster.

```sh
CLUSTER_NAME=tilt ./tools/localenv/kind.sh
```

--------------------------------

### Start Lima VM

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Alternative command to start the Lima VM.

```sh
ln -s ~/c/kvisor /tmp/kvisor
limactl start ./tools/lima-ebpf.yaml
```

--------------------------------

### Install pwru

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Downloads and installs the pwru packet tracing tool.

```sh
wget https://github.com/cilium/pwru/releases/download/v1.0.6/pwru-linux-amd64.tar.gz && \
tar -xzvf pwru-linux-amd64.tar.gz && \
mv pwru /usr/bin/ && \
rm pwru-linux-amd64.tar.gz
```

--------------------------------

### Set Up Local Development Environment

Source: https://context7.com/castai/kvisor/llms.txt

Commands to initialize a Kind cluster, start Tilt for hot-reloading, and run the dashboard UI.

```bash
# Create local Kind cluster
kind create cluster --name=kvisor --config ./e2e/kind-config.yaml
kubectl cluster-info --context kind-kvisor

# Start Tilt for hot-reload development
tilt up --context kind-kvisor

# Port-forward controller API
kubectl port-forward svc/kvisord-server 6060:80 -n kvisord

# Run dashboard UI locally
cd ui
npm install
PUBLIC_API_BASE_URL=http://localhost:6060 npm run dev
```

--------------------------------

### Install Cast AI Kvisor Fresh

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Use this command for a fresh installation of Cast AI Kvisor. Ensure you replace placeholders with your actual API key, cluster ID, and gRPC endpoint.

```bash
helm install castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent --create-namespace \
  --set castai.apiKey=<your-api-key> \
  --set castai.clusterID=<your-cluster-id> \
  --set castai.grpcAddr=<grpc-endpoint> \
  --set agent.reliabilityMetrics.enabled=true \
  --set controller.reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.operator.enabled=true \
  --set reliabilityMetrics.install.enabled=true \
  --set reliabilityMetrics.exporter.enabled=true
```

--------------------------------

### Install Kvisor with Reliability Metrics

Source: https://context7.com/castai/kvisor/llms.txt

Install Kvisor using Helm, enabling eBPF-based automatic instrumentation for collecting golden signal metrics. Ensure you replace placeholders with your actual API key, cluster ID, and desired gRPC address.

```bash
helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent --create-namespace \
  --set castai.apiKey="<your-api-key>" \
  --set castai.clusterID="<your-cluster-id>" \
  --set castai.grpcAddr="kvisor.prod-master.cast.ai:443" \
  --set agent.reliabilityMetrics.enabled=true \
  --set agent.reliabilityMetrics.obi.sizingProfile=medium \
  --set agent.reliabilityMetrics.obi.openPorts="8080,8443,8090,6379" \
  --set controller.reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.operator.enabled=true \
  --set reliabilityMetrics.install.enabled=true \
  --set reliabilityMetrics.exporter.enabled=true
```

--------------------------------

### Install kvisord via Helm

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Deploys the kvisord chart with specific resource requests and server arguments.

```sh
helm upgrade --install kvisord oci://us-east4-docker.pkg.dev/kvisor/helm-charts/kvisord \
    --version 0.7.0 \
    --namespace kvisord --create-namespace \
    --set storage.resources.requests.cpu=2 \
    --set storage.resources.requests.memory=8Gi \
    --set agent.resources.requests.cpu=100m \
    --set agent.resources.requests.memory=128Mi \
    --set server.resources.requests.cpu=2 \
    --set server.resources.requests.memory=4Gi \
    --set server.extraArgs.events-batch-size=1000 \
    --set server.extraArgs.events-batch-queue-size=30000 \
    --set-string server.extraArgs.workload-profiles-enabled=false \
```

--------------------------------

### Helm Configuration for Kvisor

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-aws.md

Example values.yaml configuration for enabling AWS cloud provider integration.

```yaml
controller:
  extraArgs:
    cloud-provider: aws
    cloud-provider-vpc-sync-enabled: true
    cloud-provider-vpc-name: "vpc-0123456789abcdef0"
    # cloud-provider-aws-region: "us-east-1"  # Recommended; required if using Transit Gateway
    # cloud-provider-vpc-sync-interval: 1h
    # cloud-provider-vpc-cache-size: 10000
    # cloud-provider-aws-use-zone-id: true

  serviceAccount:
    create: true
    name: kvisor
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::YOUR_ACCOUNT_ID:role/KvisorVPCReaderRole"
```

--------------------------------

### Setup Lima for eBPF Development

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to initialize the Lima VM environment for local eBPF development.

```sh
ln -s ~/c/kvisor /tmp/kvisor
limactl start ./tools/lima-ebpf.yaml
limactl shell lima-ebpf
cd /tmp/kvisor
devbox install
```

--------------------------------

### Test runtime on Kubernetes

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Installs a local chart to a cluster for runtime testing with eBPF events output to stdout.

```sh
helm upgrade -i castai-kvisor ./charts/kvisor/ -n castai-agent --create-namespace \
    --set image.tag=<your-pr-image-tag> \
    --set castai.enabled=false \
    --set agent.enabled=true \
    --set agent.extraArgs.ebpf-events-enabled=true
```

--------------------------------

### Install Kvisor with CAST AI Integration

Source: https://context7.com/castai/kvisor/llms.txt

Deploys Kvisor using Helm with required CAST AI API keys and cluster identifiers.

```bash
# Add CAST AI Helm repository
helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update castai-helm

# Install with CAST AI integration
helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent --create-namespace \
  --set castai.apiKey="<your-api-key>" \
  --set castai.clusterID="<your-cluster-id>" \
  --set castai.grpcAddr="kvisor.prod-master.cast.ai:443" \
  --set agent.enabled=true \
  --set controller.enabled=true \
  --set controller.extraArgs.image-scan-enabled=true \
  --set controller.extraArgs.kube-bench-enabled=true \
  --set controller.extraArgs.kube-linter-enabled=true
```

--------------------------------

### Install Standalone Kvisor with Local ClickHouse

Source: https://context7.com/castai/kvisor/llms.txt

Deploys Kvisor for local network flow monitoring without external CAST AI integration.

```bash
helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent --create-namespace \
  --set castai.enabled=false \
  --set castai.clusterID=noop \
  --set agent.enabled=true \
  --set agent.extraArgs.netflow-enabled=true \
  --set agent.extraArgs.netflow-export-interval=15s \
  --set clickhouse.enabled=true \
  --set clickhouse.persistentVolume.size=200Gi

# Verify deployment
kubectl get pods -n castai-agent
# NAME                                        READY   STATUS    RESTARTS   AGE
# castai-kvisor-agent-xxxxx                   1/1     Running   0          67s
# castai-kvisor-clickhouse-0                  2/2     Running   0          66s
# castai-kvisor-controller-xxxxxxx-xxxxx      1/1     Running   0          67s
```

--------------------------------

### Install Kvisor with Netflow

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Deploys Kvisor configured to export netflow data to a local ClickHouse instance.

```sh
helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update castai-helm

helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
    --namespace castai-agent --create-namespace \
    --set castai.enabled=false \
    --set agent.enabled=true \
    --set agent.extraArgs.netflow-enabled=true \
    --set agent.extraArgs.ebpf-events-enabled=true \
    --set agent.extraArgs.ebpf-events-policy=net_packet_dns_base \
    --set clickhouse.enabled=true
```

--------------------------------

### Install kvisor with netflows monitoring

Source: https://github.com/castai/kvisor/blob/main/GUIDE.md

Deploys kvisor via Helm with netflow monitoring enabled and local Clickhouse storage configured.

```sh
helm upgrade --install castai-kvisor castai-kvisor --repo https://castai.github.io/helm-charts \
  -n castai-agent --create-namespace \
  --reset-then-reuse-values \
  --set castai.enabled=false \
  --set castai.clusterID=noop \
  --set agent.enabled=true \
  --set agent.extraArgs.netflow-enabled=true \
  --set agent.extraArgs.netflow-export-interval=15s \
  --set clickhouse.enabled=true \
  --set clickhouse.persistentVolume.size=200Gi
```

--------------------------------

### Automated Reliability Stack Installation Script

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Execute the provided shell script to install the Kvisor reliability stack. This script handles ClickHouse Operator CRD bootstrapping and can be configured with various options for context, OBI profiles, and dynamic sizing. Use --dry-run to preview commands.

```bash
# Basic usage (existing kvisor installation, auto-detects OBI profile)
./charts/kvisor/scripts/enable-reliability-stack.sh

# With context and explicit profile
./charts/kvisor/scripts/enable-reliability-stack.sh \
  --context <kube-context> \
  --obi-profile large \
  --dynamic-sizing

# Dry-run (prints commands without executing)
./charts/kvisor/scripts/enable-reliability-stack.sh --dry-run
```

--------------------------------

### Install or Upgrade Kvisor Helm Chart

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-gcp.md

This command deploys or updates the Kvisor Helm chart in the specified namespace using the provided values file.

```bash
helm upgrade --install castai-kvisor castai-helm/castai-kvisor \
  --namespace kvisor --create-namespace --values values.yaml
```

--------------------------------

### Define tracerTaskContextT Structure

Source: https://github.com/castai/kvisor/blob/main/tools/eventcontextcodegen/testdata/definition.txt

Defines the structure for task context, capturing start time, cgroup ID, PIDs (host and container), UIDs, mount IDs, PID IDs, command name, and parent start times.

```go
type tracerTaskContextT struct {
	StartTime       uint64
	CgroupId        uint64
	Pid             uint32
	Tid             uint32
	Ppid            uint32
	HostPid         uint32
	HostTid         uint32
	HostPpid        uint32
	NodeHostPid     uint32
	Uid             uint32
	MntId           uint32
	PidId           uint32
	Comm            [16]int8
	LeaderStartTime uint64
	ParentStartTime uint64
}
```

--------------------------------

### Configure Reliability Metrics

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Base configuration for reliability metrics including ClickHouse installation and exporter settings.

```yaml
reliabilityMetrics:
  enabled: true

  # CAST AI connectivity (for the ch-exporter)
  castai:
    apiKeySecretRef: "castai-kvisor"
    # Direct value (preferred for TF-managed clusters):
    clusterID: ""
    # OR from ConfigMap (default):
    clusterIdConfigMapRef:
      name: "castai-agent-metadata"
      key: "CLUSTER_ID"
    # Telemetry server gRPC address. The ch-exporter uses this to send
    # aggregated metrics to the CAST AI mothership. Unlike the kvisor agent
    # (which auto-derives telemetry.* from castai.grpcAddr), the exporter
    # needs an explicit address.
    # EU region: "telemetry.prod-eu.cast.ai:443"
    grpcAddr: "telemetry.prod-master.cast.ai:443"

  # ClickHouse credentials
  auth:
    database: "metrics"
    username: "kvisor"
    password: "kvisor"     # Override with valueFrom.secretKeyRef for production

  # Altinity ClickHouse operator
  operator:
    enabled: false         # Set true to install operator; false if already present

  # In-cluster ClickHouse deployment (via ClickHouseInstallation CR)
  install:
    enabled: true
    image:
      repository: clickhouse/clickhouse-server
      tag: "25.12.6-alpine"
    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        memory: 2Gi
    persistence:
      size: 100Gi

  # ch-exporter sidecar
  exporter:
    enabled: true
    grpcAddr: ""           # Defaults to reliabilityMetrics.castai.grpcAddr if empty
    image:
      repository: ghcr.io/castai/kvisor/reliability-metrics-ch-exporter
      tag: "v0.3.12"
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        memory: 128Mi
    # PodMonitor for Prometheus Operator (scrapes exporter metrics on port 8080)
    podMonitor:
      enabled: false
      labels: {}
      selectorLabels: {}    # Override auto-detected pod selector
      interval: 30s
      scrapeTimeout: 10s

  # External ClickHouse (alternative to install.enabled)
  external:
    enabled: false
    address: ""            # host:port (native protocol)
    database: "metrics"
```

--------------------------------

### Enable Reliability Metrics on Existing Kvisor

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Upgrade an existing Cast AI Kvisor installation to enable reliability metrics. Use `--reset-then-reuse-values` to safely merge new configuration structures. Be aware of the CRD dependency for ClickHouseInstallation.

```bash
helm repo update castai-helm

helm upgrade castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent \
  --reset-then-reuse-values \
  --set agent.reliabilityMetrics.enabled=true \
  --set controller.reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.enabled=true \
  --set reliabilityMetrics.operator.enabled=true \
  --set reliabilityMetrics.install.enabled=true \
  --set reliabilityMetrics.exporter.enabled=true
```

--------------------------------

### Add and Update Helm Repositories

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Add the CAST AI Helm chart repository and update your local Helm repository cache. Ensure you have Helm v3.12+ installed.

```bash
helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update castai-helm
```

--------------------------------

### Run Kvisor Sizing Report Script

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Execute the sizing report script to analyze your cluster and get recommended resource settings. The script requires kubectl with read access and Python 3. It can be run with default or custom ports.

```bash
./charts/kvisor/scripts/obi-sizing-report.sh
```

```bash
./charts/kvisor/scripts/obi-sizing-report.sh --ports 8080,8443
```

```bash
./charts/kvisor/scripts/obi-sizing-report.sh --context prod-cluster --ports 8080,8443
```

--------------------------------

### View OBI Init Container Logs

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Check logs for cgroup memory limits and dynamic sizing scan results during container startup.

```bash
kubectl logs <pod> -c obi-init

# Always present (cgroup-aware GOMEMLIMIT):
# Output: "obi-init: cgroup memory limit = 805306368 bytes"
# Output: "obi-init: setting GOMEMLIMIT to 90% = 691MiB"

# With dynamicSizing enabled:
# Output: "obi-init: scanning network namespaces for listening processes..."
# Output: "obi-init: found 15 unique netns, 23 processes on ports 8080,8443,8090,6379"
# Output: "obi-init: recommended memory = 691 MiB"
```

--------------------------------

### Execute Integration Tests

Source: https://github.com/castai/kvisor/blob/main/pkg/cloudprovider/gcp/test/DEBUG.md

Set the required environment variables and run the Go test suite.

```bash
# Set environment variables
export GCP_PROJECT_ID="your-project-id"
export GCP_CREDENTIALS_FILE="$HOME/kvisor-test-sa-key.json"

# Run the tests
go test -v -run TestRefreshState ./...
```

--------------------------------

### Run OBI Sizing Report

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Identify workloads causing density spikes by analyzing replica distribution.

```bash
./charts/kvisor/scripts/obi-sizing-report.sh -p 8080,8443 | head -40
```

--------------------------------

### Create and Configure GCP Service Account

Source: https://github.com/castai/kvisor/blob/main/pkg/cloudprovider/gcp/test/DEBUG.md

Commands to create a service account, assign the Compute Network Viewer role, and generate a JSON key file.

```bash
# Create service account
gcloud iam service-accounts create kvisor-integration-test \
    --project=YOUR_PROJECT_ID \
    --display-name="Kvisor Integration Test"

# Grant permissions
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:kvisor-integration-test@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/compute.networkViewer"

# Download key
gcloud iam service-accounts keys create ~/kvisor-test-sa-key.json \
    --iam-account=kvisor-integration-test@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

--------------------------------

### Create GCP Service Account, Key, and Bind IAM Policy

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-gcp.md

This sequence creates a GCP service account, assigns it the 'compute.networkViewer' role, and generates a JSON key file for authentication.

```bash
export PROJECT_ID="your-gcp-project-id"
export SERVICE_ACCOUNT_NAME="kvisor-vpc-reader"

gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} \
  --project=${PROJECT_ID} \
  --display-name="Kvisor VPC State Reader"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --project=${PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/compute.networkViewer"

gcloud iam service-accounts keys create kvisor-sa-key.json \
  --project=${PROJECT_ID} \
  --iam-account=${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
```

--------------------------------

### Configure Existing Altinity Operator

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Configuration to use an existing Altinity ClickHouse operator instead of installing a new one.

```yaml
reliabilityMetrics:
  enabled: true
  operator:
    enabled: false     # Don't install a second operator
  install:
    enabled: true      # Deploy ClickHouse via existing operator
```

--------------------------------

### Create GCP Service Account and Bind IAM Policy

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-gcp.md

Use this command to create a GCP service account and grant it the 'compute.networkViewer' role, which is necessary for Kvisor to read network information.

```bash
export PROJECT_ID="your-gcp-project-id"
export SERVICE_ACCOUNT_NAME="kvisor-vpc-reader"

gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} \
  --project=${PROJECT_ID} \
  --display-name="Kvisor VPC State Reader"

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --project=${PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/compute.networkViewer"
```

--------------------------------

### Inspect OBI Environment Variables

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Use this command to verify the GOMEMLIMIT setting within a running OBI container.

```bash
kubectl exec <pod> -c obi -- cat /proc/1/environ | tr '\0' '\n' | grep GOMEMLIMIT
```

--------------------------------

### Configure local observability stack

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Scripts and configurations for setting up local monitoring components and Prometheus metrics scraping.

```sh
./tools/localenv/prometheus.sh
./tools/localenv/pyroscope.sh
./tools/localenv/grafana.sh
```

```sh
k port-forward svc/grafana 8080:80 -n metrics
```

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: castai-kvisor-agent
  namespace: castai-agent
spec:
  namespaceSelector:
    matchNames:
    - castai-agent
  podMetricsEndpoints:
  - honorLabels: true
    path: /metrics
    port: metrics
    scheme: http
    scrapeTimeout: 30s
  selector:
    matchLabels:
      app.kubernetes.io/name: castai-kvisor-agent
```

--------------------------------

### Disable Reliability Metrics

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Upgrade the Helm release to disable reliability metrics for the agent, controller, and globally, while keeping Kvisor installed. This preserves ClickHouse data.

```bash
helm upgrade castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent \
  --reset-then-reuse-values \
  --set agent.reliabilityMetrics.enabled=false \
  --set controller.reliabilityMetrics.enabled=false \
  --set reliabilityMetrics.enabled=false
```

--------------------------------

### Develop eBPF with Lima

Source: https://context7.com/castai/kvisor/llms.txt

Commands to set up a Linux VM on macOS for eBPF development and testing.

```bash
# Set up Lima VM with eBPF support
ln -s ~/code/kvisor /tmp/kvisor
limactl start ./tools/lima-ebpf.yaml
limactl shell lima-ebpf

# Install development dependencies
cd /tmp/kvisor
devbox install

# Generate eBPF code and run tests
cd /tmp/kvisor/pkg/ebpftracer
go generate ./...
go test -v . -exec sudo -run=TestTracer
```

--------------------------------

### Run eBPF Tracer Tests

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to generate and test the eBPF tracer within the VM.

```sh
cd /tmp/kvisor/pkg/ebpftracer
go generate ./...
go test -v . -exec sudo -run=TestTracer
```

--------------------------------

### Expose Docker Daemon to Host

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to create and use a Docker context for the Lima VM.

```sh
docker context create lima-lima-ebpf --docker "host=unix://~/.lima/lima-ebpf/sock/docker.sock"
```

```sh
docker context use lima-lima-ebpf
```

--------------------------------

### Enable All PodMonitors via --set flags

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Use these `--set` flags during `helm upgrade` to enable scraping of all components' PodMonitors.

```bash
helm upgrade castai-kvisor castai-helm/castai-kvisor \
  -n castai-agent \
  --reset-then-reuse-values \
  --set agent.reliabilityMetrics.obi.internalMetrics.enabled=true \
  --set agent.reliabilityMetrics.obi.internalMetrics.podMonitor.enabled=true \
  --set agent.reliabilityMetrics.collector.podMonitor.enabled=true \
  --set controller.reliabilityMetrics.collector.podMonitor.enabled=true \
  --set reliabilityMetrics.exporter.podMonitor.enabled=true
```

--------------------------------

### Identify required Linux capabilities

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Uses inspektor-gadget to trace and list the Linux capabilities required by the kvisor agent.

```sh
KVISOR_AGENT_KUBERNETES_NODE="kvisor-e2e-control-plane" # Replace with the kvisor agent pod name
kubectl debug --profile=sysadmin node/${KVISOR_AGENT_KUBERNETES_NODE} -it --image=ghcr.io/inspektor-gadget/ig -- \
    ig trace capabilities -c kvisor --unique -t 0
```

--------------------------------

### Create IAM Policy via AWS CLI

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-aws.md

Commands to create the Kvisor VPC reader policy in AWS.

```bash
export AWS_REGION="us-east-1"
export POLICY_NAME="KvisorVPCReaderPolicy"

cat > kvisor-vpc-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcPeeringConnections",
        "ec2:DescribeTransitGatewayAttachments",
        "ec2:DescribeTransitGatewayPeeringAttachments",
        "ec2:DescribeTransitGatewayRouteTables",
        "ec2:SearchTransitGatewayRoutes"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name ${POLICY_NAME} \
  --policy-document file://kvisor-vpc-policy.json \
  --region ${AWS_REGION}
```

--------------------------------

### Implement EventContext Helper Methods

Source: https://github.com/castai/kvisor/blob/main/tools/eventcontextcodegen/testdata/context.txt

Methods to extract flow direction, network flow type, and initiator status from the EventContext based on Retval flags.

```go
func (ctx *EventContext) GetFlowDirection() FlowDirection {
	if ctx.Retval&FlagPacketIngress > 0 && ctx.Retval&FlagPacketEgress > 0 {
		// something is broken if both ingress and egress flags are set
		return FlowDirectionUnknown
	}

	if ctx.Retval&FlagPacketIngress > 0 {
		return FlowDirectionIngress
	}
	if ctx.Retval&FlagPacketEgress > 0 {
		return FlowDirectionEgress
	}

	return FlowDirectionUnknown
}

func (ctx *EventContext) GetNetflowType() NetflowType {
	if ctx.Retval&FlagFlowTCPBegin > 0 {
		return NetflowTypeTCPBegin
	}
	if ctx.Retval&FlagFlowTCPSample > 0 {
		return NetflowTypeTCPSample
	}
	if ctx.Retval&FlagFlowTCPEnd > 0 {
		return NetflowTypeTCPEnd
	}
	return NetflowTypeUnknown
}

func (ctx *EventContext) IsSourceInitiator() bool {
	if ctx.Retval&FlagFlowSrcInitiator > 0 {
		return true
	}
	return false
}
```

--------------------------------

### Test Helm Chart Template

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to verify the Helm chart template.

```sh
helm template kvisord oci://us-east4-docker.pkg.dev/kvisor/helm-charts/kvisord --version 0.1.6
```

--------------------------------

### Clickhouse connection details

Source: https://github.com/castai/kvisor/blob/main/GUIDE.md

Configuration parameters for connecting to the local Clickhouse instance.

```text
Host: localhost
Port: 8123
Username: kvisor
Password: kvisor
Schema: kvisor
```

--------------------------------

### Create IAM Service Account via eksctl

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-aws.md

Alternative method to create a new service account with the required IAM policy attached.

```bash
export CLUSTER_NAME="your-eks-cluster"
export NAMESPACE="kvisor"
export SERVICE_ACCOUNT_NAME="kvisor"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME} --region ${AWS_REGION} --approve

eksctl create iamserviceaccount \
  --cluster=${CLUSTER_NAME} \
  --namespace=${NAMESPACE} \
  --name=${SERVICE_ACCOUNT_NAME} \
  --attach-policy-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_NAME} \
  --region=${AWS_REGION} \
  --approve
```

--------------------------------

### Configure Helm Overrides for Dynamic Sizing

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Adjust OBI sizing profiles per node pool to auto-tune memory limits.

```yaml
agent:
  reliabilityMetrics:
    enabled: true
    obi:
      sizingProfile: "xlarge"        # Set ceiling for busiest pool
      dynamicSizing: true            # Auto-tune down on quiet pools
```

--------------------------------

### Configure IAM Role for Existing Service Account

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-aws.md

Steps to associate an OIDC provider and create an IAM role for an existing Kvisor service account.

```bash
export CLUSTER_NAME="your-eks-cluster"
export NAMESPACE="kvisor"
export SERVICE_ACCOUNT_NAME="kvisor"
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME} --region ${AWS_REGION} --approve

export OIDC_PROVIDER=$(aws eks describe-cluster --name ${CLUSTER_NAME} --region ${AWS_REGION} \
  --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

cat > trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}",
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name KvisorVPCReaderRole \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name KvisorVPCReaderRole \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${POLICY_NAME}

export ROLE_ARN=$(aws iam get-role --role-name KvisorVPCReaderRole --query 'Role.Arn' --output text)
echo "Role ARN: ${ROLE_ARN}"
```

--------------------------------

### Check Migration Job Logs

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Use this command to retrieve logs for the migration job. If the job failed, investigate ClickHouse connectivity, credentials, and logs.

```bash
kubectl logs -l app.kubernetes.io/component=migrate -n castai-agent
```

--------------------------------

### Cleanup Service Account Resources

Source: https://github.com/castai/kvisor/blob/main/pkg/cloudprovider/gcp/test/DEBUG.md

Commands to list, delete keys, and remove the service account after testing is complete.

```bash
# List keys
gcloud iam service-accounts keys list \
    --iam-account=kvisor-integration-test@YOUR_PROJECT_ID.iam.gserviceaccount.com

# Delete key
gcloud iam service-accounts keys delete KEY_ID \
    --iam-account=kvisor-integration-test@YOUR_PROJECT_ID.iam.gserviceaccount.com

# Delete service account
gcloud iam service-accounts delete \
    kvisor-integration-test@YOUR_PROJECT_ID.iam.gserviceaccount.com
```

--------------------------------

### Port-forward Server API

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to expose the Kvisord server API locally.

```sh
kubectl port-forward svc/kvisord-server 6060:80 -n kvisord
```

--------------------------------

### Run E2E Tests

Source: https://context7.com/castai/kvisor/llms.txt

Executes end-to-end tests on a local Kind cluster and verifies IPv6 configuration.

```bash
# Run E2E tests on local Kind cluster
KIND_CONTEXT=kind IMAGE_TAG=local ./e2e/run.sh

# Verify IPv6 is enabled (required)
sudo sysctl net.ipv6.conf.all.disable_ipv6
# Expected: net.ipv6.conf.all.disable_ipv6 = 0
```

--------------------------------

### Connect to GKE Cluster

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to fetch credentials for the GKE cluster.

```sh
gcloud container clusters get-credentials $CLUSTER_NAME --zone us-central1-c --project $GCP_PROJECT
```

--------------------------------

### Use a Pre-existing ConfigMap for Static CIDRs

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-static-cidrs.md

Mount an external ConfigMap containing static CIDR definitions into the kvisor controller.

```yaml
controller:
  extraArgs:
    cloud-provider-static-cidrs-file: /etc/kvisor/my-cidrs/static-cidrs.yaml

  extraVolumes:
    - name: my-static-cidrs
      configMap:
        name: my-existing-configmap

  extraVolumeMounts:
    - name: my-static-cidrs
      mountPath: /etc/kvisor/my-cidrs
      readOnly: true
```

```yaml
staticCIDRMappings:
  - cidr: "10.1.0.0/24"
    zone: "us-east-1a"
    region: "us-east-1"
    name: "production-vpc"
    kind: "VPC"
    connectivityMethod: "TransitGateway"
```

--------------------------------

### Create GKE Cluster

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to provision a GKE cluster using gcloud.

```sh
export GCP_PROJECT="my-project"
export CLUSTER_NAME="my-cluster-name"
gcloud beta container --project $GCP_PROJECT \
  clusters create $CLUSTER_NAME \
  --zone "us-central1-c" \
  --cluster-version "1.25.8-gke.500" \
  --machine-type "e2-small" \
  --disk-type "pd-balanced" \
  --disk-size "100" \
  --num-nodes "2" \
  --node-locations "us-central1-c"
```

--------------------------------

### Run pwru via Docker

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Executes pwru in a privileged container to trace network packets.

```sh
docker run --privileged --rm -t --pid=host -v /sys/kernel/debug/:/sys/kernel/debug/ cilium/pwru pwru --output-tuple 'host 1.1.1.1'
```

--------------------------------

### Configure Kvisor Controller Helm Values

Source: https://context7.com/castai/kvisor/llms.txt

Configures the Kvisor controller Deployment for security scanning, CIS benchmarks, and cloud provider integration.

```yaml
controller:
  enabled: true
  replicas: 1

  # Resource allocation
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
    limits:
      memory: 512Mi

  # Listen ports
  httpListenPort: 8080
  kubeAPIListenPort: 8090
  metricsHTTPListenPort: 6060

  # Feature flags
  extraArgs:
    log-level: info

    # Image vulnerability scanning
    image-scan-enabled: "true"
    image-scan-interval: "24h"
    image-scan-parallel-scans: "5"

    # CIS Kubernetes benchmark
    kube-bench-enabled: "true"
    kube-bench-scan-interval: "24h"

    # Kubernetes linting
    kube-linter-enabled: "true"
    kube-linter-scan-interval: "1h"

    # Cloud provider integration
    cloud-provider: "aws"  # or "gcp"
    cloud-provider-vpc-sync-enabled: "true"
    cloud-provider-vpc-name: "vpc-xxxxxxxxx"
    cloud-provider-vpc-sync-interval: "1h"
```

--------------------------------

### Custom Kvisor Resource Sizing

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Use the 'custom' profile to explicitly define memory requests and limits for the agent. Ensure 'reliabilityMetrics' and 'obi' are enabled.

```yaml
agent:
  reliabilityMetrics:
    enabled: true
    obi:
      sizingProfile: "custom"
      openPorts: "8080,8443,8090,6379"
      resources:
        requests:
          memory: 400Mi
        limits:
          memory: 800Mi
```

--------------------------------

### Verify OBI Instrumentation

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

Check logs for OBI instrumentation confirmation.

```bash
POD=$(kubectl get pods -n castai-agent \
  -l app.kubernetes.io/name=castai-kvisor-agent -o name | head -1)

kubectl logs $POD -n castai-agent -c obi | grep 'instrumenting'
```

--------------------------------

### Push Helm Chart to Registry

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to package and push a Helm chart to the GCP Artifact Registry.

```sh
helm package ./charts/kvisord
helm push kvisord-0.1.6.tgz oci://us-east4-docker.pkg.dev/kvisor/helm-charts
```

--------------------------------

### Configure OBI Dynamic Sizing in Helm

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Enable dynamic sizing and set the sizing profile in your Helm values to allow OBI to scan node processes for memory recommendations.

```yaml
agent:
  reliabilityMetrics:
    enabled: true
    obi:
      sizingProfile: "xlarge"        # Set ceiling for busiest node
      dynamicSizing: true            # Scan processes + warn if limit too low
      openPorts: "8080,8443,8090,6379"
```

--------------------------------

### Generate network connections

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Tools for simulating network traffic to test flow capture.

```sh
docker run -it --rm -p 8000:8000 busybox nc -lk -p 8000
```

```sh
for i in {1..50}; do echo "hello$i" | nc -q 0 localhost 8000 & done
```

```sh
cd ./pkg/ebpftracer
NC_ADDR=localhost:8000 go test -v -count=1 . -run=TestGenerateConn
```

--------------------------------

### Configure Static CIDRs via Helm

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-static-cidrs.md

Add static CIDR mappings to the controller netflow configuration in values.yaml.

```yaml
controller:
  netflow:
    staticCIDRs:
      mappings:
        - cidr: "10.1.0.0/24"
          zone: "us-east-1a"
          region: "us-east-1"
          name: "production-vpc"
          kind: "VPC"
          connectivityMethod: "TransitGateway"

        - cidr: "10.0.0.13/32"
          zone: "us-east-1a"
          region: "us-east-1"
          name: "production-rds"
          kind: "RDS"
          connectivityMethod: "PrivateLink"

        - cidr: "10.2.0.0/16"
          region: "us-east-1"
          name: "peered-vpc"
          kind: "VPC"
          connectivityMethod: "VPCPeering"
```

--------------------------------

### Mount Tracepoint Events

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Commands to mount debugfs and list available tracepoints.

```sh
mount -t debugfs none /sys/kernel/debug
ls /sys/kernel/debug/tracing/events
```

--------------------------------

### Run E2E Tests

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Command to execute end-to-end tests on a local Kind cluster.

```sh
KIND_CONTEXT=kind IMAGE_TAG=local ./e2e/run.sh
```

--------------------------------

### Combine Static Mappings with Cloud Discovery

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-static-cidrs.md

Configure static mappings alongside cloud provider discovery, where static entries override cloud API data.

```yaml
controller:
  extraArgs:
    cloud-provider: aws
    cloud-provider-vpc-sync-enabled: true
    cloud-provider-vpc-name: "vpc-0123456789abcdef0"

  netflow:
    staticCIDRs:
      mappings:
        # This /32 overrides whatever the cloud API says about 10.0.0.13
        - cidr: "10.0.0.13/32"
          region: "us-east-1"
          name: "cross-account-rds"
          connectivityMethod: "TransitGateway"
```

--------------------------------

### Troubleshoot OOMKilled Pods

Source: https://github.com/castai/kvisor/blob/main/docs/obi-sizing.md

Identify node-level container density to determine if memory limits are exceeded.

```bash
# Check which node the pod was on
kubectl get pod <pod> -o wide

# Count containers with matching ports on that node
kubectl get pods -A --field-selector=spec.nodeName=<node>,status.phase=Running \
  -o jsonpath='{range .items[*]}{range .spec.containers[*]}{range .ports[*]}{.containerPort}{"\n"}{end}{end}{end}' \
  | grep -cE '^(8080|8443|8090|6379)$'
```

--------------------------------

### Configure Kvisor Agent Helm Values

Source: https://context7.com/castai/kvisor/llms.txt

Defines resource limits, eBPF feature flags, and security context for the Kvisor agent DaemonSet.

```yaml
agent:
  enabled: true

  # Resource allocation
  resources:
    requests:
      memory: 64Mi
    limits:
      memory: 512Mi

  # Priority class for node-critical workloads
  priorityClass: "system-node-critical"

  # Container runtime socket path
  containerdSocketPath: "/run/containerd/containerd.sock"

  # Disable containerd for CRI-O clusters
  disableContainerd: false

  # eBPF feature flags
  extraArgs:
    log-level: info

    # Network flow monitoring
    netflow-enabled: "true"
    netflow-export-interval: "15s"

    # eBPF event collection
    ebpf-events-enabled: "true"
    ebpf-events-policy: "default"

    # Container stats collection
    stats-enabled: "true"
    stats-scrape-interval: "15s"

    # Process tree tracking
    process-tree-enabled: "true"

  # Security context for eBPF capabilities
  containerSecurityContext:
    privileged: false
    runAsNonRoot: false
    allowPrivilegeEscalation: false
    capabilities:
      drop: ["ALL"]
      add: []  # Dynamically added based on features
    seccompProfile:
      type: Unconfined
    seLinuxOptions:
      level: s0
      type: spc_t

  # GPU metrics collection
  gpu:
    enabled: false
    exportInterval: "15s"
    dcgmExporter:
      image:
        repository: nvcr.io/nvidia/k8s/dcgm-exporter
        tag: "4.5.2-4.8.1-ubuntu22.04"
```

--------------------------------

### Data Batching Structures

Source: https://context7.com/castai/kvisor/llms.txt

Structures for batching security data items, including container stats, netflows, and process events.

```protobuf
// Example: Writing a data batch with container events and netflows
message WriteDataBatchRequest {
  repeated DataBatchItem items = 10;
}

message DataBatchItem {
  oneof data {
    ContainerStats container_stats = 10;
    NodeStats node_stats = 11;
    ContainerEvents container_events = 12;
    Netflow netflow = 13;
    ProcessTreeEvent process_tree = 14;
  }
}
```

--------------------------------

### Print eBPF Logs

Source: https://github.com/castai/kvisor/blob/main/DEVELOPMENT.md

Code to print logs from eBPF and the command to view them.

```c
bpf_printk("called");
```

```sh
cat /sys/kernel/debug/tracing/trace_pipe
```

--------------------------------

### Create IAM Policy for AssumeRole using AWS CLI

Source: https://github.com/castai/kvisor/blob/main/docs/cloud-aws-transit-gateway.md

This bash script creates a JSON policy file and then uses the AWS CLI to attach it to the KvisorVPCReaderRole. It cleans up the policy file afterward. Run this in the cluster account.

```bash
cat > assume-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::*:role/KvisorCrossAccountReader"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name KvisorVPCReaderRole \
  --policy-name KvisorCrossAccountAssumePolicy \
  --policy-document file://assume-policy.json

rm assume-policy.json
```

--------------------------------

### Minimal Cast AI Kvisor Configuration

Source: https://github.com/castai/kvisor/blob/main/docs/reliability-stack-installation.md

This YAML file represents the minimal configuration required for Cast AI Kvisor, including essential API credentials and enabling reliability metrics.

```yaml
castai:
  apiKey: <your-api-key>
  clusterID: <your-cluster-id>
  grpcAddr: <grpc-endpoint>

agent:
  reliabilityMetrics:
    enabled: true

controller:
  reliabilityMetrics:
    enabled: true

reliabilityMetrics:
  enabled: true
  install:
    enabled: true
  exporter:
    enabled: true
```