### Install Helm Chart with Inline Config File Source: https://github.com/minio/warp/blob/master/k8s/helm/CONFIG.md Example command to install the Warp Helm chart by providing the configuration file content inline using `--set-file`. ```bash helm install my-warp ./k8s/helm --set-file configFile=my-warp-config.yml ``` -------------------------------- ### Install Helm Chart with Custom Config File Source: https://github.com/minio/warp/blob/master/k8s/helm/CONFIG.md Example command to install the Warp Helm chart using a custom configuration file specified via `-f`. ```bash helm install my-warp ./k8s/helm -f values-configfile-example.yaml ``` -------------------------------- ### Install Warp Helm Chart with Custom Configuration Source: https://github.com/minio/warp/blob/master/k8s/helm/README.md Install the Warp Helm chart using a custom configuration file. This can be an example configuration or a user-provided YAML file. ```bash # Using the example configuration with full YAML config helm install warp helm/ -f helm/values-configfile-example.yaml # Or provide your own YAML config file helm install warp helm/ --set-file configFile=my-warp-config.yml ``` -------------------------------- ### Run GolangCI-Lint Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute the golangci-lint tool with a specified timeout and configuration file. Ensure golangci-lint is installed. ```bash # Run golangci-lint (requires installation) golangci-lint run --timeout=5m --config ./.golangci.yml ``` -------------------------------- ### Start Warp Client Source: https://github.com/minio/warp/blob/master/README.md Starts a Warp client instance. It accepts an optional host/IP to listen on. By default, it listens on 127.0.0.1:7761. Ensure clients are not run on publicly exposed ports. ```bash λ warp client [listenaddress:port] ``` -------------------------------- ### Install Warp Helm Chart with Custom Configuration Source: https://github.com/minio/warp/blob/master/k8s/README.md Install the Warp Helm chart using a custom configuration file or by setting the configFile parameter directly. ```bash # Using the example configuration file helm install warp -n warp --create namespace ./helm -f ./helm/values-configfile-example.yaml # Or provide your own YAML config file helm install warp -n warp --create namespace ./helm --set-file configFile=my-warp-config.yml ``` -------------------------------- ### Distributed Iceberg Benchmarking Setup Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Set up and run distributed Iceberg benchmarks across multiple warp clients. Start clients on separate machines and then run the benchmark from the server, specifying client ranges. ```bash # Start clients on separate machines warp client :7761 # Run distributed benchmark from server warp iceberg catalog-read \ --warp-client=client-{1...4}:7761 \ --host=catalog-server:9001 \ --access-key=minioadmin \ --secret-key=minioadmin \ --concurrent=50 \ --duration=5m ``` -------------------------------- ### Use 64-bit Atomic Operations Source: https://github.com/minio/warp/blob/master/CLAUDE.md Example demonstrating the use of `atomic.AddUint64` for 64-bit atomic operations, crucial for 32-bit architectures to avoid issues. ```go atomic.AddUint64 ``` -------------------------------- ### Warp Analysis Output Example Source: https://github.com/minio/warp/blob/master/README.md Example output showing aggregated throughput data per host. Use `--analyze.v` to enable this view. ```text Throughput by host: * http://127.0.0.1:9001: Avg: 81.48 MiB/s, 81.48 obj/s (4m59.976s) - Fastest: 86.46 MiB/s, 86.46 obj/s (1s) - 50% Median: 82.23 MiB/s, 82.23 obj/s (1s) - Slowest: 68.14 MiB/s, 68.14 obj/s (1s) * http://127.0.0.1:9002: Avg: 81.48 MiB/s, 81.48 obj/s (4m59.968s) - Fastest: 87.36 MiB/s, 87.36 obj/s (1s) - 50% Median: 82.28 MiB/s, 82.28 obj/s (1s) - Slowest: 68.40 MiB/s, 68.40 obj/s (1s) ``` -------------------------------- ### Run Warp Benchmark Source: https://github.com/minio/warp/blob/master/README.md Execute the built Warp binary with desired options to start a benchmark test. ```bash λ ./warp [options] ``` -------------------------------- ### Synchronize Distributed Benchmarks Source: https://github.com/minio/warp/blob/master/README.md When running benchmarks across multiple clients, use `--syncstart` to synchronize their start times for more reliable merging of results. The time format is 'hh:mm' in local computer time. ```bash λ warp benchmark --syncstart 14:30 ... ``` -------------------------------- ### Example Benchmark Analysis Throughput Source: https://github.com/minio/warp/blob/master/README.md Illustrates the typical analysis output for throughput, showing average, fastest, median, and slowest performance metrics. ```text Operation: GET * Average: 92.05 MiB/s, 9652.01 obj/s ``` -------------------------------- ### Install Warp Helm Chart Source: https://github.com/minio/warp/blob/master/k8s/helm/README.md Install the Warp Helm chart after configuring the values.yaml file. This command deploys the chart from the local helm directory. ```bash cd /home/warp/k8s/ helm install warp helm/ ``` -------------------------------- ### Get Objects with Multi-NIC Support Source: https://github.com/minio/warp/blob/master/README.md Use this command to fetch objects from multiple hosts, enabling automatic source-binding based on the OS routing table for each destination host. ```bash λ warp get --host=192.168.11.1:9000,192.168.12.1:9000 ... ``` -------------------------------- ### Lint and Format Code with Go Tools Source: https://github.com/minio/warp/blob/master/CLAUDE.md Utilize golangci-lint for comprehensive linting and gofmt for code formatting. Ensure golangci-lint is installed and configured. ```bash # Run golangci-lint (requires installation) golangci-lint run --timeout=5m --config ./.golangci.yml # Run go vet go vet ./... # Check formatting gofmt -d . # Format code gofmt -w . ``` -------------------------------- ### Example Segmented Throughput Analysis Source: https://github.com/minio/warp/blob/master/README.md Demonstrates the analysis of benchmark throughput divided into fixed duration segments, highlighting fastest, median, and slowest segments. ```text Throughput, split into 59 x 1s: * Fastest: 97.9MiB/s, 10269.68 obj/s * 50% Median: 95.1MiB/s, 9969.63 obj/s * Slowest: 66.3MiB/s, 6955.70 obj/s ``` -------------------------------- ### Analyze GET Operations with Detailed Timings Source: https://github.com/minio/warp/blob/master/README.md Enable detailed analysis of GET operations, including timings for the first byte received and response times for uploads. This is accessed using the --analyze.v parameter. ```bash --analyze.v ``` -------------------------------- ### Format Code with gofumpt or gofmt Source: https://github.com/minio/warp/blob/master/CLAUDE.md Format the code using gofumpt for enhanced formatting rules. If gofumpt is unavailable, fall back to gofmt and provide installation instructions for gofumpt. ```bash # Format (preferred): gofumpt -extra -w . # If gofumpt is unavailable, fall back: gofmt -w . # and surface install steps for gofumpt (e.g., go install mvdan.cc/gofumpt@latest or brew install gofumpt). ``` -------------------------------- ### Benchmark GET Operations Source: https://github.com/minio/warp/blob/master/README.md Benchmark GET operations by downloading objects within a specified duration. By default, objects are uploaded before the benchmark. This includes upload stats as PUT operations and GET operations in the analysis. ```bash Operation: GET * Average: 94.10 MiB/s, 9866.97 obj/s Throughput, split into 299 x 1s: * Fastest: 99.8MiB/s, 10468.54 obj/s * 50% Median: 94.4MiB/s, 9893.37 obj/s * Slowest: 69.4MiB/s, 7279.03 obj/s ``` -------------------------------- ### Run Mixed Benchmark with Specific Hosts Source: https://github.com/minio/warp/blob/master/README.md Example of running a mixed benchmark against multiple S3 servers, specifying hosts, access keys, secret keys, and enabling auto-termination. ```bash λ warp mixed --host=s3-server{1...8}:9000 --access-key=minio --secret-key=minio123 --autoterm ``` -------------------------------- ### Warp Per Request Statistics Example Source: https://github.com/minio/warp/blob/master/README.md Detailed per-request statistics, including averages, percentiles, and host-specific breakdowns. Enable with `--analyze.v`. ```text Operation: GET (386413). Ran 1m0s. Concurrency: 20. Hosts: 2. Requests considered: 386334: * Avg: 3ms, 50%: 3ms, 90%: 4ms, 99%: 8ms, Fastest: 1ms, Slowest: 504ms * TTFB: Avg: 3ms, Best: 1ms, 25th: 3ms, Median: 3ms, 75th: 3ms, 90th: 4ms, 99th: 8ms, Worst: 504ms * First Access: Avg: 3ms, 50%: 3ms, 90%: 4ms, 99%: 10ms, Fastest: 1ms, Slowest: 18ms * First Access TTFB: Avg: 3ms, Best: 1ms, 25th: 3ms, Median: 3ms, 75th: 3ms, 90th: 4ms, 99th: 10ms, Worst: 18ms * Last Access: Avg: 3ms, 50%: 3ms, 90%: 4ms, 99%: 7ms, Fastest: 2ms, Slowest: 10ms * Last Access TTFB: Avg: 3ms, Best: 1ms, 25th: 3ms, Median: 3ms, 75th: 3ms, 90th: 4ms, 99th: 7ms, Worst: 10ms Requests by host: * http://127.0.0.1:9001 - 193103 requests: - Avg: 3ms Fastest: 1ms Slowest: 504ms 50%: 3ms 90%: 4ms - First Byte: Avg: 3ms, Best: 1ms, 25th: 3ms, Median: 3ms, 75th: 3ms, 90th: 4ms, 99th: 8ms, Worst: 504ms * http://127.0.0.1:9002 - 193310 requests: - Avg: 3ms Fastest: 1ms Slowest: 88ms 50%: 3ms 90%: 4ms - First Byte: Avg: 3ms, Best: 1ms, 25th: 3ms, Median: 3ms, 75th: 3ms, 90th: 4ms, 99th: 8ms, Worst: 88ms Throughput: * Average: 1.57 MiB/s, 6440.36 obj/s Throughput by host: * http://127.0.0.1:9001: - Average: 0.79 MiB/s, 3218.47 obj/s - Fastest: 844.5KiB/s - 50% Median: 807.9KiB/s - Slowest: 718.9KiB/s * http://127.0.0.1:9002: - Average: 0.79 MiB/s, 3221.85 obj/s - Fastest: 846.8KiB/s - 50% Median: 811.0KiB/s - Slowest: 711.1KiB/s Throughput, split into 59 x 1s: * Fastest: 1688.0KiB/s, 6752.22 obj/s (1s, starting 12:31:40 CET) * 50% Median: 1621.9KiB/s, 6487.60 obj/s (1s, starting 12:31:17 CET) * Slowest: 1430.5KiB/s, 5721.95 obj/s (1s, starting 12:31:59 CET) ``` -------------------------------- ### Legacy Warp Configuration using warpConfiguration Source: https://github.com/minio/warp/blob/master/k8s/helm/CONFIG.md Traditional method using individual configuration fields in `values.yaml`. Suitable for basic setups where hostname is compatible with ellipsis notation. ```yaml warpConfiguration: s3ServerURL: minio-{0...3}.minio.default.svc.cluster.local:9000 s3ServerTLSEnabled: false s3ServerRegion: "us-east-1" s3AccessKey: "minio" s3SecretKey: "minio123" operationToBenchmark: get warpJobArgs: objects: 1000 obj.size: 10MiB duration: 5m0s concurrent: 10 ``` -------------------------------- ### Clone Warp Repository Source: https://github.com/minio/warp/blob/master/README.md Clone the Warp project repository from GitHub to get the source code. ```bash λ git clone https://github.com/minio/warp.git ``` -------------------------------- ### Configure Concurrent Downloads Source: https://github.com/minio/warp/blob/master/README.md Set the number of concurrent downloads for GET operations. Objects are uploaded with a specified number of concurrent prefixes, unless --noprefix is specified. ```bash --concurrent ``` -------------------------------- ### Run Mixed Operations Benchmark Source: https://github.com/minio/warp/blob/master/README.md Execute a mixed operations benchmark for a specified duration. This tests several operation types at once, with configurable distributions for GET, STAT, PUT, and DELETE operations. ```bash λ warp mixed --duration=1m ``` -------------------------------- ### Benchmark S3 Zip Extension Source: https://github.com/minio/warp/blob/master/README.md Benchmarks the MinIO s3zip extension for creating zip archives. Uploads a single zip file containing a specified number of individual files of a given size. It then downloads individual files concurrently and presents the result as a GET benchmark. This feature requires recent MinIO versions (2022 onwards). ```bash λ warp zip --obj.size=1MiB -duration=1m ``` -------------------------------- ### Build Warp from Source Source: https://github.com/minio/warp/blob/master/README.md Navigate into the cloned directory and build the Warp executable using Go. ```bash λ cd warp && go build ``` -------------------------------- ### Build and Run Warp Binary Source: https://github.com/minio/warp/blob/master/AGENTS.md Build the Warp binary and then run it with commands and options. Ensure you are in the root of the code tree. ```bash # Build the binary go build # Run the binary ./warp [command] [options] ``` -------------------------------- ### Run GolangCI-Lint (Fallback GOPATH) Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute golangci-lint using its path from GOPATH. This is a fallback if the linter is not directly in the system's PATH. ```bash # Fallback GOPATH if needed: `$(go env GOPATH)/bin/golangci-lint run -j16` $(go env GOPATH)/bin/golangci-lint run -j16 ``` -------------------------------- ### Run golangci-lint Linter Source: https://github.com/minio/warp/blob/master/CLAUDE.md Execute the golangci-lint linter using the repository's configuration. This command prefers using the linter from the system's PATH. ```bash # Prefer PATH: golangci-lint run -j16 # Fallback GOPATH if needed: $(go env GOPATH)/bin/golangci-lint run -j16 ``` -------------------------------- ### Configure Benchmark Server for Multi-NIC Clients Source: https://github.com/minio/warp/blob/master/README.md Lists all client addresses across all NICs on the benchmark server. This configuration ensures that the server correctly targets clients connected via different network interfaces. ```bash λ warp get --duration=3m \ --warp-client=192.168.11.{1...9}:7761,192.168.12.{1...9}:7761 \ --host=192.168.11.{1...8}:9000,192.168.12.{1...8}:9000 \ --access-key=minio --secret-key=minio123 ``` -------------------------------- ### List and Download Existing Objects Source: https://github.com/minio/warp/blob/master/README.md Use the --list-existing option to list objects in a bucket and download them instead of uploading random objects. Set to 0 to use all objects from the listing. Listing can be restricted by --prefix and --list-flat disables recursive listing. ```bash --list-existing ``` -------------------------------- ### Run Multipart Benchmark Source: https://github.com/minio/warp/blob/master/README.md Executes a multipart benchmark to upload parts to a single object and then tests download speed. Recommended to have `--parts` be a multiple of `--concurrent`. ```bash λ warp multipart --parts=500 --part.size=10MiB ``` -------------------------------- ### Run Benchmark with Variable Injection Source: https://github.com/minio/warp/blob/master/README.md Demonstrates how to inject variables into a YAML configuration file during benchmark execution using the -var flag. ```bash λ warp run -var VarName=Value ``` -------------------------------- ### Run Default Iceberg Catalog Read Benchmark Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Executes the default catalog read benchmark with specified host and credentials. ```bash warp iceberg catalog-read \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin ``` -------------------------------- ### Run Go Vet Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute the go vet static analysis tool across all packages in the project to find suspicious constructs. ```bash # Run go vet go vet ./... ``` -------------------------------- ### Run Benchmark from YAML Configuration Source: https://github.com/minio/warp/blob/master/README.md Execute a benchmark defined in a YAML configuration file. Values can be injected via command line. ```bash λ warp run ``` -------------------------------- ### Benchmark STAT Operations Source: https://github.com/minio/warp/blob/master/README.md Benchmark STAT operations to measure the performance of retrieving object metadata. Objects are uploaded, and then individual requests are made to get object information. Only objects per second is reported. ```bash λ warp stat --autoterm [...] ------------------- Operation: STAT * Average: 10.06 MiB/s, 1030.01 obj/s Throughput, split into 59 x 1s: * Fastest: 11.3MiB/s, 1159.69 obj/s * 50% Median: 10.3MiB/s, 1059.06 obj/s * Slowest: 6.7MiB/s, 685.26 obj/s ``` -------------------------------- ### Run Snowball Benchmark with Default Parameters Source: https://github.com/minio/warp/blob/master/README.md Executes the Snowball benchmark with default settings, uploading compressed TAR files containing duplicated objects. Useful for testing compressed uploads. ```bash λ warp snowball --duration=30s --compress ``` -------------------------------- ### Run Basic Iceberg Catalog Commits Benchmark Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Executes a basic commit benchmark for Iceberg catalog operations. Requires host, access key, and secret key. ```bash # Basic commit benchmark warp iceberg catalog-commits \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin ``` -------------------------------- ### Run Go Tests with Race Detection Source: https://github.com/minio/warp/blob/master/CLAUDE.md Execute all tests in the project with verbose output and race detection enabled. For specific packages, adjust the path accordingly. ```bash # Run all tests with race detection go test -v -race ./... # Run tests for a specific package go test -v ./pkg/bench go test -v ./pkg/aggregate ``` -------------------------------- ### Run Multipart Put Benchmark Source: https://github.com/minio/warp/blob/master/README.md Tests upload speed by creating multipart uploads, uploading specified parts, and completing the upload. Each upload splits into `--part.concurrent` threads, resulting in a total concurrency of `--concurrent` multiplied by `--part.concurrent`. ```bash λ warp multipart-put --parts 100 --part.size 5MiB ``` -------------------------------- ### Run All Go Tests with Race Detection Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute all tests in the project with the race detector enabled for thorough testing. This command should be run from the project root. ```bash # Run all tests with race detection go test -v -race ./... ``` -------------------------------- ### Configure Multiple Catalog Hosts Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Specify multiple catalog hosts for load balancing using comma-separated values or expandable patterns. ```bash --host=host1:9001,host2:9001,host3:9001 ``` ```bash --host=host{1...10}:9001 ``` -------------------------------- ### Run Fanout Benchmark with Custom Copies and Object Size Source: https://github.com/minio/warp/blob/master/README.md Executes the Fanout benchmark, copying a 512KB object to 50 locations concurrently using 8 uploads. Ideal for testing MinIO's fan-out capabilities with specific object sizes and copy counts. ```bash λ warp fanout --copies=50 --obj.size=512KiB --concurrent=8 ``` -------------------------------- ### Analyze and Compare Benchmark Results Source: https://github.com/minio/warp/blob/master/AGENTS.md Commands to analyze, compare, and merge benchmark results. Output files are Zstandard compressed CSVs. ```bash warp analyze ``` ```bash warp cmp ``` ```bash warp merge ... ``` -------------------------------- ### Format Go Code with gofumpt Source: https://github.com/minio/warp/blob/master/AGENTS.md Format Go source files using gofumpt, which is a stricter formatter than gofmt. The `-w` flag writes changes in place. ```bash # Format (preferred): `gofumpt -extra -w .` gofumpt -extra -w . ``` -------------------------------- ### Run Benchmark from YAML File Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute benchmarks defined in a YAML file. Variables can be injected using the -var flag. ```bash warp run ``` ```bash warp run file.yml -var VarName=Value ``` -------------------------------- ### Configure Multi-NIC Warp Clients Source: https://github.com/minio/warp/blob/master/README.md Sets up Warp client processes on machines with multiple Network Interface Cards (NICs). Run one `warp client` process per NIC, specifying the listen IP explicitly to ensure traffic is routed exclusively via the assigned NIC. ```bash # On each client machine (e.g. with NICs 192.168.11.2 and 192.168.12.2): warp client 192.168.11.2:7761 warp client 192.168.12.2:7761 ``` -------------------------------- ### Iceberg Write Benchmark Source: https://github.com/minio/warp/blob/master/README.md Command to run benchmarks for uploading Parquet files and committing them as Iceberg tables. ```bash warp iceberg write ``` -------------------------------- ### Format and Lint Go Code Source: https://github.com/minio/warp/blob/master/AGENTS.md A combined command to format Go code using gofumpt and then lint it using golangci-lint. This is recommended after code edits. ```bash cd && gofumpt -extra -w . && golangci-lint run -j16 ``` -------------------------------- ### Run Warp S3 Benchmark Source: https://github.com/minio/warp/blob/master/wui/static/index.html Execute the Warp benchmark tool with default parameters. This command initiates a comprehensive S3 benchmark run. ```bash warp ``` -------------------------------- ### Format Go Code with gofmt (Fallback) Source: https://github.com/minio/warp/blob/master/AGENTS.md Fallback command to format Go source files using gofmt when gofumpt is unavailable. The `-w` flag writes changes in place. ```bash # Fallback: `gofmt -w .` gofmt -w . ``` -------------------------------- ### Benchmark Apache Polaris Catalog Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Configure and run benchmarks against an Apache Polaris catalog. Requires host, OAuth2 client credentials, external catalog details, and base S3 location. ```bash warp iceberg catalog-read \ --host=polaris.example.com:8181 \ --access-key=client_id \ --secret-key=client_secret \ --external-catalog=polaris \ --catalog-name=warehouse \ --base-location=s3://bucket ``` -------------------------------- ### Warp YAML Configuration using configFile Source: https://github.com/minio/warp/blob/master/k8s/helm/CONFIG.md Provides a complete YAML configuration for Warp benchmarks. This method is recommended for full feature access and complex configurations. ```yaml configFile: | warp: api: v1 benchmark: mixed remote: region: us-east-1 access-key: 'minio' secret-key: 'minio123' host: - 'minio-{0...3}.minio.default.svc.cluster.local:9000' tls: false params: duration: 5m concurrent: 16 objects: 2500 obj: size: 10MiB ``` -------------------------------- ### Format Go Code Source: https://github.com/minio/warp/blob/master/AGENTS.md Use gofmt to automatically format Go source files in the current directory. The `-w` flag writes the changes back to the files. ```bash # Format code gofmt -w . ``` -------------------------------- ### Run Iceberg Catalog Read Benchmark with Heavy Table Reads Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Configures the catalog read benchmark to prioritize table reads by adjusting distribution flags. ```bash warp iceberg catalog-read \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin \ --table-get-distrib=50 \ --table-list-distrib=20 ``` -------------------------------- ### Analyze and Compare Benchmark Results Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Analyze, compare, and merge benchmark result files. Use 'analyze' for single file inspection, 'cmp' for comparing two runs, and 'merge' for combining results from multiple clients. ```bash # Analyze results warp analyze warp-operation-2024-01-15[120000]-AbCd.csv.zst ``` ```bash # Compare two runs warp cmp before.csv.zst after.csv.zst ``` ```bash # Merge results from multiple clients warp merge client1.csv.zst client2.csv.zst ``` -------------------------------- ### Create Warp Benchmark Job Source: https://github.com/minio/warp/blob/master/k8s/README.md Apply a warp-job.yaml file to create a benchmark job. Ensure the job definition is tailored to your specific benchmarking needs. ```bash ~ kubectl create -f warp-job.yaml ``` -------------------------------- ### Iceberg Catalog Read Benchmark Source: https://github.com/minio/warp/blob/master/README.md Command to run benchmarks specifically for Iceberg catalog read operations. ```bash warp iceberg catalog-read ``` -------------------------------- ### Benchmark PUT Operations Source: https://github.com/minio/warp/blob/master/README.md Use this to benchmark PUT operations. Objects are uploaded with specified size and concurrency until the duration elapses. Options like --noprefix, --md5, --post, and --checksum can modify behavior. ```bash Operation: PUT * Average: 10.06 MiB/s, 1030.01 obj/s Throughput, split into 59 x 1s: * Fastest: 11.3MiB/s, 1159.69 obj/s * 50% Median: 10.3MiB/s, 1059.06 obj/s * Slowest: 6.7MiB/s, 685.26 obj/s ``` -------------------------------- ### Configure Bucketed Object Sizes Source: https://github.com/minio/warp/blob/master/README.md Use a comma-separated string of colon-separated pairs (e.g., 'size:weight') for the `--obj.size` parameter to define buckets with uniform size distribution within each bucket. This preempts `--obj.randsize`. ```bash --obj.size 4096:10740,8192:1685,16384:1623 ``` -------------------------------- ### Iceberg Catalog Commits Benchmark Source: https://github.com/minio/warp/blob/master/README.md Command to run benchmarks for generating commits via property updates in Iceberg catalogs. ```bash warp iceberg catalog-commits ``` -------------------------------- ### Run Benchmark with Distributed Clients Source: https://github.com/minio/warp/blob/master/README.md Executes a benchmark using distributed Warp clients. Parameters specified apply to each client. Ensure clients have synchronized clocks for reliable results. The server coordinates benchmark runs and collects results. ```bash λ warp get --duration=3m --warp-client=client-{1...10} --host=minio-server-{1...16} --access-key=minio --secret-key=minio123 ``` -------------------------------- ### Analyze Benchmark Request Statistics Source: https://github.com/minio/warp/blob/master/README.md Use the `--analyze.v` parameter to view segmented request statistics. This command provides detailed breakdowns of throughput and first-byte times for different request size ranges. ```bash λ warp analyze --analyze.op=GET --analyze.v warp-get-2020-08-18[190338]-6Nha.csv.zst Operation: GET (78188). Concurrency: 32. Requests considered: 78123. Multiple sizes, average 1832860 bytes: Request size 1 B -> 10 KiB. Requests - 10836: * Throughput: Average: 1534.6KiB/s, 50%: 1571.9KiB/s, 90%: 166.0KiB/s, 99%: 6.6KiB/s, Fastest: 9.7MiB/s, Slowest: 1124.8B/s * First Byte: Average: 3ms, Median: 2ms, Best: 1ms, Worst: 39ms Request size 10KiB -> 1MiB. Requests - 38219: * Throughput: Average: 73.5MiB/s, 50%: 66.4MiB/s, 90%: 27.0MiB/s, 99%: 13.6MiB/s, Fastest: 397.6MiB/s, Slowest: 3.1MiB/s * First Byte: Average: 3ms, Median: 2ms, Best: 1ms, Worst: 41ms Request size 1MiB -> 10MiB. Requests - 33091: * Throughput: Average: 162.1MiB/s, 50%: 159.4MiB/s, 90%: 114.3MiB/s, 99%: 80.3MiB/s, Fastest: 505.4MiB/s, Slowest: 22.4MiB/s * First Byte: Average: 3ms, Median: 2ms, Best: 1ms, Worst: 40ms Throughput: * Average: 4557.04 MiB/s, 2604.96 obj/s (29.901s, starting 19:03:41 CEST) Throughput, split into 29 x 1s: * Fastest: 4812.4MiB/s, 2711.62 obj/s (1s, starting 19:03:41 CEST) * 50% Median: 4602.6MiB/s, 2740.27 obj/s (1s, starting 19:03:56 CEST) * Slowest: 4287.0MiB/s, 2399.84 obj/s (1s, starting 19:03:53 CEST) ``` -------------------------------- ### List Kubernetes Pods Source: https://github.com/minio/warp/blob/master/k8s/helm/README.md After successful deployment, use this command to list all pods and identify the Warp Job Pod for retrieving benchmark results. ```bash $ kubectl get pods NAME READY STATUS RESTARTS AGE warp-0 1/1 Running 0 11m warp-1 1/1 Running 0 11m warp-2 1/1 Running 0 11m warp-3 1/1 Running 0 11m warp-df9cs 0/1 Completed 0 11m ``` -------------------------------- ### Warp Helm Chart - Full YAML Configuration File Source: https://github.com/minio/warp/blob/master/k8s/README.md Use the configFile option in values.yaml to provide a complete YAML configuration for advanced control over Warp settings. ```yaml configFile: | warp: api: v1 benchmark: mixed remote: region: us-east-1 access-key: 'minioadmin' secret-key: 'minioadmin' host: - 'minio-{0...3}.minio.default.svc.cluster.local:9000' tls: true insecure: true params: duration: 5m concurrent: 16 objects: 2500 obj: size: 10MiB ``` -------------------------------- ### Run Go Tests for a Specific Package Source: https://github.com/minio/warp/blob/master/AGENTS.md Execute tests for a particular package within the project. Useful for targeted testing during development. ```bash # Run tests for a specific package go test -v ./pkg/bench go test -v ./pkg/aggregate ``` -------------------------------- ### Run Snowball Benchmark with Custom Object Count and Size Source: https://github.com/minio/warp/blob/master/README.md Configures the Snowball benchmark to upload TAR files containing a large number of small objects (1KB each) with two concurrent uploads. Useful for testing performance with many small objects. ```bash λ warp snowball --duration=60s --obj.size=1K --objs.per=1000 --concurrent=2 ``` -------------------------------- ### Configure InfluxDB Integration Source: https://github.com/minio/warp/blob/master/AGENTS.md Push real-time metrics to InfluxDB v2+ using the --influxdb flag. The connection string includes schema, token, hostname, port, bucket, org, and optional tags. ```bash --influxdb ://@://? ``` -------------------------------- ### Analyze Benchmark Data from File Source: https://github.com/minio/warp/blob/master/README.md Re-evaluates previously saved benchmark data from a specified file. Use this to analyze results after a benchmark run has completed. ```bash warp analyze (filename) ``` -------------------------------- ### Compare Benchmark Runs Source: https://github.com/minio/warp/blob/master/README.md Compare two recorded benchmark runs using the 'warp cmp' command. This helps identify differences in performance metrics between two states. The order of files does not imply chronological order. ```bash λ warp cmp warp-get-2019-11-29[125341]-7ylR.csv.zst warp-get-2019-202011-29[124533]-HOhm.csv.zst ``` -------------------------------- ### Verify Warp Pods Running Source: https://github.com/minio/warp/blob/master/k8s/README.md Check the status of Warp client pods using kubectl. Ensure all pods are in a 'Running' state before proceeding. ```bash ~ kubectl get pods -l app=warp NAME READY STATUS RESTARTS AGE warp-0 1/1 Running 0 6m53s warp-1 1/1 Running 0 6m57s warp-2 1/1 Running 0 7m8s warp-3 1/1 Running 0 7m17s ``` -------------------------------- ### Benchmark Versioned Object Operations Source: https://github.com/minio/warp/blob/master/README.md Configure the number of versions to add for each object when testing versioned listing. The default is 1. ```bash --versions=n ``` -------------------------------- ### Create Warp Client Listeners Source: https://github.com/minio/warp/blob/master/k8s/README.md Use kubectl to create Warp client listeners by applying a warp.yaml file. This is typically done to run distributed Warp benchmarks. ```bash ~ kubectl create -f warp.yaml ``` -------------------------------- ### Check Go Code Formatting Source: https://github.com/minio/warp/blob/master/AGENTS.md Use gofmt to check for formatting differences in the current directory. This command will report unformatted files without modifying them. ```bash # Check formatting gofmt -d . ``` -------------------------------- ### Warp Helm Chart - Traditional Configuration Source: https://github.com/minio/warp/blob/master/k8s/README.md Configure Warp using a simple values.yaml file for basic settings like S3 server URL, access key, and secret key. ```yaml warpConfiguration: s3ServerURL: minio-{0...3}.minio.default.svc.cluster.local:9000 s3AccessKey: "minio" s3SecretKey: "minio123" operationToBenchmark: mixed ``` -------------------------------- ### View Warp Job Pod Logs for Benchmark Results Source: https://github.com/minio/warp/blob/master/k8s/helm/README.md Retrieve the logs from the Warp Job Pod to view the benchmark results. This command shows the output of the completed job. ```bash $ kubectl logs warp-df9cs .... .... .... Operation: GET. Concurrency: 4. Hosts: 4. * Average: 448.97 MiB/s, 89.79 obj/s (4m59.883s, starting 14:01:09 UTC) Throughput by host: * http://minio-1.minio.default.svc.cluster.local:9000: Avg: 112.26 MiB/s, 22.45 obj/s (4m59.834s, starting 14:01:09 UTC) * http://minio-2.minio.default.svc.cluster.local:9000: Avg: 112.27 MiB/s, 22.45 obj/s (4m59.797s, starting 14:01:09 UTC) * http://minio-3.minio.default.svc.cluster.local:9000: Avg: 112.27 MiB/s, 22.45 obj/s (4m59.938s, starting 14:01:09 UTC) * http://minio-0.minio.default.svc.cluster.local:9000: Avg: 112.27 MiB/s, 22.45 obj/s (4m59.934s, starting 14:01:09 UTC) Aggregated Throughput, split into 299 x 1s time segments: * Fastest: 580.4MiB/s, 116.09 obj/s (1s, starting 14:01:11 UTC) * 50% Median: 471.9MiB/s, 94.38 obj/s (1s, starting 14:04:25 UTC) * Slowest: 189.4MiB/s, 37.87 obj/s (1s, starting 14:02:23 UTC) ``` -------------------------------- ### Iceberg Catalog Mixed Workload Benchmark Source: https://github.com/minio/warp/blob/master/README.md Command to run a mixed workload benchmark for Iceberg catalog operations, including reads and writes. ```bash warp iceberg catalog-mixed ``` -------------------------------- ### Run Default Mixed Iceberg Catalog Workload Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Executes a default mixed read/write workload for Iceberg catalog operations. This benchmark includes updates to tables and views. ```bash # Default mixed workload warp iceberg catalog-mixed \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin ``` -------------------------------- ### Sustained Iceberg Commit Benchmark Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Run a sustained benchmark for Iceberg table commits with options for simulating reads and uploads. Configure host, credentials, duration, concurrency, and request rates. ```bash # Sustained commits (1 every 2 sec) with 400 reads/sec warp iceberg sustained \ --host=localhost:9000 \ --access-key=minioadmin \ --secret-key=minioadmin \ --duration=1h \ --concurrent=1 \ --rps-limit=0.5 \ --simulate-read \ --read-concurrent=10 \ --read-rps-limit=400 ``` ```bash # Commit-only benchmark (no reads) warp iceberg sustained \ --host=localhost:9000 \ --access-key=minioadmin \ --secret-key=minioadmin \ --rps-limit=1 ``` ```bash # With uploads during benchmark (not just prepare) warp iceberg sustained \ --host=localhost:9000 \ --access-key=minioadmin \ --secret-key=minioadmin \ --skip-upload=false \ --duration=1h ``` -------------------------------- ### Test Partial File Requests with Range Option Source: https://github.com/minio/warp/blob/master/README.md Test the speed of partial file requests by reading objects from a random offset for a random number of bytes. This can be combined with --obj.randsize. ```bash --range ``` -------------------------------- ### Configure InfluxDB Connection Source: https://github.com/minio/warp/blob/master/README.md Use the `--influxdb` parameter or the `WARP_INFLUXDB_CONNECT` environment variable to specify the InfluxDB connection URL. Ensure all parts of the URL are correctly formatted and URL encoded if necessary. ```bash --influxdb "http://shmRUvVjk0Ig2J9qU0_g349PF6l-GB1dmwXUXDh5qd19n1Nda_K7yvSIi9tGpax9jyOsmP2dUd-md8yPOoDNHg==@127.0.0.1:8086/mybucket/myorg?mytag=myvalue" ``` -------------------------------- ### Run Iceberg Catalog Commits Benchmark for Tables Only Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Runs the commit benchmark focusing solely on table operations by disabling view updates. Set `--views-per-ns=0` to exclude views from the benchmark. ```bash # Tables only (no views) warp iceberg catalog-commits \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin \ --views-per-ns=0 ``` -------------------------------- ### Run Iceberg Catalog Commits Benchmark with Throughput Control Source: https://github.com/minio/warp/blob/master/README_ICEBERG.md Configures the commit benchmark with specific throughput for table and view updates. Use `--table-commits-throughput` and `--view-commits-throughput` to control worker distribution. ```bash # More table commits than view commits warp iceberg catalog-commits \ --host=localhost:9001 \ --access-key=minioadmin \ --secret-key=minioadmin \ --table-commits-throughput=15 \ --view-commits-throughput=5 ```