### whest dataset push examples Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Two examples: publishing with a version tag and creating a private repo. ```bash # Publish with a version tag whest dataset push ./my-eval \ --repo aicrowd/arc-whestbench-public-2026 \ --tag v1 \ --message "Bake: 10 MLPs, seed=42" ``` ```bash # Private repo whest dataset push ./my-eval \ --repo aicrowd/arc-whestbench-evals-2026 \ --tag v1 \ --private ``` -------------------------------- ### whest dataset download examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Two examples: prefetching into the HF cache and materialising an on-disk copy with `--output`. ```bash # Prefetch into the HF cache (no local copy) whest dataset download aicrowd/arc-whestbench-public-2026 \ --revision v1 ``` ```bash # Materialise an on-disk copy as well whest dataset download aicrowd/arc-whestbench-public-2026 \ --revision v1 \ --output ./eval-v1 ``` -------------------------------- ### Quick start: bake dataset on GPU Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/gpu-dataset-generation.mdx Shows three example commands for baking datasets with the torch backend: auto device detection, a production-realistic example on CUDA, and a CPU development example. The auto device command warns it takes ~4 hours on L40S and ~14 h on M3 Max, and recommends calibration first. ```bash # Auto-detect best available device (cuda > mps > cpu) # WARNING: this takes ~4 hours on L40S, ~14 h on M3 Max. Calibrate first # (see "Calibration recipe" below) before committing to a multi-hour bake. whest dataset bake \ --torch --device auto \ --n-mlps 100 --n-samples 1_000_000_000 \ --width 256 --depth 8 --seed 42 \ --output ./ground-truth # Smaller production-realistic example (10 MLPs × 10⁹ ≈ 25 min on L40S) whest dataset bake \ --torch --device cuda \ --n-mlps 10 --n-samples 1_000_000_000 \ --width 256 --depth 8 --seed 42 \ --output ./data # Develop on laptop using torch CPU (works without GPU) whest dataset bake \ --torch --device cpu \ --n-mlps 5 --n-samples 100_000 \ --width 256 --depth 8 \ --output ./dev ``` -------------------------------- ### whest submit usage examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Practical examples of submitting a pre-packaged artifact, packaging and submitting a folder or single file, and previewing a submission with --dry-run. ```bash # Submit a pre-packaged artifact whest submit ./my-submission.tar.gz ``` ```bash # Package the whole folder and submit in one step (shows a preview + confirm) whest submit --estimator . ``` ```bash # Package a single self-contained file and submit whest submit --estimator ./estimator.py ``` ```bash # Preview (dry run) without submitting whest submit --estimator . --dry-run ``` -------------------------------- ### whest package examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Four usage examples: packaging a whole folder, a single file, skipping confirmation for CI, and specifying a custom output path. ```bash # Package your whole project folder (recommended for multi-file submissions) whest package --estimator . ``` ```bash # Package a single self-contained file (ships only this file) whest package --estimator ./estimator.py ``` ```bash # Skip the folder confirmation (CI) whest package --estimator . --yes ``` ```bash # Custom output path whest package --estimator . --output ./my-submission.tar.gz --yes ``` -------------------------------- ### Example workflows for whest profile-simulation Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Two example invocations: a quick correctness check using the `quick` preset, and a full profile with JSON export using the `exhaustive` preset and `--output`. ```bash # Quick correctness check whest profile-simulation --preset quick ``` ```bash # Full profile with JSON export whest profile-simulation --preset exhaustive --output profile_results.json ``` -------------------------------- ### whest dataset inspect examples Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Examples of inspecting a local dataset directory and a remote Hugging Face Hub repository with a revision. ```bash # Local whest dataset inspect ./my-eval # Remote whest dataset inspect aicrowd/arc-whestbench-public-2026 --revision v1 ``` -------------------------------- ### whest dataset info examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Provides two example invocations of `whest dataset info`: one for a local directory and one for a remote HF Hub repo with a revision. ```bash # Local whest dataset info ./my-eval # Remote whest dataset info aicrowd/arc-whestbench-public-2026 --revision v1 ``` -------------------------------- ### Install GPU backend extra Source: https://github.com/aicrowd/whestbench/blob/main/README.md Installs the optional torch backend extra for baking large ground-truth datasets (n_samples ≥ 10⁸). ```bash pip install whestbench[gpu] ``` -------------------------------- ### whest dataset upload examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Two examples: publishing with a version tag and a private repo upload. The first uses `--tag v1` and a commit message; the second adds `--private`. ```bash # Publish with a version tag whest dataset upload ./my-eval \ --repo aicrowd/arc-whestbench-public-2026 \ --tag v1 \ --message "Bake: 10 MLPs, seed=42" ``` ```bash # Private repo whest dataset upload ./my-eval \ --repo aicrowd/arc-whestbench-evals-2026 \ --tag v1 \ --private ``` -------------------------------- ### Preview of files to bundle Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/ship-weights.md Example output from whest package showing the files that will be included with sizes and total count, followed by a confirmation prompt. Press y to confirm or use --yes to skip. ```text Packaging ./my-submission → submission-*.tar.gz Files to bundle (4 files, 1.2 MB): estimator.py (3.1 KB) layers.py (1.8 KB) utils.py (900 B) weights.npz (1.2 MB) Package these 4 files (1.2 MB)? [y/N] ``` -------------------------------- ### whest dataset bake examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Provides three example invocations of `whest dataset bake`: a full bake, a partial bake using --slice, and a GPU bake using --torch. ```bash # Full bake (10 MLPs, 10M samples each) whest dataset bake \ --n-mlps 10 --n-samples 10_000_000 \ --width 256 --depth 8 \ --output ./my-eval # Partial bake (slice 0 of 4) whest dataset bake \ --n-mlps 100 --n-samples 1_000_000_000 \ --width 256 --depth 8 \ --slice 0/4 \ --output ./partial-0 # GPU bake whest dataset bake \ --n-mlps 100 --n-samples 1_000_000_000 \ --width 256 --depth 8 \ --torch --device auto \ --output ./gpu-eval ``` -------------------------------- ### setup(self, context: SetupContext) -> None Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/estimator-contract.mdx Optional lifecycle hook called before prediction. Receives a SetupContext with configuration fields. ```APIDOC ## setup(self, context: SetupContext) -> None ### Description Optional lifecycle hook called before prediction. Receives a SetupContext with configuration fields. ### Method setup ### Signature `setup(self, context: SetupContext) -> None` ### Parameters - **context** (SetupContext) - Required - Provides configuration fields including width, depth, flop_budget, api_version, scratch_dir, and seed. ``` -------------------------------- ### Operation cost examples (table) Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/code-patterns.md Table of common flopscope operations with their FLOP costs (float32 baseline) and notes. Costs are a guide; ground truth is budget.summary(). ```python fnp.zeros((n, n)) ``` ```python fnp.ones(n) ``` ```python fnp.array(data) ``` ```python fnp.matmul(A, B) ``` ```python fnp.add(a, b) ``` ```python fnp.maximum(x, 0.0) ``` ```python fnp.sqrt(x), fnp.abs(x) ``` ```python fnp.exp(x) ``` ```python fnp.log(x) ``` ```python x ** 2, fnp.sin(x) ``` ```python fnp.transpose(W) ``` ```python fnp.reshape(x, shape) ``` ```python fnp.diag(M) ``` ```python fnp.fill_diagonal(M, v) ``` ```python fnp.outer(a, b) ``` ```python fnp.sum(x, axis=0) ``` ```python fnp.stack(rows, axis=0) ``` ```python x[0], x[:, 3] ``` ```python x[idx], x[mask] ``` ```python fnp.where(c, a, b) ``` ```python fnp.sort(x), fnp.take(x, i) ``` ```python fnp.ix_(i, j) ``` -------------------------------- ### Load weights in setup() with fnp.load Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/ship-weights.md Loads weights from an .npz file in setup() to avoid FLOPs. Uses fnp.load (pickle-free) and guards against ctx.submission_dir being None. Falls back to zero arrays for bare `whest run --estimator estimator.py`. ```python from pathlib import Path from typing import Optional import flopscope as flops import flopscope.numpy as fnp from whestbench import BaseEstimator, SetupContext from whestbench.domain import MLP class WeightedEstimator(BaseEstimator): def setup(self, ctx: SetupContext) -> None: if ctx.submission_dir is not None: # fnp.load is pickle-free (reads .npz zip archives). 0 FLOPs. data = fnp.load(Path(ctx.submission_dir) / "weights.npz") self.W1 = data["W1"] self.W2 = data["W2"] else: # Fallback for bare `whest run --estimator estimator.py` # (no packaged submission; ctx.submission_dir is None) self.W1 = fnp.zeros((4, 4)) self.W2 = fnp.zeros((4, 4)) def predict(self, mlp: MLP, budget: int) -> fnp.ndarray: # Use self.W1 / self.W2 here — already loaded, no FLOPs charged ... ``` -------------------------------- ### SetupContext fields Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/estimator-contract.md Describes the fields available in the SetupContext object passed to the setup() method. ```APIDOC ## SetupContext fields ### Description Fields available in the SetupContext object passed to the setup() method. ### Fields - **width** (int) - Neuron count for generated MLPs. - **depth** (int) - Number of layers per MLP. - **flop_budget** (int) - FLOP cap for the estimator. - **api_version** (str) - Contract version string. - **scratch_dir** (str | None) - Optional writable directory for caching. - **submission_dir** (str | None) - Read-only path to the extracted submission directory. All files shipped beside `estimator.py` are extracted here and importable as modules. Load shipped data files with `fnp.load(Path(ctx.submission_dir) / "weights.npz")` (0 FLOPs, pickle-free) or `MyModel.from_file(...)`. `None` outside the runner/grader (e.g. in plain `whest run` without a packaged submission). - **seed** (int) - Per-run seed from `--seed` (default `0`). Use in `setup()` to reproduce one-time random initialisation. ``` -------------------------------- ### whest version usage examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Two example invocations: plain `whest version` and `whest version --json`. ```bash whest version ``` ```bash whest version --json ``` -------------------------------- ### Multi-file submission layout with estimator.py, layers.py, and weights.npz Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/estimator-contract.md Shows the folder structure for a multi-file submission and the estimator.py entrypoint that imports a helper module and loads pre-trained weights in setup(). The submission_dir is None outside a packaged context, so guard with if ctx.submission_dir is not None. Loading weights with fnp.load in setup() costs 0 FLOPs. ```python # myproject/ # estimator.py ← entrypoint # layers.py ← helper module # weights.npz ← pre-trained weights # estimator.py from pathlib import Path import flopscope.numpy as fnp from layers import build_model # importable: layers.py is in the same directory class MyEstimator(BaseEstimator): def setup(self, ctx: SetupContext) -> None: if ctx.submission_dir is not None: data = fnp.load(Path(ctx.submission_dir) / "weights.npz") self.model = build_model(data["W"]) def predict(self, mlp, budget): ... ``` -------------------------------- ### setup(self, context: SetupContext) -> None Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/estimator-contract.md Optional lifecycle hook that runs before any predictions. It is executed outside the per-MLP FLOP budget and wall-time penalty, and has a hard 5-second budget per run. Use it to load precomputed artifacts, not to compute them. ```APIDOC ## setup(self, context: SetupContext) -> None ### Description Optional lifecycle hook that runs before any predictions. It is executed outside the per-MLP FLOP budget and wall-time penalty, and has a hard 5-second budget per run. Use it to load precomputed artifacts, not to compute them. ### Method setup ### Signature `setup(self, context: SetupContext) -> None` ### Parameters - **context** (SetupContext) - Required - Provides configuration and environment information for the estimator. ### Returns - **None** - This method returns nothing. ### Notes - Runs once per worker process, roughly 5-15 times per submission. - Each run has a hard 5-second budget; exceeding it fails the submission with `SETUP_TIMEOUT`. - Runs outside the per-MLP FLOP budget and `residual_wall_time_s` penalty. - Use to load precomputed work, not to perform computation. ``` -------------------------------- ### End-to-end example: bake, inspect, push, pull, run Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Complete workflow showing how to bake a dataset, inspect it locally, push it to Hugging Face Hub, pull it on another machine, and run an evaluation. Requires setting HF_TOKEN for publishing. ```bash whest dataset bake \ --n-mlps 10 --n-samples 10_000_000 \ --width 256 --depth 8 \ --output ./my-eval ``` ```bash whest dataset inspect ./my-eval ``` ```bash export HF_TOKEN=hf_... whest dataset push ./my-eval \ --repo aicrowd/arc-whestbench-public-2026 \ --tag v1 ``` ```bash whest dataset pull aicrowd/arc-whestbench-public-2026 \ --revision v1 --output ./local-copy ``` ```bash whest run --estimator ./estimator.py \ --dataset hf://aicrowd/arc-whestbench-public-2026@v1 ``` -------------------------------- ### Quick start: bake, info, and run commands Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Commands to bake a local evaluation dataset, inspect it, and run an estimator against it. Baking once makes runs deterministic and ~10x faster. ```bash # 1. Generate 10 MLPs with ground-truth statistics → ./my-eval/ whest dataset bake --n-mlps 10 --n-samples 1000 --width 64 --depth 4 \ --output ./my-eval # 2. Inspect what got written whest dataset info ./my-eval # 3. Score your estimator against the same MLPs every run whest run --estimator estimator.py --dataset ./my-eval ``` -------------------------------- ### whest dataset download output example Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Representative output from the download command, showing a progress bar and the cache location. The output includes the download duration and disk usage. ```text → Downloading hf://aicrowd/arc-whestbench-public-2026@v2-phase2 → HF cache Preflight: 3 files, 2.0 GB — not cached. ✓ Cached hf://aicrowd/arc-whestbench-public-2026@v2-phase2 in 28.9s (2.0 GB on disk) Location: ~/.cache/huggingface/hub/datasets--aicrowd--arc-whestbench-public-2026/snapshots/ ``` -------------------------------- ### Bake dataset on GPU with cuda device Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/gpu-dataset-generation.md Smaller production-realistic example (10 MLPs × 1e9 ≈ 25 min on L40S) using the cuda device explicitly. ```bash whest dataset bake \ --torch --device cuda \ --n-mlps 10 --n-samples 1_000_000_000 \ --width 256 --depth 8 --seed 42 \ --output ./data ``` -------------------------------- ### Array Creation with flopscope.numpy Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/flopscope-primer.md Shows how to create arrays using flopscope.numpy. All array creation is free (0 FLOPs). Includes zeros, eye, array, and asarray examples. ```python import flopscope as flops import flopscope.numpy as fnp x = fnp.zeros(100) # 1D zeros X = fnp.zeros((64, 100), dtype=fnp.float32) # 2D zeros, explicit dtype I = fnp.eye(100, dtype=fnp.float32) # identity matrix a = fnp.array([1.0, 2.0, 3.0]) # from list b = fnp.asarray(numpy_array) # convert from numpy (free) ``` -------------------------------- ### Example .whestignore patterns Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/ship-weights.md Add glob patterns to .whestignore (same syntax as .gitignore) to exclude files from the bundle. whest init creates a starter .whestignore when scaffolding a new project. ```text # .whestignore train.py *.log scratch/ checkpoints/ ``` -------------------------------- ### whest version command syntax Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Prints the installed whestbench version. Supports `--format rich|plain|json` and `--json` flag. ```bash whest version [--format rich|plain|json] [--json] ``` -------------------------------- ### whest dataset merge example Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Merges four partial slices into a final dataset. The result is bit-equivalent to a single-host bake with the same parameters. ```bash # After baking 4 slices on separate workers: whest dataset merge \ ./partial-0 ./partial-1 ./partial-2 ./partial-3 \ --output ./final-eval ``` -------------------------------- ### Fetch partials with scp or rsync Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/parallel-bake.md Collect partial directories from each worker onto a single machine. The scp example is simple; rsync preserves timestamps and supports resumption. ```bash # scp example (adjust hostnames and paths) scp -r worker-0:/data/partial-0 ./partial-0 scp -r worker-1:/data/partial-1 ./partial-1 scp -r worker-2:/data/partial-2 ./partial-2 scp -r worker-3:/data/partial-3 ./partial-3 # Or rsync (preserves timestamps, supports resumption) rsync -avz worker-0:/data/partial-0/ ./partial-0/ rsync -avz worker-1:/data/partial-1/ ./partial-1/ rsync -avz worker-2:/data/partial-2/ ./partial-2/ rsync -avz worker-3:/data/partial-3/ ./partial-3/ ``` -------------------------------- ### whest version JSON output Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Example JSON output from `whest version --json`. Includes `ok`, `command`, `name`, `version`, and `whestbench_version` fields. ```json { "ok": true, "command": "version", "name": "whestbench", "version": "0.2.0", "whestbench_version": "0.2.0" } ``` -------------------------------- ### Example metadata.json (CPU bake, seed_protocol 3.0) Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/dataset-format.mdx Metadata for a CPU bake using seed_protocol 3.0. No top-level 'seed' field; each MLP's input seed is stored in the parquet 'mlp_seed' column. ```json { "schema_version": "3.0", "format": "hf-datasets-parquet", "backend": "flopscope", "seed_protocol": { "name": "whestbench_explicit_per_mlp_seeds", "version": "3.0" }, "n_mlps": 10, "n_samples": 10000000, "width": 256, "depth": 8, "created_at_utc": "2026-05-25T12:00:00+00:00", "hardware": { "cpu_brand": "Intel Xeon Platinum 8480+", "cpu_count": 64, "ram_gb": 512.0 }, "whestbench_version": "0.3.0", "flopscope_version": "0.3.0" } ``` -------------------------------- ### Calibration recipe for GPU dataset generation Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/gpu-dataset-generation.md Run this once when spinning up the instance to get a precise wall-time projection for the actual N=10⁹ bake. It includes a warmup to trigger kernel compilation, then a calibration run anchored on n_mlps=10. The torch.cuda.synchronize() call is critical because CUDA ops are async; without it you'd measure dispatch time, not compute time. ```python import time from pathlib import Path import torch from whestbench.dataset_torch import create_dataset_torch # Warmup (kernel compilation, ~0.2s on cuda) create_dataset_torch( n_mlps=2, n_samples=10_000, width=256, depth=8, seed=0, output_path=Path('/tmp/warmup'), device='cuda') # Calibration anchored on n_mlps=10 to match the production setup t0 = time.perf_counter() create_dataset_torch( n_mlps=10, n_samples=1_000_000, width=256, depth=8, seed=42, output_path=Path('/tmp/cal'), device='cuda') torch.cuda.synchronize() elapsed = time.perf_counter() - t0 print(f'{elapsed:.2f}s at N=10⁶ → projected {elapsed*1000/60:.1f} min at N=10⁹') ``` -------------------------------- ### Recommended debug sequence for whest run Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Progressive debugging commands: start with a basic run, add `--debug` for tracebacks, `--fail-fast` to stop on first error, and `--runner local --format plain` for pdb/breakpoint() support. ```bash whest run --estimator ./path/to/estimator.py ``` ```bash whest run --estimator ./path/to/estimator.py --debug ``` ```bash whest run --estimator ./path/to/estimator.py --debug --fail-fast ``` ```bash whest run --estimator ./path/to/estimator.py --runner local --format plain # for pdb.set_trace() / breakpoint() ``` -------------------------------- ### Using BudgetContext with wall_time_limit_s Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/flopscope-primer.md Shows how to add a cooperative wall-clock limit in addition to the FLOP cap. The timer starts when the context is entered and is checked before and after each counted flopscope/NumPy call. If exceeded, flopscope raises TimeExhaustedError. ```python with flops.BudgetContext(flop_budget=1_000_000, wall_time_limit_s=2.0) as ctx: ... ``` -------------------------------- ### whest init Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Create starter files in a target directory. ```APIDOC ## `whest init` ### Description Create starter files in a target directory. ### Method CLI command ### Endpoint `whest init [path] [--format rich|plain|json] [--json] [--debug]` ### Parameters #### Path Parameters - **path** (string) - Optional - Target directory path. #### Options - **--format** (string) - Optional - Output format: `rich`, `plain`, or `json`. Defaults to `rich` on TTYs and `plain` otherwise. - **--json** (boolean) - Optional - Alias for `--format json`. - **--debug** (boolean) - Optional - Enable debug output. ### Request Example ```bash whest init ./my-project --format json ``` ### Response #### Success Response Starter files created in the target directory. #### Response Example ``` Created starter files in ./my-project ``` ``` -------------------------------- ### whest init Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Creates starter files in a target directory. ```APIDOC ## `whest init` ### Description Create starter files in a target directory. ### Method CLI command ### Endpoint `whest init [path] [--format rich|plain|json] [--json] [--debug]` ### Parameters #### Path Parameters - **path** (string) - Optional - Target directory for starter files. #### Options - **--format** (string) - Optional - Output format: `rich`, `plain`, or `json`. - **--json** (boolean) - Optional - Alias for `--format json`. - **--debug** (boolean) - Optional - Enable debug output. ### Request Example ```bash whest init ./my-project ``` ### Response #### Success Response Starter files created in the target directory. ``` -------------------------------- ### whest init command syntax Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Creates starter files in a target directory. The `--format` option selects output style (rich, plain, or json), and `--json` is an alias for `--format json`. ```bash whest init [path] [--format rich|plain|json] [--json] [--debug] ``` -------------------------------- ### whest doctor usage examples Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Two example invocations: an interactive first-hour check and a CI pre-flight that treats any non-OK result as a failure using `--strict --json`. ```bash # Interactive first-hour check whest doctor # CI pre-flight (treat anything that isn't OK as a failure) whest doctor --strict --json ``` -------------------------------- ### Seed setup-time randomness from ctx.seed Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/estimator-contract.md Use fnp.random.default_rng(ctx.seed) inside setup() for randomized one-time setup. The same seed is forwarded to ctx.seed for every MLP when --seed is passed. Do not call fnp.random.seed(ctx.seed) as it mutates the global RNG. ```python import flopscope.numpy as fnp def setup(self, ctx: SetupContext) -> None: self.setup_rng = fnp.random.default_rng(ctx.seed) # ... use self.setup_rng for any one-time random work ``` -------------------------------- ### whest dataset pull example Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Pulls a specific revision of a dataset to a local directory. ```bash whest dataset pull aicrowd/arc-whestbench-public-2026 \ --revision v1 \ --output ./eval-v1 ``` -------------------------------- ### whest dataset bake - representative output Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Example output from a successful bake command, showing progress and final write. ```text → Baking 20 MLPs (width=1024, depth=16, n_samples=10000) to ./my-eval ✓ Generated weights 20/20 ✓ Computed ground truth 20/20 16.1s ✓ Wrote ./my-eval (1.3 GB) ``` -------------------------------- ### Bake dataset on CPU with torch backend for development Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/gpu-dataset-generation.md Develop on laptop using torch CPU (works without GPU) with a small sample size. ```bash whest dataset bake \ --torch --device cpu \ --n-mlps 5 --n-samples 100_000 \ --width 256 --depth 8 \ --output ./dev ``` -------------------------------- ### End-to-end Whest CLI workflow: bake, inspect, upload, download, run Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md This snippet shows the complete sequence of commands to create a dataset with `whest dataset bake`, inspect it with `whest dataset info`, publish it with `whest dataset upload`, prefetch it with `whest dataset download`, and run an evaluation with `whest run`. It requires setting the `HF_TOKEN` environment variable for upload. The download step lands in the Hugging Face cache, and the run step uses a cache hit to avoid re-downloading. ```bash # 1. Bake whest dataset bake \ --n-mlps 10 --n-samples 10_000_000 \ --width 256 --depth 8 \ --output ./my-eval # 2. Inspect locally whest dataset info ./my-eval # 3. Publish export HF_TOKEN=hf_... whest dataset upload ./my-eval \ --repo aicrowd/arc-whestbench-public-2026 \ --tag v1 # 4. Prefetch on another machine (lands in the HF cache) whest dataset download aicrowd/arc-whestbench-public-2026 \ --revision v1 # 5. Run evaluation (cache hit — no re-download) whest run --estimator ./estimator.py \ --dataset hf://aicrowd/arc-whestbench-public-2026@v1 ``` -------------------------------- ### whest profile-simulation command syntax Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Shows the full command syntax with all options. The `--preset` option defaults to `standard`; `--format` defaults to `rich` on TTYs and `plain` otherwise. Use `--json` as an alias for `--format json`. ```bash whest profile-simulation [--preset super-quick|quick|standard|exhaustive] [--output ] [--format rich|plain|json] [--json] [--verbose] [--debug] ``` -------------------------------- ### metadata.json shape for multi-split dataset Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/dataset-format.md Example metadata.json with splits sub-dict, including schema_version, format, backend, seed_protocol, n_samples, width, depth, created_at_utc, hardware, splits, and default_split. ```json { "schema_version": "3.0", "format": "hf-datasets-parquet", "backend": "torch", "seed_protocol": {"name": "whestbench_explicit_per_mlp_seeds", "version": "3.0"}, "n_samples": 1000000000, "width": 256, "depth": 8, "created_at_utc": "...", "hardware": {...}, "splits": { "public": {"config": "default", "n_mlps": 50, "created_at_utc": "...", "hardware_fingerprints": [...]}, "holdout": {"config": "holdout", "n_mlps": 50, "created_at_utc": "...", "hardware_fingerprints": [...]} }, "default_split": "public" } ``` -------------------------------- ### budget.summary() and fnp.budget_summary() Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/flopscope-primer.mdx These methods return summaries of FLOP usage and timing for the current context and the session/global view, respectively. ```APIDOC ## budget.summary() and fnp.budget_summary() ### Description Returns a summary of FLOP usage and timing for the current explicit context (budget.summary()) or the accumulated session/global view (fnp.budget_summary()). ### Method Python method call ### Endpoint N/A (Python API) ### Parameters None ### Request Example ```python with flops.BudgetContext(flop_budget=10_000_000) as ctx: # ... your computations ... print(ctx.summary()) # current context only print(fnp.budget_summary()) # process/session-wide summary print(ctx.flops_used) # integer FLOP count ``` ### Response #### Success Response Returns a summary object containing the following fields: - `flops_used` (integer) - Total FLOP count used. - `wall_time_s` (float) - Total elapsed time in the context. - `flopscope_backend_time_s` (float) - Time spent inside counted flopscope numpy kernels. - `flopscope_overhead_time_s` (float) - Time spent inside flopscope's own dispatch (wrapper preambles, FLOP bookkeeping, namespace push/pop). - `residual_wall_time_s` (float) - Everything else - participant Python, GC, uninstrumented numpy. #### Response Example ```json { "flops_used": 10000000, "wall_time_s": 1.234, "flopscope_backend_time_s": 0.5, "flopscope_overhead_time_s": 0.1, "residual_wall_time_s": 0.634 } ``` ``` -------------------------------- ### Whest CLI workflow: train, validate, package, submit Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/ship-weights.md Commands to train offline, validate the estimator file, package the folder, verify the archive, dry-run the submission, and submit. Note that validate takes the estimator FILE, while package takes the folder. ```bash # 1. Train offline (not shipped) uv run python train.py # 2. Validate the estimator loads correctly (validate takes the estimator FILE) whest validate --estimator ./my-submission/estimator.py # 3. Preview and package the whole FOLDER (so helpers/subpackages ship) whest package --estimator ./my-submission # 3b. Verify the archive will pass the grader's integrity check whest validate-package ./submission-*.tar.gz # 4. Dry-run to verify what would be submitted whest submit --estimator ./my-submission --dry-run # 5. Submit whest submit --estimator ./my-submission ``` -------------------------------- ### Budget Inspection with BudgetContext and budget_summary Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/flopscope-primer.md Shows how to inspect FLOP budgets using BudgetContext and fnp.budget_summary(). The summary includes timing fields that satisfy a decomposition identity. ```python with flops.BudgetContext(flop_budget=10_000_000) as ctx: # ... your computations ... print(ctx.summary()) # current context only print(fnp.budget_summary()) # process/session-wide summary print(ctx.flops_used) # integer FLOP count ``` -------------------------------- ### metadata.json shape for multi-split dataset Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/dataset-format.mdx Example metadata.json with a `splits` sub-dict describing multiple splits. Under seed_protocol 3.0, there is no per-split `seed` field; seeds are stored in the parquet `mlp_seed` column. ```json { "schema_version": "3.0", "format": "hf-datasets-parquet", "backend": "torch", "seed_protocol": {"name": "whestbench_explicit_per_mlp_seeds", "version": "3.0"}, "n_samples": 1000000000, "width": 256, "depth": 8, "created_at_utc": "...", "hardware": {}, "splits": { "public": {"config": "default", "n_mlps": 50, "created_at_utc": "...", "hardware_fingerprints": []}, "holdout": {"config": "holdout", "n_mlps": 50, "created_at_utc": "...", "hardware_fingerprints": []} }, "default_split": "public" } ``` -------------------------------- ### whest package command Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/cli-reference.mdx Builds a submission artifact. Use --class to specify the estimator class, --requirements for dependencies, --submission-metadata for metadata, --approach for approach description, --output for the artifact path, and --format or --json for output formatting. ```bash whest package --estimator [options] ``` -------------------------------- ### Built-in standard normal PDF and CDF Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/code-patterns.md Use flopscope's built-in stats.norm.pdf and stats.norm.cdf for FLOP-tracked standard normal PDF and CDF. These are the recommended approach; all example estimators use them. ```python import flopscope as flops import flopscope.numpy as fnp phi = flops.stats.norm.pdf(x) # standard normal PDF Phi = flops.stats.norm.cdf(x) # standard normal CDF ``` -------------------------------- ### Steady-state release flow: git checkout, cz bump, uv lock, commit, tag, push Source: https://github.com/aicrowd/whestbench/blob/main/docs/release-process.md Recommended release flow for whestbench. Uses `uv run cz bump --dry-run` to preview, then `--files-only --changelog` to write version and CHANGELOG without committing or tagging, followed by `uv lock`, commit, annotated tag, and push with `--follow-tags`. Note: with no Required Reviewers on the `pypi` environment, the publish workflow runs without an approval pause. ```bash git checkout main && git pull origin main uv run cz bump --dry-run # preview the next version + CHANGELOG entry uv run cz bump --files-only --changelog # write pyproject version + CHANGELOG.md — no commit or tag yet uv lock # uv.lock records whestbench's own version; stale after every bump git add pyproject.toml CHANGELOG.md uv.lock git commit -m "bump: version → " git tag -a v -m "v" # annotated, like every existing release tag git push --follow-tags # tag push triggers the publish workflow # … the publish workflow builds, publishes to PyPI, and creates the # GitHub Release. NOTE: with no Required Reviewers on the `pypi` # environment (the current state — see "One-time setup" §2) it runs # straight through with NO approval pause. ``` -------------------------------- ### Inspect bake parameters with whest dataset info Source: https://github.com/aicrowd/whestbench/blob/main/docs/how-to/publish-to-hf-hub.md Verifies the bake parameters before uploading. This is cheap and catches any misconfiguration before it goes out. The expected output shows schema_version, format, backend, seed, and other parameters. ```bash whest dataset info ./my-bake ``` ```text WhestBench dataset schema_version: 3.0 format: hf-datasets-parquet backend: flopscope seed: 42 n_mlps: 10 n_samples: 10000000 width: 256 depth: 8 created_at_utc: 2026-05-25T12:00:00+00:00 ``` -------------------------------- ### CLI: whest dataset download Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Fetches a dataset from Hugging Face Hub. ```APIDOC ## whest dataset download ### Description Fetches a dataset from Hugging Face Hub. ### Method CLI command ### Command `whest dataset download` ### Purpose Fetch from HF ``` -------------------------------- ### Run dataset from HuggingFace Hub with whest run Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Pins a dataset to a specific revision using the hf:// URI with a tag. ```bash whest run --dataset hf://aicrowd/my-eval@v1 ``` -------------------------------- ### whest dataset info command Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Shows the syntax for printing metadata from a local directory or a HF Hub repo, with an optional --revision flag for remote repos. ```bash whest dataset info [--revision REV] ``` -------------------------------- ### Example metadata.json (legacy seed_protocol 2.0) Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/dataset-format.mdx Legacy metadata for datasets baked before protocol 3.0, using seed_protocol 2.0 with a top-level 'seed' field. These datasets continue to load correctly, but new bakes always write seed_protocol 3.0. ```json { "schema_version": "3.0", "format": "hf-datasets-parquet", "backend": "flopscope", "seed_protocol": { "name": "whestbench_seedsequence_hierarchy", "version": "2.0" }, "seed": 42, "n_mlps": 10, "n_samples": 10000000, "width": 256, "depth": 8, "created_at_utc": "2026-05-25T12:00:00+00:00", "hardware": { "cpu_brand": "Intel Xeon Platinum 8480+", "cpu_count": 64, "ram_gb": 512.0 } } ``` -------------------------------- ### Load dataset with HuggingFace datasets library Source: https://github.com/aicrowd/whestbench/blob/main/tests/fixtures/multi_split_v3.0/README.md Loads the WhestBench dataset using the HuggingFace datasets library. Returns a DatasetDict when no split is specified, or a Dataset when a split is provided. The example shows loading both splits and a single split, and accessing the 'mlp_name' field. ```python from datasets import load_dataset # Load both splits (returns DatasetDict) ds = load_dataset( "", revision="main", ) # → DatasetDict({"public": ..., "holdout": ...}) # Load one split (returns Dataset) ds = load_dataset( "", revision="main", split="public", ) print(ds[0]["mlp_name"]) ``` -------------------------------- ### Verify dataset metadata with whestbench.load_dataset Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/gpu-dataset-generation.md Use this to verify the output of a dataset baked with --torch. It loads the dataset and reads provenance from metadata.json, checking the backend, device, and torch_version. The backend will be "torch" or "flopscope". ```python import whestbench ds = whestbench.load_dataset("./ground-truth") md = whestbench.metadata(ds) backend = md.get("backend", "flopscope") # "torch" or "flopscope" device = md.get("device") # "cuda" | "mps" | "cpu" if torch torch_version = md.get("torch_version") ``` -------------------------------- ### Example metadata.json (torch CUDA bake, seed_protocol 3.0) Source: https://github.com/aicrowd/whestbench/blob/main/website/content/docs/reference/dataset-format.mdx Metadata for a torch CUDA bake using seed_protocol 3.0. Includes torch_version, device, CUDA device info, and bake_config with deterministic settings. No top-level 'seed' field; per-MLP seeds are in the parquet 'mlp_seed' column. ```json { "schema_version": "3.0", "format": "hf-datasets-parquet", "backend": "torch", "seed_protocol": { "name": "whestbench_explicit_per_mlp_seeds", "version": "3.0" }, "n_mlps": 50, "n_samples": 1000000000, "width": 256, "depth": 8, "created_at_utc": "2026-05-26T03:45:00+00:00", "hardware": { "...": "..." }, "whestbench_version": "0.3.0", "flopscope_version": "0.3.0", "torch_version": "2.3.0+cu121", "device": "cuda", "cuda_device_name": "NVIDIA L40S", "cuda_device_capability": [8, 9], "cuda_driver_version": "535.183.01", "mlps_per_batch": 16, "chunk_size": 524288, "bake_config": { "cudnn_deterministic": true, "cudnn_benchmark": false, "cublas_workspace_config": ":4096:8", "torch_use_deterministic_algorithms": false } } ``` -------------------------------- ### whest package command syntax Source: https://github.com/aicrowd/whestbench/blob/main/docs/reference/cli-reference.md Shows the basic syntax for the `whest package` command. The `--estimator` argument accepts a directory or a file, and `--output` and `--yes` are optional. ```bash whest package --estimator [--output ] [--yes|-y] [options] ``` -------------------------------- ### whest dataset info - inspect local dataset Source: https://github.com/aicrowd/whestbench/blob/main/docs/guides/datasets.md Shows a one-screen summary of a baked or downloaded dataset, including schema_version, seed_protocol, n_mlps, n_samples, hardware fingerprint, and per-split row counts. ```bash whest dataset info ./my-eval ```