### Install Critcmp with Cargo Source: https://github.com/burntsushi/critcmp/blob/master/README.md Install the critcmp command-line tool using Cargo. This is the primary method for obtaining the tool. ```bash $ cargo install critcmp ``` -------------------------------- ### Compare Critcmp Baselines from JSON Files Source: https://github.com/burntsushi/critcmp/blob/master/README.md Use exported JSON files as baselines for comparison by providing their file paths. ```bash critcmp before.json change.json ``` -------------------------------- ### List Available Critcmp Baselines Source: https://github.com/burntsushi/critcmp/blob/master/README.md Print a list of all baselines that critcmp recognizes. ```bash critcmp --baselines ``` -------------------------------- ### List (Row) Display Mode for Benchmarks Source: https://context7.com/burntsushi/critcmp/llms.txt When many baselines or long benchmark names cause the default column layout to wrap, switch to a flat list display that dedicates one section per benchmark group. This mode is useful for visualizing detailed results when the standard output becomes unwieldy. ```bash critcmp before after extra1 extra2 extra3 --list ``` ```bash # Combine with filter and threshold critcmp before after extra1 --list -f 'serialize' -t 3 ``` -------------------------------- ### Compare Multiple Critcmp Baselines Source: https://github.com/burntsushi/critcmp/blob/master/README.md Compare more than two baselines simultaneously by listing them as arguments. ```bash critcmp before change1 change2 ``` -------------------------------- ### Group Benchmarks by Regex Capture Groups Source: https://context7.com/burntsushi/critcmp/llms.txt Use a regex with capturing groups to compare benchmarks within a single baseline that have different names but share a common logical identity. The values of all capturing groups form the comparison key; everything outside captures becomes the "variant" label. ```bash # Given benchmarks "optimized/input1", "naive/input1", "optimized/input2", "naive/input2" # in baseline "benches", group by the input name: critcmp benches -g '\w+/(input\d+)' ``` ```bash # Group by a shared suffix across two algorithmic variants critcmp benches -g '(small|large)_input$' ``` -------------------------------- ### Basic Two-Baseline Comparison Source: https://context7.com/burntsushi/critcmp/llms.txt Compare two saved Criterion baselines side by side. The fastest result in each group is highlighted in green and marked with rank 1.00. ```bash # Save a baseline before a code change carpenter bench -- --save-baseline before # Make code changes, then save another baseline carpenter bench -- --save-baseline after # Compare the two baselines critcmp before after ``` -------------------------------- ### Compare More Than Two Baselines Source: https://context7.com/burntsushi/critcmp/llms.txt Compare any number of baselines simultaneously by providing them as arguments. This can be combined with filters and thresholds. ```bash # Compare four baselines at once critcmp main pr-123 pr-456 nightly # Combine with a filter and threshold critcmp main pr-123 pr-456 -f 'regex' -t 2 ``` -------------------------------- ### Group Critcmp Benchmarks by Regex Capture Groups Source: https://github.com/burntsushi/critcmp/blob/master/README.md Compare benchmarks within the same baseline that do not share identical names, by defining matching criteria using regex capture groups. All benchmarks with equivalent captured groups are included in a single comparison. ```bash critcmp benches -g '\w+/(input1)' ``` -------------------------------- ### Export a Baseline to a JSON File Source: https://context7.com/burntsushi/critcmp/llms.txt Export a baseline from the Criterion target directory into a portable JSON file for archiving or sharing. These files can later be compared directly. ```bash # Export baselines to JSON files critcmp --export before > before.json critcmp --export after > after.json # Later, compare the exported files directly (no target/ dir needed) critcmp before.json after.json # Mix file paths and named baselines critcmp before.json after ``` -------------------------------- ### Custom Target Directory for Critcmp Source: https://context7.com/burntsushi/critcmp/llms.txt By default critcmp walks up from the current directory looking for a `target/` folder. Override this for workspaces or non-standard build layouts using either the explicit flag or the CARGO_TARGET_DIR environment variable. ```bash # Explicit flag critcmp before after --target-dir /path/to/workspace/target ``` ```bash # Environment variable (also respected by Cargo itself) CARGO_TARGET_DIR=/path/to/workspace/target critcmp before after ``` -------------------------------- ### Filter Benchmarks by Regex Source: https://context7.com/burntsushi/critcmp/llms.txt Limit the comparison to only benchmark names matching a given regular expression. Non-matching benchmarks are silently excluded. ```bash # Only show benchmarks whose names contain "parse" critcmp before after -f 'parse' # Match benchmarks with names like "search/utf8" or "search/ascii" critcmp before after -f 'search/(utf8|ascii)' ``` -------------------------------- ### Basic Critcmp Usage: Compare Two Baselines Source: https://github.com/burntsushi/critcmp/blob/master/README.md Compare two saved Criterion baselines. Ensure baselines are saved using 'cargo bench -- --save-baseline ' before running this command. ```bash critcmp before change ``` -------------------------------- ### Flatten Critcmp Results to a List Source: https://github.com/burntsushi/critcmp/blob/master/README.md Display comparison results in a flattened list format instead of the default column-oriented view, useful for very large comparisons. ```bash critcmp before change1 change2 change3 change4 change5 --list ``` -------------------------------- ### Export Critcmp Baseline to JSON Source: https://github.com/burntsushi/critcmp/blob/master/README.md Export a specific baseline to a JSON file for persistent storage. This JSON file can later be used as a baseline input. ```bash critcmp --export before > before.json ``` ```bash critcmp --export change > change.json ``` -------------------------------- ### Criterion Benchmark JSON Structure Source: https://context7.com/burntsushi/critcmp/llms.txt The exported JSON format matches the 'BaseBenchmarks' struct, containing benchmark names and their associated performance estimates. ```json { "name": "before", "benchmarks": { "parse/small_input": { "baseline": "before", "fullname": "before/parse/small_input", "criterion_benchmark_v1": { "group_id": "parse", "function_id": "small_input", "value_str": null, "throughput": null, "full_id": "parse/small_input", "directory_name": "parse/small_input" }, "criterion_estimates_v1": { "mean": { "confidence_interval": { "confidence_level": 0.95, "lower_bound": 140.1, "upper_bound": 144.5 }, "point_estimate": 142.3, "standard_error": 1.1 }, "median": { "confidence_interval": { "confidence_level": 0.95, "lower_bound": 141.0, "upper_bound": 143.8 }, "point_estimate": 142.0, "standard_error": 0.9 }, "median_abs_dev": { "confidence_interval": { "confidence_level": 0.95, "lower_bound": 0.5, "upper_bound": 2.1 }, "point_estimate": 1.1, "standard_error": 0.4 }, "slope": null, "std_dev": { "confidence_interval": { "confidence_level": 0.95, "lower_bound": 0.8, "upper_bound": 1.5 }, "point_estimate": 1.1, "standard_error": 0.2 } } } } } ``` -------------------------------- ### Suppress Small Differences with a Threshold Source: https://context7.com/burntsushi/critcmp/llms.txt Hide comparisons where the performance difference is below a given percentage. Useful for cutting through noise and focusing on regressions or improvements that matter. ```bash # Only show benchmarks with more than 5% difference critcmp before after -t 5 # Only show benchmarks with more than 10% difference critcmp before after -t 10 ``` -------------------------------- ### Filter Critcmp Comparisons by Regex Source: https://github.com/burntsushi/critcmp/blob/master/README.md Limit comparisons to benchmarks matching a specific regex pattern using the -f or --filter flag. ```bash critcmp before change -f 'foo.*bar' ``` -------------------------------- ### Filter Critcmp Comparisons by Percentage Difference Source: https://github.com/burntsushi/critcmp/blob/master/README.md Hide comparisons with small differences (e.g., 5% or less) using the -t flag. ```bash critcmp before change -t 5 ``` -------------------------------- ### Control ANSI Color Output in Critcmp Source: https://context7.com/burntsushi/critcmp/llms.txt Control ANSI color output. Defaults to `auto` (color only when stdout is a TTY). Use `always` to force color even when piping, or `never` for plain text output without escape codes. ```bash critcmp before after --color always # force color even when piping ``` ```bash critcmp before after --color never # plain text, no escape codes ``` ```bash critcmp before after --color auto # default: color when TTY ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.