pytest-benchmark

https://pytest-benchmark.readthedocs.io/en/stable
Admin
A pytest plugin that provides a benchmark fixture to measure the performance of functions. It offers...
Tokens:13,816
Snippets:131
Trust Score:8.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Using benchmark.pedantic with a setup function

Source: https://pytest-benchmark.readthedocs.io/en/stable/pedantic.html

Demonstrates using a setup function to prepare arguments before each round. The setup function can return a (args, kwargs) tuple. Note that when using setup, you cannot use args, kwargs, or iterations options.

```python
def stuff(a, b, c, foo):
    pass

def test_with_setup(benchmark):
    def setup():
        # can optionally return a (args, kwargs) tuple
        return (1, 2, 3), {'foo': 'bar'}
    benchmark.pedantic(stuff, setup=setup, rounds=100)  # stuff(1, 2, 3, foo='bar') will be benchmarked

```

--------------------------------

### Example output of --between comparison

Source: https://pytest-benchmark.readthedocs.io/en/stable/comparing.html

Example output showing a table with benchmarks from two sources, including the reference source and Δ columns for percentage change.

```text
----------------------------- benchmark: 9 tests, 2 sources ------------------------------
Name (time in ns)                      0001_f41c0c7 Min  0002_8e68892 Min             ΔMin
------------------------------------------------------------------------------------------
test_getattr_thread_critical                   790.9257          245.7993           -68.9%
test_setattr_thread_critical                   899.9952          254.1521           -71.8%
...

```

--------------------------------

### Install pytest-benchmark with pip

Source: https://pytest-benchmark.readthedocs.io/en/stable/installation.html

Installs the pytest-benchmark plugin from PyPI using pip. Run this command in your terminal or command prompt.

```shell
pip install pytest-benchmark
```

--------------------------------

### pytest-benchmark compare examples

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Examples of using pytest-benchmark compare with glob patterns, run IDs, and explicit file paths. Note the special quoting to disable shell glob expansion.

```text
pytest-benchmark compare 'Linux-CPython-3.5-64bit/*'
```

```text
pytest-benchmark compare 0001
```

```text
pytest-benchmark compare /foo/bar/0001_abc.json /lorem/ipsum/0001_sir_dolor.json
```

--------------------------------

### pytest_benchmark_generate_machine_info hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Completely replace the generated machine_info. Example returns a dict with the current user.

```python
def pytest_benchmark_generate_machine_info(config):
    return {'user': getpass.getuser()}

```

--------------------------------

### Defining a class with an internal method to benchmark

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Shows a sample class with an internal method that sleeps, used as the target for the benchmark_weave patching example.

```python
class Foo(object):
    def __init__(self, arg=0.01):
        self.arg = arg

    def run(self):
        self.internal(self.arg)

    def internal(self, duration):
        time.sleep(duration)

```

--------------------------------

### Using the benchmark fixture in pytest tests

Source: https://pytest-benchmark.readthedocs.io/en/stable/index.html

Shows how to use the benchmark fixture: pass a function to benchmark to time it, and verify the result. The example includes a function to benchmark, a test that benchmarks it with default arguments, and another test that passes an argument (0.001) to the benchmarked function.

```python
def something(duration=0.000001):
    """
    Function that needs some serious benchmarking.
    """
    time.sleep(duration)
    # You may return anything you want, like the result of a computation
    return 123

def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)

    # Extra code, to verify that the run completed correctly.
    # Sometimes you may want to check the result, fast functions
    # are no good if they return incorrect results :-)
    assert result == 123

def test_my_stuff_different_arg(benchmark):
    # benchmark something, but add some arguments
    result = benchmark(something, 0.001)
    assert result == 123
```

--------------------------------

### pytest_benchmark_generate_commit_info hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Completely replace the generated commit_info. Example returns a dict with the svnversion output.

```python
def pytest_benchmark_generate_commit_info(config):
    return {'id': subprocess.check_output(['svnversion']).strip()}

```

--------------------------------

### Compare two saved runs with pytest-benchmark CLI

Source: https://pytest-benchmark.readthedocs.io/en/stable/comparing.html

Use the comparison CLI to list and compare past runs by their saved IDs. The example compares runs 0001 and 0002.

```bash
pytest-benchmark compare 0001 0002

```

--------------------------------

### Patching internal methods with benchmark_weave

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Uses the experimental benchmark_weave fixture to patch the internal method of a class. Requires aspectlib (install with pip install aspectlib or pip install pytest-benchmark[aspect]).

```python
def test_foo(benchmark):
    benchmark.weave(Foo.internal, lazy=True):
    f = Foo()
    f.run()

```

--------------------------------

### pytest_benchmark_update_json hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Adds custom fields to the output JSON. Example adds a 'foo' field.

```python
def pytest_benchmark_update_json(config, benchmarks, output_json):
    output_json['foo'] = 'bar'

```

--------------------------------

### pytest_benchmark_group_stats hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Custom grouping example. When --benchmark-group-by=special is used, groups benchmarks by a custom attribute (bench.special).

```python
@pytest.hookimpl(wrapper=True)
def pytest_benchmark_group_stats(config, benchmarks, group_by):
    outcome = yield
    if group_by == "special":  # when you use --benchmark-group-by=special
        result = defaultdict(list)
        for bench in benchmarks:
            # `bench.special` doesn't exist, replace with whatever you need
            result[bench.special].append(bench)
        outcome.force_result(result.items())

```

--------------------------------

### Prevent stripping errored benchmarks with pytest_benchmark_generate_json

Source: https://pytest-benchmark.readthedocs.io/en/stable/_modules/pytest_benchmark/hookspec.html

Implement this hook to prevent stripping benchmarks that have errors from the output. The example clears the has_error flag for all benchmarks.

```python
@pytest.hookimpl(wrapper=True)
def pytest_benchmark_generate_json(config, benchmarks, include_data, machine_info, commit_info):
    for bench in benchmarks:
        bench.has_error = False
    yield
```

--------------------------------

### Add commit message with pytest_benchmark_update_commit_info

Source: https://pytest-benchmark.readthedocs.io/en/stable/_modules/pytest_benchmark/hookspec.html

Use this hook to add something into the commit_info, like the commit message. The example adds the last git commit message.

```python
def pytest_benchmark_update_commit_info(config, commit_info):
    commit_info['message'] = subprocess.check_output(['git', 'log', '-1', '--pretty=%B']).strip()
```

--------------------------------

### py.test-benchmark CLI usage and commands

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Shows the usage of the py.test-benchmark command with its subcommands: help, list, and compare. The compare command accepts most --benchmark options except the prefix.

```text
py.test-benchmark [-h [COMMAND]] [--storage URI] [--netrc [NETRC]]
                  [--verbose]
                  {help,list,compare} ...

Commands:
    help       Display help and exit.
    list       List saved runs.
    compare    Compare saved runs.

```

--------------------------------

### Run all checks and docs builder

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Run all checks and the documentation builder with a single command.

```bash
tox
```

--------------------------------

### Create a branch for local development

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Create a new branch for your bugfix or feature.

```bash
git checkout -b name-of-your-bugfix-or-feature
```

--------------------------------

### Run all test environments in parallel

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Run all test environments in parallel using tox -p auto.

```bash
tox -p auto
```

--------------------------------

### Commit and push changes

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Stage all changes, commit with a descriptive message, and push the branch to GitHub.

```bash
git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
```

--------------------------------

### Basic benchmark usage with return value check

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Benchmarks a function by passing it to the benchmark fixture. The function returns a value that can be asserted to verify correctness after benchmarking.

```python
def something(duration=0.000001):
    """
    Function that needs some serious benchmarking.
    """
    time.sleep(duration)
    # You may return anything you want, like the result of a computation
    return 123

def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)

    # Extra code, to verify that the run completed correctly.
    # Sometimes you may want to check the result, fast functions
    # are no good if they return incorrect results :-)
    assert result == 123

```

--------------------------------

### Using benchmark.pedantic with a teardown function

Source: https://pytest-benchmark.readthedocs.io/en/stable/pedantic.html

Shows how to use a teardown function to clean up after each round. The teardown function receives the same args and kwargs as the target function.

```python
def stuff(a, b, c, foo):
    pass

def test_with_teardown(benchmark):
    def teardown():
        # cleanup the side effect of the previous bench mark round.
        pass
    benchmark.pedantic(stuff, teardown=teardown, rounds=100)

```

--------------------------------

### Clone the repository

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Clone your fork of pytest-benchmark locally using SSH.

```bash
git clone git@github.com:YOURGITHUBNAME/pytest-benchmark.git
```

--------------------------------

### pytest_benchmark_generate_machine_info(config)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to completely replace the generated machine_info. Return a dictionary with the desired machine information.

```APIDOC
## pytest_benchmark_generate_machine_info(config)

### Description
Hook to completely replace the generated machine_info. Return a dictionary with the desired machine information.

### Parameters
- **config** (object) - The pytest configuration object.

### Returns
- **dict** - The machine info to use.

### Example
```python
def pytest_benchmark_generate_machine_info(config):
    return {'user': getpass.getuser()}
```
```

--------------------------------

### Using benchmark.pedantic with args and kwargs

Source: https://pytest-benchmark.readthedocs.io/en/stable/pedantic.html

Shows how to call benchmark.pedantic with positional arguments, keyword arguments, iterations, and rounds. This mode disables automatic calibration, so you must specify iterations and rounds explicitly.

```python
def test_with_setup(benchmark):
    benchmark.pedantic(stuff, args=(1, 2, 3), kwargs={'foo': 'bar'}, iterations=10, rounds=100)

```

--------------------------------

### Compare multiple metrics and control sort order with --sort

Source: https://pytest-benchmark.readthedocs.io/en/stable/comparing.html

Use --between with multiple metrics and --sort to control the sort order of the comparison table.

```bash
pytest-benchmark compare --between=min,mean --sort=mean 0001 0002

```

--------------------------------

### Compare runs between source files with --between flag

Source: https://pytest-benchmark.readthedocs.io/en/stable/comparing.html

The --between flag pivots the table so each row is a unique benchmark, with columns for each source file's metric and the relative change. It takes a comma-separated list of metrics to compare.

```bash
pytest-benchmark compare --between=min 0001 0002

```

--------------------------------

### pytest_benchmark_generate_json(config, benchmarks, include_data, machine_info, commit_info)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to wholly customize the JSON output. Warning: improper customization may cause breakage if --benchmark-compare or --benchmark-histogram are used. By default, benchmarks with errors are stripped; to prevent this, use a wrapper hook.

```APIDOC
## pytest_benchmark_generate_json(config, benchmarks, include_data, machine_info, commit_info)

### Description
Hook to wholly customize the JSON output. Warning: improper customization may cause breakage if --benchmark-compare or --benchmark-histogram are used. By default, benchmarks with errors are stripped; to prevent this, use a wrapper hook.

### Parameters
- **config** (object) - The pytest configuration object.
- **benchmarks** (list) - List of benchmark objects.
- **include_data** (bool) - Whether to include data.
- **machine_info** (dict) - The machine info.
- **commit_info** (dict) - The commit info.

### Example
```python
@pytest.hookimpl(wrapper=True)
def pytest_benchmark_generate_json(config, benchmarks, include_data, machine_info, commit_info):
    for bench in benchmarks:
        bench.has_error = False
    yield
```
```

--------------------------------

### pytest_benchmark_scale_unit(config, unit, benchmarks, best, worst, sort)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to customize time scaling. Return a tuple (prefix, scale) for the given unit.

```APIDOC
## pytest_benchmark_scale_unit(config, unit, benchmarks, best, worst, sort)

### Description
Hook to customize time scaling. Return a tuple (prefix, scale) for the given unit.

### Parameters
- **config** (object) - The pytest configuration object.
- **unit** (str) - The measurement unit (e.g., 'seconds', 'operations').
- **benchmarks** (list) - List of benchmark objects.
- **best** (float) - The best time.
- **worst** (float) - The worst time.
- **sort** (bool) - Whether to sort.

### Returns
- **tuple** - (prefix, scale) for the unit.

### Example
```python
def pytest_benchmark_scale_unit(config, unit, benchmarks, best, worst, sort):
    if unit == 'seconds':
        prefix = ''
        scale = 1.0
    elif unit == 'operations':
        prefix = 'K'
        scale = 0.001
    else:
        raise RuntimeError("Unexpected measurement unit %r" % unit)

    return prefix, scale
```
```

--------------------------------

### Run a subset of tests

Source: https://pytest-benchmark.readthedocs.io/en/stable/contributing.html

Run a specific test or subset using pytest with the -k option.

```bash
tox -e envname -- pytest -k test_myfeature
```

--------------------------------

### pytest_benchmark_compare_machine_info hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Builtin hook that warns when the machine info differs from the saved benchmark. Use to implement custom checks or abort execution.

```python
def pytest_benchmark_compare_machine_info(config, benchmarksession, machine_info, compared_benchmark):
    if compared_benchmark["machine_info"] != machine_info:
        benchmarksession.logger.warning(
            "Benchmark machine_info is different. Current: %s VS saved: %s." % (
                format_dict(machine_info),
                format_dict(compared_benchmark["machine_info"]),
            )
    )

```

--------------------------------

### pytest_benchmark_generate_commit_info(config)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to completely replace the generated commit_info. Return a dictionary with the desired commit information.

```APIDOC
## pytest_benchmark_generate_commit_info(config)

### Description
Hook to completely replace the generated commit_info. Return a dictionary with the desired commit information.

### Parameters
- **config** (object) - The pytest configuration object.

### Returns
- **dict** - The commit info to use.

### Example
```python
def pytest_benchmark_generate_commit_info(config):
    return {'id': subprocess.check_output(['svnversion']).strip()}
```
```

--------------------------------

### pytest_benchmark_generate_json hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Wrapper hook to prevent stripping benchmarks with errors from the JSON output. Warning: improper customization may cause breakage with --benchmark-compare or --benchmark-histogram.

```python
@pytest.hookimpl(wrapper=True)
def pytest_benchmark_generate_json(config, benchmarks, include_data, machine_info, commit_info):
    for bench in benchmarks:
        bench.has_error = False
    yield

```

--------------------------------

### pytest_benchmark_compare_machine_info(config, benchmarksession, machine_info, compared_benchmark)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to implement custom checks or abort execution when comparing benchmark machine info. The builtin hook warns if machine info differs between current and saved benchmarks.

```APIDOC
## pytest_benchmark_compare_machine_info(config, benchmarksession, machine_info, compared_benchmark)

### Description
Hook to implement custom checks or abort execution when comparing benchmark machine info. The builtin hook warns if machine info differs between current and saved benchmarks.

### Parameters
- **config** (object) - The pytest configuration object.
- **benchmarksession** (object) - The benchmark session object.
- **machine_info** (dict) - The current machine info.
- **compared_benchmark** (dict) - The compared benchmark data, including its machine_info.

### Example
```python
def pytest_benchmark_compare_machine_info(config, benchmarksession, machine_info, compared_benchmark):
    if compared_benchmark["machine_info"] != machine_info:
        benchmarksession.logger.warning(
            "Benchmark machine_info is different. Current: %s VS saved: %s." % (
                format_dict(machine_info),
                format_dict(compared_benchmark["machine_info"]),
            )
        )
```
```

--------------------------------

### Storing arbitrary values in benchmark.extra_info

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Adds custom key-value pairs to the benchmark's extra_info dictionary, which are saved in the JSON output when using --benchmark-autosave or similar options.

```python
def test_my_stuff(benchmark):
    benchmark.extra_info['foo'] = 'bar'
    benchmark(time.sleep, 0.02)

```

--------------------------------

### benchmark.pedantic()

Source: https://pytest-benchmark.readthedocs.io/en/stable/pedantic.html

Runs a benchmark in pedantic mode, allowing explicit control over rounds, iterations, and setup/teardown functions. This method is used to benchmark a target function with specified arguments and options.

```APIDOC
## benchmark.pedantic()

### Description
Runs a benchmark in pedantic mode, which disables automatic calibration and allows explicit control over rounds, iterations, and setup/teardown functions.

### Method
pedantic

### Signature
benchmark.pedantic(target, args=(), kwargs=None, setup=None, rounds=1, warmup_rounds=0, iterations=1)

### Parameters
- **target** (callable) - Required - Function to benchmark.
- **args** (list or tuple) - Optional - Positional arguments to the target function.
- **kwargs** (dict) - Optional - Named arguments to the target function.
- **setup** (callable) - Optional - A function to call right before calling the target function in the first iteration of every round. The setup function can also return the arguments for the function (in case you need to create new arguments every time). Note: if you use a setup function then you cannot use the args, kwargs and iterations options.
- **teardown** (callable) - Optional - A function to call after the last iteration of every round. The teardown function receives the same args and kwargs as the target.
- **rounds** (int) - Optional - Number of rounds to run.
- **iterations** (int) - Optional - Number of iterations. The default value (1) is unsafe for benchmarking very fast functions that take under 100μs (100 microseconds).
- **warmup_rounds** (int) - Optional - Number of warmup rounds. Set to non-zero to enable warmup. Warmup will run with the same number of iterations.

### Request Example
```python
def test_with_setup(benchmark):
    benchmark.pedantic(stuff, args=(1, 2, 3), kwargs={'foo': 'bar'}, iterations=10, rounds=100)
```

### Response
No return value is documented.
```

--------------------------------

### pytest_benchmark_update_machine_info hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Adds the current user to the machine_info. Override in conftest.py to avoid warnings when machine_info differs.

```python
def pytest_benchmark_update_machine_info(config, machine_info):
    machine_info['user'] = getpass.getuser()

```

--------------------------------

### pytest_benchmark_update_machine_info(config, machine_info)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to add custom fields to the machine_info dictionary. If benchmarks are compared and machine_info is different, warnings will be shown.

```APIDOC
## pytest_benchmark_update_machine_info(config, machine_info)

### Description
Hook to add custom fields to the machine_info dictionary. If benchmarks are compared and machine_info is different, warnings will be shown.

### Parameters
- **config** (object) - The pytest configuration object.
- **machine_info** (dict) - The machine info dictionary to update.

### Example
```python
def pytest_benchmark_update_machine_info(config, machine_info):
    machine_info['user'] = getpass.getuser()
```
```

--------------------------------

### Using @pytest.mark.benchmark to set per-test options

Source: https://pytest-benchmark.readthedocs.io/en/stable/usage.html

Sets per-test options such as group, time limits, rounds, precision, timer, GC disabling, and warmup. The benchmarked function is defined inside the test and its return value is checked after the measurement.

```python
@pytest.mark.benchmark(
    group="group-name",
    min_time=0.1,
    max_time=0.5,
    min_rounds=5,
    precision=0.02,
    timer=time.time,
    disable_gc=True,
    warmup=False
)
def test_my_stuff(benchmark):
    @benchmark
    def result():
        # Code to be measured
        return time.sleep(0.000001)

    # Extra code, to verify that the run
    # completed correctly.
    # Note: this code is not measured.
    assert result is None

```

--------------------------------

### pytest_benchmark_scale_unit hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Custom time scaling. Returns prefix and scale for seconds and operations units, raises RuntimeError for unexpected units.

```python
def pytest_benchmark_scale_unit(config, unit, benchmarks, best, worst, sort):
    if unit == 'seconds':
        prefix = ''
        scale = 1.0
    elif unit == 'operations':
        prefix = 'K'
        scale = 0.001
    else:
        raise RuntimeError("Unexpected measurement unit %r" % unit)

    return prefix, scale

```

--------------------------------

### pytest_benchmark_update_json(config, benchmarks, output_json)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to add custom fields to the output JSON.

```APIDOC
## pytest_benchmark_update_json(config, benchmarks, output_json)

### Description
Hook to add custom fields to the output JSON.

### Parameters
- **config** (object) - The pytest configuration object.
- **benchmarks** (list) - List of benchmark objects.
- **output_json** (dict) - The output JSON dictionary to update.

### Example
```python
def pytest_benchmark_update_json(config, benchmarks, output_json):
    output_json['foo'] = 'bar'
```
```

--------------------------------

### pytest_benchmark_group_stats(config, benchmarks, group_by)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to customize grouping of benchmarks. Use a wrapper hook to override the grouping result.

```APIDOC
## pytest_benchmark_group_stats(config, benchmarks, group_by)

### Description
Hook to customize grouping of benchmarks. Use a wrapper hook to override the grouping result.

### Parameters
- **config** (object) - The pytest configuration object.
- **benchmarks** (list) - List of benchmark objects.
- **group_by** (str) - The grouping key (e.g., 'special' when using --benchmark-group-by=special).

### Example
```python
@pytest.hookimpl(wrapper=True)
def pytest_benchmark_group_stats(config, benchmarks, group_by):
    outcome = yield
    if group_by == "special":  # when you use --benchmark-group-by=special
        result = defaultdict(list)
        for bench in benchmarks:
            # `bench.special` doesn't exist, replace with whatever you need
            result[bench.special].append(bench)
        outcome.force_result(result.items())
```
```

--------------------------------

### pytest_benchmark_update_commit_info(config, commit_info)

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Hook to add custom fields to the commit_info dictionary.

```APIDOC
## pytest_benchmark_update_commit_info(config, commit_info)

### Description
Hook to add custom fields to the commit_info dictionary.

### Parameters
- **config** (object) - The pytest configuration object.
- **commit_info** (dict) - The commit info dictionary to update.

### Example
```python
def pytest_benchmark_update_commit_info(config, commit_info):
    commit_info['message'] = subprocess.check_output(['git', 'log', '-1', '--pretty=%B']).strip()
```
```

--------------------------------

### pytest_benchmark_update_commit_info hook implementation

Source: https://pytest-benchmark.readthedocs.io/en/stable/hooks.html

Adds the commit message to the commit_info using git log.

```python
def pytest_benchmark_update_commit_info(config, commit_info):
    commit_info['message'] = subprocess.check_output(['git', 'log', '-1', '--pretty=%B']).strip()

```
Tokens:
Raw