### Install tox Source: https://github.com/pytest-dev/pytest-xdist/blob/master/RELEASING.rst Install tox in a virtual environment to manage release tasks. ```bash $ pip install tox ``` -------------------------------- ### Install pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/index.md Install the pytest-xdist plugin using pip. This is the standard way to add the plugin to your Python environment. ```bash pip install pytest-xdist ``` -------------------------------- ### Install pytest-xdist with psutil extra Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/index.md Install pytest-xdist with the 'psutil' extra to enable CPU detection using the psutil library. This is recommended for accurate CPU count detection. ```bash pip install pytest-xdist[psutil] ``` -------------------------------- ### Manually trigger deploy workflow Source: https://github.com/pytest-dev/pytest-xdist/blob/master/RELEASING.rst Manually start the 'deploy' GitHub Actions workflow for pytest-xdist, specifying the version. This is an alternative to using tox. ```bash gh workflow run deploy.yml -R pytest-dev/pytest-xdist --ref release-X.Y.Z --field version=X.Y.Z ``` -------------------------------- ### Instantiate a Python 3.9 subprocess for tests Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/subprocess.md Use this command to start a subprocess that runs with the specified Python interpreter. Ensure the interpreter is in your system's binary lookup path. ```default pytest -d --tx popen//python=python3.9 ``` -------------------------------- ### Parametrize with Ordered List in pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/known-limitations.md A workaround for the unordered set limitation, this example shows how converting the parametrized values to a list guarantees a consistent order across workers. ```python import pytest @pytest.mark.parametrize("param", ["a", "b"]) def test_pytest_parametrize_unordered(param): pass ``` -------------------------------- ### Parametrize with Sorted Sequence in pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/known-limitations.md Another workaround for the unordered set limitation, this example demonstrates sorting the parametrized values to ensure a consistent order across workers. ```python import pytest @pytest.mark.parametrize("param", sorted({"a", "b"})) def test_pytest_parametrize_unordered(param): pass ``` -------------------------------- ### Configure Default Pytest Options in ini file Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Specify default command-line options for pytest in the `pytest` section of an ini file (e.g., `pytest.ini`, `tox.ini`, `setup.cfg`). This example sets the default number of subprocesses to 3. ```ini [pytest] addopts = -n3 ``` -------------------------------- ### Configure Default Distribution Mode in pytest.ini Source: https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst Set a default distribution mode for pytest-xdist in your pytest configuration file. This example sets the default to 'loadscope'. ```ini [pytest] addopts = --dist loadscope ``` -------------------------------- ### Run socket server for remote tests Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/remote.md Starts a socket server to listen for incoming test execution requests. This server can then be targeted by pytest for remote test execution. ```python python socketserver.py ``` -------------------------------- ### Identify pytest-xdist worker or master node Source: https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst Use these utility functions to determine if the current process is a worker or master node within a pytest-xdist setup. This is useful for conditional logic within tests or plugins. ```python import xdist if xdist.is_xdist_worker(): print("Running as a worker node") elif xdist.is_xdist_master(): print("Running as the master node") worker_id = xdist.get_xdist_worker_id() if worker_id: print(f"Worker ID: {worker_id}") ``` -------------------------------- ### Create multiple Python 3.9 subprocesses for load balancing Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/subprocess.md Prefix the `--tx` option value with a number to create multiple subprocesses. Tests will be automatically load-balanced across these processes. ```default --tx 3*popen//python=python3.9 ``` -------------------------------- ### Create Unique Database per Test Run with testrun_uid Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Utilize the `testrun_uid` fixture to generate globally unique identifiers for each test run, enabling the creation of isolated resources like databases. Requires `posix_ipc` and `setproctitle` for full functionality. ```python import pytest from posix_ipc import Semaphore, O_CREAT @pytest.fixture(scope="session", autouse=True) def create_unique_database(testrun_uid): """create a unique database for this particular test run""" database_url = f"psql://myapp-{testrun_uid}" with Semaphore(f"/{testrun_uid}-lock", flags=O_CREAT, initial_value=1): if not database_exists(database_url): create_database(database_url) @pytest.fixture() def db(testrun_uid): """retrieve unique database""" database_url = f"psql://myapp-{testrun_uid}" return database_get_instance(database_url) ``` -------------------------------- ### Run tests with logical CPU cores Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Use '-n logical' to utilize the number of logical CPU cores. This requires Python 3.13+ or the 'psutil' package. It falls back to '-n auto' if unavailable. ```bash pytest -n logical ``` -------------------------------- ### Run tests with a specific number of processes Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Specify an exact number of processes to use for parallel test execution by passing a number to the '-n' option. ```bash pytest -n 8 ``` -------------------------------- ### Run tests on multiple platforms simultaneously Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/remote.md Distributes tests to run on multiple specified platforms concurrently. Failures from all platforms are reported together. ```bash pytest --dist=each --tx=spec1 --tx=spec2 ``` -------------------------------- ### Update release files with tox Source: https://github.com/pytest-dev/pytest-xdist/blob/master/RELEASING.rst Use tox to update necessary files for a new release, specifying the version number. ```bash $ tox -e release -- X.Y.Z ``` -------------------------------- ### Distribute tests to remote SSH account Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/remote.md Synchronizes a package to a remote SSH account and distributes tests for execution. Ensure all test directories have an __init__.py file for correct module path referencing. ```bash pytest -d --rsyncdir mypkg --tx ssh=myhostpopen mypkg/tests/unit/test_something.py ``` -------------------------------- ### Parametrize with Unordered Set in pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/known-limitations.md Demonstrates how using an unordered set with pytest.mark.parametrize can lead to inconsistent test collection order across workers, causing errors. This is a known limitation. ```python import pytest @pytest.mark.parametrize("param", {"a", "b"}) def test_pytest_parametrize_unordered(param): pass ``` -------------------------------- ### Run multiple workers on remote machines via proxy Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/remote.md Creates a proxy gateway for a remote machine and launches multiple workers through it. The proxy itself does not run a worker. ```bash pytest -d --px id=my_proxy//socket=192.168.1.102:8888 --tx 5*popen//via=my_proxy ``` -------------------------------- ### Run tests via socket server Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/remote.md Executes tests using a remote socket server. Specify the server's IP address and port, and ensure the package directory is synchronized. ```bash pytest -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg ``` -------------------------------- ### Distribute Tests Across Multiple CPUs with pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/README.rst Use this command to distribute tests across multiple CPUs. pytest will spawn worker processes equal to the number of available CPUs and distribute tests randomly. ```bash pytest -n auto ``` -------------------------------- ### Configure Default Test Execution Environments in ini file Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Define default test execution environments using the `--tx` option within the `pytest` section of an ini file. This allows specifying different Python interpreters or remote execution targets. ```ini [pytest] addopts = --tx ssh=myhost//python=python3.9 --tx ssh=myhost//python=python3.6 ``` -------------------------------- ### Ensure Session-Scoped Fixture Executes Once with FileLock Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Use this fixture pattern with FileLock to ensure resource-intensive or configuration-setting fixtures execute only once across multiple pytest-xdist workers. The first worker to request the fixture produces the data, while subsequent workers read from the cached file. ```python import json import pytest from filelock import FileLock @pytest.fixture(scope="session") def session_data(tmp_path_factory, worker_id): if worker_id == "master": # not executing in with multiple workers, just produce the data and let # pytest's fixture caching do its job return produce_expensive_data() # get the temp directory shared by all workers root_tmp_dir = tmp_path_factory.getbasetemp().parent fn = root_tmp_dir / "data.json" with FileLock(str(fn) + ".lock"): if fn.is_file(): data = json.loads(fn.read_text()) else: data = produce_expensive_data() fn.write_text(json.dumps(data)) return data ``` -------------------------------- ### Disable xdist and run tests in the main process Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Use '-n 0' to disable pytest-xdist and run all tests sequentially within the main process. This is useful for debugging or when parallelization is not desired. ```bash pytest -n 0 ``` -------------------------------- ### Create Worker-Specific Log Files in pytest-xdist Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Configure pytest to create a unique log file for each worker process by leveraging the PYTEST_XDIST_WORKER environment variable within the pytest_configure hook. This is useful for debugging or tracking worker-specific test runs. ```python # content of conftest.py def pytest_configure(config): worker_id = os.environ.get("PYTEST_XDIST_WORKER") if worker_id is not None: logging.basicConfig( format=config.getini("log_file_format"), filename=f"tests_{worker_id}.log", level=config.getini("log_file_level"), ) ``` -------------------------------- ### Implement pytest_xdist_auto_num_workers hook Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Implement the pytest_xdist_auto_num_workers hook in conftest.py to programmatically determine the number of worker processes. This hook takes priority over environment variables. ```python def pytest_xdist_auto_num_workers(config): # Example: return 4 workers if user requested 'auto' if config.option.numprocesses == "auto": return 4 # Fallback to default behavior if not 'auto' or if hook doesn't specify return None ``` -------------------------------- ### Identify Worker Process with worker_id Fixture Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Use the `worker_id` fixture within tests or fixtures to determine the identity of the current xdist worker process. When xdist is disabled, it returns 'master'. ```python import pytest @pytest.fixture() def user_account(worker_id): """use a different account in each xdist worker""" return "account_%s" % worker_id ``` -------------------------------- ### Specify Rsync Directories in ini file Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/how-to.md Configure directories to be included or excluded during rsync synchronization in pytest-xdist by specifying `rsyncdirs` and `rsyncignore` in the `pytest` section of an ini file. Paths are relative to the configuration file's location. ```ini [pytest] rsyncdirs = . mypkg helperpkg rsyncignore = .hg ``` -------------------------------- ### Test function that may crash Source: https://github.com/pytest-dev/pytest-xdist/blob/master/example/boxed.txt This Python test function is designed to simulate a crashing process by sending a kill signal under specific conditions. It is intended to be run with pytest-xdist's --boxed option to observe how crashes are handled. ```python import pytest import os import time # run test function 50 times with different argument @pytest.mark.parametrize("arg", range(50)) def test_func(arg): time.sleep(0.05) # each tests takes a while if arg % 19 == 0: os.kill(os.getpid(), 15) ``` -------------------------------- ### Group fixtures with xdist_group mark Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Apply the '@pytest.mark.xdist_group' mark to pytest fixtures to group tests that depend on them. This ensures tests using the same fixture group run in the same worker. ```python @pytest.fixture( scope="session", params=[ pytest.param( "chrome", marks=pytest.mark.xdist_group("chrome"), ), pytest.param( "firefox", marks=pytest.mark.xdist_group("firefox"), ), pytest.param( "edge", marks=pytest.mark.xdist_group("edge"), ), ], ) def setup_container(): pass @pytest.mark.xdist_group(name="data-store") def test_data_store(setup_container): ... ``` -------------------------------- ### Configure auto-detected workers via environment variable Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Set the PYTEST_XDIST_AUTO_NUM_WORKERS environment variable to customize the number of processes used with '-n auto'. This setting is specific to pytest-xdist. ```bash export PYTEST_XDIST_AUTO_NUM_WORKERS=4 ``` -------------------------------- ### Group tests by custom xdist_group mark Source: https://github.com/pytest-dev/pytest-xdist/blob/master/docs/distribution.md Use the '@pytest.mark.xdist_group' decorator to ensure specific tests run within the same worker process. This is useful for tests with shared resources or state. ```python @pytest.mark.xdist_group(name="group1") def test1(): pass class TestA: @pytest.mark.xdist_group("group1") def test2(): pass ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.