Pytest Data Loader

https://github.com/yugokato/pytest-data-loader
Admin
pytest-data-loader is a pytest plugin that simplifies data-driven testing by loading, transforming,...
Tokens:21,336
Snippets:139
Trust Score:7.7
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install pytest-data-loader

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Install the plugin using pip.

```bash
pip install pytest-data-loader
```

--------------------------------

### Development Environment Commands

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Commands to initialize the development environment, install git hooks, and execute tests using uv.

```bash
uv sync                      # create .venv with all dev dependencies
uv run pre-commit install    # install git hooks (keeps uv.lock in sync)
uv run pytest tests -n auto  # run the test suite
uv run tox                   # run the full test/lint matrix
```

--------------------------------

### Use data_loader fixture for dynamic paths

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Example of using the function-scoped data_loader fixture to load data where the path depends on CLI options and test parameters.

```python
import pytest
from pytest import FixtureRequest

from pytest_data_loader import DataLoaderFixture


@pytest.fixture(scope="session")
def env(request: FixtureRequest) -> str:
    """Target environment specified by the custom --env CLI option"""
    return request.config.getoption("--env")


@pytest.mark.parametrize("filename", ["case1.json", "case2.json"])
def test_env_specific_cases(data_loader: DataLoaderFixture, env: str, filename: str):
    # Construct path from a CLI option and parametrized value
    path = f"{env}/{filename}"
    case_data = data_loader(path)
    assert isinstance(case_data, dict)
```

--------------------------------

### Load file data with @load decorator

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Uses the @load decorator to inject file content into test functions. The first example loads JSON content, while the second example loads text content and the file path.

```python
# test_something.py

from pytest_data_loader import load


@load("data", "data1.json")
def test_something1(data):
    """
    data/data1.json: '{"foo": 1, "bar": 2}'
    """
    assert data == {"foo": 1, "bar": 2}


@load(("file_path", "data"), "data2.txt")
def test_something2(file_path, data):
    """
    tests/data/data2.txt: "line1\nline2\nline3"
    """
    assert file_path.name == "data2.txt"
    assert data == "line1\nline2\nline3"
```

--------------------------------

### Parametrize files from multiple directories

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Shows how to pass a list of directories to @parametrize_dir to collect and parametrize files from multiple locations.

```python
# test_something.py

from pytest_data_loader import parametrize_dir


@parametrize_dir("data", ["images", "logos"])
def test_something(data):
    """
    data/images/: contains 3 image files
    tests/data/logos/: contains 2 logo files
    """
    assert isinstance(data, bytes)
```

```shell
$ pytest -v
================================ test session starts =================================
<snip>
collected 5 items

tests/test_something.py::test_something[images/image.gif] PASSED                [ 20%]
tests/test_something.py::test_something[images/image.jpg] PASSED                [ 40%]
tests/test_something.py::test_something[images/image.png] PASSED                [ 60%]
tests/test_something.py::test_something[logos/logo.jpg] PASSED                  [ 80%]
tests/test_something.py::test_something[logos/logo.png] PASSED                  [100%]

================================= 5 passed in 0.01s ==================================
```

--------------------------------

### Combine shared context with @load and @parametrize_dir

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Stack @load with @parametrize_dir to test structured scenarios from a directory against shared context.

```python
from pytest_data_loader import load, parametrize_dir


@load("banned_words", "banned_words.txt")
@parametrize_dir("comment", "user_comments/flagged")    # Each comment is stored as a .txt file
def test_flagged_comments_contain_banned_words(banned_words, comment):
    """Validate that flagged comments contain at least one banned word."""
    assert any(word in comment.lower() for word in banned_words)
```

--------------------------------

### Parametrize from multiple files using glob patterns

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Shows how to load and concatenate data from multiple files using a glob pattern in @parametrize.

```python
# test_something.py

from pytest_data_loader import parametrize


@parametrize("data", "*.txt")   # or ["data1.txt", "data2.txt"]
def test_something(data):
    """
    The glob pattern matches: 
      tests/data/data1.txt: "line1\nline2"
      tests/data/data2.txt: "line3\nline4"
    """
    assert data in ["line1", "line2", "line3", "line4"]
```

--------------------------------

### Parametrize files in a single directory

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Demonstrates using @parametrize_dir to load files from a directory as bytes. The test function is executed once for each file found.

```python
# test_something.py

from pytest_data_loader import parametrize_dir


@parametrize_dir("data", "images")
def test_something(data):
    """
    data/images/: contains 3 image files
    """
    # parametrized as files
    assert isinstance(data, bytes)
```

```shell
$ pytest -v
================================ test session starts =================================
<snip>
collected 3 items                                                                              

tests/test_something.py::test_something[images/image.gif] PASSED                [ 33%]
tests/test_something.py::test_something[images/image.jpg] PASSED                [ 66%]
tests/test_something.py::test_something[images/image.png] PASSED                [100%]

================================= 3 passed in 0.01s ==================================
```

--------------------------------

### Combine shared context with @load and @parametrize

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Use @load to provide shared context and @parametrize to iterate over variable inputs.

```python
from pytest_data_loader import load, parametrize


@load("prices", "prices.json")
@parametrize("order", "orders.json")
def test_order_total_matches_expected(prices, order):
    """Validate that each order total is calculated correctly using the shared price catalog."""
    total = calculate_total(order, prices)
    assert total == order["expected_total"]
```

--------------------------------

### Parametrize test functions with file data

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Demonstrates using @parametrize to load data from JSON and text files. The decorator splits file content into logical parts based on the file type.

```python
# test_something.py

from pytest_data_loader import parametrize


@parametrize("data", "data1.json")
def test_something1(data):
    """
    data/data1.json: '{"foo": 1, "bar": 2}'
    """
    # parametrized as key–value pairs
    assert data in [("foo", 1), ("bar", 2)]


@parametrize(("file_path", "data"), "data2.txt")
def test_something2(file_path, data):
    """
    tests/data/data2.txt: "line1\nline2\nline3"
    """
    # parametrized as lines
    assert file_path.name == "data2.txt"
    assert data in ["line1", "line2", "line3"]
```

--------------------------------

### Parametrize PDF pages with custom reader and processors

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Uses pypdf.PdfReader with binary mode, parametrizes over pages, and extracts text with a processor. Demonstrates advanced loader options.

```python
@parametrize(
    "page_data", 
    "test.pdf", 
    reader=pypdf.PdfReader, 
    read_options={"mode": "rb"},
    parametrizer=lambda r: r.pages,
    processor=lambda p: p.extract_text().rstrip(),
)
def test_something(page_data: str):
    ...
```

--------------------------------

### Run pytest with multiple file parametrization

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Shows the output of running pytest when using glob patterns to load multiple files.

```shell
$ pytest -v
================================ test session starts =================================
<snip>
collected 4 items

tests/test_something.py::test_something[data1.txt:part1] PASSED                 [ 25%]
tests/test_something.py::test_something[data1.txt:part2] PASSED                 [ 50%]
tests/test_something.py::test_something[data2.txt:part1] PASSED                 [ 75%]
tests/test_something.py::test_something[data2.txt:part2] PASSED                 [100%]

================================= 4 passed in 0.01s ==================================
```

--------------------------------

### Run tests with pytest

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Executes the tests using the pytest command-line interface.

```shell
$ pytest -v
================================ test session starts =================================
<snip>
collected 2 items                                                                              

tests/test_something.py::test_something1[data1.json] PASSED                     [ 50%]
tests/test_something.py::test_something2[data2.txt] PASSED                      [100%]

================================= 2 passed in 0.01s ==================================
```

--------------------------------

### @load

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

The @load data loader supports optional parameters to customize how file data is loaded and passed to test functions.

```APIDOC
## @load

### Description
Loads data from a file and passes it to the test function. Supports lazy loading, custom readers, read options, transformation callbacks, marks, and parameter IDs.

### Parameters
- **lazy_loading** (bool) - Optional - Enable or disable lazy loading
- **reader** (callable) - Optional - A file reader the plugin should use to read the file data
- **read_options** (dict) - Optional - File read options passed to `open()`. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- **onload** (callable) - Optional - A function to transform or process loaded data before passing it to the test function. Must take either one (data) or two (file path, data) arguments. When `reader` is provided, its return value becomes the data passed to `onload()`. 
- **marks** (mark or collection of marks) - Optional - Pytest mark(s) to apply to the loaded data
- **id** (string) - Optional - The parameter ID for the loaded data. If not specified, the relative or absolute file path is used

### Notes
- `onload` must take either one (data) or two (file path, data) arguments
- When `reader` is provided, its return value becomes the data passed to `onload()`

## [DECORATOR] @parametrize

### Description
Loads data from a file and parametrizes a test function with parts of the loaded data.

### Parameters
- `lazy_loading`: Enable or disable lazy loading
- `reader`: A file reader the plugin should use to read the file data
- `read_options`: File read options (as a dict) the plugin passes to `open()`. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `onload`: A function to adjust the shape of the loaded data before splitting into parts
- `parametrizer`: A function to customize how the loaded data should be split
- `filter`: A function to filter the split data parts. Only matching parts are included as test parameters
- `processor`: A function to adjust the shape of each part data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded parts. Accepts a single mark or a collection of marks applied uniformly to all parts, or a function that returns mark(s) per part data
- `ids`: Parameter IDs for the loaded parts. Accepts an iterable of ID values or a function that returns an ID per part data

### @parametrize_dir
- `lazy_loading`: Enable or disable lazy loading
- `recursive`: Recursively load files from all subdirectories of the given directory. Defaults to `False`. This option is ignored for glob patterns. Use `**` instead for recursive matching
- `reader`: A function that determines the file reader for each file path
- `read_options`: File read options the plugin passes to `open()` when reading files. Accepts a dict applied uniformly to all files, or a function that returns the options for matching file paths. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `filter`: A function to filter file paths. Only the contents of matching file paths are included as the test parameters
- `processor`: A function to adjust the shape of each loaded file's data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded files. Accepts a single mark or a collection of marks applied uniformly to all files, or a function that returns mark(s) for matching file paths
- `ids`: Parameter IDs for the loaded files. Accepts an iterable of ID values or a function that returns an ID to matching file path

> [!NOTE]
> - `filter` must take only one argument `(file_path)`
> - `reader` and callable `read_options`, `marks`, and `ids` additionally accept a two-argument form `(idx, file_path)`

### @load
- `lazy_loading`: Enable or disable lazy loading
- `reader`: A file reader the plugin should use to read the file data
- `read_options`: File read options (as a dict) the plugin passes to `open()`. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `onload`: A function to transform or process loaded data before passing it to the test function
- `marks`: Pytest mark(s) to apply to the loaded data. Accepts a single mark or a collection of marks
- `id`: The parameter ID for the loaded data. If not specified, the relative or absolute file path is used

> [!NOTE]
> `onload` must take either one (data) or two (file path, data) arguments. When `reader` is provided, 
its return value becomes the data passed to `onload()`


### @parametrize
- `lazy_loading`: Enable or disable lazy loading
- `reader`: A file reader the plugin should use to read the file data
- `read_options`: File read options (as a dict) the plugin passes to `open()`. Supports only the `mode`, `encoding`, 
`errors`, and `newline` keys
- `onload`: A function to adjust the shape of the loaded data before splitting into parts
- `parametrizer`: A function to customize how the loaded data should be split
- `filter`: A function to filter the split data parts. Only matching parts are included as test parameters
- `processor`: A function to adjust the shape of each part data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded parts. Accepts a single mark or a collection of marks applied uniformly to all 
           parts, or a function that returns mark(s) per part data
- `ids`: Parameter IDs for the loaded parts. Accepts an iterable of ID values or a function that returns an ID per part data

> [!NOTE]
> - `onload`, `parametrizer`, and `filter` must take one `(data)` or two `(file_path, data)` arguments
> - `processor` and callable `marks` and `ids` additionally accept a three-argument form `(idx, file_path, data)`, where `idx` is the zero-based post-filter position of the item
> - When `reader` is provided, its return value becomes the data passed to these callables


### @parametrize_dir
- `lazy_loading`: Enable or disable lazy loading
- `recursive`: Recursively load files from all subdirectories of the given directory. Defaults to `False`. 
This option is ignored for glob patterns. Use `**` instead for recursive matching
- `reader`: A function that determines the file reader for each file path
- `read_options`: File read options the plugin passes to `open()` when reading files. Accepts a dict applied
                  uniformly to all files, or a function that returns the options for matching file paths.
                  Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `filter`: A function to filter file paths. Only the contents of matching file paths are included as the test 
parameters
- `processor`: A function to adjust the shape of each loaded file's data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded files. Accepts a single mark or a collection of marks applied uniformly to all 
           files, or a function that returns mark(s) for matching file paths
- `ids`: Parameter IDs for the loaded files. Accepts an iterable of ID values or a function that returns an ID to 
         matching file path

> [!NOTE]
> - `filter` must take only one argument `(file_path)`
> - `reader` and callable `read_options`, `marks`, and `ids` additionally accept a two-argument form `(idx, file_path)`

### @parametrize_dir
- `lazy_loading`: Enable or disable lazy loading
- `recursive`: Recursively load files from all subdirectories of the given directory. Defaults to `False`. 
This option is ignored for glob patterns. Use `**` instead for recursive matching
- `reader`: A function that determines the file reader for each file path
- `read_options`: File read options the plugin passes to `open()` when reading files. Accepts a dict applied
                  uniformly to all files, or a function that returns the options for matching file paths.
                  Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `filter`: A function to filter file paths. Only the contents of matching file paths are included as the test 
parameters
- `processor`: A function to adjust the shape of each loaded file's data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded files. Accepts a single mark or a collection of marks applied uniformly to all 
           files, or a function that returns mark(s) for matching file paths
- `ids`: Parameter IDs for the loaded files. Accepts an iterable of ID values or a function that returns an ID to 
         matching file path

> [!NOTE]
> - `filter` must take only one argument `(file_path)`
> - `reader` and callable `read_options`, `marks`, and `ids` additionally accept a two-argument form `(idx, file_path)`

### @load_file
- `lazy_loading`: Enable or disable lazy loading
- `reader`: A file reader the plugin should use to read the file data
- `read_options`: File read options (as a dict) the plugin passes to `open()`. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `onload`: A function to transform or process loaded data before passing it to the test function
- `marks`: Pytest mark(s) to apply to the loaded data. Accepts a single mark or a collection of marks
- `id`: The parameter ID for the loaded data. If not specified, the relative or absolute file path is used

> [!NOTE]
> `onload` must take either one (data) or two (file path, data) arguments. When `reader` is provided, its return value becomes the data passed to `onload()`

### @parametrize
- `lazy_loading`: Enable or disable lazy loading
- `reader`: A file reader the plugin should use to read the file data
- `read_options`: File read options (as a dict) the plugin passes to `open()`. Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `onload`: A function to adjust the shape of the loaded data before splitting into parts
- `parametrizer`: A function to customize how the loaded data should be split
- `filter`: A function to filter the split data parts. Only matching parts are included as test parameters
- `processor`: A function to adjust the shape of each part data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded parts. Accepts a single mark or a collection of marks applied uniformly to all parts, or a function that returns mark(s) per part data
- `ids`: Parameter IDs for the loaded parts. Accepts an iterable of ID values or a function that returns an ID per part data

> [!NOTE]
> - `onload`, `parametrizer`, and `filter` must take one `(data)` or two `(file_path, data)` arguments
> - `processor` and callable `marks` and `ids` additionally accept a three-argument form `(idx, file_path, data)`, where `idx` is the zero-based post-filter position of the item
> - When `reader` is provided, its return value becomes the data passed to these callables

### @parametrize_dir
- `lazy_loading`: Enable or disable lazy loading
- `recursive`: Recursively load files from all subdirectories of the given directory. Defaults to `False`. 
This option is ignored for glob patterns. Use `**` instead for recursive matching
- `reader`: A function that determines the file reader for each file path
- `read_options`: File read options the plugin passes to `open()` when reading files. Accepts a dict applied
                  uniformly to all files, or a function that returns the options for matching file paths.
                  Supports only the `mode`, `encoding`, `errors`, and `newline` keys
- `filter`: A function to filter file paths. Only the contents of matching file paths are included as the test 
parameters
- `processor`: A function to adjust the shape of each loaded file's data before passing it to the test function
- `marks`: Pytest mark(s) for the loaded files. Accepts a single mark or a collection of marks applied uniformly to all 
           files, or a function that returns mark(s) for matching file paths
- `ids`: Parameter IDs for the loaded files. Accepts an iterable of ID values or a function that returns an ID to 
         matching file path

> [!NOTE]
> - `filter` must take only one argument `(file_path)`
> - `reader` and callable `read_options`, `marks`, and `ids` additionally accept a two-argument form `(idx, file_path)`
> - When `reader` is provided, its return value becomes the data passed to these callables



## The `data_loader` fixture

This fixture provides a way to load data from a file path at test time. It is available to all tests and can be used to load data from a file path. The fixture is function-scoped, meaning it is created and destroyed for each test function that uses it.

### Usage

```python
def test_example(data_loader):
    data = data_loader(
```

--------------------------------

### Run pytest with parametrized data

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Shows the output of running pytest on the parametrized test functions.

```shell
$ pytest -v
================================ test session starts =================================
<snip>
collected 5 items                                                                              

tests/test_something.py::test_something1[data1.json:part1] PASSED               [ 20%]
tests/test_something.py::test_something1[data1.json:part2] PASSED               [ 40%]
tests/test_something.py::test_something2[data2.txt:part1] PASSED                [ 60%]
tests/test_something.py::test_something2[data2.txt:part2] PASSED                [ 80%]
tests/test_something.py::test_something2[data2.txt:part3] PASSED                [100%]

================================= 5 passed in 0.01s ==================================
```

--------------------------------

### Per-test CSV reader with csv.reader and csv.DictReader

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Specifies file readers directly in loader options for individual tests. The first test uses csv.reader, the second uses csv.DictReader with parametrize. Both use shared read_options.

```python
# test_something.py

import csv

from pytest_data_loader import load, parametrize

read_options = {"encoding": "utf-8-sig", "newline": ""}


@load("data", "data.csv", reader=csv.reader, read_options=read_options)
def test_something1(data):
    """Load CSV file with csv.reader reader"""
    for row in data:
        assert isinstance(row, list)


@parametrize("data", "data.csv", reader=csv.DictReader, read_options=read_options)
def test_something2(data):
    """Parametrize CSV file data with csv.DictReader reader"""
    assert isinstance(data, dict)
```

--------------------------------

### Generate Cartesian product with @parametrize

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Stack multiple @parametrize decorators to automatically test all combinations of inputs.

```python
from pytest_data_loader import parametrize


@parametrize("user", "users.txt")
@parametrize("feature", "features.txt")
def test_user_feature_access_matrix(user, feature):
    """Validate access control for every user-feature combination."""
    assert can_access(user, feature)
```

--------------------------------

### Load multiple datasets with @load

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Stack multiple @load decorators to inject independent datasets into a single test function.

```python
from pytest_data_loader import load


@load("input_data", "input.json")
@load("expected_output", "expected.json")
def test_transformation_matches_expected_output(input_data, expected_output):
    """Verify that transforming input data produces the expected output."""
    assert do_something(input_data) == expected_output
```

--------------------------------

### Load test data with @load

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Use the @load decorator to inject data from a file into a test function.

```python
from pytest_data_loader import load


@load("data", "example.json")
def test_example(data):
    """
    example.json: '{"foo": 1, "bar": 2}'
    """
    assert isinstance(data, dict)
    assert data["foo"] == 1
```

--------------------------------

### Test using registered CSV reader

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Uses the @load decorator to load data.csv with the registered reader. The test function receives the parsed rows as a list.

```python
# test_something.py

from pytest_data_loader import load


@load("data", "data.csv")
def test_something(data):
    """Load CSV file with registered file reader"""
    for row in data:
        assert isinstance(row, list)
```

--------------------------------

### Create custom-scoped data loader fixture

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Use make_data_loader_fixture() to define a data loader with a scope other than function, such as session scope, in a conftest.py file.

```python
# conftest.py

import pytest

from pytest_data_loader import DataLoaderFixture, make_data_loader_fixture

session_data_loader = make_data_loader_fixture(scope="session")


@pytest.fixture(scope="session")
def api_config(session_data_loader: DataLoaderFixture, env: str) -> dict:
    return session_data_loader(f"{env}/config.json")
```

--------------------------------

### Register CSV reader in conftest.py

Source: https://github.com/yugokato/pytest-data-loader/blob/main/README.md

Registers a file reader for .csv files using pytest_data_loader.register_reader() with read_options. This applies to all tests in the same directory and subdirectories.

```python
# conftest.py

import csv

import pytest_data_loader


pytest_data_loader.register_reader(".csv", csv.reader, read_options={"newline": ""})
```
Tokens:
Raw