### Configure Remote Filesystem (SFTP Example)

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Demonstrates configuring a remote filesystem, using SFTP as an example, and associating it with a datapackage.

```python
from fsspec.implementations.sftp import SFTPFileSystem

fs = SFTPFileSystem(host="example.com", username="user")
dp = create_datapackage(fs=fs, name="remote")
```

--------------------------------

### Install Package with Development Requirements

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Install the package with development requirements for contributing to the project.

```console
$
pip install -e ".[dev]"

```

--------------------------------

### Install Package with Dev and Docs Extras

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Install the package with development and documentation extras for building documentation locally.

```console
$
pip install -e ".[dev,docs]"

```

--------------------------------

### Install Package with Development and Docs Requirements

Source: https://github.com/brightway-lca/bw_processing/blob/main/docs/content/contributing.md

Install the package with development and documentation requirements using pip.

```console
pip install -e ".[dev,docs]"
```

--------------------------------

### Install Package with Testing Requirements

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Install the package with testing requirements to run the test suite.

```console
$
pip install -e ".[testing]"

```

--------------------------------

### Install pre-commit with Pip

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Install pre-commit using pip for Git hook management.

```console
$
pip install pre-commit

```

--------------------------------

### Example: Using Default and Custom Licenses for Datapackages

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/types.md

Demonstrates creating a datapackage using the default license and overriding it with a custom license. Shows how to pass license information during datapackage creation.

```python
from bw_processing import create_datapackage, DEFAULT_LICENSES

# Use default license
dp = create_datapackage(name="data")

# Override with custom license
dp = create_datapackage(
    name="data",
    metadata={
        "licenses": [
            {
                "name": "CC-BY-4.0",
                "path": "https://creativecommons.org/licenses/by/4.0/",
                "title": "Creative Commons Attribution 4.0",
            }
        ]
    }
)
```

--------------------------------

### Load and Filter a Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/INDEX.md

This example demonstrates how to load an existing datapackage and filter its resources by attribute. It also shows how to access specific resource data and metadata.

```python
from bw_processing import load_datapackage
from pathlib import Path

dp = load_datapackage(Path("./data"))

# Get only technosphere resources
tech_only = dp.filter_by_attribute("matrix", "technosphere_matrix")

# Access a specific resource
data, metadata = dp.get_resource("my-resource.data")
```

--------------------------------

### Example: Adding a persistent array with ParamLabelSchema

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/param-labels.md

Demonstrates creating a datapackage and adding a persistent array using a ParamLabelSchema for structured parameter labels.

```python
import numpy as np
from bw_processing import (
    create_datapackage,
    INDICES_DTYPE,
    ParamLabelField,
    ParamLabelSchema,
)

dp = create_datapackage()

# Define schema for activity references
schema = ParamLabelSchema(
    fields=[
        ParamLabelField(name="name", type="string"),
        ParamLabelField(name="database", type="string"),
        ParamLabelField(name="year", type="integer", required=False),
    ],
    description="Brightway activity reference"
)

# Labels matching the schema
param_labels = [
    {"name": "electricity", "database": "ecoinvent", "year": 2020},
    {"name": "heat", "database": "ecoinvent"},
]

indices = np.array([(1, 4), (2, 5)], dtype=INDICES_DTYPE)
data = np.array([[10.0, 20.0], [40.0, 50.0]])  # 2x2 array
params = np.array([
    [25.0, 30.0],   # Temperature
    [1.0, 1.1],     # Pressure
])

dp.add_persistent_array(
    matrix="technosphere",
    indices_array=indices,
    data_array=data,
    params_array=params,
    param_labels=param_labels,
    param_label_schema=schema,
)
```

--------------------------------

### Indexing Workflow Example

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/indexing-utilities.md

Demonstrates a typical workflow for handling indices in data exchange using Brightway2. It covers creating an initial datapackage, reindexing to external IDs, and resetting to sequential indices.

```python
from pathlib import Path
from bw_processing import create_datapackage, load_datapackage, reindex, reset_index
import numpy as np

# 1. Create initial datapackage with arbitrary indices
dp = create_datapackage(dirpath=Path("./data"), name="initial")
indices = np.array([(999, 888), (777, 666)], dtype=INDICES_DTYPE)
data = np.array([1.5, 2.0])
dp.add_persistent_vector(
    matrix="technosphere",
    indices_array=indices,
    data_array=data,
)
dp.finalize_serialization()

# 2. Load and reindex to match external database
dp = load_datapackage(Path("./data"))
external_data = [
    {"name": "item_1", "id": 1},
    {"name": "item_2", "id": 2},
]
dp = reindex(dp, "mapping-meta", external_data, fields=["name"])

# 3. Reset to sequential indices if needed
dp = reset_index(dp, "final-mapping")
```

--------------------------------

### Example: Defining fields for ParamLabelSchema

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/param-labels.md

Shows how to define ParamLabelField objects for activity references, specifying names, types, and descriptions.

```python
from bw_processing import ParamLabelField, ParamLabelSchema

# Define fields for activity references
name_field = ParamLabelField(
    name="activity_name",
    type="string",
    description="Name of the activity"
)

year_field = ParamLabelField(
    name="year",
    type="integer",
    required=False,
    description="Reference year"
)
```

--------------------------------

### Example: Merging Datapackages with a Boolean Mask

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/merging-utilities.md

Demonstrates how to merge two datapackages, selecting data from the first where the mask is True and from the second where it is False. This example creates two datapackages, defines a mask, and then merges them, showing the resulting combined data.

```python
import numpy as np
from bw_processing import (
    create_datapackage,
    load_datapackage,
    merge_datapackages_with_mask,
    INDICES_DTYPE,
)
from pathlib import Path

# Create first datapackage (base scenario)
dp_base = create_datapackage(name="base")
indices1 = np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE)
data1 = np.array([1.0, 2.0])
dp_base.add_persistent_vector(
    matrix="technosphere",
    indices_array=indices1,
    data_array=data1,
    name="base-data"
)

# Create second datapackage (alternative scenario)
dp_alt = create_datapackage(name="alternative")
indices2 = np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE)
data2 = np.array([1.5, 2.5])
dp_alt.add_persistent_vector(
    matrix="technosphere",
    indices_array=indices2,
    data_array=data2,
    name="alt-data"
)

# Create mask: use first value from dp_base, second from dp_alt
mask = np.array([True, False])

# Merge
dp_merged = merge_datapackages_with_mask(
    first_dp=dp_base,
    first_resource_group_label="base-data",
    second_dp=dp_alt,
    second_resource_group_label="alt-data",
    mask_array=mask,
    metadata={"name": "merged-scenario"}
)

# Result: combined data [1.0, 2.5]
# (1.0 from base where mask=True, 2.5 from alt where mask=False)
```

--------------------------------

### Install pre-commit with Conda

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Install pre-commit using Conda or Mamba for Git hook management.

```console
# conda or mamba
$ conda install pre-commit

```

--------------------------------

### Example: Adding a persistent vector with StringLabelSchema

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/param-labels.md

Demonstrates how to create a datapackage and add a persistent vector using StringLabelSchema for parameter labels.

```python
import numpy as np
from bw_processing import (
    create_datapackage,
    INDICES_DTYPE,
    StringLabelSchema,
)

dp = create_datapackage()

schema = StringLabelSchema(
    description="Temperature and pressure measurements"
)

indices = np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE)
data = np.array([100.0, 200.0])
params = np.array([25.0, 1.013])  # temperature, pressure
labels = ["temperature_celsius", "pressure_atm"]

dp.add_persistent_vector(
    matrix="technosphere",
    indices_array=indices,
    data_array=data,
    params_array=params,
    param_labels=labels,
    param_label_schema=schema,
)
```

--------------------------------

### MatrixEntry as_dict Method Example

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/matrix-entries.md

Demonstrates converting a MatrixEntry object into a plain dictionary.

```python
entry = MatrixEntry(row=1, col=4, amount=2.5)
entry_dict = entry.as_dict()
# {'row': 1, 'col': 4, 'amount': 2.5, 'flip': False, ...}
```

--------------------------------

### Example: String Comparison and Usage of MatrixSerializeFormat Enum

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/types.md

Illustrates how to compare enum members directly with strings and how to pass an enum value when creating a datapackage. Shows both direct comparison and usage in function arguments.

```python
from bw_processing import MatrixSerializeFormat

fmt = MatrixSerializeFormat.NUMPY
if fmt == "numpy":
    print("Using numpy format")

# Use in string context
dp = create_datapackage(
    matrix_serialize_format_type=MatrixSerializeFormat.PARQUET
)
```

--------------------------------

### Example: Create and Access Matrix Indices Array

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/types.md

Demonstrates how to create a numpy array with the INDICES_DTYPE and access its 'row' and 'col' fields. Shows how to retrieve specific row and column values.

```python
import numpy as np
from bw_processing import INDICES_DTYPE

# Create indices array
indices = np.array([
    (0, 0),
    (1, 1),
    (2, 3),
], dtype=INDICES_DTYPE)

# Access fields
print(indices['row'])   # [0 1 2]
print(indices['col'])   # [0 1 3]
print(indices[0]['row']) # 0
print(indices[0]['col']) # 0
```

--------------------------------

### Example: Create and Access Uncertainty Distributions Array

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/types.md

Shows how to create a numpy array using UNCERTAINTY_DTYPE and populate it with different uncertainty distribution parameters. Demonstrates accessing the 'uncertainty_type' field.

```python
import numpy as np
from bw_processing import UNCERTAINTY_DTYPE

# Create distributions array
dists = np.zeros(3, dtype=UNCERTAINTY_DTYPE)
dists['uncertainty_type'] = [0, 2, 3]  # no uncertainty, lognormal, normal
dists['loc'] = [1.5, 1.25, 10.0]
dists['scale'] = [np.nan, 0.5, 2.0]

print(dists['uncertainty_type'])  # [0 2 3]
```

--------------------------------

### Create and Save a Simple Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/INDEX.md

Use this snippet to create a new datapackage, add persistent vector data, and finalize its serialization. Ensure you have the 'pathlib', 'bw_processing', and 'numpy' libraries installed.

```python
from pathlib import Path
from bw_processing import create_datapackage, INDICES_DTYPE
import numpy as np

dp = create_datapackage(
    dirpath=Path("./data"),
    name="my-data"
)

indices = np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE)
data = np.array([1.5, 2.0])

dp.add_persistent_vector(
    matrix="technosphere",
    indices_array=indices,
    data_array=data,
)

dp.finalize_serialization()
```

--------------------------------

### Grouping Results

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/merging-utilities.md

This example shows how results are grouped based on a boolean condition, leading to the creation of groups like 'my-data_true' and 'my-data_false'. A UserWarning may be raised in such cases.

```python
# Result: groups named "my-data_true" and "my-data_false"
# UserWarning is raised about this

```

--------------------------------

### Example Dynamic Array Interface

Source: https://github.com/brightway-lca/bw_processing/blob/main/README.md

Implements the API for dynamic arrays, including .shape and .__getitem__(). It holds a 2D numpy array and returns specified columns upon request.

```Python
import numpy as np


class ExampleArrayInterface:
    def __init__(self):
        rng = np.random.default_rng()
        self.data = rng.random((rng.integers(2, 10), rng.integers(2, 10)))

    @property
    def shape(self):
        return self.data.shape

    def __getitem__(self, args):
        if args[1] >= self.shape[1]:
            raise IndexError
        return self.data[:, args[1]]
```

--------------------------------

### Build Documentation Locally

Source: https://github.com/brightway-lca/bw_processing/blob/main/CONTRIBUTING.md

Build the project documentation using Sphinx. Use the 'docs' directory as the source and specify an output directory.

```console
# use docs as source and docs/_build as output
sphinx-build docs docs/_build

```

--------------------------------

### Create Datapackage with Sampling Combinations

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Illustrates creating datapackages with different sampling strategies: random, reproducible random, and sequential column selection.

```python
# Random sampling, different each run
dp1 = create_datapackage(name="random", sequential=False)

# Reproducible random sampling
dp2 = create_datapackage(name="reproducible", sequential=False, seed=42)

# Sequential columns
dp3 = create_datapackage(name="sequential", sequential=True)

# All combinations
dp4 = create_datapackage(name="combinatorial", combinatorial=True)
```

--------------------------------

### Configure Local Directory Filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Sets up a local directory as the filesystem backend for a datapackage.

```python
from pathlib import Path
from bw_processing import generic_directory_filesystem

fs = generic_directory_filesystem(dirpath=Path("./data"))
```

--------------------------------

### Creating and Adding Matrix Entries to Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/matrix-entries.md

Shows how to instantiate MatrixEntry objects with various parameters and add them to a Brightway datapackage.

```python
from bw_processing import MatrixEntry, MatrixName, create_datapackage

# Basic entry
entry1 = MatrixEntry(row=0, col=0, amount=1.5)

# Entry with flip
entry2 = MatrixEntry(row=1, col=1, amount=2.0, flip=True)

# Entry with rescaling and uncertainty
entry3 = MatrixEntry(
    row=2,
    col=2,
    amount=3.5,
    rescale=0.5,
    uncertainty_type=2,  # Lognormal
    loc=1.25,  # ln(3.5)
    scale=0.5,
)

# Add to datapackage
dp = create_datapackage(name="example")
dp.add_entries(
    matrix=MatrixName.technosphere,
    entries=[entry1, entry2, entry3]
)
```

--------------------------------

### Create Zip File Datapackage with Compression

Source: https://github.com/brightway-lca/bw_processing/blob/main/README.md

Demonstrates how to create a zip file filesystem for a datapackage, enabling compression by default. This is recommended for efficient storage, especially for index arrays.

```python
import zipfile
from bw_processing.io_helpers import generic_zipfile_filesystem

# Default: ZIP_DEFLATED (recommended — good compression, fast)
fs = generic_zipfile_filesystem(dirpath=some_path, filename="my.zip")

```

--------------------------------

### reset_index

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/indexing-utilities.md

Resets the numerical indices in a datapackage to sequential integers starting from zero. This is useful after filtering or merging datapackages to ensure contiguous indices.

```APIDOC
## reset_index

### Description

Reset the numerical indices in a datapackage to sequential integers starting from zero. Used after filtering or merging datapackages to ensure contiguous indices.

### Method Signature

```python
def reset_index(
    datapackage: Union[Datapackage, AbstractFileSystem],
    metadata_name: str,
) -> Datapackage
```

### Parameters

#### Required Parameters

- **datapackage** (Datapackage or AbstractFileSystem) - Datapackage or filesystem path to load.
- **metadata_name** (str) - Name of the CSV metadata resource describing the indices to reset.

### Returns

The modified `Datapackage` with sequential indices (0, 1, 2, ...).

### Details

Updates all row and column index values referenced by the metadata to new sequential values. Modifies indices in place.

### Raises

- `KeyError`: `metadata_name` not found in datapackage.
- `ValueError`: `metadata_name` is not CSV metadata.

### Example

```python
from bw_processing import reset_index

# Load a datapackage with non-contiguous indices (e.g., after filtering)
dp = load_datapackage(some_path)

# Reset indices to 0, 1, 2, ...
dp_reset = reset_index(dp, "my-metadata")

# Verify new indices
data, meta = dp_reset.get_resource("my-data.indices")
print(data['row'])  # [0, 1, 2, 3, ...]
```
```

--------------------------------

### Reset Datapackage Indices

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/indexing-utilities.md

Resets numerical indices in a datapackage to sequential integers starting from zero. Use this after filtering or merging to ensure contiguous indices.

```python
from bw_processing import reset_index

# Load a datapackage with non-contiguous indices (e.g., after filtering)
dp = load_datapackage(some_path)

# Reset indices to 0, 1, 2, ...
dp_reset = reset_index(dp, "my-metadata")

# Verify new indices
data, meta = dp_reset.get_resource("my-data.indices")
print(data['row'])  # [0, 1, 2, 3, ...]
```

--------------------------------

### Get Column Count from Iterator

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/array-utilities.md

Determine the column count from an iterable without fully consuming it. Returns a tuple of the column count and a reconstructed iterator.

```python
def get_ncols(iterator: Iterator) -> tuple[int, Iterator]:
    pass
```

--------------------------------

### Creating a Datapackage with Matrix Entries

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/matrix-entries.md

Shows how to create a Brightway Processing datapackage using the `create_datapackage_from_entries` function with multiple MatrixEntry objects for different matrices.

```python
from bw_processing import (
    MatrixEntry,
    MatrixName,
    create_datapackage_from_entries,
)

data = {
    MatrixName.technosphere: [
        MatrixEntry(row=1, col=4, amount=2.5),
        MatrixEntry(row=2, col=5, amount=7.0, flip=True),
    ],
    MatrixName.biosphere: [
        MatrixEntry(row=10, col=4, amount=0.3),
        MatrixEntry(row=11, col=4, amount=0.2),
    ],
}

dp = create_datapackage_from_entries(
    data=data,
    name="my-data",
    metadata={"author": "John Doe"}
)

# Datapackage is ready to use
dp.finalize_serialization()
```

--------------------------------

### Handle NonUnique Error

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/errors.md

Example of catching a NonUnique error, which is raised when duplicate elements are found where uniqueness is expected. This can occur in functions like `greedy_set_cover` when `raise_error=True`.

```python
from bw_processing import greedy_set_cover
from bw_processing.errors import NonUnique

data = [
    {'id': 1, 'name': 'A'},
    {'id': 2, 'name': 'A'},  # Duplicate name
]

try:
    greedy_set_cover(data, exclude=['id'])
except NonUnique:
    print("Cannot uniquely identify items")
```

--------------------------------

### Configure Directory-Based Filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Sets up a filesystem to store datapackages as uncompressed files in a specified local directory.

```python
from pathlib import Path
from bw_processing import generic_directory_filesystem, create_datapackage

fs = generic_directory_filesystem(dirpath=Path("./data"))
dp = create_datapackage(fs=fs, name="uncompressed")
```

--------------------------------

### Create On-Disk Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Initializes a datapackage that is stored on disk in a specified directory. Resources are added, and the datapackage must be finalized to save changes.

```python
from pathlib import Path

dp = create_datapackage(
    dirpath=Path("./data"),
    name="on-disk",
)

# Add resources
dp.add_persistent_vector(...)

# Must finalize to save
dp.finalize_serialization()
```

--------------------------------

### Handle InconsistentFields Error

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/errors.md

Example of catching an InconsistentFields error, which occurs when objects in a dataset have different sets of fields. This typically happens when using functions like `as_unique_attributes`.

```python
from bw_processing import as_unique_attributes
from bw_processing.errors import InconsistentFields

data = [
    {'id': 1, 'name': 'A', 'value': 1.5},
    {'id': 2, 'name': 'B'},  # Missing 'value'
]

try:
    as_unique_attributes(data)
except InconsistentFields:
    print("Objects have different fields")
```

--------------------------------

### Configure Zip File Filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Sets up a zip file as the filesystem backend for a datapackage.

```python
from pathlib import Path
from bw_processing import generic_zipfile_filesystem

fs = generic_zipfile_filesystem(
    dirpath=Path("."),
    filename="data.zip"
)
```

--------------------------------

### Configure Memory Filesystem for Testing

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Initializes an in-memory filesystem (DictFS) for testing purposes and associates it with a datapackage.

```python
from morefs.dict import DictFS

fs = DictFS()
dp = create_datapackage(fs=fs)
```

--------------------------------

### Build Documentation Locally with Sphinx

Source: https://github.com/brightway-lca/bw_processing/blob/main/docs/content/contributing.md

Build the project documentation locally using sphinx-build, specifying the source and output directories.

```console
sphinx-build docs docs/_build
```

--------------------------------

### Example Dynamic Vector Interface

Source: https://github.com/brightway-lca/bw_processing/blob/main/README.md

Implements the Python generator API for dynamic vectors. It generates a random-sized 1D numpy array with random float values on each call to __next__().

```Python
import numpy as np


class ExampleVectorInterface:
    def __init__(self):
        self.rng = np.random.default_rng()
        self.size = self.rng.integers(2, 10)

    def __next__(self):
        return self.rng.random(self.size)
```

--------------------------------

### Get unique attributes as a Pandas DataFrame

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/unique-fields.md

Use `as_unique_attributes_dataframe` to return a DataFrame containing only the columns necessary to uniquely identify each row. Supports excluding or including specific fields.

```python
def as_unique_attributes_dataframe(
    df: pd.DataFrame,
    exclude: Optional[Iterable[str]] = None,
    include: Optional[Iterable[str]] = None,
    raise_error: bool = False,
) -> pd.DataFrame
```

--------------------------------

### Format data with minimal unique attributes

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/unique-fields.md

Use `as_unique_attributes` to get a set of minimal unique attributes and a list of dictionaries containing only those fields. Optionally include specific fields.

```python
from bw_processing import as_unique_attributes

# Activity data with redundant fields
activities = [
    {'id': 1, 'name': 'Electricity', 'unit': 'kWh', 'location': 'US', 'type': 'process'},
    {'id': 2, 'name': 'Gas', 'unit': 'MJ', 'location': 'US', 'type': 'process'},
    {'id': 3, 'name': 'Electricity', 'unit': 'kWh', 'location': 'EU', 'type': 'process'},
]

# Get minimal unique attributes
attrs, formatted = as_unique_attributes(activities)
# attrs -> {'id', 'name', 'location'}
# formatted -> [
#     {'id': 1, 'name': 'Electricity', 'location': 'US'},
#     {'id': 2, 'name': 'Gas', 'location': 'US'},
#     {'id': 3, 'name': 'Electricity', 'location': 'EU'},
# ]

# Always include certain fields
attrs, formatted = as_unique_attributes(
    activities,
    include=['unit']
)
# attrs -> {'id', 'name', 'location', 'unit'}
```

--------------------------------

### Handle WrongDatatype Error for Array Dtype

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/errors.md

Example of catching a WrongDatatype error when adding a persistent vector. This specific case shows an incorrect dtype for the `flip_array` parameter, which must be boolean.

```python
import numpy as np
from bw_processing import create_datapackage, INDICES_DTYPE
from bw_processing.errors import WrongDatatype

dp = create_datapackage()
indices = np.array([(0, 0)], dtype=INDICES_DTYPE)

# Wrong: flip_array must be boolean
flip_wrong = np.array([1])  # int, not bool

try:
    dp.add_persistent_vector(
        matrix="technosphere",
        indices_array=indices,
        flip_array=flip_wrong,
    )
except WrongDatatype as e:
    print(f"Wrong dtype: {e}")
```

--------------------------------

### Create a new datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Processed data package demo.ipynb

Initializes a new, empty data package using the create_datapackage function.

```python
dp = bwp.create_datapackage()
```

--------------------------------

### Create Datapackage with Custom Metadata

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Demonstrates how to pass additional custom metadata, such as author, version, description, licenses, and sources, when creating a datapackage.

```python
dp = create_datapackage(
    name="data",
    metadata={
        "author": "John Doe",
        "version": "1.0",
        "description": "Energy data for 2020",
        "licenses": [
            {
                "name": "CC-BY-4.0",
                "path": "https://creativecommons.org/licenses/by/4.0/",
            }
        ],
        "sources": [
            {
                "title": "IEA Statistics",
                "path": "https://www.iea.org/",
            }
        ],
    }
)
```

--------------------------------

### Creating and Adding Array Entries to Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/matrix-entries.md

Demonstrates creating an ArrayEntry with row, column, data, and flip metadata, and adding it to a datapackage.

```python
import numpy as np
from bw_processing import ArrayEntry, MatrixName, create_datapackage

# Create scenario array entry
rows = np.array([0, 1, 2])
cols = np.array([10, 11, 12])
data = np.array([
    [1.0, 1.5, 2.0],     # 3 scenarios for row 0
    [2.0, 2.5, 3.0],     # 3 scenarios for row 1
    [3.0, 3.5, 4.0],     # 3 scenarios for row 2
])
flip = np.array([False, True, False])

entry = ArrayEntry(
    rows=rows,
    cols=cols,
    data=data,
    flip=flip
)

# Add to datapackage
dp = create_datapackage(name="scenarios")
dp.add_array_entries(
    matrix=MatrixName.technosphere,
    entries=[entry]
)
```

--------------------------------

### Handle ShapeMismatch Error for Array Shape

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/errors.md

Example of catching a ShapeMismatch error when adding a persistent vector. This case demonstrates an incorrect shape for the `data_array`, which should match the number of elements implied by the `indices_array`.

```python
import numpy as np
from bw_processing import create_datapackage, INDICES_DTYPE
from bw_processing.errors import ShapeMismatch

dp = create_datapackage()
indices = np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE)
data = np.array([1.0])  # Wrong: only 1 element, should be 2

try:
    dp.add_persistent_vector(
        matrix="technosphere",
        indices_array=indices,
        data_array=data,
    )
except ShapeMismatch as e:
    print(f"Shape mismatch: {e}")
```

--------------------------------

### Create Datapackage Instance

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/datapackage.md

Use `create_datapackage` to instantiate a new datapackage. It can be an in-memory instance or saved to disk, with options for compression and array handling.

```python
from pathlib import Path
from bw_processing import create_datapackage

# In-memory datapackage
dp = create_datapackage(name="my-data")

# Saved to disk with compression
dp_compressed = create_datapackage(
    dirpath=Path("./data"),
    name="compressed-data",
    compress=True
)

# Array-based with sequential column selection
dp_array = create_datapackage(
    dirpath=Path("./arrays"),
    name="scenario-data",
    sequential=True,
    seed=42
)
```

--------------------------------

### Create Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Presamples generator example.ipynb

Creates a new datapackage in the specified directory. Running this twice without changes will result in an error.

```python
dp = Datapackage.create(dirname)
```

--------------------------------

### Import Brightway Processing Library

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Presamples generator example.ipynb

Imports all necessary components from the bw_processing library.

```python
from bw_processing import *
```

--------------------------------

### generic_directory_filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/filesystem-utilities.md

Creates and returns an fsspec DirFileSystem for directory-based datapackage storage. The directory will be created if it does not exist.

```APIDOC
## generic_directory_filesystem

### Description
Create and return an fsspec `DirFileSystem` for directory-based datapackage storage.

### Method
```python
def generic_directory_filesystem(
    *, 
    dirpath: Path,
) -> DirFileSystem
```

### Parameters
#### Path Parameters
- **dirpath** (pathlib.Path) - Required - Path to the directory (created if necessary).

### Returns
A `fsspec` `DirFileSystem` instance rooted at `dirpath`.

### Raises
- `ValueError`: Parent directory does not exist.
- `AssertionError`: `dirpath` is not a `pathlib.Path`.

### Example
```python
from pathlib import Path
from bw_processing import generic_directory_filesystem, create_datapackage

# Create filesystem for a directory
dirpath = Path("./data/my-package")
fs = generic_directory_filesystem(dirpath=dirpath)

# Use with datapackage
dp = create_datapackage(fs=fs, name="stored-package")
# Files written to ./data/my-package/
```
```

--------------------------------

### Configure Zip Compression with STORED (No Compression)

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Sets up a zip filesystem with no compression (ZIP_STORED), prioritizing the fastest read and write speeds.

```python
import zipfile
from pathlib import Path
from bw_processing import generic_zipfile_filesystem

output_dir = Path("./output")

# No compression (fastest read/write)
fs = generic_zipfile_filesystem(
    dirpath=output_dir,
    filename="fast.zip",
    compression=zipfile.ZIP_STORED,
)
```

--------------------------------

### Create Datapackage with Duplicate Handling Combinations

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Shows how to create datapackages for different duplicate handling scenarios, including summing within and across resources, and overriding values.

```python
# Scenario/override package pattern:
# Base package with sum_intra=True (aggregates similar items)
dp_base = create_datapackage(
    name="base",
    sum_intra_duplicates=True,
    sum_inter_duplicates=False,
)

# Override package with specific values
dp_override = create_datapackage(
    name="override",
    sum_intra_duplicates=True,
    sum_inter_duplicates=False,
)

# When used together in matrix_utils, overrides replace base values
```

--------------------------------

### Create Directory Filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/filesystem-utilities.md

Creates an fsspec DirFileSystem for directory-based storage. The directory will be created if it doesn't exist. Use this for local file storage.

```python
from pathlib import Path
from bw_processing import generic_directory_filesystem, create_datapackage

# Create filesystem for a directory
dirpath = Path("./data/my-package")
fs = generic_directory_filesystem(dirpath=dirpath)

# Use with datapackage
dp = create_datapackage(fs=fs, name="stored-package")
# Files written to ./data/my-package/
```

--------------------------------

### Load Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Presamples generator example.ipynb

Loads an existing datapackage from the specified directory.

```python
dp = Datapackage.load("generator_metadata")
```

--------------------------------

### Core API Documentation

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/README.md

This section details the main interfaces for interacting with data packages and their entries, including creation, loading, adding data, and retrieving resources.

```APIDOC
## Datapackage Interface

### Description
Provides the main interface for managing data packages, including creation, loading, adding persistent or dynamic vectors/arrays, and retrieving or filtering resources.

### Methods
- `create_datapackage()`
- `load_datapackage()`
- `add_persistent_vector()`
- `add_persistent_array()`
- `add_dynamic_vector()`
- `add_dynamic_array()`
- `add_entries()`
- `add_array_entries()`
- `get_resource()`
- `filter_by_attribute()`
- `del_resource()`

### Reference
See `datapackage.md` for detailed documentation.
```

```APIDOC
## Matrix Entries Interface

### Description
Offers a high-level entry point for creating data packages from entries, utilizing `MatrixEntry` and `ArrayEntry` dataclasses and the `MatrixName` enum.

### Methods
- `create_datapackage_from_entries()`

### Reference
See `matrix-entries.md` for detailed documentation.
```

```APIDOC
## Array Utilities

### Description
Provides functions for creating various types of arrays, including structured arrays, regular arrays, and chunked arrays, along with helper functions for iterables.

### Methods
- `create_structured_array()`
- `create_array()`
- `create_chunked()`

### Reference
See `array-utilities.md` for detailed documentation.
```

```APIDOC
## Filesystem Utilities

### Description
Handles file input/output operations and validation for data packages, including generic filesystem backends for directories and zip files.

### Methods
- `generic_directory_filesystem()`
- `generic_zipfile_filesystem()`
- `clean_datapackage_name()`
- `safe_filename()`
- `md5()`

### Reference
See `filesystem-utilities.md` for detailed documentation.
```

```APIDOC
## Indexing Utilities

### Description
Manages index operations within data packages, including resetting and reindexing.

### Methods
- `reset_index()`
- `reindex()`

### Reference
See `indexing-utilities.md` for detailed documentation.
```

```APIDOC
## Merging Utilities

### Description
Provides functionality for merging data packages, including merging with a mask.

### Methods
- `merge_datapackages_with_mask()`

### Reference
See `merging-utilities.md` for detailed documentation.
```

```APIDOC
## Unique Fields

### Description
Facilitates feature selection by identifying unique attributes, including greedy set cover algorithms and conversions to unique attributes or dataframes.

### Methods
- `greedy_set_cover()`
- `as_unique_attributes()`
- `as_unique_attributes_dataframe()`

### Reference
See `unique-fields.md` for detailed documentation.
```

```APIDOC
## Parameter Labels

### Description
Defines schemas for parameter labels, including `StringLabelSchema`, `ParamLabelField`, and `ParamLabelSchema`, with functionality to create schemas from JSON schemas.

### Methods
- `schema_from_json_schema()`

### Reference
See `param-labels.md` for detailed documentation.
```

--------------------------------

### Create temporary directory

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Processed data package demo.ipynb

Creates a temporary directory for playing around with data packages and changes the current working directory to it.

```python
import tempfile, os
```

--------------------------------

### Filesystem Utilities

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/INDEX.md

Offers an abstraction layer for flexible storage backends, including directory and zip file systems. Provides utilities for cleaning datapackage names, creating safe filenames, and generating MD5 hashes.

```APIDOC
## Filesystem Utilities

### Description
Filesystem abstraction layer for flexible storage backends.

### Functions
- `generic_directory_filesystem()`: Create DirFileSystem for directory storage.
- `generic_zipfile_filesystem()`: Create ZipFileSystem with compression options.
- `clean_datapackage_name()`: Validate and clean datapackage names.
- `safe_filename()`: Convert strings to safe filenames.
- `md5()`: Generate MD5 hash for files.
```

--------------------------------

### Create Datapackage with Sequential Column Selection

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Configures a datapackage to select columns in a sequential order (0, 1, 2, ...). This ensures a predictable iteration pattern through array data.

```python
# Sequential columns in each array
dp = create_datapackage(
    name="ordered-scenarios",
    sequential=True,
)
```

--------------------------------

### Initialize FastParquet File Object

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Parquet speed testing.ipynb

Creates a FastParquet File object to access metadata and information about the Parquet file.

```python
fp = fastparquet.ParquetFile('biosphere.parquet')
fp
```

--------------------------------

### Load Datapackage from Filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/datapackage.md

Use `load_datapackage` to load an existing datapackage. It supports loading from directories, zip files, and with memory-mapping for large arrays.

```python
from pathlib import Path
from bw_processing import load_datapackage
from fsspec.implementations.zip import ZipFileSystem

# Load from directory
dp = load_datapackage(Path("./data/my-package"))

# Load from zip file with proxy loading
fs = ZipFileSystem("data/my-package.zip")
dp = load_datapackage(fs, proxy=True)

# Load with memory mapping for large arrays
dp_mmap = load_datapackage(Path("./data"), mmap_mode='r')
```

--------------------------------

### Retrieve and Inspect Parameter Labels and Schema

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/param-labels.md

Loads a datapackage, retrieves resource data including parameter labels and schema, and then inspects the schema if it exists. This is useful for understanding the metadata associated with a resource.

```python
# Load datapackage
dp = load_datapackage(path)

# Get labels
labels_data, meta = dp.get_resource("resource-name.param_labels")

# Access values
values = labels_data["values"]  # List of labels

# Access schema if present
if "schema" in labels_data:
    schema = schema_from_json_schema(labels_data["schema"])
    print(f"Schema type: {schema.__class__.__name__}")
    if hasattr(schema, "fields"):
        for field in schema.fields:
            print(f"  - {field.name}")
```

--------------------------------

### Iterate and print resource data

Source: https://github.com/brightway-lca/bw_processing/blob/main/dev/Processed data package demo.ipynb

Iterates through the 'first.indices' and 'first.data' resources of the datapackage and prints corresponding elements. Also prints the metadata for each resource.

```python
for x, y in zip(dp.get_resource("first.indices"), dp.get_resource("first.data")):
    print(x, y)
```

--------------------------------

### Create Datapackage with Combinatorial Column Selection

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Configures a datapackage to generate all combinations of columns from different array resources. This is useful for scenario analysis where all possible pairings are needed.

```python
# Sequential combinations of all arrays
dp = create_datapackage(
    name="scenarios",
    combinatorial=True,  # All (A, B) combinations
)
```

--------------------------------

### Reference Documentation

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/README.md

This section provides detailed information on type definitions, exception classes, and configuration options used within the bw_processing library.

```APIDOC
## Types

### Description
Defines type definitions and constants used throughout the library, including dtypes for indices and uncertainty, serialization formats, default licenses, and metadata schema documentation.

### Constants/Enums
- `INDICES_DTYPE`
- `UNCERTAINTY_DTYPE`
- `MatrixSerializeFormat` (enum)
- `DEFAULT_LICENSES`

### Reference
See `types.md` for detailed documentation.
```

```APIDOC
## Errors

### Description
Documents the hierarchy of exception classes, including `BrightwayProcessingError` and all 10 custom exception types, along with their trigger conditions.

### Exception Hierarchy
- `BrightwayProcessingError`
- [List of 10 custom exception types and their trigger conditions]

### Reference
See `errors.md` for detailed documentation.
```

```APIDOC
## Configuration

### Description
Details the various configuration options available for the library, covering datapackage creation parameters, serialization formats, array column selection policies, duplicate handling, metadata, compression, and filesystem backend configuration.

### Configuration Areas
- Datapackage creation parameters
- Serialization formats
- Array column selection policies
- Duplicate handling policies
- Metadata and compression options
- Filesystem backend configuration

### Reference
See `configuration.md` for detailed documentation.
```

--------------------------------

### Create In-Memory Datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/configuration.md

Initializes a datapackage that resides entirely in memory, suitable for temporary calculations. Finalizing serialization is not supported for in-memory datapackages.

```python
dp = create_datapackage(
    dirpath=None,  # No filesystem
    name="in-memory",
)

# Add resources
dp.add_persistent_vector(...)

# Cannot call finalize_serialization() on in-memory datapackages
# Use only for temporary calculations
```

--------------------------------

### generic_zipfile_filesystem

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/filesystem-utilities.md

Creates and returns an fsspec ZipFileSystem for zip-based datapackage storage. Supports various compression options and read/write modes.

```APIDOC
## generic_zipfile_filesystem

### Description
Create and return an fsspec `ZipFileSystem` for zip-based datapackage storage.

### Method
```python
def generic_zipfile_filesystem(
    *, 
    dirpath: Path,
    filename: str,
    write: bool = True,
    compression: int = zipfile.ZIP_DEFLATED,
    compresslevel: Optional[int] = None,
) -> ZipFileSystem
```

### Parameters
#### Path Parameters
- **dirpath** (pathlib.Path) - Required - Directory containing (or to contain) the zip file. Must exist.
- **filename** (str) - Required - Name of the zip file (e.g., "my-dp.zip").
- **write** (bool) - Optional - If True, open for writing; if False, open for reading. Defaults to True.
- **compression** (int) - Optional - Compression algorithm (ZIP_DEFLATED, ZIP_STORED, ZIP_LZMA). Defaults to `zipfile.ZIP_DEFLATED`.
- **compresslevel** (int or None) - Optional - Compression level (algorithm-dependent). Defaults to None.

### Returns
A `fsspec` `ZipFileSystem` instance.

### Raises
- `ValueError`: `dirpath` does not exist.
- `AssertionError`: `dirpath` is not a `pathlib.Path`.

### Example
```python
import zipfile
from pathlib import Path
from bw_processing import generic_zipfile_filesystem, create_datapackage

output_dir = Path("./output")

# Default compression (ZIP_DEFLATED)
fs = generic_zipfile_filesystem(
    dirpath=output_dir,
    filename="my-data.zip"
)
dp = create_datapackage(fs=fs, name="zipped-package")
# ... add resources ...
dp.finalize_serialization()
# Creates ./output/my-data.zip with good compression

# No compression (fastest write/read)
fs_fast = generic_zipfile_filesystem(
    dirpath=output_dir,
    filename="fast-data.zip",
    compression=zipfile.ZIP_STORED
)

# Maximum compression (slowest write)
fs_max = generic_zipfile_filesystem(
    dirpath=output_dir,
    filename="compact-data.zip",
    compression=zipfile.ZIP_LZMA,
    compresslevel=9
)

# Reading a zip file
fs_read = generic_zipfile_filesystem(
    dirpath=output_dir,
    filename="existing-data.zip",
    write=False
)
from bw_processing import load_datapackage
dp = load_datapackage(fs_read)
```
```

--------------------------------

### simple_graph

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/datapackage.md

Creates a datapackage from a simple dictionary of matrices, optionally using a specified filesystem and including additional metadata.

```APIDOC
## simple_graph

### Description
Create a datapackage from a simple dictionary of matrices.

### Method
`simple_graph(data: dict, fs: Optional[AbstractFileSystem] = None, **metadata) -> Datapackage`

### Parameters
#### Path Parameters
- **data** (dict) - Required - Dictionary mapping matrix names to values.
- **fs** (AbstractFileSystem or None) - Optional - Filesystem for saving. Defaults to None.
- **metadata** (dict) - Optional - Additional metadata.

### Returns
A configured `Datapackage` instance.
```

--------------------------------

### create_datapackage

Source: https://github.com/brightway-lca/bw_processing/blob/main/_autodocs/datapackage.md

Creates a new datapackage instance for adding data resources. It can be created in-memory or saved to disk, with options for naming, metadata, compression, and data handling.

```APIDOC
## create_datapackage

### Description
Creates a new datapackage instance for adding data resources.

### Method
```python
def create_datapackage(
    dirpath: Optional[Union[str, Path]] = None,
    name: Optional[str] = None,
    id_: Optional[str] = None,
    metadata: Optional[dict] = None,
    compress: bool = False,
    combinatorial: bool = False,
    sequential: bool = False,
    seed: Optional[int] = None,
    sum_intra_duplicates: bool = True,
    sum_inter_duplicates: bool = False,
    matrix_serialize_format_type: MatrixSerializeFormat = MatrixSerializeFormat.NUMPY,
    overwrite: bool = False,
) -> Datapackage
```

### Parameters
#### Path Parameters
- **dirpath** (str, Path, or None) - Optional - Directory path where the datapackage will be saved. If None, creates in-memory datapackage.
- **name** (str or None) - Optional - Human-readable name for the datapackage. Auto-generated if omitted.
- **id_** (str or None) - Optional - Unique identifier for the datapackage. Auto-generated if omitted.
- **metadata** (dict or None) - Optional - Additional metadata as a dictionary (e.g., licenses, author).
- **compress** (bool) - Optional - If True, compress the datapackage as a zip file.
- **combinatorial** (bool) - Optional - If True and multiple array resources exist, all combinations of columns are guaranteed.
- **sequential** (bool) - Optional - If True, array columns are chosen in sequence rather than randomly.
- **seed** (int or None) - Optional - Random seed for reproducible column selection in arrays.
- **sum_intra_duplicates** (bool) - Optional - If True, sum duplicate values within a single resource.
- **sum_inter_duplicates** (bool) - Optional - If True, sum overlapping data from different resources.
- **matrix_serialize_format_type** (MatrixSerializeFormat) - Optional - Serialization format for numpy arrays (NUMPY or PARQUET).
- **overwrite** (bool) - Optional - If True, overwrite existing datapackage at the same location.

### Returns
A `Datapackage` instance ready for adding resources.

### Raises
- `InvalidName`: Name contains invalid characters.
- `ValueError`: Specified format type is not recognized.

### Example
```python
from pathlib import Path
from bw_processing import create_datapackage

# In-memory datapackage
dp = create_datapackage(name="my-data")

# Saved to disk with compression
dp_compressed = create_datapackage(
    dirpath=Path("./data"),
    name="compressed-data",
    compress=True
)

# Array-based with sequential column selection
dp_array = create_datapackage(
    dirpath=Path("./arrays"),
    name="scenario-data",
    sequential=True,
    seed=42
)
```
```