### create_example_config

Source: https://bandersnatch.readthedocs.io/en/latest/bandersnatch.html

Creates an example configuration file at a specified destination path. This is useful for users to get started with configuring bandersnatch.

```APIDOC
## create_example_config

### Description
Creates an example configuration file at the specified location.

### Parameters
#### Path Parameters
- **dest** (Path) - Required - The destination path for the configuration file.
```

--------------------------------

### Create Example Configuration File

Source: https://bandersnatch.readthedocs.io/en/latest/bandersnatch.html

Creates an example configuration file at a specified destination path. This is useful for setting up initial configurations.

```python
from bandersnatch.configuration import create_example_config
from pathlib import Path

create_example_config(Path("/path/to/your/config.ini"))
```

--------------------------------

### Create Example Bandersnatch Configuration

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/configuration.html

This function creates an example configuration file at a specified destination path by copying a default example configuration file from the bandersnatch package.

```python
[docs]
def create_example_config(dest: Path) -> None:
    """Create an example configuration file at the specified location.

    :param Path dest: destination path for the configuration file.
    """
    example_source = importlib.resources.files("bandersnatch") / _example_conf_file
    try:
        shutil.copy(str(example_source), dest)
    except OSError as err:
        logger.error("Could not create config file '%s': %s", dest, err)

```

--------------------------------

### Example Bandersnatch Configuration

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

An annotated example configuration file for Bandersnatch. Use this as a reference or a starting point for your own mirror setup. It includes explanations for various settings.

```ini
[mirror]
; The directory where the mirror data will be stored.
directory = /srv/pypi

; Save JSON metadata into the web tree:
; URL/pypi/PKG_NAME/json (Symlink) -> URL/json/PKG_NAME
json = false

; Save package release files
release-files = true

; Cleanup legacy non PEP 503 normalized named simple directories
cleanup = false

; The PyPI server which will be mirrored. As of bandersnatch 7.0 the
; upstream must speak the PEP 691 JSON Simple API; this can be pypi.org,
; the test PyPI, or another bandersnatch mirror. See
; docs/mirror_from_mirror_simple_api.rst for the full procedure when
; chaining mirrors.
; master = https://test.pypi.org
; scheme for PyPI server MUST be https
master = https://pypi.org

; The network socket timeout to use for all connections. This is set to a
; somewhat aggressively low value: rather fail quickly temporarily and re-run
; the client soon instead of having a process hang infinitely and have TCP not
; catching up for ages.
timeout = 10

; The global-timeout sets aiohttp total timeout for it's coroutines
; This is set incredibly high by default as aiohttp coroutines need to be
; equipped to handle mirroring large PyPI packages on slow connections.
global-timeout = 1800

; Number of worker threads to use for parallel downloads.
; Recommendations for worker thread setting:
; - leave the default of 3 to avoid overloading the pypi master
; - official servers located in data centers could run 10 workers
; - anything beyond 10 is probably unreasonable and avoided by bandersnatch
workers = 3

; Whether to hash package indexes
; Note that package index directory hashing is incompatible with pip, and so
; this should only be used in an environment where it is behind an application
; that can translate URIs to filesystem locations.  For example, with the
; following Apache RewriteRule:
;     RewriteRule ^([^/])([^/]*)/$ /mirror/pypi/web/simple/$1/$1$2/
;     RewriteRule ^([^/])([^/]*)/([^/]+)$/ /mirror/pypi/web/simple/$1/$1$2/$3
; OR
; following nginx rewrite rules:
;     rewrite ^/simple/([^/])([^/]*)/$ /simple/$1/$1$2/ last;
;     rewrite ^/simple/([^/])([^/]*)/([^/]+)$/ /simple/$1/$1$2/$3 last;
; Setting this to true would put the package 'abc' index in simple/a/abc.
; Recommended setting: the default of false for full pip/pypi compatibility.
hash-index = false

; Format for simple API to be stored in
; Since PEP691 we have HTML and JSON
simple-format = ALL

; Whether to stop a sync quickly after an error is found or whether to continue
; syncing but not marking the sync as successful. Value should be "true" or
; "false".
stop-on-error = false

; The storage backend that will be used to save data and metadata while
; mirroring packages. By default, use the filesystem backend. Other options
; currently include: 'swift'
storage-backend = filesystem

; Setting this to "false" will disable permission management in filesystem plugin.
; This is useful in case when files in the mirror directory are not owned by
; a user who runs Bandersnatch and file permissions are managed externally -
; by using POSIX ACLs, or manually by sysadmin.
; Value should be "true" or "false", default is "true".
storage-filesystem-manage-permissions = true

; Advanced logging configuration. Uncomment and set to the location of a
; python logging format logging config file.
; log-config = /etc/bandersnatch-log.conf

; Generate index pages with absolute urls rather than relative links. This is
; generally not necessary, but was added for the official internal PyPI mirror,
; which requires serving packages from https://files.pythonhosted.org
; root_uri = https://example.com

; Number of consumers verify metadata
verifiers = 3


```

--------------------------------

### Example Tox Test Run Output

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

This is an example of the output you can expect when running tox to execute unit tests. It shows the installation of dependencies, the test session start, and the collection and execution of test cases.

```bash
$ tox
GLOB sdist-make: /Users/dhubbard/PycharmProjects/bandersnatch/setup.py
py36 create: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/py36
py36 installdeps: -rtest-requirements.txt
py36 inst: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/dist/bandersnatch-2.2.1.zip
py36 installed: apipkg==1.4,attrs==18.1.0,bandersnatch==2.2.1,certifi==2018.4.16,chardet==3.0.4,coverage==4.5.1,execnet==1.5.0,flake8==3.5.0,idna==2.6,mccabe==0.6.1,more-itertools==4.1.0,packaging==17.1,pep8==1.7.1,pluggy==0.6.0,py==1.5.3,pycodestyle==2.3.1,pyflakes==1.6.0,pyparsing==2.2.0,pytest==3.5.1,pytest-cache==1.0,pytest-codecheckers==0.2,pytest-cov==2.5.1,pytest-timeout==1.2.1,python-dateutil==2.7.3,requests==2.18.4,six==1.11.0,tox==3.0.0,urllib3==1.22,virtualenv==15.2.0,xmlrpc2==0.3.1
py36 runtests: PYTHONHASHSEED='42669967'
py36 runtests: commands[0] | pytest
=========================================================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/dhubbard/PycharmProjects/bandersnatch, inifile: pytest.ini
plugins: timeout-1.2.1, cov-2.5.1, codecheckers-0.2
timeout: 10.0s method: signal
collected 94 items

src/bandersnatch/__init__.py ..                                                                                                                                                                                                                                 [  2%]
src/bandersnatch/buildout.py ..                                                                                                                                                                                                                                 [  4%]
src/bandersnatch/log.py ..                                                                                                                                                                                                                                      [  6%]
src/bandersnatch/main.py ..                                                                                                                                                                                                                                     [  8%]
src/bandersnatch/master.py ..                                                                                                                                                                                                                                   [ 10%]
src/bandersnatch/mirror.py ..                                                                                                                                                                                                                                   [ 12%]
src/bandersnatch/package.py ..                                                                                                                                                                                                                                   [ 14%]
src/bandersnatch/release.py ..                                                                                                                                                                                                                                  [ 17%]
src/bandersnatch/utils.py ..                                                                                                                                                                                                                                   [ 19%]
src/bandersnatch/tests/conftest.py ..                                                                                                                                                                                                                           [ 21%]
src/bandersnatch/tests/test_main.py .......                                                                                                                                                                                                                     [ 28%]
src/bandersnatch/tests/test_master.py ...........                                                                                                                                                                                                               [ 40%]
src/bandersnatch/tests/test_mirror.py ....................                                                                                                                                                                                                      [ 61%]

```

--------------------------------

### Serving S3 Mirror with CloudFront

Source: https://bandersnatch.readthedocs.io/en/latest/storage_options.html

Example of how a client would install a package from an S3 mirror served via CloudFront. Note the long URL structure.

```bash
pip install -i my-s3-bucket.cloudfront.net/prefix/web/simple install django

```

--------------------------------

### Example Bandersnatch Configuration

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/mirror_configuration.md.txt

An example configuration file for bandersnatch, useful as a starting point for custom configurations. It includes annotations for clarity.

```ini
[bandersnatch]
# Set to true to disable the default plugins
# disable_default_plugins = false

# Set to true to disable the default mirror
# disable_default_mirror = false

[mirror]
# Mirror index files from this URL
# Example: http://pypi.python.org/simple
# Use a local path for testing
# Example: file:///tmp/pypi

# Directory to store mirrored files
# Example: /var/www/mirror/simple

[storage]
# Storage backend to use. Options: "mirror", "legacy"
# Default: "mirror"

[hash-index]
# Enable hash-index (see https://www.python.org/dev/peps/pep-0503/#hash-based-indexing)
# Default: false

[plugins]
# List of plugins to load
# Example: plugins = bandersnatch.plugin.ExamplePlugin

```

--------------------------------

### Install and Run Bandersnatch Mirror

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Install Bandersnatch from a local directory and then run the mirror command using a specified configuration file. Ensure you are in the 'bandersnatch' directory and have a virtual environment set up.

```bash
cd bandersnatch
/path/to/venv/bin/pip install --upgrade .
/path/to/venv/bin/bandersnatch -c src/bandersnatch/default.conf mirror

```

--------------------------------

### Initialize Storage Plugin

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Placeholder method for plugin initialization. Should contain setup code that runs once.

```python
def initialize_plugin(self) -> None:
    """
    Code to initialize the plugin
    """
    # The initialize_plugin method is run once to initialize the plugin. This should
    # contain all code to set up the plugin.
    # This method is not run in the fast path and should be used to do things like
    # indexing filter databases, etc that will speed the operation of the filter
    # and check_match methods that are called in the fast path.
    pass
```

--------------------------------

### Storage Class Initialization

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Illustrates the initialization process of the base Storage class, including configuration loading and path setup.

```python
class Storage:
    name = "storage"
    PATH_BACKEND: type[pathlib.Path] = pathlib.Path

    def __init__(
        self,
        *args: Any,
        config: configparser.ConfigParser | None = None,
        **kwargs: Any,
    ) -> None:
        self.flock_path: PATH_TYPES = ".lock"
        if config is not None:
            self.configuration = config
        else:
            self.configuration = BandersnatchConfig()
        try:
            storage_backend = self.configuration["mirror"]["storage-backend"]
        except (KeyError, TypeError):
            storage_backend = "filesystem"
        if storage_backend != self.name:
            return
        # register relevant path backends etc
        self.initialize_plugin()
        try:
            self.mirror_base_path = self.PATH_BACKEND(
                self.configuration.get("mirror", "directory")
            )
        except (configparser.NoOptionError, configparser.NoSectionError):
            self.mirror_base_path = self.PATH_BACKEND(".")
        self.web_base_path = self.mirror_base_path / "web"
        self.json_base_path = self.web_base_path / "json"
        self.pypi_base_path = self.web_base_path / "pypi"
        self.simple_base_path = self.web_base_path / "simple"
        self.executor = ThreadPoolExecutor(
            max_workers=self.configuration.getint("mirror", "workers")
        )
        self.loop = asyncio.get_event_loop()
```

--------------------------------

### Install Project Dependencies

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Installs the necessary project dependencies from requirement files into the development virtual environment. This ensures all required packages are available for development.

```bash
/path/to/venv/bin/pip install -r requirements.txt -r test-requirements.txt
```

--------------------------------

### Example Usage with CloudFront URL

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/storage_options.md.txt

Demonstrates how a bandersnatch mirror served via S3 and CloudFront might be accessed.

```shell
pip install -i my-s3-bucket.cloudfront.net/prefix/web/simple install django
```

--------------------------------

### Install and Run Bandersnatch Mirror

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/CONTRIBUTING.md.txt

Install the latest version of Bandersnatch and run it to mirror package indexes. Customize the configuration file as needed. Warning: This process can consume significant disk space.

```console
cd bandersnatch
/path/to/venv/bin/pip install --upgrade .
/path/to/venv/bin/bandersnatch -c src/bandersnatch/default.conf mirror
```

--------------------------------

### Start Existing Minio Docker Container

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Attaches to and starts a Minio Docker container that has already been created. This is an alternative to running `docker run` if the container already exists.

```bash
docker start minio --attach
```

--------------------------------

### Mirror Configuration Example

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/configuration.html

This snippet demonstrates how to retrieve and validate various mirror configuration settings, including simple format, comparison method, and download mirror options. It also handles potential errors for unsupported values.

```python
    try:
        simple_format_raw = config.get("mirror", "simple-format")
        simple_format = get_format_value(simple_format_raw)
    except ValueError as e:
        logger.error(
            f"Supplied simple-format {simple_format_raw} is not supported!"
            + " Please updare the simple-format in the [mirror] section of"
            + " your config to a supported value."
        )
        raise e

    compare_method = config.get("mirror", "compare-method")
    if compare_method not in ("hash", "stat"):
        raise ValueError(
            f"Supplied compare_method {compare_method} is not supported! Please "
            + "update compare_method to one of ('hash', 'stat') in the [mirror] "
            + "section."
        )

    download_mirror = config.get("mirror", "download-mirror")

    if download_mirror:

        logger.debug(
            "Checking config for only download from alternative download mirror"
        )
        download_mirror_no_fallback = config.getboolean(
            "mirror", "download-mirror-no-fallback"
        )
        if download_mirror_no_fallback:
            logger.info("Setting to download from mirror without fallback")
        else:
            logger.debug("Setting to fallback to original if download mirror fails")
    else:
        download_mirror_no_fallback = False
        logger.debug(
            "Skip checking download-mirror-no-fallback because dependent option"
            + "is not set in config."
        )

    cleanup = config.getboolean("mirror", "cleanup", fallback=False)

    api_method = config.get("mirror", "api-method", fallback="simple")
    if api_method not in ("simple", "xmlrpc"):
        raise ValueError(
            f"Supplied api-method {api_method} is not supported! Please "
            + "update api-method to one of ('simple', 'xmlrpc') in the [mirror] "
            + "section."
        )

    return SetConfigValues(
        json_save,
        root_uri,
        diff_file_path,
        diff_append_epoch,
        digest_name,
        storage_backend_name,
        cleanup,
        release_files_save,
        compare_method,
        download_mirror,
        download_mirror_no_fallback,
        simple_format,
        api_method,
    )
```

--------------------------------

### Mirror Bootstrap Directory Setup

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

Sets up necessary directories for the mirror, including web directories for simple, packages, and JSON metadata. It conditionally adds JSON directories if JSON saving is enabled.

```python
paths = [
            self.storage_backend.PATH_BACKEND(""),
            self.storage_backend.PATH_BACKEND("web/simple"),
            self.storage_backend.PATH_BACKEND("web/packages"),
            self.storage_backend.PATH_BACKEND("web/local-stats/days"),
        ]
        if self.json_save:
            logger.debug("Adding json directories to bootstrap")
            paths.extend(
                [
                    self.storage_backend.PATH_BACKEND("web/json"),
                    self.storage_backend.PATH_BACKEND("web/pypi"),
                ]
            )
        for path in paths:
            path = self.homedir / path
            if not path.exists():
                logger.info(f"Setting up mirror directory: {path}")
                path.mkdir(parents=True)
```

--------------------------------

### Example Tox Test Output

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/CONTRIBUTING.md.txt

This is an example of the output generated when running tox tests, showing the installation process, test collection, and execution by pytest.

```console
$ tox
GLOB sdist-make: /Users/dhubbard/PycharmProjects/bandersnatch/setup.py
py36 create: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/py36
py36 installdeps: -rtest-requirements.txt
py36 inst: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/dist/bandersnatch-2.2.1.zip
py36 installed: apipkg==1.4,attrs==18.1.0,bandersnatch==2.2.1,certifi==2018.4.16,chardet==3.0.4,coverage==4.5.1,execnet==1.5.0,flake8==3.5.0,idna==2.6,mccabe==0.6.1,more-itertools==4.1.0,packaging==17.1,pep8==1.7.1,pluggy==0.6.0,py==1.5.3,pycodestyle==2.3.1,pyflakes==1.6.0,pyparsing==2.2.0,pytest==3.5.1,pytest-cache==1.0,pytest-codecheckers==0.2,pytest-cov==2.5.1,pytest-timeout==1.2.1,python-dateutil==2.7.3,requests==2.18.4,six==1.11.0,tox==3.0.0,urllib3==1.22,virtualenv==15.2.0,xmlrpc2==0.3.1
py36 runtests: PYTHONHASHSEED='42669967'
py36 runtests: commands[0] | pytest
========================================================================================================================= test session starts =========================================================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/dhubbard/PycharmProjects/bandersnatch, inifile: pytest.ini
plugins: timeout-1.2.1, cov-2.5.1, codecheckers-0.2
timeout: 10.0s method: signal
collected 94 items

src/bandersnatch/__init__.py ..                                                                                                                                                                                                                                 [  2%]
src/bandersnatch/buildout.py ..                                                                                                                                                                                                                                 [  4%]
src/bandersnatch/log.py ..                                                                                                                                                                                                                                      [  6%]
src/bandersnatch/main.py ..                                                                                                                                                                                                                                     [  8%]
src/bandersnatch/master.py ..                                                                                                                                                                                                                                 [ 10%]
src/bandersnatch/mirror.py ..                                                                                                                                                                                                                                 [ 12%]
src/bandersnatch/package.py ..                                                                                                                                                                                                                                  [ 14%]
src/bandersnatch/release.py ..                                                                                                                                                                                                                                  [ 17%]
src/bandersnatch/utils.py ..                                                                                                                                                                                                                                  [ 19%]
src/bandersnatch/tests/conftest.py ..                                                                                                                                                                                                                           [ 21%]
src/bandersnatch/tests/test_main.py .......                                                                                                                                                                                                                     [ 28%]
src/bandersnatch/tests/test_master.py ...........                                                                                                                                                                                                               [ 40%]

```

--------------------------------

### Hash-Index Directory Structure Example

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

Illustrates the nested directory layout when hash-index is enabled, showing projects grouped under their initial letter.

```text
<mirror directory>/
└── web/
    └── simple/
        ├── b/
        │   ├── boto3/
        │   │   └── index.html
        │   └── botocore/
        │       └── index.html
        ├── c/
        │   ├── charset-normalizer/
        │   │   └── index.html
        │   ├── certifi/
        │   │   └── index.html
        │   └── cryptography/
        │       └── index.html
        ├── t/
        │   └── typing-extensions/
        │       └── index.html
        ├── ...
        └── index.html

```

--------------------------------

### Setup Storage Path for Mirroring

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

Asynchronously sets up a concrete storage path for mirroring, creating parent directories if they don't exist. It can append an epoch timestamp to the path and adjusts the path to point to a file within a directory if the configured path is a directory. Existing files at the target path are removed.

```python
async def _setup_diff_file(
    storage_plugin: Storage, configured_path: str, append_epoch: bool
) -> Path:
    # save some typing. maybe a method to add to Storage?
    async def storage_plugin_exec[T](fn: Callable[..., T], *args: Any) -> T:
        return await storage_plugin.loop.run_in_executor(
            storage_plugin.executor, fn, *args
        )

    # use the storage backend to convert an abstract path to a concrete one
    concrete_path = storage_plugin.PATH_BACKEND(configured_path)

    # create parent directory if needed
    await storage_plugin_exec(
        lambda: concrete_path.parent.mkdir(exist_ok=True, parents=True)
    )

    # adjust the file/directory name if timestamps are enabled
    if append_epoch:
        epoch = int(time.time())
        concrete_path = concrete_path.with_name(f"{concrete_path.name}-{epoch}")

    # if the path is directory, adjust to write to a file inside it
    if await storage_plugin_exec(concrete_path.is_dir):
        concrete_path = concrete_path / "mirrored-files"

    # if there is an existing file there then delete it
    if await storage_plugin_exec(concrete_path.is_file):
        concrete_path.unlink()

    return concrete_path
```

--------------------------------

### Install Bandersnatch using pip

Source: https://bandersnatch.readthedocs.io/en/latest/installation.html

Installs the latest stable version of bandersnatch within a virtual environment. This method places the executable in `bandersnatch/bin/bandersnatch` and is suitable for most users.

```bash
python3.8 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help

```

--------------------------------

### NGINX Rewrite for Hash Index

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

Example NGINX configuration to serve hash-indexed files efficiently.

```nginx
rewrite ^/simple/.+ "/simple/";
rewrite ^/legacy/.+ "/legacy/";
rewrite ^/files/.+ "/files/";
rewrite ^/packages/.+ "/packages/";
rewrite ^/old-packages/.+ "/old-packages/";
```

--------------------------------

### Run Minio Docker Container for S3 Tests

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Starts a Minio Docker container, mapping ports and mounting a volume for data persistence. This container provides an S3-compatible object storage service for testing.

```bash
docker run \
  --rm \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -v ~/tmp/minio:/data \
  minio/minio server /data --console-address ":9001"
```

--------------------------------

### get

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/master.html

Fetches data from a given path on the package repository. It handles URL construction, serial number checking, and yields the response object.

```APIDOC
## `get`

### Description
Fetches data from a given path on the package repository. It handles URL construction, serial number checking, and yields the response object.

### Parameters
- **path** (str) - The path to fetch data from.
- **required_serial** (int | None) - The expected serial number for cache validation.
- **
**kw** (Any) - Additional keyword arguments to pass to `aiohttp.ClientSession.get`.
```

--------------------------------

### Get All Packages via Simple API (PEP 691)

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/master.html

Fetches all packages using the PEP 691 Simple API JSON endpoint. Returns a dictionary mapping package names to their serial numbers.

```python
async def _all_packages_simple(self) -> dict[str, int]:
        """
        Fetch all packages using the PEP 691 Simple API JSON endpoint.
        Returns a dict mapping package names to their serial numbers.
        """
        logger.info("Fetching all packages via Simple (PEP 691 v1) API")
        simple_index = await self.fetch_simple_index()
        if not simple_index:
            return {}
        all_packages = {}
        for project in simple_index.get("projects", []):
            name = project.get("name")
            serial = project.get("_last-serial")
            if name is not None and serial is not None:
                all_packages[name] = serial
            else:
                logger.warning(
                    f"Skipping malformed project entry in simple index: {project}"
                )
        logger.debug(f"Fetched #{len(all_packages)} from simple JSON index")
        return all_packages
```

--------------------------------

### Get Storage Backend Plugins

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

A convenience function to load and return storage backend plugin objects. It defaults to the 'filesystem' backend if none is specified.

```python
def storage_backend_plugins(
    backend: str | None = "filesystem",
    config: configparser.ConfigParser | None = None,
    clear_cache: bool = False,
) -> Iterable[Storage]:
    """
    Load and return the release filtering plugin objects

    Parameters
    ==========
    backend: str
        The optional enabled storage plugin to search for
    config: configparser.ConfigParser
        The optional configparser instance to pass in
    clear_cache: bool
        Whether to clear the plugin cache

    Returns
    -------
    list of bandersnatch.storage.Storage:
        List of objects derived from the bandersnatch.storage.Storage class
    """
    return load_storage_plugins(
        STORAGE_PLUGIN_RESOURCE,
        enabled_plugin=backend,
        config=config,
        clear_cache=clear_cache,
    )
```

--------------------------------

### Run Unit Tests with Tox

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Use this command to execute unit tests. Ensure you are in the 'bandersnatch' directory and have tox installed. You can optionally add verbosity flags like -vv or specify environments like -e py3 or -e doc_build.

```bash
cd bandersnatch
/path/to/venv/bin/tox [-vv] [-e py3|doc_build]

```

--------------------------------

### Enable All Installed Filter Plugins

Source: https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html

Use this configuration to enable all filter plugins that are installed. This is the default behavior if no specific plugins are listed.

```ini
[plugins]
enabled = all

```

--------------------------------

### Install bandersnatch using pip

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/installation.md.txt

Installs the latest stable version of bandersnatch into a virtual environment. Ensure you have Python 3.10.0 or higher.

```console
python3.8 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help
```

--------------------------------

### Example Bandersnatch configuration for diff-append-epoch

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

This example shows how to configure Bandersnatch to append the current epoch time to the diff file name, allowing for time-based tracking of changes.

```ini
[mirror]
; ...
diff-file = /srv/pypi/new-files
diff-append-epoch = true
```

--------------------------------

### Install Bandersnatch in Editable Mode

Source: https://bandersnatch.readthedocs.io/en/latest/CONTRIBUTING.html

Installs the Bandersnatch package in editable mode within the virtual environment. This allows for changes in the source code to be reflected immediately without reinstallation.

```bash
/path/to/venv/bin/pip install -e .
```

--------------------------------

### Create Directory

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch_storage_plugins/filesystem.html

Creates a directory at the specified path. Supports creating parent directories if 'parents' is True and handles existing directories if 'exist_ok' is True.

```python
    def mkdir(
        self,
        path: PATH_TYPES,
        exist_ok: bool = False,
        parents: bool = False,
    ) -> None:
        """Create the provided directory"""
        if not isinstance(path, pathlib.Path):
            path = pathlib.Path(path)
        return path.mkdir(exist_ok=exist_ok, parents=parents)

```

--------------------------------

### Build BanderX Docker Image

Source: https://bandersnatch.readthedocs.io/en/latest/_sources/serving.md.txt

Build the BanderX Docker image. Navigate to the 'src/banderx' directory and run the docker build command.

```bash
cd src/banderx
docker build -t banderx .
```

--------------------------------

### Asynchronous GET Request with Stale Check

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/master.html

Performs an asynchronous GET request to a given path, optionally checking for stale data using the PYPI_LAST_SERIAL header. Yields the response object.

```python
async def get(
        self,
        path: str,
        required_serial: int | None,
        **kw: Any
    ) -> AsyncGenerator[aiohttp.ClientResponse, None]:
        logger.debug(f"Getting {path} (serial {required_serial})")
        if not path.startswith(("https://", "http://")):
            path = self.url + path
        async with self.session.get(path, **kw) as r:
            got_serial = (
                int(r.headers[PYPI_SERIAL_HEADER])
                if PYPI_SERIAL_HEADER in r.headers
                else None
            )
            await self.check_for_stale_cache(path, required_serial, got_serial)
            yield r
```

--------------------------------

### Initialize Filesystem Storage Plugin

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch_storage_plugins/filesystem.html

Initializes the filesystem storage plugin, determining whether to manage file permissions based on configuration.

```python
import contextlib
import datetime
import filecmp
import hashlib
import logging
import os
import pathlib
import shutil
import tempfile
from collections.abc import Generator
from typing import IO, Any

import filelock

from bandersnatch.storage import PATH_TYPES, StoragePlugin

logger = logging.getLogger("bandersnatch")




class FilesystemStorage(StoragePlugin):
    name = "filesystem"
    PATH_BACKEND: type[pathlib.Path] = pathlib.Path

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        super().__init__(*args, **kwargs)




    def initialize_plugin(self) -> None:
        self.manage_permissions = self.configuration.getboolean(
            "mirror", "storage-filesystem-manage-permissions", fallback=True
        )
```

--------------------------------

### Apache RewriteRule for Hash Index

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

Example Apache configuration to serve hash-indexed files efficiently.

```apache
RewriteRule ^/simple/.+ - [L,PT,QSA]
RewriteRule ^/legacy/.+ - [L,PT,QSA]
RewriteRule ^/files/.+ - [L,PT,QSA]
RewriteRule ^/packages/.+ - [L,PT,QSA]
RewriteRule ^/old-packages/.+ - [L,PT,QSA]
```

--------------------------------

### Minimal Mirror Configuration

Source: https://bandersnatch.readthedocs.io/en/latest/mirror_configuration.html

A basic configuration for mirroring PyPI index and release files. Ensure the directory exists and is writable.

```ini
[mirror]
; base destination path for mirrored files
directory = /srv/pypi

; upstream package repository to mirror
master = https://pypi.org

; parallel downloads - keep low to avoid overwhelming upstream
workers = 3

; per-request time limit
timeout = 15

; global time limit - applied to aiohttp coroutines
global-timeout = 18000

; continue syncing when an error occurs
stop-on-error = false

```

--------------------------------

### Get Mirror Directory

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Retrieves the configured mirror directory path. Defaults to '/srv/pypi' if not specified.

```python
@property
def directory(self) -> str:
    try:
        return self.configuration.get("mirror", "directory")
    except (configparser.NoOptionError, configparser.NoSectionError):
        return "/srv/pypi"
```

--------------------------------

### BandersnatchMirror Initialization

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

Initializes the BandersnatchMirror class, setting up storage backends, workers, and various configuration options for mirroring PyPI. Includes validation for the number of workers.

```python
    def __init__(
        self,
        homedir: Path,
        master: Master,
        storage_backend: str | None = None,
        stop_on_error: bool = False,
        workers: int = 3,
        hash_index: bool = False,
        json_save: bool = False,
        digest_name: str | None = None,
        root_uri: str | None = None,
        keep_index_versions: int = 0,
        diff_append_epoch: bool = False,
        diff_full_path: Path | str | None = None,
        flock_timeout: int = 1,
        diff_file_list: list[Path] | None = None,
        *, # Enforce keyword-only arguments for clarity
        cleanup: bool = False,
        release_files_save: bool = True,
        compare_method: str | None = None,
        download_mirror: str | None = None,
        download_mirror_no_fallback: bool | None = False,
        simple_format: SimpleFormat | str = "ALL",
    ) -> None:
        super().__init__(master=master, workers=workers)
        self.cleanup = cleanup

        if storage_backend:
            self.storage_backend = next(iter(storage_backend_plugins(storage_backend)))
        else:
            self.storage_backend = next(iter(storage_backend_plugins()))
        self.stop_on_error = stop_on_error
        self.loop = asyncio.get_event_loop()
        if isinstance(homedir, WindowsPath):
            self.homedir = self.storage_backend.PATH_BACKEND(homedir.as_posix())
        else:
            self.homedir = self.storage_backend.PATH_BACKEND(str(homedir))
        self.lockfile_path = self.homedir / ".lock"
        self.master = master

        # Stop soon after meeting an error. Continue without updating the
        # mirror's serial if false.
        self.stop_on_error = stop_on_error
        # Whether or not to mirror PyPI JSON metadata to disk
        self.json_save = json_save
        # Whether or not to mirror PyPI release files to disk
        self.release_files_save = release_files_save
        self.hash_index = hash_index
        # Allow configuring a root_uri to make generated index pages absolute.
        # This is generally not necessary, but was added for the official internal
        # PyPI mirror, which requires serving packages from
        # https://files.pythonhosted.org
        self.root_uri = root_uri or ""
        self.diff_append_epoch = diff_append_epoch
        self.diff_full_path = diff_full_path
        self.keep_index_versions = keep_index_versions
        self.digest_name = digest_name if digest_name else "sha256"
        self.compare_method = compare_method if compare_method else "hash"
        self.download_mirror = download_mirror
        self.download_mirror_no_fallback = download_mirror_no_fallback
        self.workers = workers
        self.diff_file_list = diff_file_list or []
        if self.workers > 10:
            raise ValueError("Downloading with more than 10 workers is not allowed.")
        # Use self. variables to pass as some defaults are defined ...
        self.simple_api = SimpleAPI(
            self.storage_backend,
            simple_format,
            self.diff_file_list,
            self.digest_name,
        )
```

--------------------------------

### Main Metadata Verification Entry Point

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/verify.html

This function initiates the metadata verification process. It configures storage backends, sets up a thread pool executor, and orchestrates the verification of all JSON metadata files found in the mirror directory.

```python
async def metadata_verify(config: ConfigParser, args: Namespace) -> int:
    """Crawl all saved JSON metadata or online to check we have all packages
    if delete - generate a diff of unowned files"""
    all_package_files: list[Path] = []

    storage_backend = next(
        iter(
            storage_backend_plugins(
                config=config,
                clear_cache=True,
                backend=config.get("mirror", "storage-backend"),
            )
        )
    )

    mirror_base_path = storage_backend.PATH_BACKEND(config.get("mirror", "directory"))
    json_base = mirror_base_path / "web" / "json"
    workers = args.workers or config.getint("mirror", "workers")
    executor = concurrent.futures.ThreadPoolExecutor(max_workers=workers)

    logger.info(f"Starting verify for {mirror_base_path} with {workers} workers")
    try:
        json_files = list(x.name for x in json_base.iterdir())
    except FileExistsError as fee:
        logger.error(f"Metadata base dir {json_base} does not exist: {fee}")
        return 2
    if not json_files:
        logger.error("No JSON metadata files found. Can not verify")
        return 3

    logger.debug(f"Found {len(json_files)} objects in {json_base}")
    logger.debug(f"Using a {workers} thread ThreadPoolExecutor")
    async with Master(
        config.get("mirror", "master"),
        config.getfloat("mirror", "timeout"),
        config.getfloat("mirror", "global-timeout", fallback=None),
    ) as master:
        await verify_producer(
            master,
            config,
            all_package_files,
            mirror_base_path,
            json_files,
            args,
            executor,
        )

    if not args.delete:
        return 0

    return await delete_unowned_files(

```

--------------------------------

### Get JSON PyPI Symlink Path

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

Returns the path for the JSON PyPI symlink for a given package name.

```python
return self.webdir / "pypi" / package_name / "json"
```

--------------------------------

### mkdir

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Creates the specified directory. Supports options for existing directories and creating parent directories.

```APIDOC
## mkdir

### Description
Create the provided directory.

### Method
`mkdir`

### Parameters
#### Path Parameters
- **path** (PATH_TYPES) - Required - The path of the directory to create.
- **exist_ok** (bool) - Optional - If True, do not raise an error if the directory already exists. Defaults to False.
- **parents** (bool) - Optional - If True, create parent directories as needed. Defaults to False.
```

--------------------------------

### Get File Size

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch_storage_plugins/filesystem.html

Retrieves the size of a file in bytes. Converts string paths to pathlib.Path objects if necessary.

```python
if not isinstance(path, pathlib.Path):
            path = pathlib.Path(path)
        return path.stat().st_size
```

--------------------------------

### Storage Plugin Methods

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Provides methods for interacting with storage backends, such as getting and setting upload times for files.

```APIDOC
## Storage

### Description
Represents a storage backend for Bandersnatch.

### Methods

#### get_upload_time

##### Description
Get the upload time of a given path.

##### Parameters
- **path** (PATH_TYPES) - Description: The path to check.

##### Returns
- **datetime.datetime**: The upload time of the path.

#### set_upload_time

##### Description
Set the upload time of a given path.

##### Parameters
- **path** (PATH_TYPES) - Required - Description: The path to set the upload time for.
- **time** (datetime.datetime) - Required - Description: The upload time to set.

##### Returns
- None
```

--------------------------------

### Prepare Version Directory and Clean Old Versions

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

Prepares the 'versions' directory for a package, creating it if it doesn't exist. It also cleans up old version files (HTML, v1_html, v1_json) by removing all but the most recent 'keep_index_versions' files.

```python
def _prepare_versions_path(self, package: Package) -> Path:
        versions_path = (
            self.storage_backend.PATH_BACKEND(str(self.simple_directory(package)))
            / "versions"
        )
        if not versions_path.exists():
            versions_path.mkdir()
        else:
            for ext in (".html", ".v1_html", ".v1_json"):
                version_files = sorted(
                    p for p in versions_path.iterdir() if p.name.endswith(ext)
                )
                version_files_to_remove = (
                    len(version_files) - self.keep_index_versions + 1
                )
                for i in range(version_files_to_remove):
                    version_files[i].unlink()

        return versions_path
```

--------------------------------

### Bandersnatch Mirror Synchronization

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/mirror.html

This snippet demonstrates the core logic for setting up a Bandersnatch mirror and synchronizing packages. It configures the master connection, initializes the BandersnatchMirror instance with various options, and then calls the synchronize method. It also includes logic for handling diff files and logging changes.

```python
storage_plugin = next(
        iter(
            storage_backend_plugins(
                config_values.storage_backend_name, config=config, clear_cache=True
            )
        )
    )

    diff_full_path: Path | None = None
    if config_values.diff_file_path:
        diff_full_path = await _setup_diff_file(
            storage_plugin,
            config_values.diff_file_path,
            config_values.diff_append_epoch,
        )

    mirror_url = config.get("mirror", "master")
    allow_non_https = config.getboolean("mirror", "allow-non-https")
    timeout = config.getfloat("mirror", "timeout")
    global_timeout = config.getfloat("mirror", "global-timeout", fallback=None)
    proxy = config.get("mirror", "proxy", fallback=None)
    storage_backend = config_values.storage_backend_name
    homedir = Path(config.get("mirror", "directory"))

    # Always reference those classes here with the fully qualified name to
    # allow them being patched by mock libraries!
    async with Master(
        mirror_url,
        timeout,
        global_timeout,
        proxy,
        allow_non_https,
        config_values.api_method,
    ) as master:
        mirror = BandersnatchMirror(
            homedir,
            master,
            storage_backend=storage_backend,
            stop_on_error=config.getboolean("mirror", "stop-on-error"),
            workers=config.getint("mirror", "workers"),
            hash_index=config.getboolean("mirror", "hash-index"),
            json_save=config_values.json_save,
            root_uri=config_values.root_uri,
            digest_name=config_values.digest_name,
            compare_method=config_values.compare_method,
            keep_index_versions=config.getint(
                "mirror", "keep_index_versions", fallback=0
            ),
            diff_append_epoch=config_values.diff_append_epoch,
            diff_full_path=diff_full_path,
            cleanup=config_values.cleanup,
            release_files_save=config_values.release_files_save,
            download_mirror=config_values.download_mirror,
            download_mirror_no_fallback=config_values.download_mirror_no_fallback,
            simple_format=config_values.simple_format,
        )
        changed_packages = await mirror.synchronize(
            specific_packages, sync_simple_index=sync_simple_index
        )

    logger.info(f"{len(changed_packages)} packages had changes")
    for package_name, changes in changed_packages.items():
        package_changes = []
        for change in changes:
            package_changes.append(mirror.homedir / change)
        mirror.diff_file_list.extend(package_changes)
        loggable_changes = [str(chg) for chg in package_changes]
        logger.debug(f"{package_name} added: {loggable_changes}")

    if diff_full_path:
        logger.info(f"Writing diff file to '{diff_full_path}'")
        # File is written in text mode; universal newlines will translate this to os.linesep
        diff_text = "\n".join([str(chg.absolute()) for chg in mirror.diff_file_list])
        await storage_plugin.loop.run_in_executor(
            storage_plugin.executor, diff_full_path.write_text, diff_text
        )

    return 0

```

--------------------------------

### Get JSON Paths

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Generates a sequence of paths for JSON metadata files, considering canonicalized and original package names.

```python
def get_json_paths(self, name: str) -> Sequence[PATH_TYPES]:
    canonicalized_name = self.canonicalize_package(name)
    paths = [
        self.json_base_path / canonicalized_name,
        self.pypi_base_path / canonicalized_name,
    ]
    if canonicalized_name != name:
        paths.append(self.json_base_path / name)
    return paths
```

--------------------------------

### Get File Lock

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/storage.html

Abstract method to retrieve the appropriate FileLock backend for a given path. Must be implemented by subclasses.

```python
def get_lock(self, path: str) -> filelock.BaseFileLock:
    """
    Retrieve the appropriate `FileLock` backend for this storage plugin

    :param str path: The path to use for locking
    :return: A `FileLock` backend for obtaining locks
    :rtype: filelock.BaseFileLock
    """
    raise NotImplementedError
```

--------------------------------

### Enable Project Size Filtering

Source: https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html

Activate the 'size_project_metadata' plugin to block downloads of projects exceeding a specified size. Configure 'max_package_size' in the 'size_project_metadata' section.

```ini
[plugins]
enabled =
    size_project_metadata

[size_project_metadata]
max_package_size = 1G

```

--------------------------------

### Combine Size Filtering with Allowlist Project Plugin

Source: https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html

Enable both 'size_project_metadata' and 'allowlist_project' for a logical AND filter. This includes projects smaller than 1GB AND present in the allowlist.

```ini
[plugins]
enabled =
    size_project_metadata
    allowlist_project

[allowlist]
packages =
    numpy
    scapy
    flask

[size_project_metadata]
max_package_size = 1G

```

--------------------------------

### Get Package Metadata

Source: https://bandersnatch.readthedocs.io/en/latest/_modules/bandersnatch/master.html

Retrieves the JSON metadata for a specific package. Handles 404 errors by raising a PackageNotFound exception.

```python
async def get_package_metadata(self, package_name: str, serial: int = 0) -> Any:
        try:
            metadata_generator = self.get(f"/pypi/{package_name}/json", serial)
            metadata_response = await metadata_generator.asend(None)
            metadata = await metadata_response.json()
            return metadata
        except aiohttp.ClientResponseError as e:
            if e.status == 404:
                raise PackageNotFound(package_name)
            raise
```