### Example Pip Install Command with S3 Mirror

Source: https://github.com/pypa/bandersnatch/blob/main/docs/storage_options.md

Demonstrates how to use a pip install command pointing to a CloudFront distribution serving an S3-hosted bandersnatch mirror. This example assumes proper CDN setup and redirects.

```shell
pip install -i my-s3-bucket.cloudfront.net/prefix/web/simple install django
```

--------------------------------

### Annotated Bandersnatch Mirror Configuration Example

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

An example configuration file for Bandersnatch, with comments explaining each setting. Use this as a reference or starting point for your own configuration.

```ini
[mirror]
; The directory where the mirror data will be stored.
directory = /srv/pypi

; Save JSON metadata into the web tree:
; URL/pypi/PKG_NAME/json (Symlink) -> URL/json/PKG_NAME
json = false

; Save package release files
release-files = true

; Cleanup legacy non PEP 503 normalized named simple directories
cleanup = false

; The PyPI server which will be mirrored. As of bandersnatch 7.0 the
; upstream must speak the PEP 691 JSON Simple API; this can be pypi.org,
; the test PyPI, or another bandersnatch mirror. See
; docs/mirror_from_mirror_simple_api.rst for the full procedure when
; chaining mirrors.
; master = https://test.pypi.org
; scheme for PyPI server MUST be https
master = https://pypi.org

; The network socket timeout to use for all connections. This is set to a
; somewhat aggressively low value: rather fail quickly temporarily and re-run
; the client soon instead of having a process hang infinitely and have TCP not
; catching up for ages.
timeout = 10

; The global-timeout sets aiohttp total timeout for it's coroutines
; This is set incredibly high by default as aiohttp coroutines need to be
; equipped to handle mirroring large PyPI packages on slow connections.
global-timeout = 1800

; Number of worker threads to use for parallel downloads.
; Recommendations for worker thread setting:
; - leave the default of 3 to avoid overloading the pypi master
; - official servers located in data centers could run 10 workers
; - anything beyond 10 is probably unreasonable and avoided by bandersnatch
workers = 3

; Whether to hash package indexes
; Note that package index directory hashing is incompatible with pip, and so
; this should only be used in an environment where it is behind an application
; that can translate URIs to filesystem locations.  For example, with the
; following Apache RewriteRule:
;     RewriteRule ^([^/])([^/]*)/$ /mirror/pypi/web/simple/$1/$1$2/
;     RewriteRule ^([^/])([^/]*)/([^/]+)$/ /mirror/pypi/web/simple/$1/$1$2/$3
; OR
; following nginx rewrite rules:
;     rewrite ^/simple/([^/])([^/]*)/$ /simple/$1/$1$2/ last;
;     rewrite ^/simple/([^/])([^/]*)/([^/]+)$/ /simple/$1/$1$2/$3 last;
; Setting this to true would put the package 'abc' index in simple/a/abc.
; Recommended setting: the default of false for full pip/pypi compatibility.
hash-index = false

; Format for simple API to be stored in
; Since PEP691 we have HTML and JSON
simple-format = ALL

; Whether to stop a sync quickly after an error is found or whether to continue
; syncing but not marking the sync as successful. Value should be "true" or
; "false".
stop-on-error = false

; The storage backend that will be used to save data and metadata while
; mirroring packages. By default, use the filesystem backend. Other options
; currently include: 'swift'
storage-backend = filesystem

; Setting this to "false" will disable permission management in filesystem plugin.
; This is useful in case when files in the mirror directory are not owned by
; a user who runs Bandersnatch and file permissions are managed externally -
; by using POSIX ACLs, or manually by sysadmin.
; Value should be "true" or "false", default is "true".
storage-filesystem-manage-permissions = true

; Advanced logging configuration. Uncomment and set to the location of a
; python logging format logging config file.
; log-config = /etc/bandersnatch-log.conf

; Generate index pages with absolute urls rather than relative links. This is
; generally not necessary, but was added for the official internal PyPI mirror,
; which requires serving packages from https://files.pythonhosted.org
; root_uri = https://example.com

; Number of consumers which verify metadata
verifiers = 3

; Number of prior simple index.html to store. Used as a safeguard against

```

--------------------------------

### Install and Run Bandersnatch Mirror

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Install Bandersnatch and run it to mirror PyPI. This process can consume significant disk space. Customize the configuration file as needed.

```console
cd bandersnatch
/path/to/venv/bin/pip install --upgrade .
/path/to/venv/bin/bandersnatch -c src/bandersnatch/default.conf mirror
```

--------------------------------

### Tox Test Execution Example

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

An example of the output when running tox tests. This shows the process from environment creation and dependency installation to the pytest test session start and collection.

```console
$ tox
GLOB sdist-make: /Users/dhubbard/PycharmProjects/bandersnatch/setup.py
py36 create: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/py36
py36 installdeps: -rtest-requirements.txt
py36 inst: /Users/dhubbard/PycharmProjects/bandersnatch/.tox/dist/bandersnatch-2.2.1.zip
py36 installed: apipkg==1.4,attrs==18.1.0,bandersnatch==2.2.1,certifi==2018.4.16,chardet==3.0.4,coverage==4.5.1,execnet==1.5.0,flake8==3.5.0,idna==2.6,mccabe==0.6.1,more-itertools==4.1.0,packaging==17.1,pep8==1.7.1,pluggy==0.6.0,py==1.5.3,pycodestyle==2.3.1,pyflakes==1.6.0,pyparsing==2.2.0,pytest==3.5.1,pytest-cache==1.0,pytest-codecheckers==0.2,pytest-cov==2.5.1,pytest-timeout==1.2.1,python-dateutil==2.7.3,requests==2.18.4,six==1.11.0,tox==3.0.0,urllib3==1.22,virtualenv==15.2.0,xmlrpc2==0.3.1
py36 runtests: PYTHONHASHSEED='42669967'
py36 runtests: commands[0] | pytest
=========================================================================================================================
platform darwin -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /Users/dhubbard/PycharmProjects/bandersnatch, inifile: pytest.ini
plugins: timeout-1.2.1, cov-2.5.1, codecheckers-0.2
timeout: 10.0s method: signal
collected 94 items

src/bandersnatch/__init__.py ..                                                                                                                                                                                                                                 [  2%]
src/bandersnatch/buildout.py ..                                                                                                                                                                                                                                 [  4%]
src/bandersnatch/log.py ..                                                                                                                                                                                                                                      [  6%]
src/bandersnatch/main.py ..                                                                                                                                                                                                                                     [  8%]
src/bandersnatch/master.py ..                                                                                                                                                                                                                                   [ 10%]
src/bandersnatch/mirror.py ..                                                                                                                                                                                                                                   [ 12%]
src/bandersnatch/package.py ..                                                                                                                                                                                                                                  [ 14%]
src/bandersnatch/release.py ..                                                                                                                                                                                                                                  [ 17%]
src/bandersnatch/utils.py ..                                                                                                                                                                                                                                    [ 19%]
src/bandersnatch/tests/conftest.py ..                                                                                                                                                                                                                           [ 21%]
src/bandersnatch/tests/test_main.py .......                                                                                                                                                                                                                     [ 28%]
src/bandersnatch/tests/test_master.py ...........                                                                                                                                                                                                               [ 40%]

```

--------------------------------

### Start Existing Minio Container

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Attaches to and starts a Minio Docker container that has already been created. Use this if the container name 'minio' is already in use.

```bash
docker start minio --attach
```

--------------------------------

### Setup Pre-commit Hooks

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Configures and runs pre-commit hooks to ensure code quality and consistency before committing changes. This step is optional but recommended.

```console
/path/to/venv/bin/pre-commit run -a
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Installs all necessary development and testing dependencies for Bandersnatch from requirement files. Ensure your virtual environment is activated.

```console
/path/to/venv/bin/pip install -r requirements.txt -r test-requirements.txt
```

--------------------------------

### Run BanderX Docker Container

Source: https://github.com/pypa/bandersnatch/blob/main/docs/serving.md

Starts the BanderX NGINX container, mounting the mirror data to the expected directory. Use this command to serve your Bandersnatch mirror.

```bash
docker run --name bandersnatch_nginx --mount type=bind,source=/data/pypi/web,target=/data/pypi/web banderx
```

--------------------------------

### Create and Upgrade Pip in a Virtual Environment

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Sets up a Python virtual environment and upgrades pip to the latest version. This is a foundational step for installing project dependencies.

```console
python3 -m venv /path/to/venv
/path/to/venv/bin/pip install --upgrade pip
```

--------------------------------

### Example diff-file and diff-append-epoch configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

This INI configuration snippet illustrates how to set the diff-file path and enable diff-append-epoch to create time-stamped diff files for tracking changes over time.

```ini
[mirror]
; ...
diff-file = /srv/pypi/new-files
diff-append-epoch = true
```

--------------------------------

### Example Connection Timeout Error

Source: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch_docker_compose/README.md

This is an example of a connection timeout error that may occur due to Docker MTU settings. It indicates a failure to connect to the PyPI repository.

```bash
ERROR: Call to list_packages_with_serial @ https://pypi.org/pypi timed out: Connection timeout to host https://pypi.org/pypi (master.py:218)
```

--------------------------------

### Installing Bandersnatch with S3 Support

Source: https://github.com/pypa/bandersnatch/blob/main/docs/storage_options.md

Install bandersnatch with the necessary dependencies for S3 support. This command ensures that the required libraries for interacting with Amazon S3 are included.

```shell
pip install bandersnatch[s3]
```

--------------------------------

### Use diff file with 7zip for archives

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

This example demonstrates using the diff file with 7zip to create split archives of new or changed files. Adjust the archive name and split size as needed.

```console
7za a -i@"/srv/pypi/mirrored-files" -spf -v100m path_to_new_zip.7z
```

--------------------------------

### Run Minio Server Docker Container

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Starts a Minio server instance using Docker, exposing ports for the server and console. This is required for running S3 integration tests.

```bash
docker run \
  --rm \
  -p 9000:9000 \
  -p 9001:9001 \
  --name minio \
  -v ~/tmp/minio:/data \
  minio/minio server /data --console-address ":9001"
```

--------------------------------

### Install bandersnatch using pip

Source: https://github.com/pypa/bandersnatch/blob/main/docs/installation.md

Installs the latest stable version of bandersnatch into a virtual environment. Ensure you have Python 3.10.0 or later.

```console
python3.8 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help
```

--------------------------------

### Run BanderX with Custom Nginx Config

Source: https://github.com/pypa/bandersnatch/blob/main/docs/serving.md

Starts the BanderX NGINX container with a custom NGINX configuration file bind-mounted. This allows for advanced NGINX settings, such as enabling hash index.

```bash
docker run --name bandersnatch_nginx --mount type=bind,source=/data/pypi/web,target=/data/pypi/web --mount type=bind,source=$PWD/nginx.conf,target=/config/nginx.conf banderx
```

--------------------------------

### Install Bandersnatch using pip

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

Installs bandersnatch within a Python virtual environment using pip. This method installs the latest stable, released version.

```shell
python3 -m venv bandersnatch
bandersnatch/bin/pip install bandersnatch
bandersnatch/bin/bandersnatch --help
```

--------------------------------

### Use diff file with rsync

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

This example shows how to use the generated diff file with rsync to copy only new or changed files to a destination. Ensure the diff file path matches your configuration.

```console
rsync -av --files-from=/srv/pypi/mirrored-files / /mnt/usb/
```

--------------------------------

### Sample Cron Job for Bandersnatch Mirror

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

This cron job configuration runs the bandersnatch mirror command every two minutes and logs its output. Ensure the 'logger' utility is installed.

```cron
    LC_ALL=en_US.utf8
    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]
```

--------------------------------

### Apache RewriteRule for hash-index

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

This Apache configuration example shows how to set up RewriteRules to correctly serve files when the hash-index option is enabled. This is necessary for web servers to translate requested paths to the actual file locations.

```default
RewriteRule ^([^/])([^/]*)/$ /mirror/pypi/web/simple/$1/$1$2/
RewriteRule ^([^/])([^/]*)/([^/]+)$/ /mirror/pypi/web/simple/$1/$1$2/$3
```

--------------------------------

### Run Bandersnatch with Docker Compose

Source: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch_docker_compose/README.md

Starts the Bandersnatch services in detached mode using Docker Compose. Ensure your docker-compose.yml is configured.

```bash
docker compose up -d
```

--------------------------------

### Example of Too Many Links Error

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

This console output illustrates an OSError encountered when a filesystem has a limitation on the number of sub-directories per directory, specifically related to 'Too many links'.

```console
    2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449
    OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'
```

--------------------------------

### setup_logging

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Sets up logging for the application.

```APIDOC
## bandersnatch.log.setup_logging(args: Any) -> StreamHandler

### Description
Sets up logging for the application.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Returns
A StreamHandler object.

### Return type
StreamHandler
```

--------------------------------

### Enable All Installed Filter Plugins

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Enables all filter plugins that are installed in the environment. This is a mandatory step for activating plugins.

```ini
[plugins]
enabled = all
```

--------------------------------

### main

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Synchronous entry point for the Bandersnatch application.

```APIDOC
## bandersnatch.main.main(loop: AbstractEventLoop | None = None) -> int

### Description
Synchronous entry point for the Bandersnatch application.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Returns
An integer representing the exit code.

### Return type
int
```

--------------------------------

### Configure API Method

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Choose the API method for fetching package lists and changelogs. 'simple' (PEP 691 v1) is the default and recommended, while 'xmlrpc' is the legacy method for compatibility.

```cfg
; api-method = simple
```

--------------------------------

### Sample Alternative Logging Configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Provides a sample INI-formatted configuration for Python's logging.config module. This can be used to customize Bandersnatch's logging behavior.

```ini
[loggers]
keys=root,file
[handlers]
keys=root,file
[formatters]
keys=common
[logger_root]
level=NOTSET
handlers=root
[logger_file]
level=INFO
handlers=file
propagate=1
qualname=bandersnatch
[formatter_common]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s
[handler_root]
class=StreamHandler
level=DEBUG
formatter=common
args=(sys.stdout,)
[handler_file]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=common
delay=False
args=('/repo/bandersnatch/banderlogfile.log', 'D', 1, 0)

```

--------------------------------

### Install Bandersnatch in Editable Mode

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Installs the Bandersnatch package in editable mode within the virtual environment. This allows for immediate reflection of code changes without reinstallation.

```console
/path/to/venv/bin/pip install -e .
```

--------------------------------

### Amazon S3 Storage Configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/storage_options.md

Configure bandersnatch to use Amazon S3 for storing package mirrors. Ensure the 's3' extra is installed (`pip install bandersnatch[s3]`). Provide S3 bucket details and optional AWS credentials or endpoint URLs.

```ini
[mirror]
# Place your s3 path here - e.g. /{bucket name}/{prefix}
directory = /my-s3-bucket/prefix
# Set storage-backend to s3
storage-backend = s3
# Provide s3 style path - e.g. /{bucket name}/{prefix}/{key}
diff-file = /your-s3-bucket/bucket-key

[s3]
# Optional Region name - can be empty if IAM are set
region_name = us-east-1
aws_access_key_id = your s3 access key
aws_secret_access_key = your s3 secret access key
# Use endpoint_url to indicate custom s3 endpoint e.g. like minio etc.
endpoint_url = endpoint url
# Optional manual signature version for compatibility
signature_version = s3v4
# Optional example for overriding parameters in Boto3 S3 calls
config_param_ServerSideEncryption = aws:kms
config_param_SSEKMSKeyId = your KMS key ID
```

--------------------------------

### Self-Sign Certificate

Source: https://github.com/pypa/bandersnatch/blob/main/src/banderx/README.md

Self-signs the generated CSR using the private key to create a self-signed certificate. This example sets the validity period to 180 days.

```bash
openssl x509 -req -days 180 -in ./src/banderx/certificate.csr -signkey ./src/banderx/private.key -out ../banderx/certificate.crt
```

--------------------------------

### async_main

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronous entry point for the Bandersnatch application.

```APIDOC
## bandersnatch.main.async_main(args: Namespace, config: ConfigParser) -> int

### Description
Asynchronous entry point for the Bandersnatch application.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Returns
An integer representing the exit code.

### Return type
int
```

--------------------------------

### Simple Format Index File Structure

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Illustrates the directory structure for generated index files when using the simple-format option, compatible with the Simple Repository API.

```text
<mirror directory>/
└── web/
    ├── packages/...
    └── simple/
        ├── index.html
        ├── index.v1_html
        ├── index.v1_json
        ├── someproject/
        │   ├── index.html
        │   ├── index.v1_html
        │   └── index.v1_json
        ├── anotherproject/
        │   ├── index.html
        │   ├── index.v1_html
        │   └── index.v1_json
        └── ...

```

--------------------------------

### Remove Minio Docker Container

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Removes the Minio Docker container. This is useful for cleaning up resources or resolving conflicts when trying to start a new container with the same name.

```bash
docker rm minio
```

--------------------------------

### Combine Size Filtering with Allowlist and Allowlist Project Plugin

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Configure size_project_metadata filtering with both an allowlist and the allowlist_project plugin. Projects are included if they are smaller than max_package_size AND are in the allowlist.

```ini
[plugins]
enabled =
    size_project_metadata
    allowlist_project

[allowlist]
packages =
    numpy
    scapy
    flask

[size_project_metadata]
max_package_size = 1G

```

--------------------------------

### Configure Package Allowlist and Blocklist with Version Specifiers

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Defines packages to be blocklisted or allowlisted based on PEP440 version specifiers. Blocklisted packages matching the specifier will not be downloaded. Allowlisted packages not matching the specifier will not be downloaded.

```ini
[plugins]
enabled =
    blocklist_project
    blocklist_release
    allowlist_project
    allowlist_release

[blocklist]
packages =
    example1
    example2>=1.4.2,<1.9,!=1.5.*,!=1.6.*

[allowlist]
packages =
    black==18.5
    ptr
```

--------------------------------

### Minimal Mirror Configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

A basic configuration for mirroring PyPI. Use this to set the destination directory, master repository, and control download parallelism and timeouts.

```ini
[mirror]
; base destination path for mirrored files
directory = /srv/pypi

; upstream package repository to mirror
master = https://pypi.org

; parallel downloads - keep low to avoid overwhelming upstream
workers = 3

; per-request time limit
timeout = 15

; global time limit - applied to aiohttp coroutines
global-timeout = 18000

; continue syncing when an error occurs
stop-on-error = false

```

--------------------------------

### Run Tox Tests

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Execute unit tests using tox. Navigate to the project directory and run the tox command. Optional arguments like -vv for verbose output or -e for specific environments (e.g., py3 or doc_build) can be used.

```console
cd bandersnatch
/path/to/venv/bin/tox [-vv] [-e py3|doc_build]
```

--------------------------------

### Master.all_packages

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously retrieves all packages available from the master.

```APIDOC
## async bandersnatch.master.Master.all_packages() -> Any

### Description
Asynchronously retrieves all packages available from the master.

### Returns
Any data structure representing all packages.

### Return type
Any
```

--------------------------------

### Configure Diff File Output

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Set a file to record downloaded files for offline systems. The file is overwritten by default, but appending the epoch date to the filename is possible.

```cfg
; diff-file = /srv/pypi/mirrored-files
; diff-append-epoch = true
```

--------------------------------

### Docker Compose Nginx Volume Configuration for HTTPS

Source: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch_docker_compose/README.md

Example Nginx volume configuration within docker-compose.yml to mount SSL certificates for HTTPS support. Ensure certificate files exist in the specified paths.

```yaml
services:
   bandersnatch_nginx:
      # ... other configurations ...
      volumes:
      # ... Other volunes ...
      - "../banderx/certificate.crt:/etc/ssl/certs/nginx.crt:ro" # SSL certificate
      - "../banderx/private.key:/etc/ssl/private/nginx.key:ro" # SSL key
```

--------------------------------

### Run Pre-commit Checks

Source: https://github.com/pypa/bandersnatch/blob/main/docs/CONTRIBUTING.md

Execute pre-commit to lint and format the codebase. Ensure you are in the bandersnatch directory and have activated your virtual environment.

```console
cd bandersnatch
/path/to/venv/bin/pre-commit -a
```

--------------------------------

### bandersnatch.storage.mkdir

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Creates the provided directory, with options for existing directories and parent directory creation.

```APIDOC
## bandersnatch.storage.mkdir

### Description
Create the provided directory.

### Parameters
- **path** (Path | str) - Required - The directory path to create.
- **exist_ok** (bool) - Optional - Defaults to False. If True, do not raise an error if the directory already exists.
- **parents** (bool) - Optional - Defaults to False. If True, create parent directories as needed.
```

--------------------------------

### Build banderx Docker Image

Source: https://github.com/pypa/bandersnatch/blob/main/src/banderx/README.md

Builds the Docker image for banderx. Ensure you are in the project root directory.

```bash
docker build --tag banderx src/banderx
```

--------------------------------

### Configure Regex Filtering for Packages and Releases

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Define regular expression patterns for filtering packages and releases. Use the [filter_regex] section with 'packages' and 'releases' keys.

```ini
[filter_regex]
packages =
    .+-evil$
releases =
    .+alpha\d$

```

--------------------------------

### Combine Size Filtering with Allowlist

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Configure size_project_metadata filtering with an allowlist to create exceptions for specific projects. Projects are included if they are smaller than max_package_size OR are in the allowlist.

```ini
[plugins]
enabled =
    size_project_metadata

[allowlist]
packages =
    numpy

[size_project_metadata]
max_package_size = 1G

```

--------------------------------

### Enable Regex Project Filtering Plugin

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Activate the regex_project plugin to filter packages using regular expressions. Add 'regex_project' to the enabled plugins list.

```ini
[plugins]
enabled =
    regex_project

```

--------------------------------

### Bandersnatch Verify Command Help

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

This command provides help for verifying and fixing your mirror's integrity. It crawls the repository to fix missed files and delete unowned ones. Use with caution.

```bash
bandersnatch verify --help
```

--------------------------------

### FilesystemStorage Class Methods

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch_storage_plugins.md

This section outlines the public methods available on the FilesystemStorage class for interacting with local file storage.

```APIDOC
## class bandersnatch_storage_plugins.filesystem.FilesystemStorage

### Description
Provides methods for interacting with local filesystem storage, including file operations, directory management, and metadata retrieval.

### Methods

#### compare_files(file1: Path | str, file2: Path | str) -> bool
Compares two files to determine if they are identical.

#### copy_file(source: Path | str, dest: Path | str) -> None
Copies a file from a source path to a destination path.

#### delete_file(path: Path | str, dry_run: bool = False) -> int
Deletes the specified file or directory recursively if necessary.

#### exists(path: Path | str) -> bool
Checks if a file or directory exists at the given path.

#### find(root: Path | str, dirs: bool = True) -> str
A helper method to simulate the 'find' command, iterating over directories and filenames relative to a root path.

#### get_file_size(path: Path | str) -> int
Returns the size of the file at the specified path.

#### get_hash(path: Path | str, function: str = 'sha256') -> str
Calculates and returns the hash (defaulting to SHA256) of the file at the given path.

#### get_lock(path: str | None = None) -> UnixFileLock
Retrieves a file lock object for the specified path, used for managing concurrent access.

#### get_upload_time(path: Path | str) -> datetime
Retrieves the upload timestamp for the file at the specified path.

#### initialize_plugin() -> None
Executes initialization code for the storage plugin.

#### is_dir(path: Path | str) -> bool
Checks if the provided path points to a directory.

#### is_file(path: Path | str) -> bool
Checks if the provided path points to a file.

#### mkdir(path: Path | str, exist_ok: bool = False, parents: bool = False) -> None
Creates a directory at the specified path, with options to ignore errors if it already exists and to create parent directories.

#### move_file(source: Path | str, dest: Path | str) -> None
Moves a file from a source path to a destination path.

#### open_file(path: Path | str, text: bool = True, encoding: str = 'utf-8') -> Generator[IO, None, None]
Opens a file and yields a context manager for iterating over its contents. Supports text or binary modes.

#### read_file(path: Path | str, text: bool = True, encoding: str = 'utf-8', errors: str | None = None) -> str | bytes
Reads and returns the contents of a file, either as a string or bytes.

#### rewrite(filepath: Path | str, mode: str = 'w', **kw: Any) -> Generator[IO, None, None]
Atomically rewrites an existing file to prevent race conditions during parallel operations.

#### rmdir(path: Path | str, recurse: bool = False, force: bool = False, ignore_errors: bool = False, dry_run: bool = False) -> int
Removes a directory, with options for recursive deletion, forceful removal, and error handling.

#### scandir(path: Path | str) -> Generator[DirEntry, None, None]
Scans the contents of a directory and yields directory entries.

#### set_upload_time(path: Path | str, time: datetime) -> None
Sets the upload timestamp for the file at the specified path.

#### update_safe(filename: Path | str, **kw: Any) -> Generator[IO, None, None]
Atomically rewrites a file, allowing clients to signal non-updates by deleting the temporary file.

#### walk(root: Path | str, dirs: bool = True) -> list[Path]
Recursively walks a directory tree starting from the root path.

#### write_file(path: Path | str, contents: str | bytes) -> None
Writes data to a file at the specified path, handling both string and byte content.
```

--------------------------------

### Configure Glob Pattern for Requirements Files

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Enables filtering based on packages listed in requirements files, using a glob pattern to match multiple files. All files matching 'requirements-*.txt' in the specified path will be considered.

```ini
[plugins]
enabled =
    project_requirements
[allowlist]
requirements_path = /requirements
requirements =
    requirements-*.txt
```

--------------------------------

### Release Files Folder Structure

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Shows the directory structure for package release files, organized by checksums, within the mirror directory.

```text
<mirror directory>/
└── web/
    ├── packages/
    │   ├── 1a/
    │   │   └── 70/
    │   │       └── e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/
    │   │           └── click-8.1.6-py3-none-any.whl
    │   ├── 8b/
    │   │   ├── 3a/
    │   │   │   └── b569b932cf737b525eb4c7a2b615ec07b102dff64f1d8a0fe52a48b911fc/
    │   │   │       └── diff-2023.12.5.tar.gz
    │   │   └── e2/
    │   │       └── 4823d9f02d2743a02e2c236f98b96b52f7a16b2bedc0e3148322dffbd06f/
    │   │           └── black-24.1.0-cp39-cp39-win_amd64.whl
    │   ├── 31/
    │   │   ├── 5f/
    │   │   │   └── ...
    │   │   └── 7a/
    │   │       └── ...
    │   └── ...
    └── simple/
        ├── click/
        ├── diff/
        ├── black/
        ├── ...
        └── index.html

```

--------------------------------

### Master.fetch_simple_index

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously fetches the simple index, returning a mapping of all project data from the PyPI Index API.

```APIDOC
## async bandersnatch.master.Master.fetch_simple_index() -> Any

### Description
Asynchronously fetches the simple index, returning a mapping of all project data from the PyPI Index API.

### Returns
Any data structure representing the PyPI Index API data.

### Return type
Any
```

--------------------------------

### Hash-Index Index File Structure

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Illustrates the folder organization for project index files when hash-index is enabled. Project index folders are grouped by the first letter of their name.

```text
<mirror directory>/
└── web/
    └── simple/
        ├── b/
        │   ├── boto3/
        │   │   └── index.html
        │   └── botocore/
        │       └── index.html
        ├── c/
        │   ├── charset-normalizer/
        │   │   └── index.html
        │   ├── certifi/
        │   │   └── index.html
        │   └── cryptography/
        │       └── index.html
        ├── t/
        │   └── typing-extensions/
        │       └── index.html
        ├── ...
        └── index.html
```

--------------------------------

### Enable Size Project Metadata Filtering

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Activate the size_project_metadata plugin to block downloads of projects exceeding a specified size. Add 'size_project_metadata' to the enabled plugins list.

```ini
[plugins]
enabled =
    size_project_metadata

[size_project_metadata]
max_package_size = 1G

```

--------------------------------

### bandersnatch.storage.storage_backend_plugins

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Loads and returns storage plugin objects, allowing specification of a backend, configuration, and cache clearing.

```APIDOC
## bandersnatch.storage.storage_backend_plugins

### Description
Loads and returns storage plugin objects, allowing specification of a backend, configuration, and cache clearing.

### Parameters
- **backend** (str) - Optional - Defaults to 'filesystem'. The optional enabled storage plugin to search for.
- **config** (configparser.ConfigParser) - Optional - The optional configparser instance to pass in.
- **clear_cache** (bool) - Optional - Whether to clear the plugin cache.

### Returns
List of objects derived from the bandersnatch.storage.Storage class.

### Return type
list of [bandersnatch.storage.Storage](#bandersnatch.storage.Storage)
```

--------------------------------

### Bandersnatch Mirror Command

Source: https://github.com/pypa/bandersnatch/blob/main/docs/index.md

Performs a one-time synchronization of the PyPI mirror. Use --force-check to ensure a full sync.

```bash
bandersnatch mirror [-h] [--force-check]
```

--------------------------------

### Watch Live Logs

Source: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch_docker_compose/README.md

Monitors the live logs for the Bandersnatch and Nginx containers. Use '-f' to follow the output in real-time.

```bash
docker compose logs -f bandersnatch
```

```bash
docker compose logs -f bandersnatch_nginx
```

--------------------------------

### bandersnatch.verify.verify

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously verifies package metadata against a master source, comparing against local files and configuration.

```APIDOC
## async bandersnatch.verify.verify(master: [Master](#bandersnatch.master.Master), config: ConfigParser, json_file: str, mirror_base_path: Path, all_package_files: list[Path], args: Namespace, executor: ThreadPoolExecutor | None = None, releases_key: str = 'releases') -> None

Asynchronously verifies package metadata against a master source.
```

--------------------------------

### Set File Comparison Method

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Configure how Bandersnatch compares files for identity. Use 'hash' for reliability or 'stat' to reduce IO workload.

```cfg
compare-method = hash
```

--------------------------------

### Run Bandersnatch Docker Image

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

Pulls the latest bandersnatch Docker image and runs it with the --help flag to display usage information. Use specific tags for production environments.

```shell
docker pull pypa/bandersnatch
docker run pypa/bandersnatch bandersnatch --help
```

--------------------------------

### Filesystem Storage Configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/storage_options.md

Configure bandersnatch to use the local filesystem for storing package mirrors. This is the default mode. Enable index hashing for better performance with a large number of packages.

```ini
[mirror]
directory = /data/pypi/mirror
storage-backend = filesystem
# Optional index hashing to store simple HTML in directories
# Recommended as PyPI has a lot of packages these days
hash-index = true
```

--------------------------------

### Configure upstream mirror URL

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_from_mirror_simple_api.md

Set the 'master' option in your bandersnatch.conf to the base URL of the upstream mirror. Bandersnatch automatically appends '/simple'.

```default
[mirror]
master = https://example-mirror.org
```

--------------------------------

### Master.url_fetch

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously fetches content from a URL and saves it to a file path.

```APIDOC
## async bandersnatch.master.Master.url_fetch(url: str, file_path: Path, executor: ProcessPoolExecutor | ThreadPoolExecutor | None = None, chunk_size: int = 65536) -> None

### Description
Asynchronously fetches content from a URL and saves it to a file path.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **url** (str) - The URL to fetch.
- **file_path** (Path) - The path to save the fetched content.
- **executor** (ProcessPoolExecutor | ThreadPoolExecutor | None) - An optional executor for concurrent operations.
- **chunk_size** (int) - The size of chunks to download (defaults to 65536).

### Returns
None
```

--------------------------------

### Enable Latest Release Plugin

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Activate the latest_release plugin to keep only a specified number of the latest releases. Add 'latest_release' to the enabled plugins list.

```ini
[plugins]
enabled =
    latest_release

[latest_release]
keep = 3
sort_by = [version|time]

```

--------------------------------

### Filter Release Files by Regex Matching on Metadata

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Filters release files for projects based on regular expression matches against their metadata. The tag 'any' combined with 'release_file.packagetype' allows filtering for specific package types like 'sdist' or 'bdist_wheel'.

```ini
[regex_release_file_metadata]
any:release_file.packagetype =
    sdist
    bdist_wheel
```

--------------------------------

### Enable Prerelease Filtering Plugin

Source: https://github.com/pypa/bandersnatch/blob/main/docs/filtering_configuration.md

Enable the prerelease_release plugin to filter out pre-release versions of packages. Add 'prerelease_release' to the enabled plugins list in the [plugins] section.

```ini
[plugins]
enabled =
    prerelease_release

```

--------------------------------

### bandersnatch.storage.load_storage_plugins

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Loads all storage plugins registered with importlib, with options to specify an enabled plugin, configuration, and cache clearing.

```APIDOC
## bandersnatch.storage.load_storage_plugins

### Description
Loads all storage plugins registered with importlib, with options to specify an enabled plugin, configuration, and cache clearing.

### Parameters
- **entrypoint_group** (str) - Required - The entrypoint group name to load plugins from.
- **enabled_plugin** (str) - Optional - The optional enabled storage plugin to search for.
- **config** (configparser.ConfigParser) - Optional - The optional configparser instance to pass in.
- **clear_cache** (bool) - Optional - Whether to clear the plugin cache.

### Returns
A list of objects derived from the Storage class.

### Return type
List of [Storage](#bandersnatch.storage.Storage)
```

--------------------------------

### Build Docker Image with Swift and Python 3.9

Source: https://github.com/pypa/bandersnatch/blob/main/MAINTAINERS.md

Builds a Docker image for Bandersnatch, enabling Swift support and specifying Python version 3.9. This command is useful for development environments requiring specific Python versions and additional build arguments.

```bash
sudo docker build --build-arg WITH_SWIFT=yes --build-arg PY_VERSION=3.9 -t pypa/bandersnatch:3.9-swift .
```

--------------------------------

### Force Full Sync with Bandersnatch

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

Use this command to force a full synchronization of your mirror with PyPI, useful for resolving inconsistencies or after moving data between file systems.

```bash
bandersnatch mirror --force-check
```

--------------------------------

### Bandersnatch Delete Command Help

Source: https://github.com/pypa/bandersnatch/blob/main/README.md

This command provides help for removing packages from your mirror. Use with caution as it is a dangerous operation.

```bash
bandersnatch delete --help
```

--------------------------------

### Build Docker Image with Default Python 3.9

Source: https://github.com/pypa/bandersnatch/blob/main/MAINTAINERS.md

Builds a Docker image for Bandersnatch using the default Python 3.9 version. This is a standard command for creating a production-ready image when specific Python versions or extra build arguments are not required.

```bash
sudo docker build -t pypa/bandersnatch:3.9 .
```

--------------------------------

### Alternative Download Source Configuration

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Configure Bandersnatch to download package metadata from one master and release files from a different download mirror. This can improve download performance or reliability.

```ini
[mirror]
directory = /srv/pypi
; Project and package metadata received from this repository
master = https://pypi.org
; Package distribution artifacts downloaded from here if possible
download-mirror = https://pypi-mirror.example.com/

```

--------------------------------

### JSON API Metadata Folder Structure

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Shows the directory layout for PyPI project metadata when the JSON API is enabled. Files like `web/json/someproject` and `web/pypi/someproject/json` contain project metadata.

```text
<mirror directory>/
├── web/
│   └── json/
│       ├── someproject
│       ├── anotherproject
│       └── ...
├── pypi/
│   ├── someproject/
│   │   └── json
│   ├── anotherproject/
│   │   └── json
│   └── ...
├── packages/
│   └── ...
└── simple/
    └── ...
```

--------------------------------

### Configure Download Mirror

Source: https://github.com/pypa/bandersnatch/blob/main/docs/mirror_configuration.md

Specify an alternative mirror for downloading packages. Bandersnatch will attempt to download from this mirror first, falling back to the master server if unsuccessful. Downloading exclusively from the mirror without fallback is possible but not recommended.

```cfg
; download-mirror = https://pypi-mirror.example.com/
; download-mirror-no-fallback = False
```

--------------------------------

### Bandersnatch Sync Command

Source: https://github.com/pypa/bandersnatch/blob/main/docs/index.md

Synchronizes specific packages with the PyPI master server. Can optionally skip updating the simple index root page.

```bash
bandersnatch sync [-h] [--skip-simple-root] package [package ...]
```

--------------------------------

### BandersnatchMirror.json_file

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Returns the path to the JSON file for a given package.

```APIDOC
## bandersnatch.mirror.BandersnatchMirror.json_file(package_name: str) -> Path

### Description
Returns the path to the JSON file for a given package.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **package_name** (str) - The name of the package.

### Returns
The path to the JSON file.

### Return type
Path
```

--------------------------------

### bandersnatch.verify.verify_producer

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously verifies package metadata from multiple JSON files against a master source and local files.

```APIDOC
## async bandersnatch.verify.verify_producer(master: [Master](#bandersnatch.master.Master), config: ConfigParser, all_package_files: list[Path], mirror_base_path: Path, json_files: list[str], args: Namespace, executor: ThreadPoolExecutor | None = None) -> None

Asynchronously verifies package metadata from multiple JSON files.
```

--------------------------------

### BandersnatchMirror.cleanup_non_pep_503_paths

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Cleans up non-PEP 503 compliant directory paths for packages, which were used for backwards compatibility.

```APIDOC
## async bandersnatch.mirror.BandersnatchMirror.cleanup_non_pep_503_paths(package: [Package](#bandersnatch.package.Package)) -> None

### Description
Cleans up non-PEP 503 compliant directory paths for packages, which were used for backwards compatibility.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **package** ([Package](#bandersnatch.package.Package)) - The package object for which to clean up paths.

### Returns
None
```

--------------------------------

### AllowListProject.filter

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch_filter_plugins.md

Checks if the plugin matches based on the package's metadata.

```APIDOC
## AllowListProject.filter

### Description
Checks if the plugin matches based on the package's metadata.

### Method
N/A (Method within a class)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **metadata** (dict) - The package's metadata.

### Response
#### Success Response (bool)
- Returns True if the values match a filter rule, False otherwise.

#### Response Example
```
True
```

```

--------------------------------

### Bandersnatch Main Command

Source: https://github.com/pypa/bandersnatch/blob/main/docs/index.md

This is the primary command for interacting with bandersnatch. It accepts various subcommands for managing PyPI mirrors.

```bash
bandersnatch [-h] [--version] [-c CONFIG] [--debug] {delete,mirror,verify,sync} ...
```

--------------------------------

### Master.get

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Asynchronously retrieves data from a given URL, optionally checking against a required serial number.

```APIDOC
## async bandersnatch.master.Master.get(path: str, required_serial: int | None, **kw: Any) -> AsyncGenerator[ClientResponse, None]

### Description
Asynchronously retrieves data from a given URL, optionally checking against a required serial number.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **path** (str) - The path to retrieve data from.
- **required_serial** (int | None) - The required serial number.
- **kw** (Any) - Additional keyword arguments.

### Returns
An asynchronous generator yielding ClientResponse objects.

### Return type
AsyncGenerator[ClientResponse, None]
```

--------------------------------

### bandersnatch.storage.open_file

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch.md

Opens a file at the given path, yielding a file context for iteration. Supports text or binary mode.

```APIDOC
## bandersnatch.storage.open_file

### Description
Yield a file context to iterate over. If text is true, open the file with ‘rb’ mode specified.

### Parameters
- **path** (Path | str) - Required - The path to the file.
- **text** (bool) - Optional - Defaults to True. If True, open in text mode; otherwise, open in binary mode.
```

--------------------------------

### AllowListProject.check_match

Source: https://github.com/pypa/bandersnatch/blob/main/docs/bandersnatch_filter_plugins.md

Checks if a package name matches against a project that is allowlisted in the configuration.

```APIDOC
## AllowListProject.check_match

### Description
Checks if a package name matches against a project that is allowlisted in the configuration.

### Method
N/A (Method within a class)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters:
- **name** (str) - The normalized package name of the package/project to check against the blocklist.

### Response
#### Success Response (bool)
- Returns True if the package name matches an allowlisted project, False otherwise.

#### Response Example
```
True
```

```