S3PyPI (gorilla-co/s3pypi)

S3PyPI

https://github.com/gorilla-co/s3pypi
Admin
S3PyPI is a CLI for creating a Python Package Repository in an S3 bucket, enabling private package...

Tokens:5,520
Snippets:69
Trust Score:7.4
Update:1 month ago
Show doc for...
Context Summary (auto-generated)
Raw
# S3PyPI

S3PyPI is a command-line interface for creating and managing a private Python Package Repository hosted in an AWS S3 bucket. It enables organizations to publish proprietary Python packages to a private repository that remains fully compatible with `pip install`, allowing seamless integration with existing Python workflows while maintaining package privacy and access control.

The tool handles package uploads, index page generation, and optional features like distributed locking via DynamoDB and basic authentication via Lambda@Edge. S3PyPI includes Terraform configurations for provisioning the required AWS infrastructure (S3 bucket, CloudFront distribution, Route 53 DNS, and optional DynamoDB tables), making it straightforward to deploy a complete private PyPI server at minimal cost.

## CLI Commands

### Upload Packages to S3

The `upload` command publishes Python distribution files (wheels, source distributions) to your S3-hosted repository and automatically updates the package index.

```bash
# Build your package first
cd /path/to/your-project/
python setup.py sdist bdist_wheel

# Upload all distributions to S3
s3pypi upload dist/* --bucket my-pypi-bucket

# Upload with a prefix (useful for organizing packages)
s3pypi upload dist/* --bucket my-pypi-bucket --prefix packages

# Upload and generate a root index listing all packages
s3pypi upload dist/* --bucket my-pypi-bucket --put-root-index

# Force overwrite existing packages
s3pypi upload dist/* --bucket my-pypi-bucket --force

# Fail if any packages already exist (CI/CD safety)
s3pypi upload dist/* --bucket my-pypi-bucket --strict

# Upload with custom S3 options (encryption, ACL)
s3pypi upload dist/* --bucket my-pypi-bucket \
    --s3-put-args='ServerSideEncryption=aws:kms,SSEKMSKeyId=1234abcd-12ab-34cd-56ef-1234567890ab'

# Use a specific AWS profile and region
s3pypi upload dist/* --bucket my-pypi-bucket --profile production --region eu-west-1

# Use custom S3 endpoint (MinIO, LocalStack, etc.)
s3pypi upload dist/* --bucket my-pypi-bucket --s3-endpoint-url http://localhost:9000

# Upload without authentication (public bucket)
s3pypi upload dist/* --bucket my-pypi-bucket --no-sign-request

# Use index.html suffix for S3 website endpoint compatibility
s3pypi upload dist/* --bucket my-pypi-bucket --index.html

# Specify custom DynamoDB locks table
s3pypi upload dist/* --bucket my-pypi-bucket --locks-table my-custom-locks
```

### Delete Packages from S3

The `delete` command removes a specific package version from the repository and updates the index accordingly.

```bash
# Delete a specific package version
s3pypi delete my-package 1.0.0 --bucket my-pypi-bucket

# Delete with prefix (if packages were uploaded with prefix)
s3pypi delete my-package 1.0.0 --bucket my-pypi-bucket --prefix packages

# Delete using specific AWS profile
s3pypi delete my-package 1.0.0 --bucket my-pypi-bucket --profile production
```

### Force Unlock DynamoDB Lock

The `force-unlock` command releases a stuck lock in DynamoDB when a previous upload was interrupted or failed without releasing its lock.

```bash
# Release a stuck lock
s3pypi force-unlock my-pypi-bucket-locks abc123def456

# Force unlock with specific AWS profile
s3pypi force-unlock my-pypi-bucket-locks abc123def456 --profile production --region us-east-1
```

## Installing Packages from S3PyPI

### Using pip with Extra Index URL

Configure pip to use your private repository alongside the public PyPI.

```bash
# Install a package from your private repository
pip install my-private-package --extra-index-url https://pypi.example.com/

# Install from a repository with a prefix
pip install my-private-package --extra-index-url https://pypi.example.com/packages/

# Install with basic auth credentials
pip install my-private-package --extra-index-url https://user:password@pypi.example.com/
```

### Permanent pip Configuration

Configure pip to always check your private repository by editing `~/.pip/pip.conf`:

```ini
[global]
extra-index-url = https://pypi.example.com/
```

For authenticated access:

```ini
[global]
extra-index-url = https://user:password@pypi.example.com/
```

## Terraform Infrastructure Setup

### Basic Terraform Configuration

Create the required AWS resources using the provided Terraform module.

```hcl
# terraform/config.auto.tfvars
region = "eu-west-1"
bucket = "my-pypi-bucket"
domain = "pypi.example.com"
```

```bash
# Initialize and deploy
cd s3pypi/terraform/
terraform init
terraform apply
```

### Advanced Terraform Module Usage

Include S3PyPI as a module in your own Terraform configuration with all features enabled.

```hcl
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

module "s3pypi" {
  source = "github.com/gorilla-co/s3pypi//terraform/modules/s3pypi"

  bucket = "my-pypi-bucket"
  domain = "pypi.example.com"

  # Use wildcard certificate (*.example.com)
  use_wildcard_certificate = true

  # Enable DynamoDB locking for concurrent uploads
  enable_dynamodb_locking = true

  # Enable basic authentication via Lambda@Edge
  enable_basic_auth = true

  providers = {
    aws.us_east_1 = aws.us_east_1
  }
}
```

## Basic Authentication Management

### Adding Users for Basic Auth

Use the provided script to create users with passwords stored securely in AWS Systems Manager Parameter Store.

```bash
# Add a user interactively (prompts for password)
cd basic_auth/
./put_user.py pypi.example.com alice

# Add user with password from stdin (CI/CD pipelines)
echo "mysecretpassword" | ./put_user.py pypi.example.com alice --password-stdin

# Generate a random secure password
./put_user.py pypi.example.com alice --random-password 32

# Overwrite existing user
./put_user.py pypi.example.com alice --overwrite

# Custom salt length for password hashing
./put_user.py pypi.example.com alice --salt-nbytes 64
```

## Python API Usage

### Programmatic Package Upload

Use the S3PyPI core module directly in Python scripts for automated workflows.

```python
from pathlib import Path
from s3pypi.core import Config, upload_packages, delete_package
from s3pypi.storage import S3Config

# Configure S3 connection
cfg = Config(
    s3=S3Config(
        bucket="my-pypi-bucket",
        prefix="packages",           # Optional prefix
        profile="production",        # AWS profile
        region="eu-west-1",
        locks_table="my-locks",      # DynamoDB table for locking
        put_kwargs={                 # Extra S3 PutObject args
            "ServerSideEncryption": "aws:kms",
            "SSEKMSKeyId": "1234abcd-..."
        }
    )
)

# Upload packages
dist_files = [
    Path("dist/mypackage-1.0.0.tar.gz"),
    Path("dist/mypackage-1.0.0-py3-none-any.whl")
]
upload_packages(
    cfg,
    dist_files,
    put_root_index=True,  # Update root index
    strict=False,         # Don't fail on existing files
    force=False           # Don't overwrite existing files
)

# Delete a package version
delete_package(cfg, name="mypackage", version="1.0.0")
```

### Working with Package Indexes

Parse and generate PyPI-compatible HTML index pages.

```python
from s3pypi.index import Index, Hash
from pathlib import Path

# Parse an existing index page
html_content = """
<!DOCTYPE html>
<html>
  <body>
    <a href="mypackage-1.0.0.tar.gz#sha256=abc123">mypackage-1.0.0.tar.gz</a>
  </body>
</html>
"""
index = Index.parse(html_content)
print(index.filenames)
# {'mypackage-1.0.0.tar.gz': Hash(name='sha256', value='abc123')}

# Create a new index
new_index = Index()
new_index.filenames["mypackage-1.0.0.tar.gz"] = Hash.of("sha256", Path("dist/mypackage-1.0.0.tar.gz"))
new_index.filenames["mypackage-1.0.0-py3-none-any.whl"] = Hash.of("sha256", Path("dist/mypackage-1.0.0-py3-none-any.whl"))

# Generate HTML
print(new_index.to_html())
```

### Distribution Parsing Utilities

Parse distribution filenames to extract package name and version.

```python
from s3pypi.core import parse_distribution_id, normalize_package_name

# Parse distribution filenames
dist = parse_distribution_id("hello_world-1.0.0-py3-none-any.whl")
print(f"Name: {dist.name}, Version: {dist.version}")
# Name: hello_world, Version: 1.0.0

dist = parse_distribution_id("my-package-2.3.4.tar.gz")
print(f"Name: {dist.name}, Version: {dist.version}")
# Name: my_package, Version: 2.3.4

# Normalize package names (PEP 503)
print(normalize_package_name("My.Package_Name"))  # my-package-name
print(normalize_package_name("company---test.1")) # company-test-1
```

## IAM Policy Configuration

### Minimum Required IAM Permissions

Configure IAM policy for users or CI/CD pipelines that need to upload packages.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-pypi-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::my-pypi-bucket"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/my-pypi-bucket-locks"
    }
  ]
}
```

## Migration from S3PyPI 0.x to 1.x/2.x

### Import Existing Resources into Terraform

Migrate from CloudFormation-based deployment to Terraform.

```bash
# Initialize Terraform
cd terraform/
terraform init

# Import existing S3 bucket
terraform import module.s3pypi.aws_s3_bucket.pypi my-pypi-bucket

# Import existing CloudFront distribution
terraform import module.s3pypi.aws_cloudfront_distribution.cdn EDFDVBD6EXAMPLE

# Apply configuration
terraform apply
```

### Migrate S3 Index Objects

Rename index.html files to directory-style keys for CloudFront OAI compatibility.

```bash
# Run migration script
python scripts/migrate-s3-index.py my-pypi-bucket
```

For legacy S3 website endpoint compatibility (not recommended):

```bash
# Upload with legacy options
s3pypi upload dist/* --bucket my-pypi-bucket --index.html --s3-put-args='ACL=public-read'
```

## Summary

S3PyPI provides a complete solution for hosting private Python package repositories on AWS infrastructure. The primary use cases include enterprise environments requiring private package distribution, CI/CD pipelines that need to publish internal libraries, and organizations seeking a low-cost alternative to hosted PyPI services. The CLI supports all common workflows including uploading packages with SHA-256 checksums, managing package versions, and handling concurrent uploads through DynamoDB-based locking.

Integration with existing Python tooling is seamless through standard pip configuration using `--extra-index-url`. The Terraform modules enable infrastructure-as-code deployment with optional features like basic authentication via Lambda@Edge and HTTPS via CloudFront with ACM certificates. For automation, the Python API provides programmatic access to all functionality, making it straightforward to integrate package publishing into build pipelines, release automation scripts, or custom deployment workflows.