Mypy Boto3 Builder

https://github.com/youtype/mypy_boto3_builder
Admin
Generates type annotations for boto3, boto3-stubs, types-aiobotocore, and types-aioboto3 projects,...
Tokens:6,951
Snippets:78
Trust Score:6.8
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Setup and Dependencies - Install uv and sync dependencies

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/CLAUDE.md

Commands to install the uv package manager and install project dependencies with all extras and dev dependencies.

```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dependencies
uv sync --all-extras --dev
```

--------------------------------

### Install types-aiobotocore with pip

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/pre_build.md

Install the pre-built type stubs for aiobotocore. The lite version omits session.create_client overloads to save memory but requires explicit type annotations. Remember to also install mypy or pyright.

```bash
python -m pip install 'types-aiobotocore[essential]'

# Lite version does not provide session.create_client overloads
# it is more RAM-friendly, but requires explicit type annotations
python -m pip install 'types-aiobotocore-lite[essential]'

# do not forget to install mypy or pyright
```

--------------------------------

### Parsing flow example

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Shows how to parse a service model using ServicePackageParser. It creates a session, gets the service model for 's3', and parses it to obtain a ServicePackage structure.

```python
service_model = boto3.Session().get_service_model('s3')
parser = ServicePackageParser(service_model)
service_package = parser.parse()  # Returns ServicePackage structure
```

--------------------------------

### Build type annotations via CLI

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/how_to_build.md

Install dependencies and run the builder to generate wheel packages for specified services.

```bash
# Install preferred version of `boto3`
python -m pip install boto3==1.35.71 botocore==1.35.71

# Install `mypy-boto3-builder`
python -m pip install mypy-boto3-builder

# Build all packages in typings directory
python -m mypy_boto3_builder ./typings --product types-boto3 types-boto3-services --output-type wheel

# Or specify required services explicitly
python -m mypy_boto3_builder ./typings --product types-boto3 types-boto3-services --output-type wheel -s ec2 s3

# Install custom `types-boto3` packages
python -m pip install ./typings/*.whl
```

--------------------------------

### Install types-boto3 with pip

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/pre_build.md

Install the pre-built type stubs for boto3 and botocore. The lite version omits session.client/resource overloads to save memory but requires explicit type annotations. Remember to also install mypy or pyright.

```bash
python -m pip install 'types-boto3[essential]'

# Lite version does not provide session.client/resource overloads
# it is more RAM-friendly, but requires explicit type annotations
python -m pip install 'types-boto3-lite[essential]'

# do not forget to install mypy or pyright
```

--------------------------------

### Install types-aioboto3 with pip

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/pre_build.md

Install the pre-built type stubs for aioboto3. The lite version omits session.client/resource overloads to save memory but requires explicit type annotations. Remember to also install mypy or pyright.

```bash
python -m pip install 'types-aioboto3[essential]'

# Lite version does not provide session.client/resource overloads
# it is more RAM-friendly, but requires explicit type annotations
python -m pip install 'types-aioboto3-lite[essential]'

# do not forget to install mypy or pyright
```

--------------------------------

### Building and Testing - Build, pre-commit, integration, unit tests, type checking, linting

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/CLAUDE.md

Commands to build the main package, run pre-commit checks, integration tests, unit tests, type checking with mypy and pyright, and linting/formatting with ruff.

```bash
# Build the main package
./scripts/build.sh

# Run pre-commit checks (formatting, linting, type checking)
./scripts/before_commit.sh

# Run integration tests
python scripts/integration.py

# Run unit tests
pytest

# Type checking with mypy
mypy mypy_boto3_builder

# Type checking with pyright
pyright mypy_boto3_builder

# Linting and formatting with ruff
ruff check mypy_boto3_builder
ruff format mypy_boto3_builder
```

--------------------------------

### Run builder with uvx

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/how_to_build.md

Use uvx to run the builder interactively or with a specific boto3 version.

```bash
# run builder and chat with him :)
uvx mypy_boto3_builder

# set library version explicitly for better results
uvx --with 'boto3==1.35.71' mypy-boto3-builder
```

--------------------------------

### Project Commands - Generate type stubs and run CI workflows

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/CLAUDE.md

Commands to generate type stubs interactively, generate specific services, and run CI workflows for integration and sanity checks.

```bash
# Generate type stubs interactively
uvx mypy_boto3_builder

# Generate specific services
python -m mypy_boto3_builder output_dir --services s3 ec2 --product types-boto3

# Run CI workflows
python scripts/ci.py integration
python scripts/ci.py sanity
```

--------------------------------

### Template Rendering: PackageWriter

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Renders the service package to Python code using PackageWriter.write_service_package().

```python
# Render to Python code:
writer = PackageWriter(output_path)
writer.write_service_package(package, templates_path)
```

--------------------------------

### CLI command structure

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Shows the command structure for the mypy-boto3-builder CLI, including the main command with options and the interactive chat subcommand.

```text
mypy-boto3-builder
├── --product <product_name> --services <AWS SDK services> # Generate type stubs
└── chat             # Interactive configuration
```

--------------------------------

### Generate stubs with Docker

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/how_to_build.md

Run the Docker container to generate stubs into a local output directory, with options for specific services or boto3 versions.

```bash
mkdir output

# generate stubs for all services
docker run -v `pwd`/output:/output -ti mypy_boto3_builder_stable

# generate stubs for s3 service
docker run -v `pwd`/output:/output -ti mypy_boto3_builder_stable -s s3

# generate stubs for a specific boto3 version
docker run -e BOTO3_VERSION=1.35.71 BOTOCORE_VERSION=1.35.71 -v `pwd`/output:/output -ti mypy_boto3_builder_stable
```

--------------------------------

### Pull and tag Docker image

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/how_to_build.md

Download the latest stable Docker image and tag it for local use.

```bash
docker pull docker.pkg.github.com/youtype/mypy_boto3_builder/mypy_boto3_builder_stable:latest
docker tag docker.pkg.github.com/youtype/mypy_boto3_builder/mypy_boto3_builder_stable:latest mypy_boto3_builder
```

--------------------------------

### Parsing Phase: ServicePackageParser

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Parses a service model to create a service package. Uses session.get_service_model() and ServicePackageParser.parse().

```python
# For each service:
service_model = session.get_service_model(service_name)
parser = ServicePackageParser(service_model, service_name)
service_package = parser.parse()
```

--------------------------------

### Service Discovery: get_available_service_names()

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Discovers available AWS services using botocore. Returns a list of service names like 's3', 'ec2', 'lambda'.

```python
# Discover available AWS services
service_names = get_available_service_names()
# Returns: ['s3', 'ec2', 'lambda', ...]
```

--------------------------------

### Generation Phase: Boto3Generator

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Transforms a service package into a target format using Boto3Generator.generate_service_package().

```python
# Transform to target format:
generator = Boto3Generator()
package = generator.generate_service_package(service_package)
```

--------------------------------

### Parallel Processing: Sequential vs ThreadPoolExecutor

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Shows current sequential processing and a potential parallel approach using ThreadPoolExecutor for independent services.

```python
# Current: Sequential
for service in services:
    generate_product(product)
```

```python
# Potential: Parallel
with ThreadPoolExecutor() as executor:
    futures = [executor.submit(generate_product, s) for s in products]
```

--------------------------------

### Template Caching: get_template with lru_cache

Source: https://github.com/youtype/mypy_boto3_builder/blob/main/docsmd/architecture.md

Caches templates per process to avoid re-parsing Jinja2. Uses @lru_cache(maxsize=None) decorator.

```python
@lru_cache(maxsize=None)
def get_template(template_path: Path) -> Template:
    return jinja_env.get_template(str(template_path))
```
Tokens:
Raw