### Install Torchlayers using pip

Source: https://github.com/szymonmaszke/torchlayers/blob/master/README.md

Provides instructions for installing the torchlayers library using pip. It shows commands for installing the latest release and the nightly build, specifying the `--user` flag for local installation.

```shell
pip install --user torchlayers
```

```shell
pip install --user torchlayers-nightly
```

--------------------------------

### Install Torchlayers using Docker

Source: https://github.com/szymonmaszke/torchlayers/blob/master/README.md

Details how to install and run the torchlayers library using Docker. It includes commands for pulling CPU and GPU-enabled images from Docker Hub and provides an example for a quick CPU start. Users need to install nvidia-docker for GPU support.

```shell
docker pull szymonmaszke/torchlayers:18.04
```

--------------------------------

### Build PyTorch Layer or Module with Example Input - torchlayers

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.html

The `build` function in torchlayers compiles a PyTorch module by providing an example input. This is particularly useful for shape inference and removing overhead. It supports a `post_build` hook for actions like weight initialization after shape inference.

```python
import torch
import torchlayers as tl

class _MyModuleImpl(torch.nn.Linear):
    def post_build(self):
        # You can do anything here really
        torch.nn.init.eye_(self.weights)

MyModule = tl.infer(_MyModuleImpl)
```

--------------------------------

### Torchlayers Conv Example: Building a Classifier

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

This example demonstrates how to use the torchlayers.Conv module to build a simple image classifier. It showcases the sequential application of convolutional layers, activation functions, global max pooling, and a linear layer. The `Conv` layers are initialized with varying output channels, and default parameters like `kernel_size` and `padding='same'` are utilized.

```python
import torchlayers as tl
import torch

class Classifier(tl.Module):
    def __init__(self, out_shape):
        super().__init__()
        self.conv1 = tl.Conv(64, 6)
        self.conv2 = tl.Conv(128)
        self.conv3 = tl.Conv(256)
        self.pooling = tl.GlobalMaxPool()
        self.dense = tl.Linear(out_shape)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        return self.dense(self.pooling(x))
```

--------------------------------

### Build Function: Instantiate PyTorch Model with Shape Inference

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Compiles a PyTorch model by inferring shapes from example input using `torchlayers.build()`. This function returns a standard PyTorch module ready for training or deployment. It requires the model class inheriting from `tl.Module` and an example input tensor for shape inference.

```python
import torch
import torchlayers as tl

class Classifier(tl.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = tl.Conv(64, kernel_size=6)  # in_channels inferred
        self.conv2 = tl.Conv(128, kernel_size=3)
        self.conv3 = tl.Conv(256, kernel_size=3, padding=1)
        self.pooling = tl.GlobalMaxPool()
        self.dense = tl.Linear(10)  # in_features inferred

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        return self.dense(self.pooling(x))

# Build model with example input
model = tl.build(Classifier(), torch.randn(1, 3, 32, 32))

# Now use like any PyTorch model
optimizer = torch.optim.Adam(model.parameters())
output = model(torch.randn(16, 3, 32, 32))  # (16, 10)
```

--------------------------------

### L1/L2 Regularization: Weight Decay via Hooks

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Illustrates applying L1 or L2 regularization to specific model parameters using backward hooks. The example shows how to regularize only the weights of a Linear layer and demonstrates the process with SGD optimizer and loss backpropagation.

```python
import torch
import torchlayers as tl

# Regularize only weights of Linear layer
regularized_model = tl.Sequential(
    tl.Conv(64),
    tl.ReLU(),
    tl.GlobalAvgPool(),
    tl.L2(tl.Linear(30), weight_decay=1e-5, name="weight"),  # Only "weight" parameter
)

model = tl.build(regularized_model, torch.randn(1, 3, 32, 32))

# L2 regularization automatically applied during backward
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
for _ in range(10):
    optimizer.zero_grad()
    output = model(torch.randn(16, 3, 32, 32))
    loss = output.sum()
    loss.backward()  # L2 penalty added to gradients here
    optimizer.step()

# Can also use L1 regularization
l1_layer = tl.L1(tl.Linear(30), weight_decay=1e-5)
```

--------------------------------

### ChannelSplit Layer Example - PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Illustrates how to use the ChannelSplit layer to divide the output channels of a convolutional layer. The layer takes a percentage `p` to determine the split ratio and an optional `dim` to specify the splitting dimension (defaulting to channel dimension). The example shows splitting the output of a Conv layer and returning only one part.

```python
import torchlayers as tl

class Net(tl.Module):
    def __init__(self):
        super().__init__()
        self.layer = tl.Conv(256, groups=16)
        self.splitter = tl.ChannelSplit(0.5)

    def forward(x):
        outputs = self.layer(x)
        half, rest = self.splitter(outputs)
        return half
```

--------------------------------

### ChannelShuffle Layer Example - PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Demonstrates the usage of the ChannelShuffle layer within a PyTorch Sequential model. This layer reshuffles output channels from a group convolution to improve knowledge transfer between layers. It requires the number of groups used in the preceding convolutional layer as input.

```python
import torchlayers as tl

model = tl.Sequential(
    tl.Conv(64),
    tl.Swish(),
    tl.Conv(128, groups=16),
    tl.ChannelShuffle(groups=16),
    tl.Conv(256),
    tl.GlobalMaxPool(),
    tl.Linear(10),
)
```

--------------------------------

### Inference Mode: All Layers Used

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Example of setting a model to inference mode using model.eval() and performing a forward pass with random input data. This is standard practice for consistent output during inference.

```python
model.eval()
output_eval = model(torch.randn(16, 3, 32, 32))
```

--------------------------------

### Build a Basic Classifier with torchlayers

Source: https://github.com/szymonmaszke/torchlayers/blob/master/README.md

Demonstrates how to define a simple convolutional neural network classifier using torchlayers. It showcases the automatic inference of input features for `tl.Linear` and `tl.Conv2d` layers when an example input is provided to `tl.build`. This approach simplifies layer definition by omitting the need to manually specify input dimensions.

```python
import torchlayers as tl
import torch


class Classifier(tl.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = tl.Conv2d(64, kernel_size=6)
        self.conv2 = tl.Conv2d(128, kernel_size=3)
        self.conv3 = tl.Conv2d(256, kernel_size=3, padding=1)
        # New layer, more on that in the next example
        self.pooling = tl.GlobalMaxPool()
        self.dense = tl.Linear(10)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        return self.dense(self.pooling(x))

# Pass model and any example inputs afterwards
clf = tl.build(Classifier(), torch.randn(1, 3, 32, 32))
```

--------------------------------

### Initialize Documentation UI Components with jQuery

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/genindex.html

jQuery ready function that initializes multiple UI components for the TorchLayers documentation page. Binds mobile menu, table of contents, PyTorch anchors, side menus, scroll anchors, and navigation highlighting. Additionally, adds CSS classes to links containing code blocks for styling purposes.

```JavaScript
$(document).ready(function() {
  mobileMenu.bind();
  mobileTOC.bind();
  pytorchAnchors.bind();
  sideMenus.bind();
  scrollToAnchor.bind();
  highlightNavigation.bind();
  // Add class to links that have code blocks, since we cannot create links in code blocks
  $("article.pytorch-article a span.pre").each(function(e) {
    $(this).closest("a").addClass("has-code");
  });
})
```

--------------------------------

### Initialize AvgPoolNd Layer (Python)

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/pooling.html

Initializes the AvgPoolNd layer with specified parameters for average pooling. It takes kernel_size, stride, padding, ceil_mode, and count_include_pad as arguments. The stride defaults to kernel_size if not provided. This layer is designed to work with PyTorch tensors.

```python
def __init__(
        self,
        kernel_size: int = 2,
        stride: int = None,
        padding: int = 0,
        ceil_mode: bool = False,
        count_include_pad: bool = True,
    ):
        super().__init__(
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            ceil_mode=ceil_mode,
            count_include_pad=count_include_pad,
        )
```

--------------------------------

### Instantiate and Build AutoEncoder Module

Source: https://github.com/szymonmaszke/torchlayers/blob/master/README.md

Demonstrates how to instantiate the AutoEncoder model and build it using a dummy input tensor. The `tl.build` function initializes the model, preparing it for training or inference. This is a common pattern for setting up PyTorch models with custom layers.

```python
autoencoder = tl.build(AutoEncoder(), torch.randn(1, 3, 256, 256))
```

--------------------------------

### MaxPool Class Documentation

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.pooling.html

Provides detailed information about the MaxPool class, including its initialization parameters, the types of pooling it performs based on input tensor shapes, and its return values.

```APIDOC
## Class torchlayers.pooling.MaxPool

### Description

Performs a max operation across the first dimension of a `torch.Tensor`. Depending on the shape of the input `torch.Tensor` (3D, 4D, or 5D, including batch), it automatically uses `torch.nn.MaxPool1D`, `torch.nn.MaxPool2D`, or `torch.nn.MaxPool3D` respectively.

### Parameters

#### Initialization Parameters

- **kernel_size** (_int_, optional) - The size of the window to take a max over. Default: `2`
- **stride** (_int_, optional) - The stride of the window. Default value is `kernel_size`.
- **padding** (_int_, optional) - Implicit zero padding to be added on both sides. Default: `0`
- **dilation** (_int_, optional) - Parameter controlling the stride of elements in the window. Default: `1`
- **return_indices** (_bool_, optional) - If `True`, will return the max indices along with the outputs. Useful for `torch.nn.MaxUnpool` later. Default: `False`
- **ceil_mode** (_bool_, optional) - When True, will use `ceil` instead of `floor` to compute the output shape. Default: `False`

### Returns

- **torch.Tensor**: A tensor with the same shape as the input, but with pooled values. The output shape will be `(batch, features)` for 2D output.

### Usage

Depending on the shape of the passed `torch.Tensor` (1D, 2D, or 3D), either 1D, 2D, or 3D pooling will be used for 3D, 4D, and 5D input shapes respectively (batch dimension included).

### Example

```python
import torch
import torchlayers

# Example with a 4D tensor (batch, channels, height, width)
input_tensor = torch.randn(2, 3, 10, 10)
max_pool_layer = torchlayers.pooling.MaxPool(kernel_size=2, stride=2)
output_tensor = max_pool_layer(input_tensor)
print(output_tensor.shape) # Expected: torch.Size([2, 3, 5, 5])

# Example with a 5D tensor (batch, channels, depth, height, width)
input_tensor_3d = torch.randn(2, 3, 5, 10, 10)
max_pool_layer_3d = torchlayers.pooling.MaxPool(kernel_size=2, stride=2)
output_tensor_3d = max_pool_layer_3d(input_tensor_3d)
print(output_tensor_3d.shape) # Expected: torch.Size([2, 3, 4, 5, 5])
```
```

--------------------------------

### Implement ResNet Residual Connections with torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Illustrates the use of torchlayers.Residual module for implementing ResNet-style skip connections. Examples include a basic residual block with group convolutions and a residual block with a projection layer to handle channel mismatches.

```python
import torch
import torchlayers as tl

# ResNet-style block with group convolution and channel shuffle
resnet_block = tl.Residual(
    tl.Sequential(
        tl.Conv(64, groups=16),  # Group convolution
        tl.ReLU(),
        tl.GroupNorm(num_groups=4),
        tl.Conv(64, groups=16),
        tl.ChannelShuffle(groups=16),  # Shuffle channels between groups
        tl.ReLU(),
    )
)

# With projection when channels change
resnet_projection = tl.Residual(
    tl.Sequential(
        tl.Conv(128),
        tl.ReLU(),
        tl.Conv(128),
    ),
    projection=tl.Conv(128, kernel_size=1),  # 1x1 conv to match channels
)

model = tl.Sequential(
    tl.Conv(64),
    resnet_block,
    resnet_projection,
    tl.GlobalAvgPool(),
    tl.Linear(10),
)

model = tl.build(model, torch.randn(1, 3, 32, 32))
```

--------------------------------

### Build PyTorch Module with Torchlayers

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers.html

The 'build' function compiles a PyTorch module, optimizing it for performance and removing overhead. It also supports a 'post_build' hook for actions like weight initialization. This function is intended to be used after creating a module with torchlayers and performing shape inference.

```python
import torch
import torchlayers as tl

# Example usage with a custom module having a post_build method
class _MyModuleImpl(torch.nn.Linear):
    def post_build(self):
        # You can do anything here really
        torch.nn.init.eye_(self.weights)

MyModule = tl.infer(_MyModuleImpl)

# Instantiate and build the module
module_instance = MyModule(in_features=10, out_features=5)
built_module = tl.build(module_instance)

```

--------------------------------

### ConvPixelShuffle Upsampling with ICNR Initialization in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.upsample.html

Implements a 2D convolution followed by PixelShuffle for learnable upsampling. It utilizes ICNR weight initialization to provide a starting point similar to nearest neighbor upsampling. This class currently supports 4D input tensors (batch, channels, height, width) due to limitations in PyTorch's PixelShuffle implementation.

```python
import torch
import torch.nn as nn
from typing import Tuple, Union, Callable

class ConvPixelShuffle(nn.Module):
    """Two dimensional convolution with ICNR initialization followed by PixelShuffle.

    Increases `height` and `width` of `input` tensor by scale, acts like learnable upsampling.
    Due to ICNR weight initialization of `convolution` it has similar starting point to
    nearest neighbour upsampling.

    `kernel_size` got a default value of `3`, `upscale_factor` got a default value of `2`

    Note:
        Currently only `4D` input is allowed (`[batch, channels, height, width]`),
        due to [`torch.nn.PixelShuffle`](https://pytorch.org/docs/stable/nn.html#torch.nn.PixelShuffle)
        not supporting `1D` or `3D` versions. See [this PyTorch PR](https://github.com/pytorch/pytorch/pull/6340/files)
        for example of dimension-agnostic implementation.

    Args:
        in_channels (int): Number of channels in the input image
        out_channels (int): Number of channels produced after PixelShuffle
        upscale_factor (int, optional): Factor to increase spatial resolution by. Default: `2`
        kernel_size (int or tuple, optional): Size of the convolving kernel. Default: `3`
        stride (int or tuple, optional): Stride of the convolution. Default: 1
        padding (int or tuple, optional): Zero-padding added to both sides of the input. Default: 0
        padding_mode (string, optional): Accepted values `zeros` and `circular` Default: `zeros`
        dilation (int or tuple, optional): Spacing between kernel elements. Default: 1
        groups (int, optional): Number of blocked connections from input channels to output channels. Default: 1
        bias (bool, optional): If `True`, adds a learnable bias to the output. Default: `True`
        initializer (Callable[[torch.Tensor], torch.Tensor], optional): Custom weight initializer.
    """
    def __init__(self,
                 in_channels: int,
                 out_channels: int,
                 upscale_factor: int = 2,
                 kernel_size: int = 3,
                 stride: int = 1,
                 padding: Union[Tuple[int, int], int, str] = 'same',
                 padding_mode: str = 'zeros',
                 dilation: int = 1,
                 groups: int = 1,
                 bias: bool = True,
                 initializer: Callable[[torch.Tensor], torch.Tensor] = None):
        super().__init__()

        self.upscale_factor = upscale_factor
        self.out_channels = out_channels

        # Correct number of output channels for PixelShuffle
        # PixelShuffle rearranges channels, so the convolution's output channels
        # must be a multiple of upscale_factor^2.
        if out_channels % (upscale_factor ** 2) != 0:
            raise ValueError(f"out_channels must be divisible by upscale_factor squared ({upscale_factor**2})")

        self.conv = nn.Conv2d(
            in_channels, out_channels, kernel_size, stride, padding, dilation, groups, bias, padding_mode
        )

        # Apply ICNR initialization if no custom initializer is provided
        if initializer is None:
            self.init_weights_icnr()
        else:
            # Use the provided custom initializer
            nn.init.apply_(self.conv.weight, initializer)
            if self.conv.bias is not None:
                nn.init.zeros_(self.conv.bias)

        self.pixel_shuffle = nn.PixelShuffle(upscale_factor)

    def init_weights_icnr(self):
        """Initialize weights using the ICNR method."""
        # ICNR initialization for the convolution weights
        # Based on the paper: https://arxiv.org/abs/1707.02937
        ni = self.conv.in_channels
        nf = self.conv.out_channels
        ks = self.conv.kernel_size[0] if isinstance(self.conv.kernel_size, tuple) else self.conv.kernel_size
        factor = self.upscale_factor
        subpixel_kernel_size = ks + (ks - 1) * (factor - 1)

        cnn_weight = self.conv.weight.data
        act = (nn.LeakyReLU(0.2, inplace=True))(torch.zeros(1, ni, ks, ks))
        output_dim = nf * (ks**2)
        # Subpixel weight initialization
        weight = torch.zeros(nf // (factor**2), ni, ks, ks, device=cnn_weight.device)
        nn.init.kaiming_uniform_(weight, a=0.2)

        weight = weight.view(nf // (factor**2), ni, ks, ks)
        up_kernel = weight.repeat(1, 1, factor, factor)
        up_kernel = up_kernel.view(nf // (factor**2), ni, ks, factor, ks, factor)
        up_kernel = up_kernel.permute(0, 1, 2, 4, 3, 5).contiguous()
        up_kernel = up_kernel.view(nf // (factor**2), ni, ks, ks * factor * factor)
        up_kernel = up_kernel.view(nf // (factor**2), ni, ks * factor, ks * factor)
        up_kernel = up_kernel.repeat(factor, 1, 1, 1)
        up_kernel = up_kernel.view(nf, ni, ks, ks * (factor**2))
        up_kernel = up_kernel.permute(0, 1, 3, 2).contiguous()
        up_kernel = up_kernel.view(nf, ni, factor * ks, factor * ks)

        self.conv.weight.data.copy_(up_kernel)

        # Bias initialization
        if self.conv.bias is not None:
            torch.nn.init.zeros_(self.conv.bias.data)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Forward pass of the ConvPixelShuffle module.

        Args:
            x (torch.Tensor): Input tensor with shape [batch, channels, height, width].

        Returns:
            torch.Tensor: Output tensor after convolution and PixelShuffle.
        """
        if x.dim() != 4:
            raise ValueError(f"Input tensor must be 4D, but got {x.dim()}D")

        x = self.conv(x)
        x = self.pixel_shuffle(x)
        return x


```

--------------------------------

### Separable Convolution Layer Initialization (Python)

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

Initializes a SeparableConv layer, which performs depthwise and pointwise convolutions sequentially. It preserves all input dimensions except channels and supports various kernel sizes, strides, and padding modes. Note that the 'same' padding mode is restricted to odd values for kernel_size, dilation, and stride.

```python
import typing

class SeparableConv(_dev_utils.modules.Representation):
    """Separable convolution layer.

    This mode preserves all dimensions except channels.

    `kernel_size` got a default value of `3`.

    .. note::
                **IMPORTANT**: `same` currently works only for odd values of `kernel_size`,
                `dilation` and `stride`. If any of those is even you should explicitly pad
                your input asymmetrically with `torch.functional.pad` or a-like.

    Parameters
    ----------
    in_channels : int
        Number of channels in the input image
    out_channels : int
        Number of channels produced by the convolution
    kernel_size : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Size of the convolving kernel. User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `3`
    stride : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Stride of the convolution. User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `3`
    padding : Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional
        Padding added to both sides of the input. String "same" can be used with odd
        `kernel_size`, `stride` and `dilation`
        User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `same`
    dilation : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Spacing between kernel elements. String "same" can be used with odd
        `kernel_size`, `stride` and `dilation`
        User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `1`
    bias : bool, optional
        If ```True```, adds a learnable bias to the output. Default: ```True``
    padding_mode : string, optional
        Accepted values `zeros` and `circular` Default: `zeros`

    """

    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        kernel_size=3,
        stride=1,
        padding="same",
        dilation=1,
        bias: bool = True,
        padding_mode: str = "zeros",
    ):
        super().__init__()

        self.in_channels: int = in_channels
        self.out_channels: int = out_channels
        self.kernel_size: typing.Union[
            int, typing.Tuple[int, int], typing.Tuple[int, int, int]
        ] = kernel_size
        self.stride: typing.Union[
            int, typing.Tuple[int, int], typing.Tuple[int, int, int]
        ] = stride
        self.padding: typing.Union[
            str, int, typing.Tuple[int, int], typing.Tuple[int, int, int]
        ] = padding
        self.dilation: typing.Union[
            int, typing.Tuple[int, int], typing.Tuple[int, int, int]
        ] = dilation
        self.bias: bool = bias
        self.padding_mode: str = padding_mode

        self.depthwise = Conv(
            in_channels=in_channels,
            out_channels=in_channels,
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            dilation=dilation,
            groups=in_channels,
            bias=bias,
            padding_mode=padding_mode,
        )

        self.pointwise = Conv(
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=1,
            stride=1,
            padding=0,
            dilation=1,
            groups=1,
            bias=False,
            padding_mode=padding_mode,
        )

```

--------------------------------

### Initialize Mobile and PyTorch Functionality

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/regularization.html

Initializes various JavaScript functionalities for the torchlayers documentation website upon document ready. This includes binding mobile menus, table of contents, PyTorch anchors, side menus, scroll-to-anchor behavior, and navigation highlighting. It also adds a 'has-code' class to links containing code blocks for styling purposes.

```javascript
$(document).ready(function() {
  mobileMenu.bind();
  mobileTOC.bind();
  pytorchAnchors.bind();
  sideMenus.bind();
  scrollToAnchor.bind();
  highlightNavigation.bind();

  // Add class to links that have code blocks, since we cannot create links in code blocks
  $("article.pytorch-article a span.pre").each(function(e) {
    $(this).closest("a").addClass("has-code");
  });
})
```

--------------------------------

### TorchScript Compilation for Production Deployment with Torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Demonstrates how to export a torchlayers model to TorchScript for optimized C++ deployment. This involves defining a model, scripting it using `torch.jit.script`, saving it, and then loading it for inference. Dependencies include torch and torchlayers.

```python
import torch
import torchlayers as tl

# Define and build model
model = tl.Sequential(
    tl.Conv(64),
    tl.ReLU(),
    tl.MaxPool(),
    tl.Conv(128),
    tl.ReLU(),
    tl.GlobalAvgPool(),
    tl.Linear(10),
)

model = tl.build(model, torch.randn(1, 3, 32, 32))

# Script and save for production
scripted_model = torch.jit.script(model)
torch.jit.save(scripted_model, "production_model.pt")

# Load and use in production
loaded_model = torch.jit.load("production_model.pt")
loaded_model.eval()

# Inference
with torch.no_grad():
    output = loaded_model(torch.randn(16, 3, 32, 32))
    predictions = output.argmax(dim=1)
    print(f"Predictions: {predictions}")
```

--------------------------------

### Initialize Conv Layer with Parameters

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

This Python code initializes a convolutional layer (Conv) within the torchlayers library. It takes various arguments to define the layer's behavior, including input/output channels, kernel size, stride, padding, dilation, groups, bias, and padding mode. The 'same' padding option is noted to have specific requirements.

```python
class Conv(_Conv):
    """Standard convolution layer.

    Based on input shape it either creates 1D, 2D or 3D convolution (for inputs of shape
    3D, 4D, 5D including batch as first dimension).

    Otherwise acts exactly like PyTorch's Convolution, see
    `documentation <https://pytorch.org/docs/stable/nn.html#convolution-layers>`__.

    Default argument for `kernel_size` was added equal to `3`.

    .. note::
                **IMPORTANT**: `same` currently works only for odd values of `kernel_size`,
                `dilation` and `stride`. If any of those is even you should explicitly pad
                your input asymmetrically with `torch.functional.pad` or a-like.

    Parameters
    ----------
    in_channels : int
        Number of channels in the input image
    out_channels : int
        Number of channels produced by the convolution
    kernel_size : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Size of the convolving kernel. User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `3`
    stride : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Stride of the convolution. User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `3`
    padding : Union[str, int, Tuple[int, int], Tuple[int, int, int]], optional
        Padding added to both sides of the input. String "same" can be used with odd
        `kernel_size`, `stride` and `dilation`
        User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `same`
    dilation : Union[int, Tuple[int, int], Tuple[int, int, int]], optional
        Spacing between kernel elements. String "same" can be used with odd
        `kernel_size`, `stride` and `dilation`
        User can specify `int` or 2-tuple (for `Conv2d`)
        or 3-tuple (for `Conv3d`). Default: `1`
    groups : int, optional
        Number of blocked connections from input channels to output channels. Default: 1
    bias : bool, optional
        If ```True```, adds a learnable bias to the output. Default: ```True``
    padding_mode : string, optional
        Accepted values `zeros` and `circular` Default: `zeros`

    """

    def __init__(
        self,
        in_channels: int,
        out_channels: int,
        kernel_size=3,
        stride=1,
        padding="same",
        dilation=1,
        groups: int = 1,
        bias: bool = True,
        padding_mode: str = "zeros",
    ):
        super().__init__(
            module_name="Conv",
            in_channels=in_channels,
            out_channels=out_channels,
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            dilation=dilation,
            groups=groups,
            bias=bias,
            padding_mode=padding_mode,
        )

```

--------------------------------

### Model Summary: Keras-Style Architecture Visualization

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Provides a way to visualize the model architecture in a Keras-style format, showing layer types, input/output shapes, parameter counts, and buffer information. The summary can be generated before and after building the model, with options to customize displayed columns.

```python
import torch
import torchlayers as tl

model = tl.Sequential(
    tl.Conv(64),
    tl.ReLU(),
    tl.BatchNorm(),
    tl.MaxPool(),
    tl.Conv(128),
    tl.ReLU(),
    tl.Dropout(),
    tl.GlobalMaxPool(),
    tl.Linear(10),
)

# Before build - shows ? for uninfered dimensions
print(model)

# Build and get detailed summary
model = tl.build(model, torch.randn(1, 3, 32, 32))
summary = tl.summary(model, torch.randn(1, 3, 32, 32))

# Print full summary with all columns
print(summary)

# Print custom summary (without inputs and buffers columns)
print(summary.string(inputs=False, buffers=False))

# Available columns: layers, inputs, outputs, params, buffers
print(summary.string(layers=True, outputs=True, params=True))
```

--------------------------------

### Utilize PolyNet and MPoly Modules with torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Showcases the Poly and MPoly modules from torchlayers for implementing PolyNet-style architectures. Poly applies a single module multiple times, while MPoly applies different modules sequentially, both with identity connections. WayPoly applies modules in parallel.

```python
import torch
import torchlayers as tl

# Poly: Output = I + F(I) + F(F(I)) + F(F(F(I)))
poly_model = tl.Sequential(
    tl.Conv(64),
    tl.ReLU(),
    tl.Poly(tl.InvertedResidualBottleneck(), order=3),  # Apply 3 times
    tl.MaxPool(),
    tl.Poly(tl.Conv(128), order=2),
    tl.GlobalAvgPool(),
    tl.Linear(10),
)

# MPoly: Different modules in sequence
# Output = I + F1(I) + F2(F1(I)) + F3(F2(F1(I)))
mpoly_model = tl.MPoly(
    tl.Conv(64),
    tl.Conv(64),
    tl.Conv(64),
)

# WayPoly: Different modules in parallel
# Output = I + F1(I) + F2(I) + F3(I)
waypoly_model = tl.WayPoly(
    tl.Fire(128),
    tl.Fire(128),
    tl.Fire(128),
)

model = tl.build(poly_model, torch.randn(1, 3, 32, 32))
```

--------------------------------

### InvertedResidualBottleneck Initialization in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

Initializes the InvertedResidualBottleneck module, a versatile block inspired by MobileNetV2 and extended with Squeeze-Excitation. It handles channel expansion, depthwise convolution, activation functions, batch normalization, and optional Squeeze-Excitation blocks. Parameters allow customization of hidden channels, activation types, and Squeeze-Excitation configurations. Dependencies include torch, normalization, and typing.

```python
def __init__(
        self,
        in_channels: int,
        hidden_channels: int = None,
        activation=None,
        batchnorm: bool = True,
        squeeze_excitation: bool = True,
        squeeze_excitation_hidden: int = None,
        squeeze_excitation_activation=None,
        squeeze_excitation_sigmoid=None,
    ):
        def _add_batchnorm(block, channels):
            if batchnorm:
                block.append(normalization.BatchNorm(channels))
            return block

        super().__init__()

        # Argument assignments
        self.in_channels: int = in_channels
        self.hidden_channels: int = hidden_channels if hidden_channels is not None else in_channels * 4
        self.activation: typing.Callable[
            [torch.Tensor], torch.Tensor
        ] = torch.nn.ReLU6() if activation is None else activation
        self.batchnorm: bool = batchnorm
        self.squeeze_excitation: bool = squeeze_excitation
        self.squeeze_excitation_hidden: int = squeeze_excitation_hidden
        self.squeeze_excitation_activation: typing.Callable[
            [torch.Tensor], torch.Tensor
        ] = squeeze_excitation_activation
        self.squeeze_excitation_sigmoid: typing.Callable[
            [torch.Tensor], torch.Tensor
        ] = squeeze_excitation_sigmoid

        # Initial expanding block
        initial = torch.nn.Sequential(
```

--------------------------------

### ConvTranspose Layer Initialization (Python)

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Initializes a ConvTranspose layer. This layer supports learnable bias and different padding modes. It requires the number of input and output channels, and optionally accepts kernel size, stride, padding, output padding, dilation, groups, bias, and padding mode.

```python
from torchlayers.convolution import ConvTranspose

# Example initialization for a 2D transposed convolution
conv_transpose_2d = ConvTranspose(
    in_channels=64,
    out_channels=128,
    kernel_size=3,
    stride=2,
    padding=1,
    bias=True,
    padding_mode='zeros'
)
```

--------------------------------

### Implement Squeeze-and-Excitation Channel Attention with torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Demonstrates the implementation of a residual block incorporating the Squeeze-and-Excitation (SE) channel attention mechanism using torchlayers. It shows how to create a SE-aware convolutional block and integrate it into an SE-ResNet style model.

```python
import torch
import torchlayers as tl

# Residual block with SE attention
se_block = tl.Sequential(
    tl.Conv(64),
    tl.ReLU(),
    tl.Conv(64),
    tl.SqueezeExcitation(hidden=16),  # Squeeze to 16, excite back to 64
    tl.ReLU(),
)

# SE-ResNet style
model = tl.Sequential(
    tl.Conv(64),
    tl.Residual(se_block),
    tl.MaxPool(),
    tl.GlobalAvgPool(),
    tl.Linear(10),
)

model = tl.build(model, torch.randn(1, 3, 32, 32))
```

--------------------------------

### ChannelSplit class definition and initialization in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

Defines a convenience layer that splits input tensors along a specified dimension using a percentage parameter. The __init__ method validates that the split ratio p is between 0 and 1, stores the ratio and dimension parameters for later use in forward pass.

```python
class ChannelSplit(_dev_utils.modules.Representation):
    def __init__(self, p: float, dim: int = 1):
        super().__init__()
        if not 0.0 < p < 1.0:
            raise ValueError(
                "Ratio of small expand fire module has to be between 0 and 1."
            )
        self.p: float = p
        self.dim: int = dim
```

--------------------------------

### ConvPixelShuffle post_build weight initialization

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/upsample.html

Initializes the weights of the convolution layer after the module is built using ICNR initialization. This method is called after the layer construction to apply the ICNR weight initialization technique to the convolution weights for artifact-free sub-pixel convolution.

```python
def post_build(self):
    """Initialize weights after layer was built."""
    self.icnr_initialization(self.convolution.weight.data)
```

--------------------------------

### Swish and HardSwish Activation Functions in Torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Demonstrates the usage of Swish and HardSwish activation functions within a torchlayers Sequential model. It also shows their functional forms for direct tensor operations. Dependencies include torch and torchlayers.

```python
import torch
import torchlayers as tl

# Swish: x * sigmoid(beta * x)
swish_model = tl.Sequential(
    tl.Conv(64),
    tl.Swish(beta=1.0),  # Default beta=1.0
    tl.Conv(128),
    tl.Swish(),
)

# HardSwish: efficient approximation used in MobileNetV3
hardswish_model = tl.Sequential(
    tl.Conv(32),
    tl.HardSwish(),
    tl.InvertedResidualBottleneck(32, activation=tl.HardSwish()),
    tl.Conv(64),
    tl.HardSwish(),
)

# Can also use functional forms
x = torch.randn(16, 64, 32, 32)
y = tl.swish(x, beta=1.5)
z = tl.hard_swish(x)

model = tl.build(hardswish_model, torch.randn(1, 3, 224, 224))
```

--------------------------------

### Apply Swish activation with configurable beta in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.activations.html

Applies the Swish activation function element-wise with an optional beta parameter that multiplies the sigmoid component. Default beta is 1.0 for standard Swish behavior. Accepts a torch.Tensor and returns the activated tensor.

```python
import torch
import torchlayers

tensor = torch.randn(3, 4)
activated = torchlayers.activations.swish(tensor, beta=1.0)
```

--------------------------------

### SqueezeExcitation Class Definition

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Creates a SqueezeExcitation module that learns channel-wise excitation maps. Takes in_channels (required), hidden layer size (optional, defaults to 1/16 of in_channels), and activation function (optional, defaults to ReLU). The module performs channel attention by squeezing, exciting, and rescaling input features.

```python
class torchlayers.convolution.SqueezeExcitation(
    in_channels: int,
    hidden: int = None,
    activation=None,
    sigmoid=None
)
```

--------------------------------

### Build SqueezeNet Fire Module Architecture with torchlayers

Source: https://context7.com/szymonmaszke/torchlayers/llms.txt

Constructs a SqueezeNet-inspired architecture utilizing the Fire module, which features an efficient squeeze-expand design. The code shows how to stack multiple Fire modules with varying channel configurations and apply pooling and linear layers to form a complete model.

```python
import torch
import torchlayers as tl

# SqueezeNet-inspired architecture
squeezenet = tl.Sequential(
    tl.Conv(96, kernel_size=7, stride=2),
    tl.ReLU(),
    tl.MaxPool(kernel_size=3, stride=2),
    # Fire module: squeeze to hidden_channels, expand to out_channels
    tl.Fire(out_channels=128, hidden_channels=16, p=0.5),  # p=0.5: half 1x1, half 3x3
    tl.Fire(out_channels=128, hidden_channels=16),
    tl.Fire(out_channels=256, hidden_channels=32),
    tl.MaxPool(kernel_size=3, stride=2),
    tl.Fire(out_channels=256, hidden_channels=32),
    tl.Fire(out_channels=384, hidden_channels=48),
    tl.Fire(out_channels=384, hidden_channels=48),
    tl.Fire(out_channels=512, hidden_channels=64),
    tl.MaxPool(kernel_size=3, stride=2),
    tl.Fire(out_channels=512, hidden_channels=64),
    tl.GlobalAvgPool(),
    tl.Linear(1000),
)

model = tl.build(squeezenet, torch.randn(1, 3, 224, 224))
```

--------------------------------

### Channel Shuffle Layer Initialization (Python)

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

Initializes a ChannelShuffle layer, designed to improve knowledge transfer in grouped convolutions by reshuffling output channels. This layer requires the number of groups used in the preceding convolutional layer as an argument.

```python
class ChannelShuffle(_dev_utils.modules.Representation):
    """Shuffle output channels.

    When using group convolution knowledge transfer between next layers is reduced
    (as the same input channels are convolved with the same output channels).

    This layer reshuffles output channels via simple `reshape` in order to mix the representation
    from separate groups and improve knowledge transfer.

    Originally proposed by Xiangyu Zhang et. al in:
    `ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices <https://arxiv.org/abs/1707.01083>`__

    Example::

        import torchlayers as tl

        model = tl.Sequential(
            tl.Conv(64),
            tl.Swish(),
            tl.Conv(128, groups=16),
            tl.ChannelShuffle(groups=16),
            tl.Conv(256),
            tl.GlobalMaxPool(),
            tl.Linear(10),
        )

    Parameters
    ----------
    groups : int
        Number of groups used in the previous convolutional layer.

    """

    def __init__(self, groups: int):
        super().__init__()

```

--------------------------------

### MPoly Class - Apply Multiple Modules Sequentially with Summation PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

PyTorch module that applies multiple modules sequentially and sums all intermediate outputs with identity. Implements the equation I + F_1 + F_1(F_0) + ... + F_N(F_{N-1}...F_0). Each module output becomes input to the next, creating a chain of polynomial operations originally from PolyNet.

```python
class MPoly(torch.nn.Module):
    def __init__(self, *poly_modules: torch.nn.Module):
        super().__init__()
        self.poly_modules: torch.nn.Module = torch.nn.ModuleList(poly_modules)

    def forward(self, inputs):
        outputs = [self.poly_modules[0](inputs)]
        for module in self.poly_modules[1:]:
            outputs.append(module(outputs[-1]))
        return torch.stack([inputs] + outputs, dim=0).sum(dim=0)
```

--------------------------------

### Dense connection class initialization in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/_modules/torchlayers/convolution.html

Initializes a Dense module that implements dense connections by concatenating input and output channels instead of adding. Accepts a module and dimension parameter for channel concatenation, used in DenseNet-style architectures.

```python
class Dense(torch.nn.Module):
    def __init__(self, module: torch.nn.Module, dim: int = 1):
        super().__init__()
```

--------------------------------

### Expand Poly layer computation manually in PyTorch

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Demonstrates the manual expansion of what a Poly layer with order=3 computes internally. Shows how the module applies a shared convolution layer sequentially, collecting outputs at each level, and then sums the original output with all intermediate results to produce the final output.

```python
import torchlayers as tl
import torch

x = torch.randn(1, 3, 32, 32)

first_convolution = tl.Conv(64)
output = first_convolution(x)

shared_convolution = tl.Conv(64)
first_level = shared_convolution(output)
second_level = shared_convolution(first_level)
third_level = shared_convolution(second_level)

# That's what tl.Poly would return
final = output + first_level + second_level + third_level
```

--------------------------------

### SqueezeExcitation Module Usage with Residual Block

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

Demonstrates how to use the SqueezeExcitation class within a Residual block for channel-wise attention. SqueezeExcitation learns excitation maps by squeezing input channels via average pooling, passing through non-linear layers, and rescaling via sigmoid before multiplying with the original input. This implements the Squeeze-and-Excitation Networks architecture.

```python
import torchlayers as tl

# Assume only 128 channels can be an input in this case
block = tl.Residual(tl.Conv(128), tl.SqueezeExcitation(), tl.Conv(128))
```

--------------------------------

### Implement ResNet-like Block using torchlayers.Residual

Source: https://github.com/szymonmaszke/torchlayers/blob/master/docs/packages/torchlayers.convolution.html

This snippet demonstrates how to create a ResNet-like block using the `torchlayers.Residual` module. It defines a sequential model within the residual connection and utilizes `torchlayers.infer` for module inference. The block expects input channels and applies a series of convolutional and ReLU layers.

```python
import torch
import torchlayers as tl

# ResNet-like block
class _BlockImpl(tl.Module):
    def __init__(self, in_channels: int):
        self.block = tl.Residual(
            tl.Sequential(
                tl.Conv(in_channels),
                tl.ReLU(),
                tl.Conv(4 * in_channels),
                tl.ReLU(),
                tl.Conv(in_channels),
            )
        )

    def forward(self, x):
        return self.block(x)

Block = tl.infer(_BlockImpl)
```