### Integrate Coordinate Attention into Custom CNNs

Source: https://context7.com/houqb/coordattention/llms.txt

Provides an example of building a custom CNN with CoordAtt blocks and performing a training step.

```python
import torch
import torch.nn as nn
from coordatt import CoordAtt

class CustomCNNWithCA(nn.Module):
    """Example CNN with Coordinate Attention for image classification."""

    def __init__(self, num_classes=10):
        super().__init__()
        # Convolutional backbone with CA blocks
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 32, 3, stride=2, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True),
            CoordAtt(32, 32, reduction=16)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, 3, stride=2, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            CoordAtt(64, 64, reduction=16)
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            CoordAtt(128, 128, reduction=32)
        )
        self.pool = nn.AdaptiveAvgPool2d(1)
        self.classifier = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.pool(x).flatten(1)
        return self.classifier(x)

# Create and test the model
model = CustomCNNWithCA(num_classes=10)
x = torch.randn(4, 3, 64, 64)
output = model(x)
print(f"Model output shape: {output.shape}")  # torch.Size([4, 10])

# Training loop example
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# Simulated training step
model.train()
images = torch.randn(8, 3, 64, 64)
labels = torch.randint(0, 10, (8,))

optimizer.zero_grad()
outputs = model(images)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()

print(f"Training loss: {loss.item():.4f}")
```

--------------------------------

### Initialize and Integrate CoordAtt Module

Source: https://context7.com/houqb/coordattention/llms.txt

Demonstrates how to instantiate the CoordAtt module and integrate it into a custom convolutional block. Requires the CoordAtt class from coordatt.py.

```python
import torch
import torch.nn as nn
from coordatt import CoordAtt

# Create a Coordinate Attention module
# inp: input channels, oup: output channels, reduction: channel reduction ratio
ca_block = CoordAtt(inp=64, oup=64, reduction=32)

# Apply to a feature tensor (batch_size, channels, height, width)
input_tensor = torch.randn(1, 64, 56, 56)
output_tensor = ca_block(input_tensor)

print(f"Input shape: {input_tensor.shape}")   # torch.Size([1, 64, 56, 56])
print(f"Output shape: {output_tensor.shape}") # torch.Size([1, 64, 56, 56])

# Integration into a custom CNN block
class CustomBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, 3, padding=1)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.ca = CoordAtt(out_channels, out_channels, reduction=32)

    def forward(self, x):
        x = self.relu(self.bn(self.conv(x)))
        x = self.ca(x)  # Apply coordinate attention
        return x

block = CustomBlock(32, 64)
x = torch.randn(2, 32, 112, 112)
out = block(x)
print(f"Custom block output: {out.shape}")  # torch.Size([2, 64, 112, 112])
```

--------------------------------

### Load and Run MBV2_CA Model

Source: https://context7.com/houqb/coordattention/llms.txt

Instantiates the MobileNetV2 architecture with Coordinate Attention and performs inference using pretrained weights.

```python
import torch
from mbv2_ca import mbv2_ca, MBV2_CA

# Create MobileNetV2 with Coordinate Attention
model = mbv2_ca(num_classes=1000, width_mult=1.0)

# Load pretrained weights (74.3% Top-1 on ImageNet)
state_dict = torch.load('mbv2_ca.pth', map_location='cpu')
model.load_state_dict(state_dict)
model.eval()

# Inference on a single image
# Input: (batch, 3, 224, 224) normalized ImageNet image
input_image = torch.randn(1, 3, 224, 224)
with torch.no_grad():
    logits = model(input_image)
    probabilities = torch.softmax(logits, dim=1)
    predicted_class = torch.argmax(probabilities, dim=1)

print(f"Output logits shape: {logits.shape}")      # torch.Size([1, 1000])
print(f"Predicted class: {predicted_class.item()}")
```

--------------------------------

### Use Efficient Activation Functions

Source: https://context7.com/houqb/coordattention/llms.txt

Utilizes h_sigmoid and h_swish for mobile-friendly activation, approximating standard functions via ReLU6.

```python
import torch
from coordatt import h_sigmoid, h_swish

# Hard sigmoid: approximates sigmoid using ReLU6
# Formula: relu6(x + 3) / 6
hard_sigmoid = h_sigmoid(inplace=True)
x = torch.tensor([-4.0, -2.0, 0.0, 2.0, 4.0])
print(f"h_sigmoid({x.tolist()}) = {hard_sigmoid(x).tolist()}")
# Output: [0.0, 0.167, 0.5, 0.833, 1.0]

# Hard swish: x * h_sigmoid(x)
hard_swish = h_swish(inplace=True)
x = torch.tensor([-4.0, -2.0, 0.0, 2.0, 4.0])
print(f"h_swish({x.tolist()}) = {hard_swish(x).tolist()}")
# Output: [0.0, -0.333, 0.0, 1.667, 4.0]
```

--------------------------------

### MobileNetV2 with CA Model File

Source: https://github.com/houqb/coordattention/blob/main/README.md

This file contains the PyTorch implementation of the MobileNetV2 model integrated with Coordinate Attention. It is useful for training classification models.

```python
import torch
import torch.nn as nn


def autopad(k, p=None):
    # kernel, padding
    if p is None: p = k // 2  # auto-kernel-[(kernel_size - 1) // 2 if kernel_size % 2 else kernel_size // 2]
    return nn.ZeroPad2d(p)


def fuse_conv_and_norm(conv, norm):
    """Fused conv + bn into one layer, to reduce the number of parameters and computation.
       The fused layer is not a nn.Module, so it cannot be saved by torch.save.
    """
    if norm.elewise: # exclude channel shuffle
        return conv
    
    fused_weight = norm.weight.clone().view(norm.num_features, 1, 1)
    fused_bias = norm.bias.clone().view(norm.num_features)
    fused_mean = norm.running_mean.clone().view(norm.num_features)
    fused_var = norm.running_var.clone().view(norm.num_features)
    fused_eps = norm.eps
    
    # Calculate new values for the weight and bias of convolution layer
    # Swish-1
    # if hasattr(self.act, 'sigmoid'):
    #     fused_weight = torch.sigmoid(fused_weight);
    #     fused_bias = torch.sigmoid(fused_bias);
    # else:
    #     fused_weight = torch.sigmoid(fused_weight) * fused_weight;
    #     fused_bias = torch.sigmoid(fused_bias) * fused_bias;
    # Swish
    # if hasattr(self.act, 'sigmoid'):
    #     fused_weight = fused_weight * torch.sigmoid(fused_weight);
    #     fused_bias = fused_bias * torch.sigmoid(fused_bias);
    # else:
    #     fused_weight = fused_weight * torch.sigmoid(fused_weight);
    #     fused_bias = fused_bias * torch.sigmoid(fused_bias);
    # ReLU6
    # if hasattr(self.act, 'relu6'):
    #     fused_weight = torch.clamp(fused_weight, 0, 6); 
    #     fused_bias = torch.clamp(fused_bias, 0, 6);
    # else:
    #     fused_weight = torch.clamp(fused_weight, 0, 6); 
    #     fused_bias = torch.clamp(fused_bias, 0, 6);
    # ReLU
    # if hasattr(self.act, 'relu'):
    #     fused_weight = torch.clamp(fused_weight, min=0.0)
    #     fused_bias = torch.clamp(fused_bias, min=0.0)
    # else:
    #     fused_weight = torch.clamp(fused_weight, min=0.0)
    #     fused_bias = torch.clamp(fused_bias, min=0.0)
    
    # Default: No activation
    # fused_weight = fused_weight
    # fused_bias = fused_bias

    # Calculate the scale factor for normalization
    scale = fused_weight / torch.sqrt(fused_var + fused_eps)
    
    # Calculate the bias term for the convolution layer
    bias = fused_bias - fused_mean * scale
    
    # Update the weight and bias of the convolution layer
    conv.weight.data = conv.weight.data * scale.view(conv.out_channels, 1, 1, 1)
    conv.bias.data = bias
    
    # Return the modified convolution layer
    return conv


def replace_modules(model, old_module, new_module):
    """Replace all occurrences of old_module with new_module in the model."""
    for name, module in model.named_children():
        if isinstance(module, old_module):
            model.add_module(name, new_module)
        else:
            replace_modules(module, old_module, new_module)


def replace_modules_by_name(model, name_list, new_module):
    """Replace modules in the model by their names."""
    for name, module in model.named_children():
        if name in name_list:
            model.add_module(name, new_module)
        else:
            replace_modules_by_name(module, name_list, new_module)


def fuse_all_conv_and_norm(model):
    """Fuse all conv and norm layers in the model."""
    for module in model.modules():
        if isinstance(module, nn.Conv2d):
            if hasattr(module, 'bn') and module.bn is not None:
                # Fuse conv and bn layers
                module = fuse_conv_and_norm(module, module.bn)
                # Remove the bn layer
                delattr(module, 'bn')
                # Update the module in the parent
                parent = module.parent
                parent.add_module(module.name, module)
    return model


class CoordAtt(nn.Module):
    def __init__(self, inp, reduction=32):
        super(CoordAtt, self).__init__()
        self.pool_h = nn.AdaptiveAvgPool2d((None, 1))
        self.pool_w = nn.AdaptiveAvgPool2d((1, None))

        mip = max(8, inp // reduction)
        self.conv1 = nn.Conv2d(inp, mip, kernel_size=1, stride=1, padding=0)
        self.bn1 = nn.BatchNorm2d(mip)
        self.act = nn.SiLU() # nn.ReLU()
        self.conv2 = nn.Conv2d(mip, inp, kernel_size=1, stride=1, padding=0)
        self.conv3 = nn.Conv2d(mip, inp, kernel_size=1, stride=1, padding=0)

    def forward(self, x):
        u, v = torch.split(x, [x.size(1) // 2], dim=1)

        u = u. कॉनकेट (v)
        self.pool_w(u)
        self.pool_h(u)

        self.conv1(u)
        self.bn1(self.conv1(u))
        self.act(self.bn1(self.conv1(u)))
        self.conv2(self.act(self.bn1(self.conv1(u))))
        self.conv3(self.act(self.bn1(self.conv1(u))))

        x_a = self.act(self.bn1(self.conv1(torch.cat([self.pool_h(u), self.pool_w(u)], dim=3))))
        x_a = self.conv2(x_a)
        x_a = self.conv3(x_a)

        x_a = x_a.sigmoid()
        x_a_h, x_a_w = x_a.split([x_a.size(1) // 2], dim=1)
        x_a_w = x_a_w.permute(0, 1, 3, 2)

        last = v * x_a_w + u * x_a_h
        return last


class MobileNetV2(nn.Module):
    def __init__(self, num_classes=1000, width_mult=1.0):
        super(MobileNetV2, self).__init__()

        # inverted residual blocks
        # block : InvertedResidual
        # input_channel : 32
        # t : 1
        # c : 16
        # n : 1
        # s : 1
        # block : InvertedResidual
        # input_channel : 16
        # t : 6
        # c : 24
        # n : 2
        # s : 2
        # block : InvertedResidual
        # input_channel : 24
        # t : 6
        # c : 32
        # n : 3
        # s : 2
        # block : InvertedResidual
        # input_channel : 32
        # t : 6
        # c : 64
        # n : 4
        # s : 2
        # block : InvertedResidual
        # input_channel : 64
        # t : 6
        # c : 96
        # n : 3
        # s : 1
        # block : InvertedResidual
        # input_channel : 96
        # t : 6
        # c : 160
        # n : 3
        # s : 2
        # block : InvertedResidual
        # input_channel : 160
        # t : 6
        # c : 320
        # n : 1
        # s : 1

        features = []

        # first layer
        input_channel = int(32 * width_mult)
        layer1 = nn.Sequential(nn.Conv2d(3, input_channel, 3, stride=2, padding=1, bias=False), nn.BatchNorm2d(input_channel), nn.SiLU())
        features.append(layer1)

        # inverted residual blocks
        # t: expansion factor
        # c: number of output channels
        # n: number of blocks
        # s: stride in first block
        inverted_residual_setting = [
            # t, c, n, s
            [1, 16, 1, 1],
            [6, 24, 2, 2],
            [6, 32, 3, 2],
            [6, 64, 4, 2],
            [6, 96, 3, 1],
            [6, 160, 3, 2],
            [6, 320, 1, 1],
        ]

        for t, c, n, s in inverted_residual_setting:
            output_channel = int(c * width_mult)
            for i in range(n):
                stride = s if i == 0 else 1
                # Add CoordAtt block
                if output_channel == 320:
                    features.append(InvertedResidual(input_channel, output_channel, stride, t, CoordAtt(output_channel)))
                else:
                    features.append(InvertedResidual(input_channel, output_channel, stride, t, CoordAtt(output_channel)))
                input_channel = output_channel
        
        # last cnn layer
        output_channel = ceil(640 * width_mult)
        features.append(nn.Sequential(nn.Conv2d(input_channel, output_channel, 1, stride=1, padding=0, bias=False), nn.BatchNorm2d(output_channel), nn.SiLU()))

        self.features = nn.Sequential(*features)

        # final classifier
        self.classifier = nn.Sequential(nn.Dropout(0.2), nn.Linear(output_channel, num_classes))

        self._initialize_weights()

    def forward(self, x):
        x = self.features(x)
        x = x.mean([2, 3])  # global average pooling
        x = self.classifier(x)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out')
                if m.bias is not None:
                    nn.init.zeros_(m.bias)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.ones_(m.weight)
                nn.init.zeros_(m.bias)
            elif isinstance(m, nn.Linear):
                nn.init.normal_(m.weight, 0, 0.01)
                if m.bias is not None:
                    nn.init.zeros_(m.bias)


def make_model(args, add_module_by_name=None):
    model = MobileNetV2(num_classes=args.num_classes, width_mult=args.width_mult)
    return model


class InvertedResidual(nn.Module):
    def __init__(self, inp, oup, stride, expand_ratio, attention_module=None):
        super(InvertedResidual, self).__init__()
        self.stride = stride
        assert stride in [1, 2]

        hidden_dim = int(round(inp * expand_ratio))
        self.use_res_connect = self.stride == 1 and inp == oup

        layers = []
        if expand_ratio != 1:
            # pw linear
            layers.append(nn.Conv2d(inp, hidden_dim, 1, 1, 0, bias=False))
            layers.append(nn.BatchNorm2d(hidden_dim))
            layers.append(nn.SiLU())
        
        # dw
        layers.append(nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False))
        layers.append(nn.BatchNorm2d(hidden_dim))
        layers.append(nn.SiLU())

        # attention
        if attention_module is not None:
            layers.append(attention_module)

        # pw-linear
        layers.append(nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False))
        layers.append(nn.BatchNorm2d(oup))

        self.conv = nn.Sequential(*layers)

    def forward(self, x):
        if self.use_res_connect:
            return x + self.conv(x)
        else:
            return self.conv(x)


from math import ceil

```

--------------------------------

### Use InvertedResidual with Coordinate Attention

Source: https://context7.com/houqb/coordattention/llms.txt

Shows how to instantiate and use the InvertedResidual block, which integrates Coordinate Attention into the MobileNetV2 structure.

```python
import torch
from mbv2_ca import InvertedResidual

# Create an inverted residual block
# Parameters: input_channels, output_channels, stride, expand_ratio
block = InvertedResidual(inp=32, oup=64, stride=2, expand_ratio=6)

# Process feature maps
x = torch.randn(1, 32, 56, 56)
out = block(x)
print(f"Input: {x.shape}")   # torch.Size([1, 32, 56, 56])
print(f"Output: {out.shape}") # torch.Size([1, 64, 28, 28])

# With stride=1 and same channels, residual connection is used
block_residual = InvertedResidual(inp=64, oup=64, stride=1, expand_ratio=6)
x = torch.randn(1, 64, 28, 28)
out = block_residual(x)
print(f"With residual - Input: {x.shape}, Output: {out.shape}")
# Both torch.Size([1, 64, 28, 28]), output = input + conv(input)
```

--------------------------------

### Extract Features from a Pretrained Model

Source: https://context7.com/houqb/coordattention/llms.txt

Demonstrates how to create a feature extractor by removing the classifier from a pretrained model.

```python
class FeatureExtractor(torch.nn.Module):
    def __init__(self, pretrained_model):
        super().__init__()
        self.features = pretrained_model.features
        self.conv = pretrained_model.conv
        self.avgpool = pretrained_model.avgpool

    def forward(self, x):
        x = self.features(x)
        x = self.conv(x)
        x = self.avgpool(x)
        return x.flatten(1)

backbone = FeatureExtractor(model)
features = backbone(input_image)
print(f"Feature vector shape: {features.shape}")  # torch.Size([1, 1280])
```

--------------------------------

### Activation Functions: h_sigmoid and h_swish

Source: https://context7.com/houqb/coordattention/llms.txt

Efficient activation functions that approximate standard sigmoid and swish functions using ReLU6, optimized for mobile deployment.

```APIDOC
## Activation Functions

### Description
Provides `h_sigmoid` and `h_swish` for computationally efficient non-linearities.

### Methods
- **h_sigmoid(inplace=True)**: Approximates sigmoid using relu6(x + 3) / 6.
- **h_swish(inplace=True)**: Approximates swish using x * h_sigmoid(x).
```

--------------------------------

### Class: MBV2_CA

Source: https://context7.com/houqb/coordattention/llms.txt

A complete MobileNetV2 architecture with Coordinate Attention integrated into inverted residual blocks.

```APIDOC
## Class MBV2_CA

### Description
MobileNetV2 model architecture enhanced with Coordinate Attention modules for improved feature representation.

### Parameters
- **num_classes** (int) - Required - Number of output classes for classification.
- **width_mult** (float) - Optional - Multiplier for scaling the network width.

### Usage Example
```python
from mbv2_ca import mbv2_ca
model = mbv2_ca(num_classes=1000, width_mult=1.0)
model.load_state_dict(torch.load('mbv2_ca.pth'))
```
```

--------------------------------

### Class: CoordAtt

Source: https://context7.com/houqb/coordattention/llms.txt

The CoordAtt class implements the coordinate attention mechanism, which can be integrated into any convolutional neural network to capture long-range dependencies.

```APIDOC
## Class CoordAtt

### Description
A module that performs adaptive average pooling along spatial dimensions, concatenates features, and applies convolutions to generate horizontal and vertical attention maps.

### Parameters
- **inp** (int) - Required - Number of input channels.
- **oup** (int) - Required - Number of output channels.
- **reduction** (int) - Optional - Channel reduction ratio for the bottleneck dimension.

### Usage Example
```python
from coordatt import CoordAtt
ca_block = CoordAtt(inp=64, oup=64, reduction=32)
output = ca_block(input_tensor)
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.