### Example: Multi-Branch HyperConnections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/multi_input_variants.md

Demonstrates setting up HyperConnections with multiple specialized branches and configuring it to use two distinct inputs per branch. This setup is useful for ensemble-like processing or complex architectures.

```python
import torch
from torch import nn
from hyper_connections.hyper_connections_with_multi_branch_inputs import HyperConnections

# Create multiple specialized branches
branches = [
    nn.Linear(512, 512),           # Attention-like
    nn.Sequential(
        nn.Linear(512, 2048),
        nn.GELU(),
        nn.Linear(2048, 512)
    ),                              # FFN-like
]

# Multi-branch hyper-connections
hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=branches,
    num_branch_inputs=2,  # Two distinct inputs, one per branch
    layer_index=0
)

# Process
residual = torch.randn(8, 1024, 512)  # Pre-expanded
residual = hc(residual)
```

--------------------------------

### Minimal Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/README.md

A basic usage example demonstrating the initialization and application of HyperConnections.

```APIDOC
## Minimal Example

### Description
Demonstrates the minimal setup for using hyper-connections with stream expansion and reduction.

### Code
```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)
branch = nn.Linear(512, 512)
hc = init(branch=branch)

x = torch.randn(2, 1024, 512)
x = expand(x)
x = hc(x)
x = reduce(x)
```
```

--------------------------------

### Usage: Manual Hyper Connections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/README.md

This snippet illustrates a manual setup for hyper-connections, where the hyper-connection object is instantiated directly. It shows how to get the branch input and the residual addition function separately.

```python
import torch
from torch import nn

# a single branch layer

branch = nn.Linear(512, 512)

# before

residual = torch.randn(2, 1024, 512)

residual = branch(residual) + residual

# after, say 4 streams in paper

from hyper_connections import get_init_and_expand_reduce_stream_functions

init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)

# 1. instantiate hyper connection with correct number of streams (4 in this case) - or use the init function above

hyper_conn = init_hyper_conn(dim = 512)

# 2. expand to 4 streams

residual = expand_stream(residual)

# 3. forward your residual into hyper connection for the branch input + add residual function (learned betas)

branch_input, add_residual = hyper_conn(residual)

branch_output = branch(branch_input)

residual = add_residual(branch_output)

# or you can do it in one line as so -> residual = hyper_conn.decorate_branch(branch)(residual)

# 4. reduce 4 streams with a summation, this has to be done after your for loop trunk

residual = reduce_stream(residual)
```

--------------------------------

### Standard 4-Stream Hyper-Connections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Demonstrates the standard initialization, expansion, and reduction of 4 streams with a specified dimension. This is the typical setup for multi-stream processing.

```python
import torch
from torch import nn
from hyper_connections import get_init_and_expand_reduce_stream_functions

# Standard 4-stream setup
init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)

# Initialize with a branch module
branch = nn.Linear(512, 512)
hc = init(branch=branch)

# Process residual stream
residual = torch.randn(2, 1024, 512)
residual = expand(residual)  # (8, 1024, 512)
residual = hc(residual)
residual = reduce(residual)  # (2, 1024, 512)
```

--------------------------------

### Install Hyper Connections Library

Source: https://github.com/lucidrains/hyper-connections/blob/main/README.md

Install the hyper-connections library using pip. This command fetches and installs the latest stable version.

```bash
$ pip install hyper-connections
```

--------------------------------

### Basic Single-Block Hyper-Connections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Initializes hyper-connections for a single transformation block. Recommended for a typical performance/cost tradeoff.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

# Initialize with 4 streams (typical for good performance/cost tradeoff)
init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)

# Wrap a single transformation
branch = nn.Linear(512, 512)
hc = init(branch=branch)
```

--------------------------------

### Usage: Automated Hyper Connections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/README.md

This snippet demonstrates the automated setup for hyper-connections using helper functions. It involves wrapping a branch layer, expanding the residual stream, forwarding through the wrapped branch, and reducing the streams.

```python
import torch
from torch import nn

# a single branch layer

branch = nn.Linear(512, 512)

# before

residual = torch.randn(2, 1024, 512)

residual = branch(residual) + residual

# after, say 4 streams in paper

from hyper_connections import get_init_and_expand_reduce_stream_functions

init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)

# 1. wrap your branch function

hyper_conn_branch = init_hyper_conn(dim = 512, branch = branch)

# 2. expand to 4 streams, this must be done before your trunk, typically a for-loop with many branch functions

residual = expand_stream(residual)

# 3. forward your residual as usual into the wrapped branch function(s)

residual = hyper_conn_branch(residual)

# 4. reduce 4 streams with a summation, this has to be done after your for-loop trunk

residual = reduce_stream(residual)
```

--------------------------------

### Input Tensor Format Example (Sequence)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Example of the standard input tensor format for a sequence, with batch size, sequence length, and feature dimension.

```text
(2, 1024, 512)  # batch=2, sequence_length=1024, features=512
```

--------------------------------

### Channel-First HyperConnections Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Demonstrates the setup and usage of the channel-first HyperConnections variant with a convolutional branch. It processes image-like tensors in (B, C, H, W) format.

```python
import torch
from torch import nn
from hyper_connections.hyper_connections_channel_first import HyperConnections, get_init_and_expand_reduce_stream_functions

# Setup expansion
init, expand, reduce = get_init_and_expand_reduce_stream_functions(4)

# Create a simple convolutional branch
branch = nn.Conv2d(256, 256, kernel_size=3, padding=1)

# Channel-first hyper-connections
hc = init(
    dim=256,
    branch=branch,
    layer_index=0
)

# Process image-like tensors (B, C, H, W)
x = torch.randn(2, 256, 32, 32)  # 2 images, 256 channels, 32x32 spatial
x = expand(x)  # (8, 256, 32, 32)
x = hc(x)
x = reduce(x)  # (2, 256, 32, 32)

assert x.shape == (2, 256, 32, 32)
```

--------------------------------

### HyperConnections Forward Pass Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/HyperConnections.md

Demonstrates how to use the HyperConnections class with an optional branch module. This example shows setting up the connections, expanding the input across streams, performing the forward pass, and reducing the streams back to a single output.

```python
import torch
from torch import nn
from hyper_connections import get_init_and_expand_reduce_stream_functions

# Setup with 4 streams
init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)

# Create hyper connection wrapper around a branch
branch = nn.Linear(512, 512)
hyper_conn_branch = init_hyper_conn(dim=512, branch=branch)

# Expand input to 4 streams (batch B becomes 4*B)
residual = torch.randn(2, 1024, 512)
residual = expand_stream(residual)  # Shape: (8, 1024, 512)

# Forward through hyper-connected branch
residual = hyper_conn_branch(residual)  # Automatic routing and branch execution
residual = reduce_stream(residual)  # Sum 4 streams back to 1, shape: (2, 1024, 512)

```

--------------------------------

### Basic Hyper-Connections Usage Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Demonstrates the recommended basic usage for most users, involving importing and initializing hyper-connections with a linear branch. Requires PyTorch.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)
branch = nn.Linear(512, 512)
hc = init(branch=branch)
```

--------------------------------

### Example Usage of Expand/Reduce Functions (Channel-First)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Demonstrates how to use the expand and reduce functions for the channel-first variant to adjust tensor dimensions for stream processing.

```python
from hyper_connections.hyper_connections_channel_first import get_expand_reduce_stream_functions

expand, reduce = get_expand_reduce_stream_functions(4)

x = torch.randn(2, 256, 32, 32)
x = expand(x)  # (8, 256, 32, 32)
x = reduce(x)  # (2, 256, 32, 32)
```

--------------------------------

### Example Addition and Multiplication Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Simple example implementations of functions that can be used as 'depth_residual_fn'.

```python
def add(x, y):
    return x + y

def multiply(x, y):
    return x * y
```

--------------------------------

### StreamEmbed Example: With Expansion

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/StreamEmbed.md

Demonstrates creating and using StreamEmbed with automatic batch expansion. The layer will expand the batch dimension before adding embeddings.

```python
# Or with automatic expansion
stream_embed_expand = StreamEmbed(
    num_streams=4,
    dim=512,
    expand_to_streams=True
)

residual = torch.randn(2, 1024, 512)
residual = stream_embed_expand(residual)  # Expands to (8, 1024, 512) and adds embeddings

```

--------------------------------

### AttentionPoolReduceStream Forward Pass Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/AttentionPoolReduceStream.md

Demonstrates how to use AttentionPoolReduceStream in a forward pass. This example shows creating the module, applying it to a tensor representing multiple streams, and comparing its output to a simple summation reduction.

```python
import torch
from hyper_connections import AttentionPoolReduceStream

# Create attention pool reducer
attn_reduce = AttentionPoolReduceStream(
    num_streams=4,
    dim=512
)

# Use after hyper-connection layer
residual = torch.randn(8, 1024, 512)  # 8 = 2 batch * 4 streams
residual = attn_reduce(residual)  # Shape: (2, 1024, 512)

# Compare to simple summation
from einops.layers.torch import Reduce
sum_reduce = Reduce('(b s) ... -> b ...', 'sum', s=4)
residual_sum = sum_reduce(residual)  # Same output shape, different operation
```

--------------------------------

### Minimal HyperConnections Initialization and Usage

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/README.md

This snippet demonstrates the minimal setup for using HyperConnections, including initialization, stream expansion, applying a branch module, and stream reduction. It requires importing necessary functions and PyTorch modules.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)
branch = nn.Linear(512, 512)
hc = init(branch=branch)

x = torch.randn(2, 1024, 512)
x = expand(x)
x = hc(x)
x = reduce(x)
```

--------------------------------

### Example: Decorating a Branch with Residual Connection

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/Residual.md

Illustrates how to use the `decorate_branch` method to wrap a standard PyTorch module (like nn.Linear) with the residual connection logic.

```python
from hyper_connections import Residual
import torch
from torch import nn

residual_conn = Residual()
branch = nn.Linear(512, 512)

# Decorate the branch
decorated = residual_conn.decorate_branch(branch)

x = torch.randn(2, 1024, 512)
y = decorated(x)  # Equivalent to branch(x) + x
```

--------------------------------

### Input Tensor Format Example (2D Spatial)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Example of the standard input tensor format for 2D spatial data, including batch size, height, width, and feature dimension.

```text
(8, 32, 32, 256)  # batch=8, height=32, width=32, features=256
```

--------------------------------

### Usage Pattern: Replacing Standard Reduce

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/AttentionPoolReduceStream.md

Illustrates how to integrate AttentionPoolReduceStream into a model by replacing the standard reduce function. This example shows setting up standard stream functions and then substituting the attention-based reduction.

```python
from hyper_connections import get_expand_reduce_stream_functions
from hyper_connections.hyper_connections import AttentionPoolReduceStream
import torch
from torch import nn

# Get standard expand/reduce
init, expand, reduce = get_expand_reduce_stream_functions(4, dim=512)

# Replace the reduce function with attention pooling
branch = nn.Linear(512, 512)
hc = init(dim=512, branch=branch)

# Create attention-based reduce
attn_reduce = AttentionPoolReduceStream(4, dim=512)

residual = torch.randn(2, 1024, 512)
residual = expand(residual)
residual = hc(residual)
residual = attn_reduce(residual)  # Attention-weighted reduction instead of sum
```

--------------------------------

### Checking Installed Hyper-Connections Version

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Provides methods to check the installed version of the 'hyper-connections' package. This is useful for ensuring compatibility with specific features or bug fixes.

```python
import hyper_connections
print(hyper_connections.__version__)  # If available
```

```python
import pkg_resources
version = pkg_resources.get_distribution("hyper-connections").version
print(version)
```

--------------------------------

### Fractional Stream Hyper-Connections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Illustrates the setup for fractional stream connections, as described in the frac-connections paper. This allows for a single stream to be internally divided into multiple feature fractions.

```python
# With fractions (frac-connections paper)
init_frac, expand_frac, reduce_frac = get_init_and_expand_reduce_stream_functions(
    1,  # single stream
    num_fracs=4,  # but 4 feature fractions
    dim=512
)

hc_frac = init_frac(branch=branch)
residual = expand_frac(residual)
residual = hc_frac(residual)
residual = reduce_frac(residual)
```

--------------------------------

### Get Init, Expand, and Reduce Functions (Channel-First)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Convenience function to obtain the necessary functions for initializing, expanding, and reducing streams in the channel-first variant. Used for creating hyper-connected convolutional blocks.

```python
from hyper_connections.hyper_connections_channel_first import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

init, expand, reduce = get_init_and_expand_reduce_stream_functions(4)

# Create hyper-connected conv block
branch = nn.Sequential(
    nn.Conv2d(256, 512, 3, padding=1),
    nn.ReLU(),
    nn.Conv2d(512, 256, 3, padding=1)
)

hc = init(dim=256, branch=branch)

x = torch.randn(2, 256, 32, 32)
x = expand(x)
x = hc(x)
x = reduce(x)
```

--------------------------------

### Channel-First Hyper-Connections Usage Example for CNNs

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Shows how to use hyper-connections with the channel-first variant for convolutional networks, specifying dimensions and a Conv2d branch. Requires PyTorch.

```python
from hyper_connections.hyper_connections_channel_first import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

init, expand, reduce = get_init_and_expand_reduce_stream_functions(4)
branch = nn.Conv2d(256, 256, 3, padding=1)
hc = init(dim=256, branch=branch)
```

--------------------------------

### Example: Residual with Branch

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/Residual.md

Demonstrates using the Residual class with an explicit branch module. The output is the sum of the branch output and the original input.

```python
import torch
from torch import nn
from hyper_connections import Residual

# With branch
branch = nn.Linear(512, 512)
residual_conn = Residual(branch=branch)

x = torch.randn(2, 1024, 512)
y = residual_conn(x)  # Equivalent to branch(x) + x
```

--------------------------------

### Get Expand/Reduce Functions for Channel-First

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Retrieves expand and reduce functions specifically designed for the channel-first data format. These functions are compatible with the channel-first layout.

```python
def get_expand_reduce_stream_functions(
    num_streams: int,
    disable: bool = False
) -> tuple[Module, Module]
```

--------------------------------

### StreamEmbed Example: No Expansion

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/StreamEmbed.md

Demonstrates creating and using StreamEmbed without automatic batch expansion. The input residual tensor must already have the stream dimension expanded.

```python
import torch
from hyper_connections import StreamEmbed

# Create stream embeddings (no expansion)
stream_embed = StreamEmbed(
    num_streams=4,
    dim=512,
    expand_to_streams=False
)

# Use with pre-expanded residuals
residual = torch.randn(8, 1024, 512)  # 8 = 2 batch * 4 streams
residual = stream_embed(residual)  # Adds learned per-stream embeddings

```

--------------------------------

### Specify Additional Input Paths for HyperConnections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/multi_input_variants.md

Examples demonstrating how to specify secondary input residual streams using integers, strings, or tuples for positional and keyword arguments.

```python
# Input comes from second positional argument (*args[0])
additional_input_paths = [1]

# Inputs come from second positional arg and keyword 'residual3'
additional_input_paths = [1, 'residual3']

# Second positional arg has different dimension (256 instead of dim)
additional_input_paths = [(1, 256), ('aux', 128)]
```

--------------------------------

### Example: Residual without Branch (Manual Control)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/Residual.md

Shows how to use the Residual class without a predefined branch, allowing for manual control over adding the residual. The forward method returns the input and a function to apply the residual addition.

```python
# Without branch (manual control)
residual_conn_manual = Residual()
x = torch.randn(2, 1024, 512)
branch_input, add_residual = residual_conn_manual(x)

# branch_input is just x
y = add_residual(branch(branch_input))  # Equivalent to branch(x) + x
```

--------------------------------

### Manifold-Constrained Hyper-Connections Setup

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Configures hyper-connections with tighter constraints for better optimization using manifold-constrained functions. Adjust sinkhorn iterations and domain for stability.

```python
from hyper_connections.manifold_constrained_hyper_connections import get_init_and_expand_reduce_stream_functions as mc_get_init

# Use tighter constraints for better optimization
init, expand, reduce = mc_get_init(
    num_streams=4,
    dim=512,
    sinkhorn_iters=30,  # More iterations = tighter constraint
    log_domain_sinkhorn=True,  # Better numerical stability
    num_dynamic_alpha_proposals=3  # Average 3 proposals
)

branch = nn.Linear(512, 512)
hc = init(branch=branch, dropout=0.1)
```

--------------------------------

### ManifoldConstrainedHyperConnections Forward Pass Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/ManifoldConstrainedHyperConnections.md

Demonstrates a forward pass using ManifoldConstrainedHyperConnections with a specified branch and increased Sinkhorn iterations for tighter constraints.

```python
import torch
from torch import nn
from hyper_connections import ManifoldConstrainedHyperConnections, mHC

# Using the short alias
hc = mHC(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    sinkhorn_iters=30  # More iterations for tighter constraint
)

residual = torch.randn(8, 1024, 512)  # Pre-expanded
residual = hc(residual)

```

--------------------------------

### Get Initialization and Expand/Reduce Stream Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

A convenience function that returns an initialization function and the expand/reduce pair. This is the primary entry point for most use cases. It supports auto-disabling for single-path networks.

```python
def get_init_and_expand_reduce_stream_functions(
    num_streams: int,
    num_fracs: int = 1,
    dim: int | None = None,
    add_stream_embed: bool = False,
    disable: bool | None = None
) -> tuple[Callable, Module, Module]:
    pass
```

--------------------------------

### GatedResidual Example (Coarse and Fine Gating)

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/residual_updates.md

Illustrates the usage of GatedResidual for both coarse (single gate per position) and fine (per-element gates) gating mechanisms within HyperConnections. The fine_gate parameter controls the granularity of the gating.

```python
from hyper_connections.residuals import GatedResidual
from hyper_connections import HyperConnections
import torch.nn as nn

# Coarse gating: single value per position
gate_coarse = GatedResidual(dim=512, fine_gate=False)

hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    depth_residual_fn=lambda x, r: gate_coarse(x, r)
)

# Fine gating: per-element gates
gate_fine = GatedResidual(dim=512, fine_gate=True)
```

--------------------------------

### Multi-Layer Transformer Block with Hyper-Connections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Sets up hyper-connections for a multi-layer transformer, utilizing learnable per-stream representations. This example shows separate initialization for attention and feedforward branches.

```python
# Create hyper-connections factory
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    num_streams=4,
    dim=512,
    add_stream_embed=True  # Learnable per-stream representations
)

class TransformerBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.attn_hc = init(
            dim=512,
            branch=nn.MultiheadAttention(512, 8),
            layer_index=0  # First stream for attention
        )
        self.ff_hc = init(
            dim=512,
            branch=nn.Sequential(
                nn.Linear(512, 2048),
                nn.GELU(),
                nn.Linear(2048, 512)
            ),
            layer_index=1  # Second stream for feedforward
        )
    
    def forward(self, x):
        x = expand(x)
        x = self.attn_hc(x)
        x = self.ff_hc(x)
        x = reduce(x)
        return x
```

--------------------------------

### Integrating Hyper-Connections in a Transformer Block

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Provides an example of how to incorporate hyper-connections within a standard Transformer block. It shows the initialization of attention and feed-forward layers using hyper-connections.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, dim=512, num_heads=8, num_streams=4):
        super().__init__()
        init, self.expand, self.reduce = get_init_and_expand_reduce_stream_functions(
            num_streams, dim=dim
        )
        
        # Multiple branches in hyper-connections
        self.attn = init(dim=dim, branch=nn.MultiheadAttention(dim, num_heads))
        self.ff = init(dim=dim, branch=nn.Sequential(
            nn.Linear(dim, dim * 4),
            nn.GELU(),
            nn.Linear(dim * 4, dim)
        ))
    
    def forward(self, x):
        x = self.expand(x)
        x = self.attn(x)
        x = self.ff(x)
        x = self.reduce(x)
        return x
```

--------------------------------

### Get Stream Initialization and Expansion/Reduction Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/HyperConnections.md

Use this convenience function to obtain callables for initializing HyperConnections and for expanding tensors to multiple streams and reducing them back. It is useful for setting up residual stream management within a model.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch
from torch import nn

# Get initialization and stream functions
init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)

# Initialize with branch
branch = nn.Linear(512, 512)
hc_branch = init(branch=branch)

# Use in a loop
residual = torch.randn(2, 1024, 512)
residual = expand(residual)
for _ in range(12):  # 12 transformer blocks
    residual = hc_branch(residual)
residual = reduce(residual)
```

--------------------------------

### Comparing Hyper-Connections vs. Residual Connections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Illustrates how to configure the model to use either full hyper-connections or fall back to standard residual connections. This allows for easy comparison of performance and behavior.

```python
def get_model(use_hyper_connections=True):
    init, expand, reduce = get_init_and_expand_reduce_stream_functions(
        4 if use_hyper_connections else 1,
        dim=512
    )
    
    branch = nn.Linear(512, 512)
    hc = init(branch=branch)
    
    return hc, expand, reduce

# Same code works with both configurations
hc, expand, reduce = get_model(use_hyper_connections=True)  # Full hyper-connections
hc, expand, reduce = get_model(use_hyper_connections=False)  # Falls back to Residual
```

--------------------------------

### GRUGatedResidual Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/residual_updates.md

Demonstrates how to use GRUGatedResidual as a custom depth_residual_fn within HyperConnections. Ensure the dimension matches the residual dimension.

```python
import torch
from hyper_connections.residuals import GRUGatedResidual
from hyper_connections import HyperConnections

# Use as depth_residual_fn
gru_gate = GRUGatedResidual(dim=512)

hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    depth_residual_fn=lambda x, r: gru_gate(x, r)
)
```

--------------------------------

### Initialize Hyperconnections with Additional Input Paths

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/multi_input_variants.md

Initialize Hyperconnections with a main branch and two additional input paths of specified dimensions. This is useful when your network requires multiple distinct input streams.

```python
branch = nn.Linear(512, 512)
hc = init(
    dim=512,
    branch=branch,
    additional_input_paths=[(1, 256), (2, 128)]  # Two additional inputs
)
```

--------------------------------

### Stream Configuration Type Hints

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Illustrates type hints for stream-related configuration parameters. These include the number of residual streams, feature fractions, and input views.

```python
num_residual_streams: int      # Number of parallel streams
num_fracs: int = 1             # Feature fractions
num_input_views: int = 1       # Multiple input views per stream
```

--------------------------------

### Manifold-Constrained Hyper-Connections Usage Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Illustrates using the manifold-constrained variant for better stability, including setting Sinkhorn iterations. Requires PyTorch.

```python
from hyper_connections import ManifoldConstrainedHyperConnections
from hyper_connections.manifold_constrained_hyper_connections import get_init_and_expand_reduce_stream_functions

init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    4, dim=512, sinkhorn_iters=30
)
hc = init(branch=branch)
```

--------------------------------

### Initialize Hyperconnections for Capacity

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Maximize routing expressiveness by using more streams, fractions, enabling stream embeddings, and multiple input views.

```python
# Maximize routing expressiveness
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    num_streams=8,  # Many streams
    num_fracs=4,    # Also fractions
    dim=512,
    add_stream_embed=True  # Learnable identities
)

hc = init(
    branch=branch,
    num_input_views=2,  # Multiple inputs
    layer_index=0
)
```

--------------------------------

### OrthogonalResidualUpdate Example

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/residual_updates.md

Shows how to integrate OrthogonalResidualUpdate into the HyperConnections framework as a custom depth_residual_fn. This method enhances training stability by updating the residual only in directions orthogonal to itself.

```python
from hyper_connections.residuals import OrthogonalResidualUpdate
from hyper_connections import HyperConnections
import torch.nn as nn

# Use orthogonal updates
ortho_update = OrthogonalResidualUpdate(double_precision=True)

hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    depth_residual_fn=lambda x, r: ortho_update(x, r)
)
```

--------------------------------

### Initialize and Use a Multi-Branch Network

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/multi_input_variants.md

Set up and use a multi-branch Hyperconnections network with multiple specialized branches. This configuration is suitable for complex architectures requiring parallel processing paths.

```python
from hyper_connections.hyper_connections_with_multi_branch_inputs import get_init_and_expand_reduce_stream_functions as multi_branch_get_init

init, expand, reduce = multi_branch_get_init(4)

# Multiple specialized branches
branches = [
    nn.Linear(512, 512),  # Path 1
    nn.Linear(512, 512),  # Path 2
]

hc = init(
    dim=512,
    branch=branches,
    num_branch_inputs=2
)

x = torch.randn(2, 1024, 512)
x = expand(x)
x = hc(x)
x = reduce(x)
```

--------------------------------

### Activation and Constraint Configuration Type Hints

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Defines type hints for activation and constraint-related configuration parameters. Parameters include activation function choice, dropout rate, and Sinkhorn iterations.

```python
tanh: bool = True              # Activation function
dropout: float = 0.0           # Dropout rate
sinkhorn_iters: int = 20       # Constraint iterations
log_domain_sinkhorn: bool      # Numerical stability
```

--------------------------------

### Module Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/README.md

Convenience APIs for initialization of modules.

```APIDOC
## Module Functions

### Description
Convenience APIs for initialization.

### Functions
- `get_expand_reduce_stream_functions(num_streams, dim, **kwargs)`: Returns functions to expand to and reduce from multiple streams.
- `get_init_and_expand_reduce_stream_functions(num_streams, dim, **kwargs)`: Returns a complete initialization pipeline including expand and reduce functions.
```

--------------------------------

### Manual Dimension Initialization

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Shows how to manually specify the dimension during initialization when it's not pre-filled. This is useful when the dimension is not implicitly known.

```python
# Manual initialization without dim pre-fill
init_manual, expand_m, reduce_m = get_init_and_expand_reduce_stream_functions(4)
hc_manual = init_manual(dim=512, branch=branch)  # Must provide dim
```

--------------------------------

### Python 3.10+ Type Hints for Initialization

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Demonstrates the use of modern Python type hints with '|' for unions in a class constructor. This style is preferred for Python 3.10 and later.

```python
def __init__(
    self,
    num_residual_streams: int,
    *,
    dim: int,
    branch: Module | None = None,
    layer_index: int | None = None,
    tanh: bool = True,
    channel_first: bool = False,
    dropout: float = 0.0,
    ...
)
```

--------------------------------

### Using Residual as Fallback

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/Residual.md

Demonstrates how to use the Residual class when hyper-connections are explicitly disabled. The `init` function returns a partial application of Residual in this mode.

```python
from hyper_connections import get_init_and_expand_reduce_stream_functions
import torch
from torch import nn

# Disable hyper-connections (falls back to Residual)
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    num_streams=1,  # Single stream
    disable=True  # Explicitly disable
)

# init will be a partial application of Residual instead of HyperConnections
branch = nn.Linear(512, 512)
residual_conn = init(branch=branch)

residual = torch.randn(2, 1024, 512)
residual = expand(residual)  # Identity operation
residual = residual_conn(residual)  # Simple residual connection
residual = reduce(residual)  # Identity operation
```

--------------------------------

### Initialize Hyperconnections for Stability

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Use constraints and regularization by increasing Sinkhorn iterations, enabling log domain Sinkhorn for numerical stability, and increasing dynamic alpha proposals for averaging.

```python
# Use constraints and regularization
init, expand, reduce = mc_get_init(
    num_streams=4,
    dim=512,
    sinkhorn_iters=50,  # Tight constraints
    log_domain_sinkhorn=True,  # Numerical stability
    num_dynamic_alpha_proposals=5  # Proposal averaging
)

hc = init(
    branch=branch,
    dropout=0.1,  # Regularization
    tanh=True  # Gating
)
```

--------------------------------

### Get Stream Dimension Management Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/ManifoldConstrainedHyperConnections.md

Returns expand and reduce functions for stream dimension management. Useful for managing feature dimensions across multiple streams.

```python
@staticmethod
def get_expand_reduce_stream_functions(
    num_streams: int,
    add_stream_embed: bool = False,
    dim: int | None = None,
    disable: bool = False
) -> tuple[Module, Module]
```

--------------------------------

### Residual Block with Channel-First Hyper Connections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Example of implementing a residual block using the channel-first variant for convolutional neural networks. It utilizes the expand, hyper-connected module, and reduce functions.

```python
import torch.nn as nn
from hyper_connections.hyper_connections_channel_first import get_init_and_expand_reduce_stream_functions

class ResidualBlock(nn.Module):
    def __init__(self, channels=256, num_streams=4):
        super().__init__()
        init, self.expand, self.reduce = get_init_and_expand_reduce_stream_functions(num_streams)
        
        self.hc = init(
            dim=channels,
            branch=nn.Sequential(
                nn.Conv2d(channels, channels, 3, padding=1),
                nn.BatchNorm2d(channels),
                nn.ReLU(),
                nn.Conv2d(channels, channels, 3, padding=1),
                nn.BatchNorm2d(channels)
            )
        )
    
    def forward(self, x):
        x = self.expand(x)
        x = self.hc(x)
        x = self.reduce(x)
        return x
```

--------------------------------

### Architecture Configuration Type Hints

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/types.md

Shows type hints for architecture-specific configuration parameters. This includes the feature dimension, an optional branch module, and data format.

```python
dim: int                       # Feature dimension
branch: Module | None = None   # Branch module
channel_first: bool = False    # Data format
```

--------------------------------

### get_init_and_expand_reduce_stream_functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/ManifoldConstrainedHyperConnections.md

Convenience static method for initialization with manifold constraints. It returns initialization, expand, and reduce functions.

```APIDOC
## get_init_and_expand_reduce_stream_functions

### Description
Convenience function for initialization with manifold constraints.

### Method
`@staticmethod get_init_and_expand_reduce_stream_functions`

### Parameters
#### Path Parameters
- `num_streams` (int) - Required - Number of streams
- `num_fracs` (int) - Optional - Default: 1 - Feature fractions
- `dim` (int | None) - Optional - Default: None - Optional feature dimension
- `add_stream_embed` (bool) - Optional - Default: False - Use stream embeddings
- `disable` (bool | None) - Optional - Default: None - Auto-disable if num_streams==1 and num_fracs==1
- `sinkhorn_iters` (int) - Optional - Default: 20 - Sinkhorn iterations (passed via **kwargs)
- `**kwargs` - Any - Additional arguments passed to ManifoldConstrainedHyperConnections

### Returns
Tuple of (init_fn, expand_fn, reduce_fn)

### Example
```python
from hyper_connections.manifold_constrained_hyper_connections import get_init_and_expand_reduce_stream_functions
import torch
from torch import nn

# Use manifold-constrained version
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    4,
    dim=512,
    sinkhorn_iters=30  # Tighter constraint
)

branch = nn.Linear(512, 512)
hc = init(branch=branch)

residual = torch.randn(2, 1024, 512)
residual = expand(residual)
residual = hc(residual)
residual = reduce(residual)
```
```

--------------------------------

### Initialize Hyperconnections for Speed

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Minimize parameters and computation by using fewer streams and disabling embeddings and dropout.

```python
# Minimize parameters and computation
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    num_streams=2,  # Fewer streams
    dim=512,
    add_stream_embed=False  # No embeddings
)

hc = init(
    branch=branch,
    dropout=0.0  # Skip dropout
)
```

--------------------------------

### Project File Organization

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/INDEX.md

Overview of the directory structure for the HyperConnections library, detailing the purpose of each file and directory.

```text
output/
├── INDEX.md                              # This file
├── imports_guide.md                      # Where to import from
├── types.md                              # Type definitions
├── configuration.md                      # Constructor options
└── api-reference/
    ├── HyperConnections.md               # Main class
    ├── ManifoldConstrainedHyperConnections.md  # mHC variant
    ├── Residual.md                       # Base class
    ├── StreamEmbed.md                    # Learnable embeddings
    ├── AttentionPoolReduceStream.md      # Attention reduction
    ├── residual_updates.md               # Alternative updates
    ├── module_functions.md               # Convenience functions
    ├── multi_input_variants.md           # Multi-input/branch
    └── channel_first_variant.md          # CNN optimization
```

--------------------------------

### Get Expand and Reduce Stream Functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/module_functions.md

Provides functions to expand input to multiple streams and reduce them back. Useful for parallel processing within a network. Supports optional learnable stream embeddings and a disable option for identity functions.

```python
def get_expand_reduce_stream_functions(
    num_streams: int,
    add_stream_embed: bool = False,
    dim: int | None = None,
    disable: bool = False
) -> tuple[Module, Module]:
    pass
```

```python
import torch
from torch import nn
from hyper_connections import get_expand_reduce_stream_functions

# Basic expand/reduce
expand, reduce = get_expand_reduce_stream_functions(4)

x = torch.randn(2, 1024, 512)  # batch=2
x = expand(x)  # (8, 1024, 512) - batch multiplied by 4
x = reduce(x)  # (2, 1024, 512) - summed back

# With learnable stream embeddings
expand_embed, reduce_embed = get_expand_reduce_stream_functions(
    4,
    add_stream_embed=True,
    dim=512
)

x = torch.randn(2, 1024, 512)
x = expand_embed(x)  # Expands to (8, 1024, 512) and adds embeddings
x = reduce_embed(x)

# Disable (no-op)
expand_noop, reduce_noop = get_expand_reduce_stream_functions(4, disable=True)
x_out = expand_noop(x)  # Returns x unchanged
```

--------------------------------

### Initialize Hyper-Connections with Manifold Constraints

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/ManifoldConstrainedHyperConnections.md

Convenience function for initialization with manifold constraints. Returns initialization, expansion, and reduction functions.

```python
@staticmethod
def get_init_and_expand_reduce_stream_functions(
    num_streams: int,
    num_fracs: int = 1,
    dim: int | None = None,
    add_stream_embed: bool = False,
    disable: bool | None = None,
    sinkhorn_iters: int = 20,
    **kwargs
) -> tuple[Callable, Module, Module]
```

```python
from hyper_connections.manifold_constrained_hyper_connections import get_init_and_expand_reduce_stream_functions
import torch
from torch import nn

# Use manifold-constrained version
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    4,
    dim=512,
    sinkhorn_iters=30  # Tighter constraint
)

branch = nn.Linear(512, 512)
hc = init(branch=branch)

residual = torch.randn(2, 1024, 512)
residual = expand(residual)
residual = hc(residual)
residual = reduce(residual)
```

--------------------------------

### Forward Pass with Multi-Input HyperConnections and Custom Branch

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/multi_input_variants.md

Demonstrates a full forward pass using HyperConnections with multiple inputs and a custom branch module, including input expansion and output reduction.

```python
import torch
from torch import nn
from hyper_connections.hyper_connections_with_multi_input_streams import HyperConnections

class CustomModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(512, 512)
        self.linear2 = nn.Linear(256, 512)
        self.linear3 = nn.Linear(128, 512)
    
    def forward(self, x, second, *, third):
        # Combine multiple inputs
        return self.linear1(x) + self.linear2(second) + self.linear3(third)

# Create multi-input hyper-connections
branch = CustomModule()
hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=branch,
    additional_input_paths=[
        (1, 256),      # Second positional arg, dimension 256
        ('third', 128) # Keyword arg 'third', dimension 128
    ]
)

# Setup expansion
from hyper_connections.hyper_connections_with_multi_input_streams import get_init_and_expand_reduce_stream_functions
_, expand, reduce = get_init_and_expand_reduce_stream_functions(4)

# Process with multiple inputs
residual = torch.randn(2, 1024, 512)      # Primary
residual2 = torch.randn(2, 1024, 256)     # Second input
residual3 = torch.randn(2, 1024, 128)     # Third input

# Expand all inputs
residual = expand(residual)    # (8, 1024, 512)
residual2 = expand(residual2)  # (8, 1024, 256)
residual3 = expand(residual3)  # (8, 1024, 128)

# Forward through hyper-connections
residual = hc(residual, residual2, third=residual3)

# Reduce primary output
residual = reduce(residual)  # (2, 1024, 512)
residual2 = reduce(residual2)  # (2, 1024, 256)
residual3 = reduce(residual3)  # (2, 1024, 128)
```

--------------------------------

### Import Standard Variant Components

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Import core hyper-connections components for sequence/transformer data in channel-last format. Includes core classes and utilities.

```python
# From hyper_connections/hyper_connections.py
from hyper_connections import (
    HyperConnections,
    Residual,
    get_expand_reduce_stream_functions,
    get_init_and_expand_reduce_stream_functions
)

# Utilities
from hyper_connections.hyper_connections import (
    RMSNorm,
    StreamEmbed,
    AttentionPoolReduceStream
)
```

--------------------------------

### MVSplitResidualUpdate Prepare Method

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/residual_updates.md

A compatibility method for MVSplitResidualUpdate that returns the residual input twice and an empty dictionary.

```python
def prepare(self, residual: Tensor) -> tuple[Tensor, Tensor, dict]:
```

--------------------------------

### Instantiate MVSplitResidualUpdate and HyperConnections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/residual_updates.md

Demonstrates how to instantiate MVSplitResidualUpdate and use it as the depth_residual_fn in HyperConnections for stable deep networks.

```python
from hyper_connections.residuals import MVSplitResidualUpdate
from hyper_connections import HyperConnections
import torch.nn as nn

# Mean-variance split for stable deep networks
mv_update = MVSplitResidualUpdate(
    dim=512,
    init_alpha=0.01,
    init_beta=0.1
)

hc = HyperConnections(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    depth_residual_fn=lambda x, r: mv_update(x, r, **{})
)
```

--------------------------------

### get_init_and_expand_reduce_stream_functions Constructor

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/configuration.md

Provides convenience functions for initializing and managing hyper-connections streams. Supports optional feature dimension, stream embeddings, and explicit disabling.

```python
init, expand, reduce = get_init_and_expand_reduce_stream_functions(
    num_streams,
    num_fracs=1,
    dim=None,
    add_stream_embed=False,
    disable=None
)
```

--------------------------------

### Channel-First HyperConnections Constructor

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/channel_first_variant.md

Initializes the channel-first HyperConnections variant. Ensure channel_first is set to True for this variant.

```python
def __init__(
    self,
    num_residual_streams: int,
    *,
    dim: int,
    branch: Module | None = None,
    layer_index: int | None = None,
    tanh: bool = True,
    channel_first: bool = True,
    dropout: float = 0.0
)
```

--------------------------------

### Manifold-Constrained Hyper-Connections Initialization

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/INDEX.md

Initializes `ManifoldConstrainedHyperConnections` with specified parameters, including sinkhorn iterations for constraint enforcement. Requires `nn` from `torch`.

```python
from hyper_connections import ManifoldConstrainedHyperConnections as mHC

hc = mHC(
    num_residual_streams=4,
    dim=512,
    branch=nn.Linear(512, 512),
    sinkhorn_iters=30
)
```

--------------------------------

### HyperConnections Constructor Parameters

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/HyperConnections.md

Details the parameters for initializing the HyperConnections class, including the number of residual streams, feature dimension, and options for branch integration, activation, and connection types.

```python
def __init__(
    self,
    num_residual_streams: int,
    *,
    dim: int,
    branch: Module | None = None,
    layer_index: int | None = None,
    tanh: bool = True,
    channel_first: bool = False,
    dropout: float = 0.0,
    add_branch_out_to_residual: bool = True,
    num_input_views: int = 1,
    depth_residual_fn: Callable = add,
    num_fracs: int = 1
)

```

--------------------------------

### Usage Pattern with get_expand_reduce_stream_functions

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/api-reference/StreamEmbed.md

Shows how StreamEmbed is integrated into the hyper-connection pipeline using helper functions. The `expand` function returned by `get_expand_reduce_stream_functions` is an instance of StreamEmbed with `expand_to_streams=True`.

```python
from hyper_connections import get_expand_reduce_stream_functions
import torch
from torch import nn

# Get functions with stream embeddings enabled
init, expand, reduce = get_expand_reduce_stream_functions(
    4,
    add_stream_embed=True,
    dim=512
)

branch = nn.Linear(512, 512)
hc = init(dim=512, branch=branch)

residual = torch.randn(2, 1024, 512)
residual = expand(residual)  # Expands and adds stream embeddings
residual = hc(residual)
residual = reduce(residual)

```

--------------------------------

### Correct Usage of Init Function for HyperConnections

Source: https://github.com/lucidrains/hyper-connections/blob/main/_autodocs/imports_guide.md

Ensure the HyperConnections model is initialized using the 'init' function returned by 'get_init_and_expand_reduce_stream_functions'. Directly instantiating HyperConnections without this function is incorrect.

```python
init, expand, reduce = get_init_and_expand_reduce_stream_functions(4, dim=512)
hc = init(branch=branch)  # Correct
```