### Install the package

Source: https://github.com/lucidrains/taylor-series-linear-attention/blob/main/README.md

Install the library via pip.

```bash
$ pip install taylor-series-linear-attention
```

--------------------------------

### Install Taylor Series Linear Attention

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Install the library using pip. For causal attention support, install the additional CUDA dependency.

```bash
pip install taylor-series-linear-attention
```

```bash
pip install pytorch-fast-transformers
```

--------------------------------

### Initialize and use causal attention

Source: https://github.com/lucidrains/taylor-series-linear-attention/blob/main/README.md

Usage for autoregressive tasks. Requires installing pytorch-fast-transformers and setting causal to True.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

attn = TaylorSeriesLinearAttn(
  dim = 512,
  dim_head = 16,
  heads = 16,
  causal = True,        # set this to True
  rotary_emb = True     # rotary embeddings
)

x = torch.randn(1, 8192, 512)

out = attn(x)

assert x.shape == out.shape
```

--------------------------------

### Implement Vision Transformer with ViT

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

A complete Vision Transformer setup for image classification tasks using linear attention for improved efficiency.

```python
import torch
from taylor_series_linear_attention import ViT

# Initialize Vision Transformer
model = ViT(
    image_size=256,     # Input image size (256x256)
    patch_size=16,      # Patch size (16x16 patches)
    num_classes=1000,   # Number of output classes
    dim=512,            # Transformer dimension
    depth=12,           # Number of transformer blocks
    ff_mult=4,          # Feed-forward expansion factor
    heads=16,           # Number of attention heads
    channels=3,         # Input channels (RGB)
    dim_head=8,         # Per-head dimension
    dropout=0.1,        # Attention dropout
    emb_dropout=0.1     # Embedding dropout
)

# Input batch of images
images = torch.randn(4, 3, 256, 256)

# Forward pass returns class logits
logits = model(images)
assert logits.shape == (4, 1000)  # torch.Size([4, 1000])

# For inference
model.eval()
with torch.no_grad():
    predictions = model(images).argmax(dim=-1)
    print(predictions)  # tensor([class_idx, ...])
```

--------------------------------

### Configure TaylorSeriesLinearAttn

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Initialize the core attention module with various architectural options like causal masking, multi-query attention, and token shifting.

```python
attn = TaylorSeriesLinearAttn(
    dim=512,                      # Model dimension
    dim_head=16,                  # Per-head dimension
    heads=16,                     # Number of attention heads
    causal=False,                 # Causal masking for autoregressive
    one_headed_kv=True,           # Multi-query attention (shared K/V)
    rotary_emb=False,             # Rotary positional embeddings
    combine_heads=True,           # Project heads back to dim
    gate_value_heads=True,        # Learnable gates per value head
    prenorm=True,                 # Apply RMSNorm before attention
    shift_tokens=True,            # Token shifting from RWKV
    dropout=0.1,                  # Dropout probability
    remove_even_power_dups=True   # Optimize symmetric power terms
)

x = torch.randn(2, 2048, 512)
mask = torch.ones((2, 2048)).bool()

out = attn(x, mask=mask)
assert out.shape == x.shape
```

--------------------------------

### Initialize and use cross-attention

Source: https://github.com/lucidrains/taylor-series-linear-attention/blob/main/README.md

Usage for cross-attention by providing context tensors and masks.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

attn = TaylorSeriesLinearAttn(
  dim = 512,
  dim_head = 16,
  heads = 16
)

x = torch.randn(1, 1024, 512)

context = torch.randn(1, 65536, 512)
context_mask = torch.ones((1, 65536)).bool()

out = attn(x, context = context, mask = context_mask)

assert x.shape == out.shape
```

--------------------------------

### Initialize and use self-attention

Source: https://github.com/lucidrains/taylor-series-linear-attention/blob/main/README.md

Standard usage for self-attention with input tensors and masks.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

attn = TaylorSeriesLinearAttn(
  dim = 512,
  dim_head = 16,
  heads = 16
)

x = torch.randn(1, 4096, 512)
mask = torch.ones((1, 4096)).bool()

out = attn(x, mask = mask)

assert x.shape == out.shape
```

--------------------------------

### Initialize and Use Self Attention

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Initialize the TaylorSeriesLinearAttn module for self-attention and perform a forward pass with an optional mask. The output shape matches the input shape.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

# Initialize self-attention module
attn = TaylorSeriesLinearAttn(
    dim=512,          # Input/output dimension
    dim_head=16,      # Dimension per attention head
    heads=16          # Number of attention heads
)

# Create input tensor (batch_size=1, seq_len=4096, dim=512)
x = torch.randn(1, 4096, 512)

# Optional attention mask (batch_size=1, seq_len=4096)
mask = torch.ones((1, 4096)).bool()

# Forward pass with mask
out = attn(x, mask=mask)

# Output shape matches input shape
assert out.shape == x.shape  # torch.Size([1, 4096, 512])
```

--------------------------------

### Initialize and Use Cross Attention

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Initialize the TaylorSeriesLinearAttn module for cross-attention. This allows attending to a separate context sequence, useful for encoder-decoder architectures. The output shape matches the query input shape.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

# Initialize cross-attention module
attn = TaylorSeriesLinearAttn(
    dim=512,
    dim_head=16,
    heads=16
)

# Query input (shorter sequence)
x = torch.randn(1, 1024, 512)

# Context to attend to (can be very long due to linear complexity)
context = torch.randn(1, 65536, 512)
context_mask = torch.ones((1, 65536)).bool()

# Forward pass with context
out = attn(x, context=context, mask=context_mask)

# Output shape matches query input shape
assert out.shape == x.shape  # torch.Size([1, 1024, 512])
```

--------------------------------

### Build Custom Transformer Blocks

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Integrate TaylorSeriesLinearAttn into custom PyTorch modules as a drop-in replacement for standard attention layers.

```python
import torch
import torch.nn as nn
from taylor_series_linear_attention import TaylorSeriesLinearAttn

class LinearAttentionTransformerBlock(nn.Module):
    def __init__(self, dim=512, heads=8, dim_head=64, ff_mult=4, dropout=0.1):
        super().__init__()
        self.norm1 = nn.LayerNorm(dim)
        self.attn = TaylorSeriesLinearAttn(
            dim=dim,
            heads=heads,
            dim_head=dim_head,
            dropout=dropout
        )
        self.norm2 = nn.LayerNorm(dim)
        self.ff = nn.Sequential(
            nn.Linear(dim, dim * ff_mult),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(dim * ff_mult, dim),
            nn.Dropout(dropout)
        )

    def forward(self, x, mask=None):
        x = x + self.attn(self.norm1(x), mask=mask)
        x = x + self.ff(self.norm2(x))
        return x

# Stack multiple blocks
blocks = nn.ModuleList([
    LinearAttentionTransformerBlock(dim=512, heads=8, dim_head=16)
    for _ in range(6)
])

x = torch.randn(2, 4096, 512)
for block in blocks:
    x = block(x)

print(x.shape)  # torch.Size([2, 4096, 512])
```

--------------------------------

### Caching for Autoregressive Inference

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Demonstrates how to use the TaylorSeriesLinearAttn module with caching for efficient autoregressive inference. The key-value states are cached to avoid recomputation when generating tokens one at a time.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

attn = TaylorSeriesLinearAttn(
    dim=512,
    dim_head=16,
    heads=16,
    causal=True,
    rotary_emb=True
)

# Initial forward pass - process prompt and get cache
prompt = torch.randn(1, 100, 512)
out, cache = attn(prompt, return_cache=True)

# Generate tokens one at a time using cache
for _ in range(50):
    # Single token input
    new_token = torch.randn(1, 1, 512)

    # Use cache for efficient generation
    out, cache = attn(new_token, cache=cache, return_cache=True)

    # out.shape is (1, 1, 512) - single token output
```

--------------------------------

### Process Vision Data with ChannelFirstTaylorSeriesLinearAttn

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Use this wrapper to handle channel-first tensor formats common in image and video processing tasks.

```python
import torch
from taylor_series_linear_attention import ChannelFirstTaylorSeriesLinearAttn

# Initialize for image processing
attn = ChannelFirstTaylorSeriesLinearAttn(
    dim=256,
    dim_head=16,
    heads=8
)

# Image input: (batch, channels, height, width)
image = torch.randn(2, 256, 32, 32)
out = attn(image)
assert out.shape == image.shape  # torch.Size([2, 256, 32, 32])

# Video input: (batch, channels, frames, height, width)
video = torch.randn(1, 256, 16, 32, 32)
out = attn(video)
assert out.shape == video.shape  # torch.Size([1, 256, 16, 32, 32])
```

--------------------------------

### Initialize and Use Causal Attention

Source: https://context7.com/lucidrains/taylor-series-linear-attention/llms.txt

Initialize the TaylorSeriesLinearAttn module for causal attention with rotary embeddings. Causal attention is suitable for autoregressive language modeling. No mask is needed for causal attention.

```python
import torch
from taylor_series_linear_attention import TaylorSeriesLinearAttn

# Initialize causal attention with rotary embeddings
attn = TaylorSeriesLinearAttn(
    dim=512,
    dim_head=16,
    heads=16,
    causal=True,       # Enable causal masking
    rotary_emb=True    # Add rotary positional embeddings
)

# Input sequence for autoregressive processing
x = torch.randn(1, 8192, 512)

# Forward pass (no mask needed for causal attention)
out = attn(x)

assert out.shape == x.shape  # torch.Size([1, 8192, 512])
```

--------------------------------

### Academic Citations

Source: https://github.com/lucidrains/taylor-series-linear-attention/blob/main/README.md

BibTeX entries for relevant research papers.

```bibtex
@inproceedings{Arora2023ZoologyMA,
  title   = {Zoology: Measuring and Improving Recall in Efficient Language Models},
  author  = {Simran Arora and Sabri Eyuboglu and Aman Timalsina and Isys Johnson and Michael Poli and James Zou and Atri Rudra and Christopher R'e},
  year    = {2023},
  url     = {https://api.semanticscholar.org/CorpusID:266149332}
}
```

```bibtex
@inproceedings{Keles2022OnTC,
  title   = {On The Computational Complexity of Self-Attention},
  author  = {Feyza Duman Keles and Pruthuvi Maheshakya Wijewardena and Chinmay Hegde},
  booktitle = {International Conference on Algorithmic Learning Theory},
  year    = {2022},
  url     = {https://api.semanticscholar.org/CorpusID:252198880}
}
```

```bibtex
@article{Shazeer2019FastTD,
  title   = {Fast Transformer Decoding: One Write-Head is All You Need},
  author  = {Noam M. Shazeer},
  journal = {ArXiv},
  year    = {2019},
  volume  = {abs/1911.02150}
}
```

```bibtex
@inproceedings{Peng2023RWKVRR,
  title   = {RWKV: Reinventing RNNs for the Transformer Era},
  author  = {Bo Peng and Eric Alcaide and Quentin G. Anthony and Alon Albalak and Samuel Arcadinho and Stella Biderman and Huanqi Cao and Xin Cheng and Michael Chung and Matteo Grella and G Kranthikiran and Xuming He and Haowen Hou and Przemyslaw Kazienko and Jan Kocoń and Jiaming Kong and Bartlomiej Koptyra and Hayden Lau and Krishna Sri Ipsit Mantri and Ferdinand Mom and Atsushi Saito and Xiangru Tang and Bolun Wang and Johan Sokrates Wind and Stansilaw Wozniak and Ruichong Zhang and Zhenyuan Zhang and Qihang Zhao and Peng Zhou and Jian Zhu and Rui Zhu},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year    = {2023},
  url     = {https://api.semanticscholar.org/CorpusID:258832459}
}
```

```bibtex
@inproceedings{Katharopoulos2020TransformersAR,
  title   = {Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention},
  author  = {Angelos Katharopoulos and Apoorv Vyas and Nikolaos Pappas and Franccois Fleuret},
  booktitle = {International Conference on Machine Learning},
  year    = {2020},
  url     = {https://api.semanticscholar.org/CorpusID:220250819}
}
```

```bibtex
@misc{buckman2024,
  author    = {Buckman, Jacob and Gelada, Carles and Zhang, Sean},
  publisher = {Manifest AI},
  title     = {Symmetric {Power} {Transformers}},
  date      = {2024-08-15},
  langid    = {en}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.