### Initialize AcceptVideoSwin Encoder

Source: https://context7.com/lucidrains/srt-h/llms.txt

Configure the Swin Transformer-based video encoder for temporal processing.

```python
import torch
from SRT_H.SRT_H import AcceptVideoSwin

# Initialize Swin video encoder
video_model = AcceptVideoSwin(
    hub_url='SharanSMenon/swin-transformer-hub',
    model_name='swin_tiny_patch4_window7_224',
    dim_model=768,
    max_time_seq_len=8  # Maximum frames
)
```

--------------------------------

### Initialize ACT with Tactile Input Support

Source: https://context7.com/lucidrains/srt-h/llms.txt

Initializes the ACT model to support tactile sensing using the ViTacFormer architecture. It enables bidirectional cross-attention fusion between visual and tactile modalities.

```python
import torch
from SRT_H import ACT

# Initialize ACT with tactile input support
act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    dim_tactile_input=37,
    tactile_self_attn_depth=2,
    tactile_image_fusion_cross_attn_depth=2
)

batch_size = 3
state_tokens = torch.randn(batch_size, 512, 512)
joint_state = torch.randn(batch_size, 17)
tactile_input = torch.randn(batch_size, 16, 37)  # Tactile sensor readings
actions = torch.randn(batch_size, 16, 20)

# Training with tactile input
loss = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    tactile_input=tactile_input,
    actions=actions
)
loss.backward()
```

--------------------------------

### Perform Inference with Tactile Input

Source: https://context7.com/lucidrains/srt-h/llms.txt

Execute action sampling using state tokens, joint states, and tactile sensor data.

```python
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    tactile_input=tactile_input
)
```

--------------------------------

### Implement ACT with Flow Policy

Source: https://context7.com/lucidrains/srt-h/llms.txt

Enable flow matching for expressive action generation as an alternative to DETR-style decoding.

```python
import torch
from SRT_H import ACT

# Initialize ACT with flow policy decoder
act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    flow_policy=True  # Use flow matching instead of DETR queries
)

state_tokens = torch.randn(3, 512, 512)
joint_state = torch.randn(3, 17)
actions = torch.randn(3, 16, 20)

# Training with flow matching loss
loss = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    actions=actions
)
loss.backward()

# Sampling uses the flow model's iterative denoising
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state
)
```

--------------------------------

### Initialize and Train Basic ACT Model

Source: https://context7.com/lucidrains/srt-h/llms.txt

Initializes the Action Chunking Transformer (ACT) model for training. Pass ground truth actions to compute the loss. Requires visual state embeddings and robot joint state.

```python
import torch
from SRT_H import ACT

# Basic ACT model initialization
act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    dim_action=20,
    heads=8,
    encoder_depth=6,
    decoder_depth=6
)

# Training: pass state tokens, joint state, and ground truth actions
batch_size = 3
state_tokens = torch.randn(batch_size, 512, 512)  # Visual state embeddings
joint_state = torch.randn(batch_size, 17)          # Current joint positions
actions = torch.randn(batch_size, 16, 20)          # Ground truth action sequence

loss = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    actions=actions
)
loss.backward()
```

--------------------------------

### Initialize and Train HighLevelPolicy

Source: https://context7.com/lucidrains/srt-h/llms.txt

Configure the HighLevelPolicy with Swin Transformer encoding and perform training or inference.

```python
import torch
from SRT_H.SRT_H import HighLevelPolicy

# Initialize high-level policy
high_level_policy = HighLevelPolicy(
    dim_language_embed=768,        # Language embedding dimension (DistilBERT)
    transformer=dict(
        dim=768,
        attn_dim_head=64,
        heads=8,
        depth=4
    ),
    task_loss_weight=0.4,
    is_corrective_loss_weight=0.3,
    corrective_motion_loss_weight=0.3
)

batch_size = 3
dim = high_level_policy.dim

# Video input (batch, channels, time, height, width)
video = torch.randn(batch_size, 3, 2, 224, 224)

# Task embeddings (pre-computed from language model)
num_tasks = 17
task_embeds = torch.randn(batch_size, num_tasks, dim)
task_labels = torch.randint(0, num_tasks, (batch_size,))

# Corrective motion embeddings
num_corrective_motions = 31
correct_motion_embeds = torch.randn(batch_size, num_corrective_motions, dim)
correct_motion_labels = torch.randint(0, num_corrective_motions, (batch_size,))

# Binary labels indicating if corrective action is needed
is_corrective_labels = torch.randint(0, 2, (batch_size,))

# Training: returns total loss and breakdown
loss, (task_loss, is_corrective_loss, correct_motion_loss) = high_level_policy(
    video,
    task_embeds=task_embeds,
    task_labels=task_labels,
    is_corrective_labels=is_corrective_labels,
    correct_motion_embeds=correct_motion_embeds,
    correct_motion_labels=correct_motion_labels
)
loss.backward()

# Inference: get embeddings without labels
pred_task_embed, is_corrective_embed, pred_correct_motion_embed = high_level_policy(video)
```

--------------------------------

### Initialize and use DETRActionDecoder

Source: https://context7.com/lucidrains/srt-h/llms.txt

Implements DETR-style action prediction using learnable queries and cross-attention decoding.

```python
import torch
from torch import nn
from x_transformers import Encoder
from SRT_H.SRT_H import DETRActionDecoder

# Create a decoder transformer
decoder = Encoder(
    dim=512,
    depth=4,
    heads=8,
    cross_attend=True,
    use_rmsnorm=True
)

# Initialize DETR action decoder
action_decoder = DETRActionDecoder(
    decoder=decoder,
    dim=512,
    dim_action=20,
    action_chunk_len=16,
    action_loss_fn=nn.L1Loss()  # Loss function for action prediction
)

# Encoded context from the encoder
encoded = torch.randn(3, 100, 512)
mask = torch.ones(3, 100, dtype=torch.bool)
actions = torch.randn(3, 16, 20)

# Training: compute action loss
loss = action_decoder(encoded, actions, mask=mask)
loss.backward()

# Inference: sample actions
sampled_actions = action_decoder.sample(encoded, mask)
# Shape: (3, 16, 20)
```

--------------------------------

### Initialize and use FlowActionDecoder

Source: https://context7.com/lucidrains/srt-h/llms.txt

Implements flow matching (diffusion policy) for action generation.

```python
import torch
from x_transformers import Encoder
from SRT_H.SRT_H import FlowActionDecoder

# Create a decoder transformer
decoder = Encoder(
    dim=512,
    depth=4,
    heads=8,
    cross_attend=True,
    use_rmsnorm=True
)

# Initialize flow action decoder
flow_decoder = FlowActionDecoder(
    decoder=decoder,
    dim=512,
    dim_action=20,
    action_chunk_len=16
)

# Encoded context
encoded = torch.randn(3, 100, 512)
mask = torch.ones(3, 100, dtype=torch.bool)
actions = torch.randn(3, 16, 20)

# Training: compute flow matching loss
loss = flow_decoder(encoded, actions, mask=mask)
loss.backward()

# Inference: sample via iterative denoising
sampled_actions = flow_decoder.sample(encoded, mask)
# Shape: (3, 16, 20)
```

--------------------------------

### Inference with Basic ACT Model

Source: https://context7.com/lucidrains/srt-h/llms.txt

Performs inference using the ACT model to sample actions. Does not require ground truth actions, returning predicted action chunks.

```python
# Inference: sample actions without passing ground truth
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state
)
# Returns: torch.Tensor of shape (3, 16, 20) - predicted action chunks
```

--------------------------------

### Apply Action Normalization

Source: https://context7.com/lucidrains/srt-h/llms.txt

Stabilize training by providing dataset statistics for automatic action normalization and denormalization.

```python
import torch
from SRT_H import ACT

# Compute action statistics from your dataset
action_mean = torch.randn(20)  # Mean of actions per dimension
action_std = torch.randn(20).abs() + 0.1  # Std of actions per dimension
action_norm_stats = torch.stack([action_mean, action_std])  # Shape: (2, 20)

# Initialize ACT with action normalization
act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    dim_action=20,
    action_norm_stats=action_norm_stats  # Normalizes during training, denormalizes during inference
)

state_tokens = torch.randn(3, 512, 512)
joint_state = torch.randn(3, 17)
actions = torch.randn(3, 16, 20)  # Raw actions (will be normalized internally)

loss = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    actions=actions
)
loss.backward()

# Inference: returned actions are automatically denormalized
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state
)  # Returns denormalized actions
```

--------------------------------

### Configure ACT with Language Conditioning

Source: https://context7.com/lucidrains/srt-h/llms.txt

Integrate clinician feedback using FiLM conditioning with DistilBERT or pre-computed embeddings.

```python
import torch
from SRT_H import ACT
from SRT_H.SRT_H import DistilBert

# Initialize ACT with language conditioning
act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    lang_condition_model=DistilBert()  # Use DistilBERT for language encoding
)

batch_size = 3
video = torch.randn(batch_size, 3, 2, 224, 224)
joint_state = torch.randn(batch_size, 17)
actions = torch.randn(batch_size, 16, 20)

# Clinician feedback during surgery
feedback = [
    "that looks ok, please proceed",
    "you forgot to clip the cystic artery",
    "stop, that is the common bile duct, not the cystic duct"
]

# Training with language feedback
loss = act(
    video=video,
    joint_state=joint_state,
    feedback=feedback,
    actions=actions
)
loss.backward()

# Alternative: pass pre-computed language embeddings
lang_condition = torch.randn(batch_size, 768)  # DistilBERT dimension
loss = act(
    video=video,
    joint_state=joint_state,
    lang_condition=lang_condition,
    actions=actions
)
```

--------------------------------

### Initialize ACT with Custom Vision Transformer for Video Input

Source: https://context7.com/lucidrains/srt-h/llms.txt

Initializes the ACT model to process raw video input using a custom Vision Transformer (ViT) as the image model. Supports frame dropout regularization during training.

```python
import torch
from SRT_H import ACT
from vit_pytorch import ViT
from vit_pytorch.extractor import Extractor

# Create a custom Vision Transformer as the image model
vit = ViT(
    image_size=256,
    patch_size=32,
    num_classes=1000,
    dim=1024,
    depth=6,
    heads=16,
    mlp_dim=2048,
    dropout=0.1,
    emb_dropout=0.1
)
image_model = Extractor(vit, return_embeddings_only=True)

# Initialize ACT with custom image model
act = ACT(
    image_model=image_model,
    image_model_dim_emb=1024,
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    max_num_image_frames=32,
    dropout_video_frame_prob=0.07
)

# Process video input directly (batch, channels, time, height, width)
video = torch.randn(3, 3, 2, 224, 224)  # 2 frames of 224x224 RGB video
joint_state = torch.randn(3, 17)
actions = torch.randn(3, 16, 20)

loss = act(
    video=video,
    joint_state=joint_state,
    actions=actions
)
loss.backward()
```

--------------------------------

### Retrieve Detailed Loss Breakdown

Source: https://context7.com/lucidrains/srt-h/llms.txt

Monitor training progress by returning individual loss components like reconstruction and KL divergence.

```python
import torch
from SRT_H import ACT

act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    vae_kl_loss_weight=1.0  # Weight for VAE KL divergence loss
)

state_tokens = torch.randn(3, 512, 512)
joint_state = torch.randn(3, 17)
actions = torch.randn(3, 16, 20)

# Get total loss and breakdown
total_loss, loss_breakdown = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    actions=actions,
    return_loss_breakdown=True
)

# loss_breakdown is a namedtuple with:
# - action_recon: Action reconstruction loss (L1 or flow loss)
# - vae_kl_div: KL divergence loss for the style vector VAE
print(f"Total Loss: {total_loss.item():.4f}")
print(f"Action Reconstruction: {loss_breakdown.action_recon.item():.4f}")
print(f"VAE KL Divergence: {loss_breakdown.vae_kl_div.item():.4f}")
```

--------------------------------

### Retrieve Commands with CommandsIndexer

Source: https://context7.com/lucidrains/srt-h/llms.txt

Build an ANN index for surgical commands and retrieve the closest matches for a given embedding.

```python
import torch
from SRT_H.SRT_H import CommandsIndexer, DistilBert

# Define surgical command vocabulary
commands = [
    "grasp the cystic duct",
    "clip the cystic artery",
    "dissect the gallbladder",
    "retract the liver",
    "cauterize the tissue",
    "release the grasper"
]

# Build command index with embeddings
indexer = CommandsIndexer(
    commands=commands,
    model=DistilBert(),       # Language model for encoding
    embed_batch_size=32,
    embed_on_device='cuda'    # Device for embedding computation
)

# Query with a predicted embedding
query_embed = torch.randn(2, 768)  # Batch of 2 query embeddings

# Get closest command embeddings
closest_embeds = indexer(query_embed)
# Shape: (2, 768)

# Get closest command strings
closest_embeds, command_strings = indexer(query_embed, return_strings=True)
# command_strings: list of closest command strings
```

--------------------------------

### Perform Inference with Custom Style Vector

Source: https://context7.com/lucidrains/srt-h/llms.txt

Apply a custom style vector to the action sampling process.

```python
custom_style = torch.ones(512)  # Custom style vector
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state,
    style_vector=custom_style
)
# Shape: (3, 16, 20)
```

--------------------------------

### Perform Inference with Custom Style Vector

Source: https://context7.com/lucidrains/srt-h/llms.txt

Control generated actions during inference by providing a custom style vector.

```python
import torch
from SRT_H import ACT

act = ACT(
    dim=512,
    dim_joint_state=17,
    action_chunk_len=16,
    dim_style_vector=512  # Style vector dimension
)

# Train the model first...
state_tokens = torch.randn(3, 512, 512)
joint_state = torch.randn(3, 17)

# Inference with default style (zeros)
sampled_actions = act(
    state_tokens=state_tokens,
    joint_state=joint_state
)
```

--------------------------------

### Encode Clinician Feedback with DistilBert

Source: https://context7.com/lucidrains/srt-h/llms.txt

Use the DistilBert class to convert text feedback into embeddings.

```python
import torch
from SRT_H.SRT_H import DistilBert

# Initialize DistilBERT encoder
distilbert = DistilBert(
    hf_path="distilbert/distilbert-base-uncased",
    dim=768  # Output embedding dimension
)

# Encode clinician feedback
texts = [
    "proceed with the dissection",
    "stop immediately",
    "move the instrument slightly to the left"
]

# Get language embeddings (uses CLS token)
embeddings = distilbert(texts)
# Shape: (3, 768) - one embedding per text input
```

--------------------------------

### Process video for embeddings

Source: https://context7.com/lucidrains/srt-h/llms.txt

Extracts flattened temporal-spatial tokens from a video tensor using a video model.

```python
# Process video (batch, channels, time, height, width)
video = torch.randn(2, 3, 4, 224, 224)  # 4 frames

# Get video embeddings
embeddings = video_model(video)
# Shape: (2, time*patches, 768) - flattened temporal-spatial tokens
```

--------------------------------

### Extract Visual Features with EfficientNetImageModel

Source: https://context7.com/lucidrains/srt-h/llms.txt

Use EfficientNet to extract spatial feature embeddings from surgical images.

```python
import torch
from SRT_H.SRT_H import EfficientNetImageModel

# Initialize EfficientNet image encoder
image_model = EfficientNetImageModel(
    hub_url='NVIDIA/DeepLearningExamples:torchhub',
    model_name='nvidia_efficientnet_b0',
    dim=1280  # Output embedding dimension
)

# Process images (batch, channels, height, width)
images = torch.randn(4, 3, 224, 224)

# Get spatial feature embeddings
embeddings = image_model(images)
# Shape: (4, num_patches, 1280) - spatial tokens from EfficientNet
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.