### Install LyCORIS from PyPI or Source

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Install the LyCORIS library using pip from PyPI or by cloning the source repository and installing locally.

```bash
# From PyPI
pip install lycoris-lora

# From source
git clone https://github.com/KohakuBlueleaf/LyCORIS
cd LYCORIS
pip install .
```

--------------------------------

### Install LyCORIS from Source

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Clone the LyCORIS repository and install it from source. This is useful for development or if you need the latest changes.

```bash
git clone https://github.com/KohakuBlueleaf/LyCORIS
cd LyCORIS
pip install .
```

--------------------------------

### Run kohya training with TOML config files

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Execute kohya's training script for Lycoris modules using TOML configuration files for both training and dataset settings. Example configuration files are available in the project's example directory.

```bash
accelerate launch train_network.py \
  --config_file example_configs/training_configs/kohya/loha_config.toml \
  --dataset_config example_configs/training_configs/kohya/dataset_config.toml

```

--------------------------------

### Run kohya training with command line arguments

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Use this command to train Lycoris modules for SD models with kohya's script using command line arguments. Ensure 'accelerate' is installed and configured.

```bash
accelerate launch train_network.py \
  --network_module lycoris.kohya \
  --network_dim "DIM_FOR_LINEAR" --network_alpha "ALPHA_FOR_LINEAR"\
  --network_args "conv_dim=DIM_FOR_CONV" "conv_alpha=ALPHA_FOR_CONV" \
  "dropout=DROPOUT_RATE" "algo=locon" \

```

--------------------------------

### TOML: Kohya Training Configuration for LoKr

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Configure Lycoris for kohya-ss/sd-scripts training using a TOML file. This example sets up LoKr with specific dimensions, alpha, and network module settings, alongside optimizer configurations.

```toml
# lokr_config.toml – full LoKr config
[Basics]
pretrained_model_name_or_path = "/path/to/model"
train_data_dir                = "/path/to/dataset"
max_train_epochs              = 2

[Network_setup]
network_dim   = 100000
network_alpha = 1

[LyCORIS]
network_module = "lycoris.kohya"
network_args   = ["preset=attn-mlp", "algo=lokr", "factor=6"]

[Optimizer]
train_batch_size = 8
optimizer_type   = "AdamW8bit"
unet_lr          = 2e-4
text_encoder_lr  = 2e-4

```

--------------------------------

### Batch HCP Convert: Specify Network Path

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/Conversion-scripts.md

Provides an example of specifying multiple network paths for conversion, including individual files.

```bash
--network_path /path/to/ckpt_folder /path/to/ckpt_folder2/unet-oft-5000.safetensors /path/to/ckpt_folder2/text_encoder-oft-5000.safetensors /path/to/ckpt_folder2/unet-loha-5000.safetensors
```

--------------------------------

### Bash: Kohya Training with Lycoris CLI Arguments

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Integrate Lycoris into kohya-ss/sd-scripts training by specifying `lycoris.kohya` as the network module via CLI. This example configures LoHa on attention and MLP layers with a dropout rate.

```bash
# CLI: LoHa on attention+MLP layers
accelerate launch train_network.py \
  --pretrained_model_name_or_path /path/to/sd-model \
  --train_data_dir /path/to/dataset \
  --network_module lycoris.kohya \
  --network_dim 16 \
  --network_alpha 8 \
  --network_args "algo=loha" "preset=attn-mlp" "dropout=0.05"

```

--------------------------------

### Functional API Usage Example

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Demonstrates how to use Lycoris functional APIs for generating weights and integrating them into forward passes for both weight and activation differences. Note that specific arguments for weight_diff and bypass_forward_diff may vary by algorithm.

```python
from lycoris.functional import xxx
weights = xxx.weight_gen(org_weight)

def forward_with_diff_weight(x, org_weight, weights):
    return org_forward(x, org_weight + xxx.weight_diff(*weights))

def forward_with_diff_activation(x, org_weight, weights):
    org_out = org_forward(x, org_weight)
    return org_out + xxx.bypass_forward_diff(x, org_out, *weights)
```

--------------------------------

### Bash: Merge LyCORIS Adapter into Base Checkpoint

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Merge a LyCORIS adapter back into a base checkpoint using the `merge.py` tool. This example specifies SDXL compatibility, device, data type, and adapter weight.

```bash
# Merge a LyCORIS adapter back into the base checkpoint
python3 tools/merge.py \
  --is_sdxl \
  --device cuda \
  --dtype fp16 \
  --weight 1.0 \
  /path/to/base_model.safetensors \
  output_locon.safetensors \
  merged_model

```

--------------------------------

### Install LyCORIS via Pip

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Install the LyCORIS package using pip for use in your Python environment.

```bash
pip install lycoris-lora
```

--------------------------------

### Bash: Launch Kohya Training with TOML Config

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Launch the kohya training script using a specified TOML configuration file for Lycoris settings and a separate dataset configuration file.

```bash
accelerate launch train_network.py \
  --config_file lokr_config.toml \
  --dataset_config dataset_config.toml

```

--------------------------------

### Exclude specific layers

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Applies presets to all layers matching '.*' but excludes layers named 'final_layer' or starting with 'time_embedding'.

```python
LycorisNetwork.apply_preset({
    "target_name": [".*"],
    "exclude_name": ["final_layer", "time_embedding.*"],
})
```

--------------------------------

### Convert Models to and from Bundle Format

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Use this script for converting models to and from the bundle format, which is particularly useful for pivotal tuning. It allows specifying paths for network checkpoints, embeddings, and destination directory, with verbose output options.

```bash
python3 batch_bundle_convert.py \
  --network_path /path/to/sd-webui-ssd/models/Lora  \
  --emb_path /path/to/ckpts \
  --dst_dir /path/to/sd-webui-ssd/models/Lora/bundle \
  --to_bundle --verbose 2 
```

--------------------------------

### Apply Multiple Adapters Simultaneously with Stacked Wrappers

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Demonstrates applying multiple Lycoris adapters to a model sequentially and selectively removing them. `apply_to()` and `restore()` are stack-safe, ensuring correct behavior when managing multiple adapters.

```python
import torch, torch.nn as nn
from lycoris import create_lycoris, LycorisNetwork

model = nn.Sequential(nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 64))

LycorisNetwork.apply_preset({"target_name": [".*0.*"]})
net_a = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="lokr")
net_a.apply_to()

LycorisNetwork.apply_preset({"target_name": [".*2.*"]})
net_b = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="loha")
net_b.apply_to()

x = torch.randn(4, 128)
model.eval()
with torch.no_grad():
    out_both = model(x)

# Remove only net_b while keeping net_a
net_b.restore()
with torch.no_grad():
    out_a_only = model(x)

# Remove net_a, verify full restoration
net_a.restore()
with torch.no_grad():
    out_base = model(x)

# Restoring in reverse order fully recovers base model
assert not torch.allclose(out_both, out_base)
print("Stacked and selectively removed adapters successfully.")
```

--------------------------------

### Apply Lycoris preset and create network

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

This Python code demonstrates how to import Lycoris components, apply a preset for network targeting, create a Lycoris network instance with specified dimensions and algorithm, and apply it to a PyTorch model. The model will then run with the Lycoris net applied.

```python
from lycoris import create_lycoris, LycorisNetwork

LycorisNetwork.apply_preset(
    {"target_name": [".*attn.*"]}
)
lycoris_net = create_lycoris(
    your_model, 
    1.0, 
    linear_dim=16, 
    linear_alpha=2.0, 
    algo="lokr"
)
lycoris_net.apply_to()

# after apply_to(), your_model() will run with LyCORIS net
lycoris_param = lycoris_net.parameters()
forward_with_lyco = your_model(x)

```

--------------------------------

### Python: Create Lycoris Network with TOML Preset

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Create a Lycoris network instance using a specified TOML configuration file for algorithm presets. Ensure the `unet` object is defined before calling `create_lycoris`.

```python
from lycoris import create_lycoris, LycorisNetwork

lycoris_net = create_lycoris(
    unet,
    multiplier=1.0,
    linear_dim=16,
    linear_alpha=8.0,
    algo="lora",                             # default algo for unspecified modules
    preset="example_configs/preset_configs/custom.toml",
)
lycoris_net.apply_to()

```

--------------------------------

### Stacked Wrappers

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Demonstrates how to apply multiple adapters simultaneously using `apply_to()` and `restore()`. These methods are stack-safe, ensuring that each wrapper only affects its own layers and `restore()` correctly removes only its contribution.

```APIDOC
## Stacked Wrappers — Apply multiple adapters simultaneously

### Description
`apply_to()` and `restore()` are stack-safe: each wrapper patches only the layers it owns, and `restore()` peels back exactly that wrapper's contribution, leaving other wrappers intact.

### Method Signature
- `apply_to()`: Applies the adapter to the model.
- `restore()`: Removes the adapter's contribution from the model.

### Request Example
```python
# Apply net_a
LycorisNetwork.apply_preset({"target_name": [".*0.*"]})
net_a = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="lokr")
net_a.apply_to()

# Apply net_b
LycorisNetwork.apply_preset({"target_name": [".*2.*"]})
net_b = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="loha")
net_b.apply_to()

# Remove only net_b while keeping net_a
net_b.restore()

# Remove net_a, verify full restoration
net_a.restore()
```
```

--------------------------------

### Load LyCORIS Adapter from Weights

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use `create_lycoris_from_weights` to load a `LycorisNetwork` from a saved `.safetensors` or `.pt` file. The adapter is automatically applied to the provided base model.

```python
import torch
import torch.nn as nn
from lycoris import create_lycoris_from_weights

# Assume `model` is the same base model used during training
model = nn.Linear(256, 256)

# Load from safetensors (or .pt)
lycoris_net, weights_sd = create_lycoris_from_weights(
    multiplier=1.0,
    file="my_adapter.safetensors",
    module=model,
    # weights_sd=None  # Pass a pre-loaded dict to skip file I/O
)

lycoris_net.apply_to()
model.eval()
with torch.no_grad():
    out = model(torch.randn(4, 256))
    print(out.shape)  # torch.Size([4, 256])

# Adjust strength at inference time
lycoris_net.set_multiplier(0.7)
```

--------------------------------

### Run HCP-Diffusion training

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Launch HCP-Diffusion training for Lycoris modules using the specified command and YAML configuration file. Note that HCP-Diffusion support has been dropped in Lycoris 3.0.0.

```bash
accelerate launch -m hcpdiff.train_ac_single \
  --cfg example_configs/training_configs/hcp/hcp_diag_oft.yaml

```

--------------------------------

### Save and Load Weights

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Demonstrates how to save network weights to a safetensors file with fp16 precision and optional metadata, and how to reload them.

```APIDOC
## Save and Load Weights

### Description
Saves network weights to a safetensors file with fp16 precision and optional metadata. Also shows how to reload the weights.

### Save Weights
```python
net.save_weights(
    "adapter.safetensors",
    dtype=torch.float16,
    metadata={"ss_network_dim": "8", "ss_network_alpha": "1.0"},
)
```

### Load Weights
```python
state = net.load_weights("adapter.safetensors")
print(state)  # {} if perfectly clean, otherwise lists missing/unexpected keys
```
```

--------------------------------

### Configure LyCORIS Layer Targeting with Presets

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use the `LycorisNetwork.apply_preset` class method to set global targeting rules for subsequent `create_lycoris` calls. Supports module class names, name patterns, and algorithm overrides.

```python
from lycoris import LycorisNetwork
```

--------------------------------

### LycorisBaseModule Methods

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Lists the instance methods for LycorisBaseModule, covering operations like applying, restoring, merging, and modifying weights.

```APIDOC
## Class: `LycorisBaseModule`

### Methods

* `apply_to`
* `restore`
* `merge_to`
* `get_diff_weight`
* `get_merged_weight`
* `apply_max_norm`
* `bypass_forward_diff`
* `bypass_forward`
* `parametrize_forward`
* `forward`
```

--------------------------------

### Bash: Convert Models to/from Bundle Format

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use `batch_bundle_convert.py` to convert LyCORIS models to or from the bundle format, typically used for pivotal tuning. This command allows specifying paths for network adapters, embeddings, destination directory, and conversion direction.

```bash
# Convert to/from bundle format (for pivotal tuning)
python3 tools/batch_bundle_convert.py \
  --network_path /path/to/sd-webui/models/Lora \
  --emb_path /path/to/embeddings \
  --dst_dir /path/to/bundle_output \
  --to_bundle \
  --verbose 2

```

--------------------------------

### LycorisNetwork.apply_preset

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

A class method to configure global targeting rules for layer selection in subsequent `create_lycoris` calls. It supports targeting by module class names, name patterns (regex/fnmatch), algorithm overrides, and exclusion lists.

```APIDOC
## `LycorisNetwork.apply_preset` — Configure which layers to target

Class method that sets the global targeting rules used by the next `create_lycoris` call. Supports module class names, regex/fnmatch name patterns, per-module algorithm overrides, and exclusion lists.

```python
from lycoris import LycorisNetwork


```

--------------------------------

### Bash: Convert HCP-Diffusion Outputs to SD-WebUI Format

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Convert checkpoints from HCP-Diffusion format to the SD-WebUI compatible format using `batch_hcp_convert.py`. Options include specifying the network path, destination directory, output prefix, and automatic alpha scaling.

```bash
# Convert HCP-Diffusion outputs to sd-webui format
python3 tools/batch_hcp_convert.py \
  --network_path /path/to/hcp_ckpts \
  --dst_dir /path/to/sd-webui/models/Lora \
  --output_prefix my_model \
  --auto_scale_alpha \
  --to_webui

```

--------------------------------

### Lycoris Network Wrappers

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Introduces the wrapper classes for integrating Lycoris algorithms into PyTorch networks, including stacking and restoration capabilities.

```APIDOC
## Wrappers

* `LycorisNetwork`: the wrapper class to patch any pytorch modules to apply LYCORIS algorithms.
* `create_lycoris`: see example
* `create_lycoris_from_weights`: see example

`LycorisNetwork.apply_to()` can be invoked multiple times on the same module with different wrapper instances. Each wrapper is stacked on top of the previous one, and calling `restore()` on a wrapper removes only its own contribution while keeping earlier wrappers active.
```

--------------------------------

### Create LyCORIS Wrapper for PyTorch Module

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use `create_lycoris` to wrap any PyTorch `nn.Module` with a LyCORIS adapter. Configure targeting rules with `LycorisNetwork.apply_preset`. The adapter parameters are trained while the base model is frozen.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from lycoris import create_lycoris, LycorisNetwork

# Define or load any PyTorch model
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

model = SimpleNet()

# Target only layers whose names match the regex ".*fc.*"
LycorisNetwork.apply_preset({"target_name": [".*fc.*"]})

lycoris_net = create_lycoris(
    model,
    multiplier=1.0,      # Scale factor applied during forward pass
    linear_dim=16,       # Rank for linear layers
    linear_alpha=8.0,    # Alpha for linear layers (scale = alpha/dim)
    algo="lokr",         # Algorithm: lora, loha, lokr, dylora, full, diag-oft, boft, tlora
    factor=8,            # LoKr-specific: controls Kronecker factorization
    conv_dim=8,          # Rank for conv layers (defaults to linear_dim)
    dropout=0.05,        # Dropout on adapter activations
)

# Patch the model in-place; from here model() uses the adapter
lycoris_net.apply_to()

print(f"Adapter modules: {len(lycoris_net.loras)}")
# Adapter modules: 3

# Only train adapter parameters, keep base model frozen
model.requires_grad_(False)
optimizer = torch.optim.AdamW(lycoris_net.parameters(), lr=1e-3)

x = torch.randn(32, 784)
y_target = torch.randint(0, 10, (32,)) 

for step in range(5):
    optimizer.zero_grad()
    out = model(x)
    loss = F.cross_entropy(out, y_target)
    loss.backward()
    optimizer.step()
    print(f"step {step}: loss={loss.item():.4f}")
# step 0: loss=2.3041
# step 4: loss=2.2918
```

--------------------------------

### Batch Bundle Convert: To Bundle Format

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/Conversion-scripts.md

Converts models to bundle format, combining network checkpoints with their corresponding embeddings. Use `--to_bundle` for this conversion.

```bash
python batch_bundle_convert.py \
--network_path /path/to/sd-webui-ssd/models/Lora  \
--emb_path /path/to/ckpts \
--dst_dir /path/to/sd-webui-ssd/models/Lora/bundle \
--to_bundle --verbose 2 
```

--------------------------------

### Preset System

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Utilizes built-in named presets for Stable Diffusion, simplifying the configuration of adapters for common training scenarios. These presets can be passed via `preset=` in `create_lycoris` or as `network_args` in kohya scripts.

```APIDOC
## Preset system — Built-in named presets for Stable Diffusion

### Description
Built-in string presets for common SD training scenarios, passed via `preset=` in `create_lycoris` or as `network_args` in kohya scripts.

### Available Presets
- `"full"` - all UNet + CLIP layers (default)
- `"full-lin"` - full but skip conv layers
- `"attn-mlp"` - all transformer blocks (kohya default)
- `"attn-only"` - attention layers only
- `"unet-transformer-only"` - UNet transformer blocks only
- `"unet-convblock-only"` - ResBlock, Upsample, Downsample only

### Example Usage
```python
# Example: attention-only LoHa
lycoris_net = create_lycoris(
    unet,
    multiplier=1.0,
    linear_dim=16,
    linear_alpha=8.0,
    algo="loha",
    preset="attn-only",
)
lycoris_net.apply_to()
```
```

--------------------------------

### create_lycoris

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Creates a LyCORIS wrapper for any PyTorch module, applying the specified adapter algorithm to targeted layers. The wrapper's parameters are trained while the base model remains frozen.

```APIDOC
## `create_lycoris` — Create a LyCORIS wrapper for any PyTorch module

The primary entry point for wrapping any `nn.Module` with a LyCORIS adapter. It reads the current preset from `LycorisNetwork` class-level state and instantiates the appropriate algorithm modules for all targeted layers. Returns a `LycorisNetwork` object whose parameters are trained while the base model is frozen.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from lycoris import create_lycoris, LycorisNetwork

# Define or load any PyTorch model
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

model = SimpleNet()

# Target only layers whose names match the regex ".*fc.*"
LycorisNetwork.apply_preset({"target_name": [".*fc.*"]})

lycoris_net = create_lycoris(
    model,
    multiplier=1.0,      # Scale factor applied during forward pass
    linear_dim=16,       # Rank for linear layers
    linear_alpha=8.0,    # Alpha for linear layers (scale = alpha/dim)
    algo="lokr",         # Algorithm: lora, loha, lokr, dylora, full, diag-oft, boft, tlora
    factor=8,            # LoKr-specific: controls Kronecker factorization
    conv_dim=8,          # Rank for conv layers (defaults to linear_dim)
    dropout=0.05,        # Dropout on adapter activations
)

# Patch the model in-place; from here model() uses the adapter
lycoris_net.apply_to()

print(f"Adapter modules: {len(lycoris_net.loras)}")
# Adapter modules: 3

# Only train adapter parameters, keep base model frozen
model.requires_grad_(False)
optimizer = torch.optim.AdamW(lycoris_net.parameters(), lr=1e-3)

x = torch.randn(32, 784)
y_target = torch.randint(0, 10, (32,))

for step in range(5):
    optimizer.zero_grad()
    out = model(x)
    loss = F.cross_entropy(out, y_target)
    loss.backward()
    optimizer.step()
    print(f"step {step}: loss={loss.item():.4f}")
# step 0: loss=2.3041
# step 4: loss=2.2918
```
```

--------------------------------

### Preset System for Stable Diffusion Adapters

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Applies a built-in preset for configuring Lycoris adapters, such as 'attn-only' for attention layers. Presets simplify the configuration for common Stable Diffusion training scenarios.

```python
from lycoris import create_lycoris, LycorisNetwork

# Load your Stable Diffusion UNet
# from diffusers import UNet2DConditionModel
# unet = UNet2DConditionModel.from_pretrained(...)

# Available built-in presets:
#   "full"                  – all UNet + CLIP layers (default)
#   "full-lin"              – full but skip conv layers
#   "attn-mlp"              – all transformer blocks (kohya default)
#   "attn-only"             – attention layers only
#   "unet-transformer-only" – UNet transformer blocks only
#   "unet-convblock-only"   – ResBlock, Upsample, Downsample only

# Example: attention-only LoHa
lycoris_net = create_lycoris(
    unet,
    multiplier=1.0,
    linear_dim=16,
    linear_alpha=8.0,
    algo="loha",
    preset="attn-only",
)
lycoris_net.apply_to()
```

--------------------------------

### Batch HCP Convert: Network Extensions

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/Conversion-scripts.md

Allows specifying custom file extensions for input network files during conversion.

```bash
--network_ext .safetensors .pt
```

--------------------------------

### Functional API for Low-Level Algorithm Primitives

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Utilizes `lycoris.functional` to compute adapter weight deltas without the overhead of the wrapper class. Demonstrates `bypass_forward_diff` for direct computation and `diff_weight` for materializing the delta.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from lycoris.functional import loha, lokr

org_model = nn.Linear(128, 128)

# Generate adapter weights (initialized so delta ≈ 0)
lokr_weights = lokr.weight_gen(org_model.weight)
loha_weights = loha.weight_gen(org_model.weight)

x = torch.randn(4, 128)
base_out = org_model(x)

# Option 1: bypass_forward_diff — compute ΔWx without materializing ΔW
out_lokr = base_out + lokr.bypass_forward_diff(x, base_out, *lokr_weights)
out_loha = base_out + loha.bypass_forward_diff(x, base_out, *loha_weights)

# Option 2: diff_weight — materialize ΔW and add to base weight
out_lokr_w = F.linear(x, org_model.weight + lokr.diff_weight(*lokr_weights), org_model.bias)
out_loha_w = F.linear(x, org_model.weight + loha.diff_weight(*loha_weights), org_model.bias)

# At init, delta should be ≈ 0
print(F.mse_loss(base_out, out_lokr).item())   # ~0.0
print(F.mse_loss(base_out, out_loha).item())   # ~0.0
print(F.mse_loss(base_out, out_lokr_w).item()) # ~0.0
print(F.mse_loss(base_out, out_loha_w).item()) # ~0.0
```

--------------------------------

### Bash: Extract LoCon Adapter from Dreambooth Model

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use the `extract_locon.py` tool to extract a LoCon adapter from a Dreambooth model checkpoint. This command specifies SDXL compatibility, device, safetensors format, and adapter dimensions.

```bash
# Extract a LoCon adapter from a dreambooth model vs its base
python3 tools/extract_locon.py \
  --is_sdxl \
  --device cuda \
  --safetensors \
  --linear_dim 16 \
  --conv_dim 8 \
  /path/to/base_model.safetensors \
  /path/to/finetuned_model.safetensors \
  output_locon

```

--------------------------------

### Save and Load Weights with fp16 and Metadata

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Saves network weights in safetensors format with fp16 precision and optional metadata. Reloads weights and prints a dictionary indicating missing or unexpected keys.

```python
net.save_weights(
    "adapter.safetensors",
    dtype=torch.float16,
    metadata={"ss_network_dim": "8", "ss_network_alpha": "1.0"},
)

state = net.load_weights("adapter.safetensors")
print(state)
```

--------------------------------

### Convert Models between HCP and sd-webui Format

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

This script facilitates the conversion of LyCORIS models trained with HCP-Diffusion for use in sd-webui. It supports automatic scaling of alpha and conversion to webui format.

```bash
python3 batch_hcp_convert.py \
  --network_path /path/to/ckpts \
  --dst_dir /path/to/stable-diffusion-webui/models/Lora \
  --output_prefix something \
  --auto_scale_alpha --to_webui
```

--------------------------------

### create_lycoris_from_weights

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Reconstructs a LycorisNetwork from a saved weights file (.safetensors or .pt) without requiring the original training configuration. It automatically detects the algorithm used.

```APIDOC
## `create_lycoris_from_weights` — Load a LyCORIS adapter from a saved file

Reconstructs a `LycorisNetwork` from a `.safetensors` or `.pt` weights file without needing the original training configuration. Automatically detects the algorithm stored in the state dict via `get_module`.

```python
import torch
import torch.nn as nn
from lycoris import create_lycoris_from_weights

# Assume `model` is the same base model used during training
model = nn.Linear(256, 256)

# Load from safetensors (or .pt)
lycoris_net, weights_sd = create_lycoris_from_weights(
    multiplier=1.0,
    file="my_adapter.safetensors",
    module=model,
    # weights_sd=None  # Pass a pre-loaded dict to skip file I/O
)

lycoris_net.apply_to()
model.eval()
with torch.no_grad():
    out = model(torch.randn(4, 256))
    print(out.shape)  # torch.Size([4, 256])

# Adjust strength at inference time
lycoris_net.set_multiplier(0.7)
```
```

--------------------------------

### Extract LoCon from Dreambooth Model

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Use this script to extract LoCon from a Dreambooth model along with its base model. Use --help for more detailed options.

```bash
python3 extract_locon.py <settings> <base_model> <db_model> <output>
```

```bash
python3 extract_locon.py --help
```

```bash
usage: extract_locon.py [-h] [--is_v2] [--is_sdxl] [--device DEVICE] [--mode MODE] [--safetensors] [--linear_dim LINEAR_DIM]
                        [--conv_dim CONV_DIM] [--linear_threshold LINEAR_THRESHOLD] [--conv_threshold CONV_THRESHOLD]
                        [--linear_ratio LINEAR_RATIO] [--conv_ratio CONV_RATIO] [--linear_quantile LINEAR_QUANTILE]
                        [--conv_quantile CONV_QUANTILE] [--use_sparse_bias] [--sparsity SPARSITY] [--disable_cp]
                        base_model db_model output_name
```

--------------------------------

### Kohya Wrapper

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Specific wrapper for compatibility with kohya-ss/sd-scripts.

```APIDOC
## Kohya Wrapper

* the specialized wrapper for kohya-ss/sd-scripts.
```

--------------------------------

### LycorisNetwork.apply_to

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Registers adapter modules and patches base model layers to enable adapter forward passes. This method can be called multiple times to stack adapters.

```APIDOC
## LycorisNetwork.apply_to — Activate the adapter on the base model

Registers all adapter modules as submodules of the `LycorisNetwork` and patches the base model's layers so that forward passes run through the adapter. Can be called multiple times with different wrappers to stack adapters.

```python
from lycoris import create_lycoris, LycorisNetwork
import torch.nn as nn

model = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 32))

LycorisNetwork.apply_preset({"target_name": [".*0.*"]})  # first linear
net1 = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="lora")
net1.apply_to()

LycorisNetwork.apply_preset({"target_name": [".*2.*"]})  # second linear
net2 = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="loha")
net2.apply_to()

print(f"net1 modules: {len(net1.loras)}, net2 modules: {len(net2.loras)}")
# net1 modules: 1, net2 modules: 1
```
```

--------------------------------

### Merge LyCORIS Model Back to Checkpoint

Source: https://github.com/kohakublueleaf/lycoris/blob/main/README.md

Merge your trained LyCORIS model back into a base checkpoint model. Use --help for additional parameters.

```bash
python3 merge.py <settings> <base_model> <lycoris_model> <output>
```

```bash
python3 merge.py --help
```

```bash
usage: merge.py [-h] [--is_v2] [--is_sdxl] [--device DEVICE] [--dtype DTYPE] [--weight WEIGHT] base_model lycoris_model output_name
```

--------------------------------

### TOML Configuration for Per-Module Algorithm Assignment

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Use this TOML configuration to specify different algorithms and hyperparameters for various layer types within the same training run. It allows fine-grained control over module-specific settings.

```toml
# example_configs/preset_configs/custom.toml
enable_conv = true

unet_target_module = [
  "Transformer2DModel",
  "ResnetBlock2D",
  "Downsample2D",
  "Upsample2D",
]
unet_target_name = ["conv_in", "conv_out"]

text_encoder_target_module = ["CLIPAttention", "CLIPMLP"]

[module_algo_map]
  [module_algo_map.CrossAttention]  # Full-rank LoKr for attention
    algo    = "lokr"
    dim     = 10000000000000
    factor  = 64

  [module_algo_map.FeedForward]     # Full-rank LoKr for MLP
    algo    = "lokr"
    dim     = 100000000000
    factor  = 12

  [module_algo_map.ResnetBlock2D]   # Low-rank LoRA + tucker for ResBlocks
    algo       = "lora"
    dim        = 4
    alpha      = 1
    use_tucker = true

  [module_algo_map.CLIPAttention]
    algo  = "lora"
    dim   = 8
    alpha = 1

```

--------------------------------

### Permanently bake adapter into base weights

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Adds adapter weight delta directly into base model parameters, eliminating adapter overhead. The `precise` flag uses a stored snapshot to avoid floating-point drift.

```python
import torch, torch.nn as nn
from lycoris import create_lycoris, LycorisNetwork

model = nn.Linear(64, 64)
LycorisNetwork.apply_preset({"target_module": ["Linear"]})
net = create_lycoris(model, 1.0, linear_dim=8, linear_alpha=4.0, algo="loha")
net.apply_to()

# Train net here ...

# Bake weights: standard (fast, small drift over many cycles)
net.restore()
net.merge_to(weight=1.0)

# Bake weights: precise (uses stored snapshot, no numerical drift)
net.merge_to(weight=1.0, precise=True)

# Verify: adapter is now baked, restore() brings back the pre-merge state
model.eval()
x = torch.randn(4, 64)
with torch.no_grad():
    out = model(x)
print(out.shape)  # torch.Size([4, 64])
```

--------------------------------

### Lycoris Module Functions

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Describes the top-level functions for module creation and management, including determining algorithms and constructing modules from weights.

```APIDOC
## Module Functions

* `get_module`: determine the algorithm and extract corresponding weights from state dict.
* `make_module`: based on given algorithm and weights to construct modules.
```

--------------------------------

### Python: Lycoris with bitsandbytes Quantized Layers

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Apply Lycoris adapters to bitsandbytes 4-bit or 8-bit quantized layers using `bypass_mode=True`. This requires loading the model with `load_in_4bit=True` and applying presets for target layers.

```python
from transformers import AutoModelForCausalLM
from lycoris import create_lycoris, LycorisNetwork

# Load a 4-bit quantized LLM
model = AutoModelForCausalLM.from_pretrained(
    "KBlueLeaf/DanTagGen",
    load_in_4bit=True,
)

LycorisNetwork.apply_preset({"target_name": [".*proj.*"]})

lycoris_net = create_lycoris(
    model,
    multiplier=1.0,
    linear_dim=16,
    linear_alpha=2.0,
    algo="lokr",
    factor=8,
    bypass_mode=True,  # required for bnb quantized layers
)
lycoris_net.apply_to()

print(lycoris_net)
# LycorisNetwork with LokrModule adapters on all proj layers

```

--------------------------------

### LycorisBaseModule Class Methods

Source: https://github.com/kohakublueleaf/lycoris/blob/main/docs/API.md

Provides an overview of the class methods available for LycorisBaseModule, used for module parametrization and state management.

```APIDOC
## Class: `LycorisBaseModule`

### Class Methods

* `classmethod parametrize`
* `classmethod algo_check`
* `classmethod extract_state_dict`
* `classmethod make_module_from_state_dict`
```

--------------------------------

### Apply Max Norm Regularization to Clamp Adapter Weight Norms

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Clamps adapter weight norms to a specified maximum value to prevent weight explosion during training. Typically called once per optimizer step. Requires LycorisNetwork to be applied to the model.

```python
import torch, torch.nn as nn
from lycoris import create_lycoris, LycorisNetwork

model = nn.Linear(128, 128)
LycorisNetwork.apply_preset({"target_module": ["Linear"]})
net = create_lycoris(model, 1.0, linear_dim=32, linear_alpha=16.0, algo="lora")
net.apply_to()

optimizer = torch.optim.AdamW(net.parameters(), lr=1e-3)
x = torch.randn(8, 128)

for step in range(3):
    optimizer.zero_grad()
    loss = model(x).pow(2).mean()
    loss.backward()
    optimizer.step()

    # Clamp norms above 1.0 on the target device
    scaled, avg_norm, max_norm = net.apply_max_norm_regularization(
        max_norm_value=1.0, device="cpu"
    )
    print(f"step {step}: scaled={scaled}, avg_norm={avg_norm:.4f}, max_norm={max_norm:.4f}")
```

--------------------------------

### Adjust adapter strength at runtime

Source: https://context7.com/kohakublueleaf/lycoris/llms.txt

Updates the scale factor applied to the adapter's weight delta during the forward pass, enabling dynamic control of adapter contribution without reloading weights.

```python
from lycoris import create_lycoris, LycorisNetwork
import torch, torch.nn as nn

model = nn.Linear(128, 128)
LycorisNetwork.apply_preset({"target_module": ["Linear"]})
net = create_lycoris(model, 1.0, linear_dim=16, linear_alpha=8.0, algo="lokr")
net.apply_to()

x = torch.randn(1, 128)
model.eval()

for strength in [0.0, 0.5, 1.0, 1.5]:
    net.set_multiplier(strength)
    with torch.no_grad():
        out = model(x)
    print(f"multiplier={strength}: output norm={out.norm().item():.4f}")
# multiplier=0.0: output norm=1.2843
# multiplier=0.5: output norm=1.2843
# multiplier=1.0: output norm=1.2843  (init delta is ~0 so no visible change here)
# multiplier=1.5: output norm=1.2843
```