### Setup Data Feeder Pipeline for Training

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Configures the data feeder pipeline for training a sentiment analysis model. This involves extracting tensors, creating one-hot encodings, batching, truncating, and moving data to the GPU.

```swift
var trainData = dataFromDisk(filePath: trainListFile)
// Extract tensors from ImdbText struct.
trainData["tensor"] = trainData["main", ImdbText.self].map(\.tensor)
trainData["mask"] = trainData["main", ImdbText.self].map(\.mask)
trainData["c"] = trainData["main", ImdbText.self].map(\.c)
// Create one hot tensor out of the scalar.
trainData["oneHot"] = trainData["c", Int.self].toOneHot(Float32.self, count: 2)

let deviceCount = DeviceKind.GPUs.count

// Batching tensors together. 
var batchedTrainData = trainData["tensor", "mask", "oneHot"].combine(size: batchSize, repeating: deviceCount)
for i in 0..<deviceCount {
  batchedTrainData["truncTensor_", i] = batchedTrainData["tensor_", i]!.toTruncate(batchedTrainData["mask_", i]!)
  batchedTrainData["squaredMask_", i] = batchedTrainData["mask_", i]!.toOneSquared(maxLength: maxLength)
  // Move the tensors from CPU to GPU.
  let toGPUTrain = batchedTrainData["truncTensor_", i, "oneHot_", i, "squaredMask_", i].toGPU(i)
  batchedTrainData["tensorGPU_", i] = toGPUTrain["truncTensor_", i]
  batchedTrainData["oneHotGPU_", i] = toGPUTrain["oneHot_", i]
  batchedTrainData["squaredMaskGPU_", i] = toGPUTrain["squaredMask_", i]
}
```

--------------------------------

### Configure Adam and SGD Optimizers in Swift

Source: https://context7.com/liuliu/s4nnc/llms.txt

Demonstrates the configuration of Adam and SGD optimizers with Nesterov momentum. Ensure parameters are assigned to the optimizer before starting the training loop. The `adam.step()` call automatically increments the internal step counter.

```swift
import NNC

let graph = DynamicGraph()

// --- Adam optimizer example ---
var adam = AdamOptimizer(
  graph,
  rate: 1e-4,
  betas: (0.9, 0.999),
  decay: 0,
  epsilon: 1e-8
)
ului.parameters = [myModel.parameters]

// --- SGD with Nesterov momentum ---
var sgd = SGDOptimizer(
  graph,
  nesterov: true,
  rate: 1e-3,
  scale: 1,
  decay: 1e-4,
  momentum: 0.9,
  dampening: 0
)
sgd.parameters = [myModel.parameters]

// --- Training loop ---
for step in 0..<1000 {
  let xBatch: DynamicGraph.Tensor<Float32> = graph.variable(.GPU(0), .NC(32, 16))
  xBatch.rand()
  let target: DynamicGraph.Tensor<Float32> = graph.variable(.GPU(0), .NC(32, 2))
  target.full(0)

  let pred = myModel(inputs: xBatch)[0].as(of: Float32.self)
  let lossLayer = MSELoss()
  let loss = lossLayer(pred, target: target)
  loss.backward(to: [xBatch])
  adam.step()
  // adam.step automatically increments internal step counter
}
```

--------------------------------

### Initialize DynamicGraph Tensor Variable

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Initializes a tensor variable within a DynamicGraph, which can be used for computations and automatic differentiation. Setup requires a DynamicGraph instance.

```swift
let graph = DynamicGraph()
let variable: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .HWC(1, 1, 2))
```

--------------------------------

### Create and Manipulate Tensors in Swift

Source: https://context7.com/liuliu/s4nnc/llms.txt

Demonstrates creating CPU and GPU tensors, initializing from arrays, reshaping, permuting, slicing, and checking for NaN values. Requires NNC import.

```swift
import NNC

// Create a CPU Float32 tensor with shape [2, 3] (NCHW format, NC dims)
var cpuTensor = Tensor<Float32>(.CPU, .NC(2, 3))
cpuTensor[0, 0] = 1.0
cpuTensor[0, 1] = 2.0
cpuTensor[0, 2] = 3.0
cpuTensor[1, 0] = 4.0
cpuTensor[1, 1] = 5.0
cpuTensor[1, 2] = 6.0

// Initialize from a Swift array
let fromArray = Tensor<Float32>([Float32](1...6), .CPU, .NC(2, 3))

// Reshape without copying (zero-copy alias)
let reshaped = cpuTensor.reshaped(.C(6))
print(reshaped[0], reshaped[5])  // 1.0  6.0

// Permute dimensions: [2, 3] -> [3, 2]
let permuted = cpuTensor.permuted(1, 0)
print(permuted.shape)  // [3, 2]

// Transfer to GPU 0 (requires CUDA or Metal)
let gpuTensor = cpuTensor.toGPU(0)
let backOnCPU = gpuTensor.toCPU()

// Range slicing
let row0: Tensor<Float32> = cpuTensor[0, 0..<3]

// Check for NaN (useful after training steps)
print(cpuTensor.isNaN)  // false
```

--------------------------------

### Create DataFrame with custom Swift struct mapping

Source: https://context7.com/liuliu/s4nnc/llms.txt

Shows how to initialize a DataFrame to hold custom Swift structs, enabling structured data loading and processing.

```swift
// --- Iterate with custom type mapping ---
struct Sample { var label: Int; var data: Tensor<Float32> }
var typedDf = DataFrame(from: [Sample]())  // custom Swift struct support
```

--------------------------------

### Build DataFrame from filenames and process images

Source: https://context7.com/liuliu/s4nnc/llms.txt

Demonstrates building a DataFrame from image filenames, lazily loading and processing them into tensors. Only the currently iterated column is loaded into memory.

```swift
import NNC

// --- Build a pipeline from filenames ---
var df = DataFrame(from: ["img1.png", "img2.png", "img3.png"])
df["image"] = df["0"]!.toLoadImage()  // decode JPEG/PNG on demand
df["resized"] = df["image"]!.toResizeImage(224, 224)
df["tensor"] = df["resized", Tensor<Float32>.self].toTensor(.NHWC)

for tensor in df["tensor", Tensor<Float32>.self] {
  // Only one image is in memory at a time
  print(tensor.shape)  // [1, 224, 224, 3]
}
```

--------------------------------

### Save model with 4-bit quantization using Store

Source: https://context7.com/liuliu/s4nnc/llms.txt

Demonstrates saving a model to the Store with 4-bit quantization enabled for compact storage. The `.q4p` codec is specified.

```swift
// --- Save with 4-bit quantization ---
graph.openStore("quantized.db", codec: [.q4p]) { store in
  store.write("model_q4", model: myModel)
}
```

--------------------------------

### Save and load models and tensors using Store

Source: https://context7.com/liuliu/s4nnc/llms.txt

Shows how to use DynamicGraph.Store for persisting models and tensors to an SQLite database. Supports loading back into existing variables.

```swift
import NNC

let graph = DynamicGraph()
let myModel = buildModel()

// --- Save model and tensors ---
graph.openStore("checkpoint.db") { store in
  store.write("model_v1", model: myModel)

  let embedding: DynamicGraph.Tensor<Float32> = graph.variable(.CPU, .NC(10000, 512))
  embedding.rand()
  store.write("embedding", variable: embedding)
}

// --- Load back ---
graph.openStore("checkpoint.db") { store in
  store.read("model_v1", model: myModel)

  // Explicit type + device
  let loadedEmb = store.read("embedding", type: Float32.self, format: .NCHW, 
                              shape: [10000, 512], of: .CPU)
  print(loadedEmb?.shape ?? "not found")
}
```

--------------------------------

### Load and Process Images with DataFrame

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Demonstrates efficient image loading and processing using DataFrame. Only one image is loaded into memory at a time, and it's freed when the next is pulled.

```swift
let df = DataFrame(from: [filename1, filename2, filename3])
df["image"] = df["0"]!.toLoadImage()
for tensor in df["image", Tensor<UInt8>.self] {
  print(tensor)
}
```

--------------------------------

### TensorBoard Integration with SummaryWriter

Source: https://context7.com/liuliu/s4nnc/llms.txt

Log scalar metrics, histograms, and images to TensorBoard event files using SummaryWriter. Extend with SummaryWriter+NNC to log tensor values directly. Ensure the log directory exists before writing.

```swift
import NNC
import NNCTensorBoard

let writer = SummaryWriter(logDirectory: "./logs", comment: "run1")

// Log scalars during training
for step in 0..<1000 {
  let trainLoss: Float = computeLoss()
  let valAcc: Float = evaluate()

  writer.addScalar("Loss/train", trainLoss, step: step)
  writer.addScalar("Accuracy/val", valAcc, step: step)
}

// Log a tensor as a histogram
let weightTensor: DynamicGraph.Tensor<Float32> = myModel.parameters
  .detach(.CPU).first?.value as? DynamicGraph.Tensor<Float32> ?? graph.variable(.CPU, .C(1))
writer.addHistogram("weights/layer0", weightTensor, step: 500)

try writer.close()
// Launch: tensorboard --logdir ./logs
```

--------------------------------

### DynamicGraph for Automatic Differentiation in Swift

Source: https://context7.com/liuliu/s4nnc/llms.txt

Shows how to use DynamicGraph for tensor operations and automatic differentiation. Tensors are created as variables or constants within the graph. Requires NNC import.

```swift
import NNC

let graph = DynamicGraph()

// Create typed tensor variables on CPU
let x: DynamicGraph.Tensor<Float32> = graph.variable(.CPU, .C(1))
let y: DynamicGraph.Tensor<Float32> = graph.variable(.CPU, .C(1))
x[0] = 3.0
y[0] = 4.0

// Element-wise operations using Swift operators
let sum = x .+ y          // 7.0
let product = x .* y      // 12.0
let diff = y .- x         // 1.0

// Automatic differentiation
x.requiresGrad = true
let z = x .* x .+ y  // z = x^2 + y
z.backward(to: [x])
let dzdx = DynamicGraph.Tensor<Float32>(x.grad!)[0]
print(dzdx)  // 6.0  (dz/dx = 2x = 6)

// Constants cannot accumulate gradients
let c = graph.constant(Tensor<Float32>([1, 2, 3], .CPU, .C(3)))

// Disable gradient tracking for inference
let result = graph.withNoGrad {
  x .* y
}

// Inspect graph health
let stats = graph.statistics
print("variables: \(stats.variables), computations: \(stats.computations)")

// Control logging
DynamicGraph.logLevel = .error
```

--------------------------------

### Train Model with SGDOptimizer

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Trains a defined model using the SGDOptimizer. This involves setting up the optimizer, defining input and target tensors, calculating loss, performing backpropagation, and updating model parameters with `sgd.step()`.

```swift
let sgd = SGDOptimizer(graph, nesterov: false, rate: 0.0001, scale: 1, decay: 0, momentum: 0.9, dampening: 0)
sgd.parameters = [twoLayerModel.parameters]
for _ 0..<100 {
  let x: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  x[0] = 1
  let target: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  target[0] = 0
  let z = twoLayerModel(inputs: x)
  let binaryLoss = SigmoidBinaryCrossEntropyLoss()
  let loss = binaryLoss(z, target: target)
  loss[0].backward(to: [x])
  sgd.step()
}
```

--------------------------------

### Batch and move DataFrame tensors to GPU

Source: https://context7.com/liuliu/s4nnc/llms.txt

Illustrates batching DataFrame tensors and moving them to the GPU for accelerated processing. The `combine` method is used for batching.

```swift
// --- Batch and move to GPU ---
let batchSize = 32
var batched = df["tensor"].combine(size: batchSize, repeating: 1)
let toGPU = batched["tensor_0"]!.toGPU(0)
batched["tensorGPU"] = toGPU
```

--------------------------------

### Tensor Initialization

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Initializes a raw tensor that resides either on CPU or GPU with a given dimensions. Alternatively, you can initialize a tensor from a native Swift array.

```APIDOC
## Tensor

### Description
Initializes a raw tensor that resides either on CPU or GPU with a given dimensions. Alternatively, you can initialize a tensor from native Swift array. Basic usage looks like this:

### Initializers
```swift
public struct Tensor<Element> {
  init(_ kind: DeviceKind, _ shapeFormat: TensorShapeFormat)
  init<S: Sequence>(_ sequence: S, _ kind: DeviceKind, _ shapeFormat: TensorShapeFormat) where S.Element == Element
}
```

### Usage Example
```swift
var tensor = Tensor<Float>(.CPU, .HWC(1, 1, 2))
tensor[0, 0, 0] = 1
tensor[0, 0, 1] = 2
```

### Functionalities
Limited functionalities are associated with raw tensors. Mostly, you can only `reshaped` or `toGPU` / `toCPU`.
```

--------------------------------

### Load DataFrame from CSV

Source: https://context7.com/liuliu/s4nnc/llms.txt

Shows how to load data from a CSV file into a DataFrame, with support for automatically using the header row.

```swift
// --- Load from CSV (multi-core reader) ---
var csvDf = DataFrame(fromCSV: "data.csv", automaticUseHeader: true)
print(csvDf.columns)  // ["col1", "col2", ...]
```

--------------------------------

### S4NNc Training Loop with Adam Optimizer

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

This Swift code demonstrates the training loop for the S4NNc model using an Adam optimizer. It initializes the graph, variables, and optimizer, then iterates through epochs and batches to train the model. Accuracy is calculated and printed periodically.

```swift
let graph = DynamicGraph()

let vocabVec: DynamicGraph.Group<DynamicGraph.Tensor<Float32>> = DynamicGraph.Group((0..<deviceCount).map { graph.variable(.GPU($0), .NC(vocabSize, embeddingSize)) })
let seqVec: DynamicGraph.Group<DynamicGraph.Tensor<Float32>> = DynamicGraph.Group((0..<deviceCount).map { graph.variable(.GPU($0), .NC(maxLength, embeddingSize)) })
vocabVec.rand(-1...1)
seqVec.rand(-1...1)
var adamOptimizer = AdamOptimizer(graph, rate: 0.0001, betas: (0.9, 0.98), decay: 0, epsilon: 1e-9)
adamOptimizer.parameters = [vocabVec, seqVec, transformer.parameters]
var overallAccuracy = 0.0
for epoch in 0..<10 {
  batchedTrainData.shuffle()
  var columns = [String]()
  for i in 0..<deviceCount {
    columns += ["tensorGPU_\(i)", "oneHotGPU_\(i)", "squaredMaskGPU_\(i)"]
  }
  let computeStream = StreamContext(.GPU(0))
  for (i, batch) in batchedTrainData[columns].enumerated() {
    adamOptimizer.rate = 0.0001 * min(Float(adamOptimizer.step - 1) / (10000.0 / Float(batchSize)), 1) * Float(deviceCount)
    let tensorGPU = (0..<deviceCount).map { batch[$0 * 3] as! Tensor<Int32> }
    let oneHotGPU = (0..<deviceCount).map { batch[$0 * 3 + 1] as! Tensor<Float32> }
    let squaredMaskGPU = (0..<deviceCount).map { batch[$0 * 3 + 2] as! Tensor<Int32> }
    let batchLength = tensorGPU[0].dimensions[1]
    let output = graph.withStream(computeStream) { () -> DynamicGraph.Group<DynamicGraph.AnyTensor> in
      let wordIndices = graph.variable(tensorGPU.reshaped(.C(batchSize * batchLength)))
      let wordVec = Functional.indexSelect(input: vocabVec, index: wordIndices)
      var seqIndicesCPU = Tensor<Int32>(.CPU, .C(batchSize * batchLength))
      for i in 0..<batchSize {
        for j in 0..<batchLength {
          seqIndicesCPU[i * batchLength + j] = Int32(j)
        }
      }
      let seqIndicesGPU = (0..<deviceCount).map { seqIndicesCPU.toGPU($0) }
      let seqIndices = graph.constant(seqIndicesGPU)
      let posVec = Functional.indexSelect(input: seqVec, index: seqIndices)
      let selectVec = wordVec + posVec
      let inputVec = selectVec.reshaped(.CHW(batchSize, batchLength, embeddingSize))
      let masked = graph.constant(squaredMaskGPU.reshaped(.CHW(batchSize, batchLength, batchLength)))
      let output = transformer(inputs: inputVec, masked)[0]
      let softmaxLoss = SoftmaxCrossEntropyLoss()
      let target = graph.variable(oneHotGPU)
      let loss = softmaxLoss(output, target: target)
      loss.backward(to: [vocabVec, seqVec])
      adamOptimizer.step()
      return output
    }
    computeStream.joined()
    var correct = 0
    for k in 0..<deviceCount {
      let oneHot = oneHotGPU[k].toCPU()
      let output = DynamicGraph.Tensor<Float32>(output[k]).toCPU()
      for i in 0..<batchSize {
        let truth = oneHot[i, 1] > oneHot[i, 0]
        let prediction = output[i, 1] > output[i, 0]
        if truth == prediction {
          correct += 1
        }
      }
    }
    let accuracy = Double(correct) / Double(batchSize * deviceCount)
    overallAccuracy = overallAccuracy * 0.9 + accuracy * 0.1
    if adamOptimizer.step % 50  == 0 {
      print("epoch \(epoch) (\(i)/\(batchedTrainData.count)), training accuracy \(overallAccuracy)")
    }
  }
}
```

--------------------------------

### Store and Retrieve Tensors and Models with SQLite

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Shows how to use s4nnc's SQLite-based storage for key-value persistence of tensors, variables, and models.

```swift
graph.openStore("filePath") { store in
  let aTensor = store.read("a")
  store.write("b", variable: z)
  store.write("2layer", model: twoLayerModel)
}
```

--------------------------------

### Load tensor into existing variable with Store

Source: https://context7.com/liuliu/s4nnc/llms.txt

Illustrates loading a tensor from the Store directly into a pre-allocated variable, which can be more efficient than creating a new tensor.

```swift
// --- Read tensor into existing variable (avoids allocation) ---
graph.openStore("checkpoint.db") { store in
  let target: DynamicGraph.Tensor<Float32> = graph.variable(.GPU(0), .NC(10000, 512))
  store.read("embedding", variable: target, codec: [.zip])
}
```

--------------------------------

### Shuffle DataFrame batches

Source: https://context7.com/liuliu/s4nnc/llms.txt

Demonstrates shuffling the order of batches within a DataFrame, useful for randomizing data order between training epochs.

```swift
// --- Shuffle between epochs ---
batched.shuffle()
```

--------------------------------

### Initialize Raw Tensor in Swift

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Initializes a raw tensor on CPU or GPU with specified dimensions. Can also be initialized from a native Swift sequence.

```swift
public struct Tensor<Element> {
  init(_ kind: DeviceKind, _ shapeFormat: TensorShapeFormat)
  init<S: Sequence>(_ sequence: S, _ kind: DeviceKind, _ shapeFormat: TensorShapeFormat) where S.Element == Element
}
```

```swift
var tensor = Tensor<Float>(.CPU, .HWC(1, 1, 2))
tensor[0, 0, 0] = 1
tensor[0, 0, 1] = 2
```

--------------------------------

### Model Definition and Training

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

s4nnc provides stateful `Model` that contains trainable parameters, allowing for complex computation units and training. Models can be trained using optimizers.

```APIDOC
## Model and Optimizer

### Description
Computations on `DynamicGraph` with tensor variables are stateless. s4nnc also provided stateful `Model` that contains trainable parameters. You can use `Model` to construct complex computation unit and train them.

### Model Definition Example
```swift
func TwoLayerLinearModel() {
  let x = Input()
  let y = Dense(count: 2)(x)
  let z = Dense(count: 1)(y)
  return Model([x], [z])
}

let twoLayerModel = TwoLayerLinearModel()
let z = twoLayerModel(inputs: x)
print(z)
```

### Training Example
You can train the model with optimizers.

```swift
let sgd = SGDOptimizer(graph, nesterov: false, rate: 0.0001, scale: 1, decay: 0, momentum: 0.9, dampening: 0)
sgd.parameters = [twoLayerModel.parameters]
for _ 0..<100 {
  let x: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  x[0] = 1
  let target: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  target[0] = 0
  let z = twoLayerModel(inputs: x)
  let binaryLoss = SigmoidBinaryCrossEntropyLoss()
  let loss = binaryLoss(z, target: target)
  loss[0].backward(to: [x])
  sgd.step()
}
```

### Recommendation
Because `Model` can express complex computations statically, it is recommended to have most of your computations expressed as `Model`.
```

--------------------------------

### DynamicGraph Tensor Initialization and Operations

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

DynamicGraph is where most computations are associated with tensors. It operates on tensor variables/constants, not raw tensors. Tensor variables can participate in computations and automatic differentiation.

```APIDOC
## DynamicGraph

### Description
`DynamicGraph` is where you associate most computations with tensors. The `DynamicGraph` operates on tensor variables / constants, not the raw tensors. Initializing a tensor variable / constant is very similar to initializing a raw tensor.

### Initialization Example
```swift
let graph = DynamicGraph()
let variable: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .HWC(1, 1, 2))
```

### Computation Example
A tensor variable can participate computations, for example:

```swift
let x: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
let y: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
x[0] = 2
y[0] = -1
let z = x .* y
print(z[0])
```

### Automatic Differentiation Example
Because these are tensor variables, you can also do automatic differentiation:

```swift
x.requiresGrad = true
z.backward(to: [x])
print(DynamicGraph.Tensor<Float>(x.grad!)[0])
```

### Memory Management
Tensor variables memory management is automatic. If there is no reference to it (as defined by no automatic differentiation requires the given tensor variable's participation), the memory will be freed. Hence, unlike PyTorch, you don't need to worry about `no_grad` annotation most of the time.
```

--------------------------------

### Build dynamic transformer model with ModelBuilder

Source: https://context7.com/liuliu/s4nnc/llms.txt

Uses ModelBuilder to create a transformer model whose architecture adapts to input sequence length. Parameters are shared across different sequence lengths.

```swift
import NNC

let graph = DynamicGraph()

// Build a transformer that adapts to sequence length
let transformer = ModelBuilder { (seqLen: Int, inputs: [DynamicGraph_Any]) -> Model in
  let x = Input()
  let attn = SelfAttention(heads: 8, embeddingSize: 512, querySize: 64, outputSize: 512)(x)
  let norm = LayerNorm(epsilon: 1e-5, axis: [2])(attn)
  return Model([x], [norm], name: "transformer_\(seqLen)")
}

// First call builds the model for seqLen=128
let input128: DynamicGraph.Tensor<Float32> = graph.variable(.GPU(0), .CHW(4, 128, 512))
let out128 = transformer(inputs: input128)[0]

// Second call reuses parameters but adapts to seqLen=256
let input256: DynamicGraph.Tensor<Float32> = graph.variable(.GPU(0), .CHW(4, 256, 512))
let out256 = transformer(inputs: input256)[0]
```

--------------------------------

### Define a Two-Layer MLP Model in Swift

Source: https://context7.com/liuliu/s4nnc/llms.txt

Defines a reusable two-layer Multi-Layer Perceptron model. Supports features like gradient checkpointing, parameter filtering, model copying, and sequential composition. Compile the model before loading parameters from a store.

```swift
import NNC

// --- Define a two-layer MLP ---
func TwoLayerMLP(hiddenSize: Int, outputSize: Int) -> Model {
  let x = Input()
  let h = Dense(count: hiddenSize)(x)
  let hActivated = ReLU()(h)
  let out = Dense(count: outputSize)(hActivated)
  return Model([x], [out])
}

let graph = DynamicGraph()
let mlp = TwoLayerMLP(hiddenSize: 64, outputSize: 2)

// Forward pass
let inputVar: DynamicGraph.Tensor<Float32> = graph.variable(.CPU, .NC(4, 16))
inputVar.rand()
let output = mlp(inputs: inputVar)[0].as(of: Float32.self)
print(output.shape)  // [4, 2]

// Enable gradient checkpointing (trades compute for memory)
mlp.gradientCheckpointing = true

// Access / filter parameters by name
let weights = mlp.parameters.filter { $0.contains("weight") }

// Copy model structure (without parameters)
let mlpCopy = mlp.copied()

// Sequential model composition
let encoderDecoder = Model([encoder, decoder])

// Compile without running (for parameter pre-loading)
mlp.compile(inputs: inputVar)

// Load from store after compiling
graph.openStore("checkpoint.db") {
  store.read("mlp", model: mlp)
}
```

--------------------------------

### Group Tensor Variables for Parallel Computation

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Illustrates grouping multiple tensor variables for simultaneous computation, which can be beneficial for data parallelism across multiple GPUs.

```swift
let xGroup = DynamicGraph.Group(x0, x1)
let yGroup = DynamicGraph.Group(y0, y1)
let zGroup = xGroup .* yGroup
```

--------------------------------

### Asynchronous GPU Computation with StreamContext

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Utilizes StreamContext to explicitly manage asynchronous streams for GPU computation, improving efficiency. The computation is only guaranteed to be available after `joined()` is called.

```swift
let computeStream = StreamContext(.GPU(0))
var z: DynamicGraph.Tensor<Float>? = nil
graph.withStream(computeStream) {
  let x: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  let y: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
  x[0] = 2
  y[0] = -1
  z = x .+ y
}
computeStream.joined()
print(z[0]) // Result only available after joined the computeStream
```

--------------------------------

### Automatic Differentiation with AutoGrad

Source: https://context7.com/liuliu/s4nnc/llms.txt

Trigger reverse-mode automatic differentiation using the .backward(to:) method on a loss tensor. Gradients accumulate in the .grad property of source variables. Multiple calls to .backward sum gradients for accumulation before optimizer steps.

```swift
import NNC

let graph = DynamicGraph()

// Simple quadratic: loss = sum((w * x - y)^2)
let w: DynamicGraph.Tensor<Float32> = graph.variable(.CPU, .C(4))
w.full(1.0)
w.requiresGrad = true

let x: DynamicGraph.Tensor<Float32> = graph.constant(
  Tensor<Float32>([1, 2, 3, 4], .CPU, .C(4))
)
let y: DynamicGraph.Tensor<Float32> = graph.constant(
  Tensor<Float32>([2, 4, 6, 8], .CPU, .C(4))
)

let pred = w .* x
let diff = pred .- y
let loss = (diff .* diff).reduced(.sum, axis: [0])

// Backward: compute dL/dw
loss.backward(to: [w])
let grad = DynamicGraph.Tensor<Float32>(w.grad!)
print([grad[0], grad[1], grad[2], grad[3]])
// [-2.0, -8.0, -18.0, -32.0]  (gradient = -2*(y-w*x)*x)

// Gradient accumulation (call backward multiple times before step)
for _ in 0..<4 {
  let miniLoss = computeMiniBatchLoss()
  miniLoss.backward(to: [w])
}
var sgd = SGDOptimizer(graph, nesterov: false, rate: 0.01, scale: 0.25,
                       decay: 0, momentum: 0, dampening: 0)
sgd.parameters = [w]
sgd.step()  // applies accumulated gradients, then zeroes them
```

--------------------------------

### Implement Mixed-Precision Training with GradScaler

Source: https://context7.com/liuliu/s4nnc/llms.txt

Utilizes GradScaler for automatic mixed-precision training to prevent gradient underflow with Float16. The scaler dynamically adjusts its scale factor, backing off on NaN gradients and growing on clean steps. The `scaler.step()` method handles gradient updates and scale adjustments.

```swift
import NNC

let graph = DynamicGraph()
var scaler = GradScaler(
  scale: 65536,
  growthFactor: 2,
  backoffFactor: 0.5,
  growthInterval: 2_000
)
var adam = AdamOptimizer(graph, rate: 1e-4)
adam.parameters = [myModel.parameters]
var optimizers = [adam]

for _ in 0..<steps {
  let loss = computeLoss()       // returns DynamicGraph.Tensor<Float16>
  let scaledLoss = scaler.scale(loss)
  scaledLoss.backward(to: myModel.parameters)
  scaler.step(&optimizers)       // skips update and backs off scale on NaN
}
print("current scale: \(scaler.scale)")
```

--------------------------------

### Multi-GPU Data Parallelism with DynamicGraph.Group

Source: https://context7.com/liuliu/s4nnc/llms.txt

Use DynamicGraph.Group to hold synchronized tensor variables across multiple GPUs for element-wise operations. Assign Group parameters to an optimizer for transparent data parallelism.

```swift
import NNC

let graph = DynamicGraph()
let deviceCount = DeviceKind.GPUs.count  // e.g. 4

// Create synchronized variables across all GPUs
let vocabEmbed = DynamicGraph.Group(
  (0..<deviceCount).map { graph.variable(.GPU($0), .NC(32000, 512)) }
)
vocabEmbed.rand(-0.02...0.02)

// Adam optimizer with grouped parameters
var adam = AdamOptimizer(graph, rate: 1e-4)
adam.parameters = [vocabEmbed, myTransformer.parameters]

// Batched inputs per GPU (one sub-batch per device)
let batchPerGPU = 8
let inputGroup = DynamicGraph.Group(
  (0..<deviceCount).map { gpu -> DynamicGraph.Tensor<Int32> in
    let t = graph.variable(.GPU(gpu), .C(batchPerGPU))
    // fill with token ids...
    return t
  }
)

// indexSelect operates on all GPUs simultaneously
let wordVecs = Functional.indexSelect(input: vocabEmbed, index: inputGroup)
let loss = computeLoss(wordVecs)
loss.backward(to: [vocabEmbed])
adam.step()
```

--------------------------------

### Define Two-Layer Linear Model

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Defines a simple two-layer linear model using s4nnc's `Model` and `Dense` layers. Input and output tensors are represented by `Input()` and `Dense` respectively.

```swift
func TwoLayerLinearModel() {
  let x = Input()
  let y = Dense(count: 2)(x)
  let z = Dense(count: 1)(y)
  return Model([x], [z])
}

let twoLayerModel = TwoLayerLinearModel()
let z = twoLayerModel(inputs: x)
print(z)
```

--------------------------------

### Automatic Differentiation with DynamicGraph

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Enables automatic differentiation for a tensor variable by setting `requiresGrad` to true. The gradient can then be computed and accessed via the `grad` property after calling `backward()`.

```swift
x.requiresGrad = true
z.backward(to: [x])
print(DynamicGraph.Tensor<Float>(x.grad!)[0])
```

--------------------------------

### DynamicGraph Tensor Computation

Source: https://github.com/liuliu/s4nnc/blob/main/README.md

Performs element-wise multiplication between two tensor variables in a DynamicGraph. The result is a new tensor variable.

```swift
let x: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
let y: DynamicGraph.Tensor<Float> = graph.variable(.CPU, .C(1))
x[0] = 2
y[0] = -1
let z = x .* y
print(z[0])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.