### Import ParameterSchedulers

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Imports the ParameterSchedulers library, which is necessary for using any of its scheduling functionalities.

```julia
using ParameterSchedulers
```

--------------------------------

### Use Stateful Schedules with next!

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Introduces the Stateful wrapper to create mutable schedules. The `next!` function advances the schedule's internal state and returns the next parameter value.

```julia
using ParameterSchedulers: Stateful, next!
s = Exp(start = 0.1, decay = 0.8)
stateful_s = Stateful(s)
println("s: ", next!(stateful_s))
println("s: ", next!(stateful_s))
println(stateful_s)
```

--------------------------------

### ParameterSchedulers.jl Decay Schedule Setup

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/basic-schedules.md

Initializes the ParameterSchedulers.jl library for working with decay schedules. This is a common setup step before defining or using decay-based scheduling functions.

```Julia
using ParameterSchedulers # hide
```

--------------------------------

### Reset Stateful Schedule State

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Demonstrates how to reset the internal iteration state of a Stateful schedule using the `reset!` function, allowing it to be iterated from the beginning again.

```julia
using ParameterSchedulers: Stateful, next!, reset!
s = Exp(start = 0.1, decay = 0.8)
stateful_s = Stateful(s)
reset!(stateful_s)
println("s: ", next!(stateful_s))
```

--------------------------------

### Attempt to Call Stateful Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Illustrates that Stateful schedules, designed for mutable iteration via `next!`, cannot be directly called like their immutable counterparts. This will result in an error.

```julia
using ParameterSchedulers: Stateful
s = Exp(start = 0.1, decay = 0.8)
stateful_s = Stateful(s)
try
    stateful_s(1)
catch e
    println(e)
end
```

--------------------------------

### Iterate with an Exponential Decay Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Shows how to use a schedule within a for-loop to iterate through parameter values. Be cautious with infinite iterators like Exp, as they can lead to infinite loops.

```julia
s = Exp(start = 0.1, decay = 0.8)
for (i, param) in enumerate(s)
    (i > 10) && break
    println("s($i): ", param)
end
```

--------------------------------

### ParameterSchedulers.jl Cyclic Schedule Setup

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/basic-schedules.md

Initializes the ParameterSchedulers.jl library for working with cyclic schedules. This setup is necessary before defining or utilizing cyclic scheduling mechanisms.

```Julia
using ParameterSchedulers #hide
```

--------------------------------

### Call an Exponential Decay Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/getting-started.md

Demonstrates creating an exponential decay schedule and calling it at specific iterations to retrieve parameter values. Schedules are immutable by default.

```julia
s = Exp(start = 0.1, decay = 0.8)
println("s(1): ", s(1))
println("s(5): ", s(5))
```

--------------------------------

### Triangular Wave Cyclic Schedule Example

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/basic-schedules.md

Illustrates the triangular wave cyclic schedule. It includes a custom `tricycle` function and shows how to instantiate and verify the `Triangle` schedule type from ParameterSchedulers.jl.

```Julia
tricycle(period, t) = (2 / π) * abs(asin(sin(π * (t - 1) / period)))
s = Triangle(l0 = 0.1, l1 = 0.4, period = 2)
println(
    "abs(l0 - l1) * g(1) + min(l0, l1) == s(1): ",
    abs(0.1 - 0.4) * tricycle(2, 1) + min(0.1, 0.4) == s(1)
)
```

--------------------------------

### Shifted Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/warmup-schedules.md

Shows how to use the `Shifted` wrapper to adjust the starting point of a schedule. This is useful when the subsequent schedule needs to begin at a specific phase or value, such as starting a `Triangle` schedule at its peak after a sine warm-up.

```Julia
using ParameterSchedulers
using UnicodePlots

min_lr = 1e-6
initial_lr = 1e-2
warmup = 20
total_iters = 100

# shift the Triangle by half a period + 1 to start at the peak
tri = Shifted(Triangle(l0 = min_lr, l1 = initial_lr, period = 10), 6)
s = WarmupSin(min_lr, initial_lr, warmup, total_iters, tri)
t = 1:50 |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### Exponential Decay Schedule Example

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/basic-schedules.md

Demonstrates the usage of the exponential decay schedule. It defines a custom exponential decay function and verifies its output against the built-in `Exp` schedule type from ParameterSchedulers.jl.

```Julia
expdecay(decay, t) = decay^(t - 1)
s = Exp(start = 0.1, decay = 0.8)
println("l g(1) == s(1): ", 0.1 * expdecay(0.8, 1) == s(1))
```

--------------------------------

### Custom WarmupLinear Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/warmup-schedules.md

Defines a custom `WarmupLinear` constructor using `Sequence` to combine a linear ramp (`Triangle`) for warm-up with a subsequent schedule (e.g., `Exp`). This allows for a complete learning rate strategy from start to finish.

```Julia
using ParameterSchedulers
using UnicodePlots

min_lr = 1e-6
initial_lr = 1e-2
warmup = 20
total_iters = 100

WarmupLinear(startlr, initlr, warmup, total_iters, schedule) =
    Sequence(Triangle(l0 = startlr, l1 = initlr, period = 2 * warmup) => warmup,
             schedule => total_iters)

s = WarmupLinear(min_lr, initial_lr, warmup, total_iters, Exp(initial_lr, 0.8))
t = 1:total_iters |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### Composing Schedules: SinDecay10 Example

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Illustrates creating a custom schedule (`SinDecay10`) by composing existing schedules using `ComposedSchedule`. This example shows how to decay the amplitude of a `Sin` schedule exponentially over time.

```julia
function SinDecay10(range, offset, period)
    parameters = (Step(range, 0.1, period), offset, period)
    ComposedSchedule(Sin(range, offset, period), parameters)
end

SinDecay10(0.5, 0.1, 5)
```

--------------------------------

### Create Step Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Creates an exponential decay schedule using ParameterSchedulers.jl. This schedule decays by a factor `decay` at specified `step_sizes`. It takes `start` value, `decay` factor, and `step_sizes` as input.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Step(start = 1.0, decay = 0.8, step_sizes = [2, 3, 2]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### LinearLR to Sequence (with Triangle)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's LinearLR, which linearly adjusts the learning rate from a start factor to an end factor over total iterations, to ParameterSchedulers.jl's `Sequence` with a `Triangle` schedule.

```Julia
Sequence(Triangle(lr * start_factor, lr * end_factor, 2 * total_iters) => total_iters, lr => nepochs)
```

```Python
LinearLR(_, start_factor, end_factor, total_iters)
```

--------------------------------

### Create Exponential Decay Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements an exponential decay schedule where the value decreases by a `decay` factor at each iteration. It requires a `start` value and the `decay` rate.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Exp(start = 1.0, decay = 0.5) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### Interpolating Schedules with Time Step

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Demonstrates using `Interpolator` to map iteration states to a continuous time scale, useful for scenarios like ODE solvers where the time step (`dt`) is a factor. This example sets a parameter for the first half of a simulation duration and a different one for the second half.

```Julia
using ParameterSchedulers # hide

dt = 1e-3 # simulation time step in seconds
T = 2 # simulate 2 seconds
# our parameter is 1e-2 for the first half of the simulation
# then it drops to 1e-3 for the second half of the simulation
# we interpolate at a rate of dt
s = Interpolator(Sequence(1e-2 => cld(T, dt) / 2, 1e-3 => cld(T, dt) / 2), dt)
```

--------------------------------

### Create Inverse Decay Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Applies an inverse decay schedule based on the formula `(1 + t * decay)^degree`. This schedule provides a flexible way to control the rate of decay over time. It requires `start`, `decay` rate, and `degree`.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Inv(start = 1.0, decay = 0.1, degree = -0.5) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### Create Polynomial Decay Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a polynomial decay schedule where the parameter decreases according to a polynomial function of degree `degree`. It requires a `start` value, the `degree` of the polynomial, and the `max_iter` (total iterations).

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Poly(start = 1.0, degree = 2, max_iter = t[end]) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### ComposedSchedule with Custom Composition Function

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Demonstrates using the three-argument form of `ComposedSchedule` to provide a custom composition function (`compose_fn`). This allows for more control over how schedule parameters are updated, as shown in this example where the `Sin` schedule is rebuilt with new parameter values.

```julia
function SinDecay10(range, offset, period)
    parameters = (Step(range, 0.1, period), offset, period)
    ComposedSchedule(Sin(range, offset, period), parameters) do schedule, parameter_values
        @show schedule
        @show parameter_values
        Sin(parameter_values...)
    end
end
```

--------------------------------

### Implement Lambda Schedule in Julia

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/interface.md

Demonstrates implementing a custom `Lambda` schedule by wrapping a function `f`. The schedule value at iteration `t` is `f(t)`. This example shows defining the struct, the callable interface `(schedule::Lambda)(t)`, and the iteration interface `Base.iterate` and `Base.IteratorEltype`.

```julia
using ParameterSchedulers
using ParameterSchedulers: AbstractSchedule

struct Lambda{T} <: AbstractSchedule{missing}
    f::T
end

(schedule::Lambda)(t) = schedule.f(t)

Base.iterate(schedule::Lambda, t = 1) = schedule(t), t + 1

# since the eltype is unknown, we indicate it
Base.IteratorEltype(::Type{<:Lambda}) = Base.EltypeUnknown()
```

--------------------------------

### Sine Warm-up Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/warmup-schedules.md

Demonstrates creating a sine wave warm-up schedule using the `Sin` schedule combined with `Sequence`. Similar to the linear ramp, it ramps up the learning rate over a specified warm-up period before transitioning to another schedule.

```Julia
using ParameterSchedulers
using UnicodePlots

WarmupSin(startlr, initlr, warmup, total_iters, schedule) =
    Sequence(Sin(l0 = startlr, l1 = initlr, period = 2 * warmup) => warmup,
             schedule => total_iters)

s = WarmupSin(min_lr, initial_lr, warmup, total_iters, Exp(initial_lr, 0.8))
t = 1:total_iters |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### OneCycle Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/warmup-schedules.md

Demonstrates the use of the `OneCycle` convenience constructor for a one-cycle cosine learning rate schedule. It shows how to create the schedule and plot its output over a specified number of steps.

```Julia
using ParameterSchedulers
using UnicodePlots

nsteps = 10
maxval = 1f-1
onecycle = OneCycle(10, 1f-1; percent_start = 0.4)

t = 1:nsteps |> collect
lineplot(t, onecycle.(t); border = :none)
```

--------------------------------

### Linear Warm-up Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/warmup-schedules.md

Illustrates creating a linear warm-up schedule by using half a period of the `Triangle` schedule. This approach ramps the learning rate from a minimum value to an initial maximum value over a specified number of warmup iterations.

```Julia
using ParameterSchedulers
using UnicodePlots

min_lr = 1e-6 # don't actually start with lr = 0
initial_lr = 1e-2
warmup = 20 # warmup for 20 epochs

ramp = Triangle(l0 = min_lr, l1 = initial_lr, period = 2 * warmup)

t = 1:warmup |> collect
lineplot(t, ramp.(t); border = :none)
```

--------------------------------

### Passing Scheduler to Flux.train! with Flux optimizers

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/optimizers.md

Demonstrates passing a `ParameterSchedulers.Scheduler` to `Flux.train!` for simplified training loop management.

```Julia
s = Inv(start = 1e-1, degree = 2, decay = 0.2)
opt = Scheduler(Descent, s)
opt_st = Flux.setup(opt, m)
loss(m, x, y) = Flux.mse(m(x), y)
for epoch in 1:nepochs
    sched_step = opt_st.layers[1].weight.state.t
    println("epoch: $epoch, sched state: $sched_step")
    Flux.train!(loss, m, data, opt_st)
end
```

--------------------------------

### Looping Arbitrary Function Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Shows how to use the `Loop` constructor with an arbitrary function (e.g., `log`) to create a repeating schedule. The resulting schedule is plotted.

```Julia
using ParameterSchedulers # hide
using UnicodePlots

s = Loop(log, 10)
t = 1:25 |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### ML Use Case: Training with Parameter Schedules

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Demonstrates integrating ParameterSchedulers with Flux and Optimisers for a machine learning task. It shows how to set up a scheduler with a model and iterate through training epochs and mini-batches, logging the scheduler's step.

```julia
using Flux
using Optimisers
using ParameterSchedulers: Scheduler

nepochs = 3
data = [(Flux.rand32(4, 10), rand([-1, 1], 1, 10)) for _ in 1:3]
m = Chain(Dense(4, 4, tanh), Dense(4, 1, tanh))
s = Interpolator(Sequence(1f-2 => 1, Exp(1f-2, 2f0) => 2), length(data))
opt = Scheduler(Optimisers.Descent, s)
opt_st = Flux.setup(opt, m)
for epoch in 1:nepochs
    for (i, (x, y)) in enumerate(data)
        global opt_st, m
        step = opt_st.layers[1].weight.state.t
        println("epoch: $epoch, batch: $i, sched step = $step")
        g = Flux.gradient(m -> Flux.mse(m(x), y), m)[1]
        opt_st, m = Flux.update!(opt_st, m, g)
    end
end
```

--------------------------------

### CyclicLR (triangular) to Triangle

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's CyclicLR with a triangular base learning rate policy to ParameterSchedulers.jl's `Triangle` scheduler.

```Julia
Triangle(base_lr, max_lr, step_size)
```

```Python
CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular', _, None)
```

--------------------------------

### Using ParameterSchedulers with Flux.jl Optimizer

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

This snippet demonstrates how to integrate ParameterSchedulers.jl with Flux.jl optimizers. It shows the creation of an optimizer using the `Scheduler` function, pairing a `Momentum` optimizer with an `Exp` learning rate decay schedule.

```julia
using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Momentum, Exp(start = 1e-2, decay = 0.8))
```

--------------------------------

### Using Scheduler with Flux optimizers

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/optimizers.md

Wraps a schedule with `ParameterSchedulers.Scheduler` to integrate seamlessly with Flux optimizers, simplifying parameter adjustment during training.

```Julia
using ParameterSchedulers: Scheduler

nepochs = 3
s = Inv(start = 1e-1, degree = 2, decay = 0.2)
opt = Scheduler(Descent, s)
opt_st = Flux.setup(opt, m)
for epoch in 1:nepochs
    for (i, (x, y)) in enumerate(data)
        global opt_st, m
        sched_step = opt_st.layers[1].weight.state.t
        println("epoch: $epoch, batch: $i, sched state: $sched_step")
        g = Flux.gradient(m -> Flux.mse(m(x), y), m)[1]
        opt_st, m = Flux.update!(opt_st, m, g)
    end
end
```

--------------------------------

### Sequence with Manual Steps

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Demonstrates using `Sequence` to define a schedule by manually specifying parameter values and the number of steps for each value. The resulting schedule is plotted.

```Julia
using ParameterSchedulers # hide
using UnicodePlots

s = Sequence(1e-1 => 5, 5e-2 => 4, 3.4e-3 => 10)
t = 1:20 |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### Looping Exponential Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Demonstrates creating a schedule that repeats an exponential decay pattern every 10 iterations. The schedule is then visualized using UnicodePlots.

```Julia
using ParameterSchedulers # hide
using UnicodePlots

s = Loop(Exp(start = 0.1, decay = 0.4), 10)
t = 1:25 |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### Epoch-based learning rate adjustment with Flux

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/optimizers.md

Adjusts the learning rate on an epoch basis by zipping a schedule with epochs and iterating through batches within each epoch.

```Julia
nepochs = 6
s = Step(start = 1e-1, decay = 0.2, step_sizes = [3, 2, 1])
for (eta, epoch) in zip(s, 1:nepochs)
    global opt_st
    adjust!(opt_st, eta)
    for (i, (x, y)) in enumerate(data)
        global m
        g = Flux.gradient(m -> Flux.mse(m(x), y), m)[1]
        opt_st, m = Flux.update!(opt_st, m, g)
        println("epoch: $epoch, batch: $i, opt state: $(opt_st.layers[1].weight.rule)")
    end
end
```

--------------------------------

### Utility Functions Documentation

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/general.md

Documentation for utility functions within the ParameterSchedulers package, specifically those located in the utils.jl file. This section covers helper functions that support the core scheduling functionalities.

```APIDOC
ParameterSchedulers.utils.jl
  - Documentation for utility functions in the ParameterSchedulers package.
  - Covers helper functions that do not belong to the core Scheduler or Stateful interfaces.
  - Examples might include functions for creating, manipulating, or inspecting schedulers and their states.
```

--------------------------------

### Sequence of Schedules

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Illustrates creating a schedule that transitions between different schedule types (`Triangle` and `Exp`) over specified intervals. The combined schedule is then plotted.

```Julia
using ParameterSchedulers # hide
using UnicodePlots

nepochs = 50
s = Sequence([Triangle(l0 = 0.0, l1 = 0.5, period = 5), Exp(start = 0.5, decay = 0.5)],
             [nepochs ÷ 2, nepochs ÷ 2])

t = 1:nepochs |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### LambdaLR / MultiplicativeLR to lr_lambda

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's LambdaLR and MultiplicativeLR, which apply a multiplicative factor to the learning rate based on a lambda function, to ParameterSchedulers.jl's `lr_lambda`.

```Julia
lr_lambda
```

```Python
LambdaLR(_, lr_lambda)
MultiplicativeLR(_, lr_lambda)
```

--------------------------------

### CosineAnnealingLR to CosAnneal

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Translates PyTorch's CosineAnnealingLR and Tensorflow's CosineDecay to ParameterSchedulers.jl's `CosAnneal`, implementing a cosine annealing schedule for the learning rate.

```Julia
CosAnneal(lr, eta_min, T_0, false)
```

```Python
CosineAnnealingLR(_, T_max, eta_min)
CosineDecay(lr, T_max, eta_min)
```

--------------------------------

### Sequence with Generator

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Shows how to construct a `Sequence` using a generator expression for the schedules, allowing for dynamic schedule generation. The output is then plotted.

```Julia
using ParameterSchedulers # hide
using UnicodePlots

s = Sequence(2 / t for t in 1:10)
t = 1:50 |> collect
lineplot(t, s.(t); border = :none)
```

--------------------------------

### MultiStepLR to Step (with milestones)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's MultiStepLR, which reduces the learning rate at specific milestones, to ParameterSchedulers.jl's `Step` scheduler using a list of milestones.

```Julia
Step(lr, gamma, milestones)
```

```Python
MultiStepLR(_, milestones, gamma)
```

--------------------------------

### CyclicLR (arbitrary scaling) to Arbitrary looping schedules

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

References ParameterSchedulers.jl's support for arbitrary looping schedules, corresponding to PyTorch's CyclicLR with custom scaling functions.

```Julia
See [Arbitrary looping schedules](@ref)
```

```Python
CyclicLR(_, base_lr, max_lr, step_size, step_size, _, _, scale_fn)
```

--------------------------------

### Create TriangleExp Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Creates a triangle wave schedule with an exponential decay applied to its amplitude. The amplitude decreases at a rate specified by `decay` over each `period`. It requires `l0`, `l1`, `period`, and `decay`.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### Stateful Operations Documentation

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/general.md

Documentation for stateful operations within ParameterSchedulers, including the Stateful type and the next! and reset! functions. These are crucial for managing the progression and resetting of scheduling states.

```APIDOC
ParameterSchedulers.Stateful
  - Documentation for the Stateful type.
  - Represents schedulers that maintain an internal state.
  - Likely defines the interface for state management.

ParameterSchedulers.next!(scheduler, args...)
  - Advances the scheduler to its next state.
  - Modifies the scheduler in-place.
  - Returns the next state or value.
  - Parameters:
    - scheduler: The scheduler instance to advance.
    - args...: Additional arguments required by the specific scheduler.

ParameterSchedulers.reset!(scheduler, args...)
  - Resets the scheduler to its initial state.
  - Modifies the scheduler in-place.
  - Parameters:
    - scheduler: The scheduler instance to reset.
    - args...: Additional arguments required for resetting.
```

--------------------------------

### CyclicLR (exp_range) to TriangleExp

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's CyclicLR with an exponential range policy to ParameterSchedulers.jl's `TriangleExp` scheduler.

```Julia
TriangleExp(base_lr, max_lr, step_size, gamma)
```

```Python
CyclicLR(_, base_lr, max_lr, step_size, step_size, 'exp_range', gamma, None)
```

--------------------------------

### SequentialLR to Sequence (with milestones)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's SequentialLR, which applies a sequence of schedulers at specified milestones, to ParameterSchedulers.jl's `Sequence` scheduler.

```Julia
Sequence(schedulers, milestones)
```

```Python
SequentialLR(_, schedulers, milestones)
```

--------------------------------

### Create One Cycle Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a one-cycle cosine learning rate schedule, a technique that can speed up training. It requires the total number of `nsteps` and the `maxval` for the cycle. This schedule increases then decreases the parameter.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = OneCycle(10, 1.0) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### ConstantLR to Sequence (with factor)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Translates PyTorch's ConstantLR, which applies a constant factor for a specified number of iterations, to ParameterSchedulers.jl's `Sequence` scheduler.

```Julia
Sequence(lr * factor => total_iters, lr => nepochs)
```

```Python
ConstantLR(_, factor, total_iters)
```

--------------------------------

### Scheduler Type Documentation

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/general.md

Documentation for the base Scheduler type in ParameterSchedulers. This likely includes its definition, purpose, and common usage patterns within the package.

```APIDOC
ParameterSchedulers.Scheduler
  - Documentation for the base Scheduler type.
  - Describes the fundamental interface for all schedulers in the package.
  - Expected to cover abstract methods or properties that concrete scheduler types must implement.
```

--------------------------------

### CosineAnnealingRestarts to CosAnneal (with restarts)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps PyTorch's CosineAnnealingRestarts and Tensorflow's CosineDecayRestarts to ParameterSchedulers.jl's `CosAnneal` with restart functionality, allowing periodic restarts of the cosine annealing cycle.

```Julia
CosAnneal(lr, eta_min, T_0)
```

```Python
CosineAnnealingRestarts(_, T_0, 1, eta_min)
CosineDecayRestarts(lr, T_0, 1, 1, eta_min)
```

--------------------------------

### CosineAnnealingRestarts (with T_mult) to CosAnneal variants

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Covers PyTorch's CosineAnnealingRestarts with a multiplier for restart period and Tensorflow's CosineDecayRestarts, mapping them to advanced `CosAnneal` variants in ParameterSchedulers.jl.

```Julia
See [below](@ref "Cosine annealing variants")
```

```Python
CosineAnnealingRestarts(_, T_0, T_mult, eta_min)
CosineDecayRestarts(lr, T_0, T_mult, 1, alpha)
CosineDecayRestarts(lr, T_0, T_mult, m_mul, alpha)
```

--------------------------------

### Sin Function (Julia)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a sine wave function for parameter scheduling. It takes parameters for initial offset (l0), amplitude (l1), and period.

```julia
using UnicodePlots, ParameterSchedulers

t = 1:10 |> collect
s = Sin(l0 = 0.0, l1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
```

--------------------------------

### ReduceLROnPlateau style schedules

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

References ParameterSchedulers.jl's implementation for schedules that reduce the learning rate based on a monitored metric, similar to PyTorch's ReduceLROnPlateau.

```Julia
See [below](@ref "`ReduceLROnPlateau` style schedules")
```

```Python
ReduceLROnPlateau(_, mode, factor, patience, threshold, 'abs', 0)
```

--------------------------------

### Stateful iteration for schedules with Flux

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/optimizers.md

Uses `ParameterSchedulers.Stateful` to advance a schedule with every batch, useful for nested loops where epoch restarts are not desired.

```Julia
nepochs = 3
s = ParameterSchedulers.Stateful(Inv(start = 1e-1, decay = 0.2, degree = 2))
for epoch in 1:nepochs
    for (i, (x, y)) in enumerate(data)
        global opt_st, m
        adjust!(opt_st, ParameterSchedulers.next!(s))
        g = Flux.gradient(m -> Flux.mse(m(x), y), m)[1]
        opt_st, m = Flux.update!(opt_st, m, g)
        println("epoch: $epoch, batch: $i, opt state: $(opt_st.layers[1].weight.rule)")
    end
end
```

--------------------------------

### Cyclic Schedule API Reference Generation

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/cyclic.md

This entry describes the directive used to generate the API reference for cyclic schedules. It specifies the modules and pages to be included in the documentation.

```APIDOC
```@autodocs
Modules = [ParameterSchedulers]
Pages = ["cyclic.jl"]
```

This directive instructs the documentation generator to include API documentation from the `ParameterSchedulers` module, specifically focusing on the content within the `cyclic.jl` file. It serves as a placeholder for the actual generated API reference content.
```

--------------------------------

### Create Triangle Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Generates a triangle wave pattern for parameter scheduling. The schedule cycles between `l0` and `l1` over a given `period`. This is useful for cyclical learning rates or other periodic adjustments.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = Triangle(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### StepLR / ExponentialDecay to Step

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Corresponds to PyTorch's StepLR and Tensorflow's ExponentialDecay (with specific parameters) to ParameterSchedulers.jl's `Step` scheduler, which reduces the learning rate by a factor at specified intervals.

```Julia
Step(lr, gamma, step_size)
```

```Python
StepLR(_, step_size, gamma)
ExponentialDecay(lr, step_size, gamma, True)
```

--------------------------------

### Decay Learning Rate with ComposedSchedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Demonstrates decaying the learning rate by a multiplier `m_mul` using `ComposedSchedule` and `Step` for cosine annealing. This approach composes a base cosine annealing schedule with a step function to modify the learning rate.

```julia
# decay learning rate by m_mul
s = ComposedSchedule(CosAnneal(range, offset, period),
                     (Step(range, m_mul, period), offset, period))
```

--------------------------------

### Iterating during training with Flux

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/optimizers.md

Adjusts the learning rate of the optimizer before each batch of training by zipping a schedule with training data.

```Julia
using Flux, ParameterSchedulers
using Optimisers: Descent, adjust!

data = [(Flux.rand32(4, 10), rand([-1, 1], 1, 10)) for _ in 1:3]
m = Chain(Dense(4, 4, tanh), Dense(4, 1, tanh))
opt = Descent()
opt_st = Flux.setup(opt, m)
s = Exp(start = 1e-1, decay = 0.2)

for (eta, (x, y)) in zip(s, data)
    global opt_st, m
    adjust!(opt_st, eta)
    g = Flux.gradient(m -> Flux.mse(m(x), y), m)[1]
    opt_st, m = Flux.update!(opt_st, m, g)
    println("opt state: ", opt_st.layers[1].weight.rule)
end
```

--------------------------------

### ParameterSchedulers Complex Schedule API Generation

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/complex.md

This entry describes the configuration for generating API documentation for the ParameterSchedulers Julia package. It specifies the modules and pages to be documented and applies filters to exclude specific components like Stateful, next!, and reset!.

```APIDOC
```@autodocs
Modules = [ParameterSchedulers]
Pages = ["complex.jl"]
Filter = f -> !(f === ParameterSchedulers.Stateful) &&
              !(f === ParameterSchedulers.next!) &&
              !(f === ParameterSchedulers.reset!)
```
```

--------------------------------

### SinExp Function (Julia)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a sine wave function with an exponentially decaying amplitude. The decay rate is controlled by the 'decay' parameter.

```julia
using UnicodePlots, ParameterSchedulers

t = 1:10 |> collect
s = SinExp(l0 = 0.0, l1 = 1.0, period = 2, decay = 0.8)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
```

--------------------------------

### ExponentialLR to Exp

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Corresponds to PyTorch's ExponentialLR and Tensorflow's ExponentialDecay (without step decay) to ParameterSchedulers.jl's `Exp` scheduler, which decays the learning rate by a fixed gamma factor each step.

```Julia
Exp(lr, gamma)
```

```Python
ExponentialLR(_, gamma)
ExponentialDecay(lr, 1, gamma, False)
```

--------------------------------

### Implement Decay2 Schedule in Julia

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/interface.md

Illustrates creating a `Decay2` schedule that halves the parameter value each iteration. It subtypes `AbstractSchedule{false}` as an infinite schedule and defines its behavior via `(schedule::Decay2)(t) = schedule.λ / 2^(t - 1)`.

```julia
using ParameterSchedulers
using ParameterSchedulers: AbstractSchedule

# we subtype AbstractSchedule{IsFinite} with IsFinite == false
# this is because this is an infinite schedule
struct Decay2{T<:Number} <: AbstractSchedule{false}
    λ::T
end

(schedule::Decay2)(t) = schedule.λ / 2^(t - 1)
```

--------------------------------

### FluxML ParameterSchedulers Decay Schedule API

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/api/decay.md

This section provides the API reference for decay schedules implemented in the FluxML ParameterSchedulers Julia package. It details the available functions, types, and configurations for managing learning rate decay and other scheduling mechanisms.

```APIDOC
Project: /fluxml/parameterschedulers.jl
Content:
# Decay schedule API reference

```@autodocs
Modules = [ParameterSchedulers]
Pages = ["decay.jl"]
```

This directive indicates that the documentation for the `ParameterSchedulers` Julia module, specifically focusing on the `decay.jl` file, should be generated. The output will include detailed API information for decay scheduling functionalities, such as function signatures, parameter descriptions, return types, and usage examples, as defined within the module's source code.
```

--------------------------------

### CyclicLR (triangular2) to TriangleDecay2

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Corresponds to PyTorch's CyclicLR with a triangular2 policy (decaying amplitude) to ParameterSchedulers.jl's `TriangleDecay2` scheduler.

```Julia
TriangleDecay2(base_lr, max_lr, step_size)
```

```Python
CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular2', _, None)
```

--------------------------------

### Create TriangleDecay2 Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a triangle wave function where the amplitude halves every `period`. This schedule combines cyclical behavior with an exponential decay of the cycle's range. It requires `l0`, `l1`, and `period`.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = TriangleDecay2(l0 = 0.0, l1 = 1.0, period = 2) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### Plotting Simulation Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/tutorials/complex-schedules.md

Generates a time range for a simulation and plots the schedule's output against it. Useful for visualizing schedule behavior over time.

```julia
trange = dt:dt:T |> collect
lineplot(trange, s.(trange); border = :none)
```

--------------------------------

### Create Cosine Annealing Schedule with ParameterSchedulers.jl

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Applies cosine annealing to schedule parameter values over a specified `period`. This method is useful for learning rate scheduling in deep learning. It requires `l0` (initial value), `l1` (final value), and `period`.

```Julia
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = CosAnneal(l0 = 0.0, l1 = 1.0, period = 4) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```

--------------------------------

### Implement Square Wave Schedule in Julia

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/interface.md

Shows how to implement a cyclic `Square` wave schedule. It alternates between two values (`λ0`, `λ1`) over a specified `period`. The implementation uses the modulo operator on `t` to determine the current value.

```julia
using ParameterSchedulers
using ParameterSchedulers: AbstractSchedule

struct Square{T<:Number, S<:Integer} <: AbstractSchedule{false}
    λ0::T
    λ1::T
    period::S
end

(schedule::Square{T})(t) where T =
    (mod(t - 1, schedule.period) < schedule.period / 2) ? schedule.λ1 : schedule.λ0
```

--------------------------------

### ExponentialDecay (with steps) to Interpolator(Exp)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps Tensorflow's ExponentialDecay with specified steps to ParameterSchedulers.jl's `Interpolator` combined with `Exp`, allowing decay at custom intervals.

```Julia
Interpolator(Exp(lr, gamma), steps)
```

```Python
ExponentialDecay(lr, steps, gamma, False)
```

--------------------------------

### InverseTimeDecay to Inv

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Maps Tensorflow's InverseTimeDecay (with fixed or decaying step) to ParameterSchedulers.jl's `Inv` scheduler, which decays the learning rate by a factor of 1/t.

```Julia
Inv(lr, decay_rate, 1)
```

```Python
InverseTimeDecay(lr, 1, decay_rate, False)
```

--------------------------------

### AbstractSchedule Interface in Julia

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/interface.md

Details the `ParameterSchedulers.AbstractSchedule` base type and its type parameters (`true`, `false`, `missing`, `T`) which automatically implement parts of the iteration interface based on schedule finiteness.

```julia
```@docs
ParameterSchedulers.AbstractSchedule
```
```

--------------------------------

### SinDecay2 Function (Julia)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/README.md

Implements a sine wave function where the amplitude is halved every specified period. It's useful for creating decaying oscillatory behaviors.

```julia
using UnicodePlots, ParameterSchedulers

t = 1:10 |> collect
s = SinDecay2(l0 = 0.0, l1 = 1.0, period = 2)
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
```

--------------------------------

### Increase Period with Sequence and InfiniteArrays

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Shows how to increase the period of a cosine annealing schedule by a fixed multiple `t_mul`. It utilizes `Sequence` with a generator expression over `OneToInf` from `InfiniteArrays.jl` to create an infinite sequence of schedules.

```julia
using InfiniteArrays: OneToInf

# increase period by factor t_mul
e = Exp(period, t_mul)
s = Sequence(CosAnneal(range, offset, e(t)) for t in OneToInf(),
             e(t) for t in OneToInf())
```

--------------------------------

### ReduceLROnPlateau Style Schedule

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Implements a ReduceLROnPlateau-like schedule by combining a base schedule (e.g., Exponential decay) with a predicate function using `ParameterSchedulers.Stateful`. The `advance` keyword argument allows dynamic schedule updates based on a condition, such as a plateau in accuracy.

```julia
# the code below is written to match
# ReduceLROnPlateau(_, 'max', factor, patience, threshold, 'abs', 0)
# we also assume accuracy_func() is an accuracy metric that's already given for our model

# this is done to match ReduceLROnPlateau
# but it could be any schedule
s = Exp(lr, factor)
predicate = Flux.plateau(accuracy_func, patience; min_dist = threshold)
ParameterSchedulers.Stateful(s; advance = predicate)
```

--------------------------------

### InverseTimeDecay (with decay_step) to Interpolator(Inv)

Source: https://github.com/fluxml/parameterschedulers.jl/blob/main/docs/src/cheatsheet.md

Translates Tensorflow's InverseTimeDecay with a specified decay step to ParameterSchedulers.jl's `Interpolator` combined with `Inv` for custom decay intervals.

```Julia
Interpolator(Inv(lr, decay_rate, 1), decay_step)
```

```Python
InverseTimeDecay(lr, decay_step, decay_rate, False)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.