Optimization.jl

https://docs.sciml.ai/Optimization/stable/
Admin
Optimization.jl is a unified optimization package that provides a uniform interface to over 25...
Tokens:52,081
Snippets:507
Trust Score:9.1
Update:5 months ago
Show doc for...
Tokens:
Raw
### Installation

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/optim

Instructions on how to install the OptimizationOptimJL package.

```APIDOC
## Installation: OptimizationOptimJL.jl

To use this package, install the OptimizationOptimJL package:

```julia
import Pkg;
Pkg.add("OptimizationOptimJL");
```
```

--------------------------------

### Install OptimizationODE.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ode

Use the Pkg.add command to install the OptimizationODE package.

```julia
using Pkg
Pkg.add("OptimizationODE")
```

--------------------------------

### Install OptimizationQuadDIRECT.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/quaddirect

Install the OptimizationQuadDIRECT package by adding it from a URL. Note that QuadDIRECT.jl itself needs to be installed separately as it is not a registered package.

```julia
import Pkg;
Pkg.add(url = "https://github.com/SciML/Optimization.jl",
    subdir = "lib/OptimizationQuadDIRECT");
```

```julia
\] add https://github.com/timholy/QuadDIRECT.jl.git
```

--------------------------------

### Install OptimizationBBO.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/blackboxoptim

Install the OptimizationBBO package to use BlackBoxOptim.jl.

```julia
import Pkg;
Pkg.add("OptimizationBBO");
```

--------------------------------

### Install OptimizationIpopt.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Install the OptimizationIpopt package using the Julia package manager.

```julia
import Pkg;
Pkg.add("OptimizationIpopt");
```

--------------------------------

### Install OptimizationOptimJL.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/optim

Install the OptimizationOptimJL package to use Optim.jl's algorithms.

```julia
import Pkg;
Pkg.add("OptimizationOptimJL");
```

--------------------------------

### Install OptimizationNLPModels Package

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlpmodels

Install the OptimizationNLPModels package to translate an NLPModel.

```julia
import Pkg;
Pkg.add("OptimizationNLPModels")
```

--------------------------------

### Install NLPModelsTest and Ipopt

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlpmodels

Install NLPModelsTest for problem definitions and Ipopt for solving optimization problems.

```julia
import Pkg;
Pkg.add("NLPModelsTest", "Ipopt")
```

--------------------------------

### Enable Warm Starting for Ipopt

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Enable warm starting by setting `warm_start_init_point = "yes"` to reuse solutions from previous similar problems.

```julia
IpoptOptimizer(warm_start_init_point = "yes")
```

--------------------------------

### Install SimpleOptimization.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/simpleoptimization

Install the SimpleOptimization package using the Julia package manager.

```julia
import Pkg;
Pkg.add("SimpleOptimization");
```

--------------------------------

### Install OptimizationSophia.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/sophia

Install the OptimizationSophia package using the Julia package manager.

```julia
using Pkg
Pkg.add("OptimizationSophia")
```

--------------------------------

### Install OptimizationManopt.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/manopt

Install the OptimizationManopt package to use the Optimization.jl interface with Manopt.jl solvers.

```julia
import Pkg;
Pkg.add("OptimizationManopt");
```

--------------------------------

### Install OptimizationSpeedMapping.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/speedmapping

Install the OptimizationSpeedMapping package using the Julia package manager.

```julia
import Pkg;
Pkg.add("OptimizationSpeedMapping");
```

--------------------------------

### Install OptimizationPolyalgorithms.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/polyopt

Install the OptimizationPolyalgorithms package using the Julia package manager.

```julia
import Pkg;
Pkg.add("OptimizationPolyalgorithms");
```

--------------------------------

### Install OptimizationSciPy.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/scipy

Install the OptimizationSciPy.jl package using the Julia package manager.

```julia
import Pkg
Pkg.add("OptimizationSciPy")
```

--------------------------------

### Installation of OptimizationNLopt.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlopt

Instructions on how to add the OptimizationNLopt package to your Julia environment.

```APIDOC
## Installation: OptimizationNLopt.jl
To use this package, install the OptimizationNLopt package:
```julia
import Pkg;
Pkg.add("OptimizationNLopt");
```
```

--------------------------------

### Install OptimizationMOI.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/mathoptinterface

Install the OptimizationMOI package to use MathOptInterface with Optimization.jl. This is the first step to integrate external solvers.

```julia
import Pkg;
Pkg.add("OptimizationMOI");
```

--------------------------------

### Install OptimizationNOMAD.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nomad

Install the OptimizationNOMAD package using the Julia package manager.

```julia
import Pkg;
Pkg.add("OptimizationNOMAD");
```

--------------------------------

### Basic PyCMA Optimization Example

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/pycma

Demonstrates solving the Rosenbrock problem using PyCMAOpt within the Optimization.jl framework. The rosenbrock function and initial parameters are defined, followed by problem setup and the solve call.

```julia
using OptimizationPyCMA

rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
_p = [1.0, 100.0]
l1 = rosenbrock(x0, _p)
f = OptimizationFunction(rosenbrock)
prob = OptimizationProblem(f, x0, _p, lb = [-1.0, -1.0], ub = [0.8, 0.8])
sol = solve(prob, PyCMAOpt())
```

--------------------------------

### Initial Solve with BlackBoxOptim

Source: https://docs.sciml.ai/Optimization/stable/tutorials/remakecomposition

Performs an initial solve using BlackBoxOptim's BBO_adaptive_de_rand_1_bin solver for exploration. This step aims to find a good starting point for the subsequent refinement.

```julia
res1 = solve(prob, BBO_adaptive_de_rand_1_bin(), maxiters = 4000)

@show res1.objective
```

--------------------------------

### Example: Optimizing Rosenbrock Function with NLopt.LN_NELDERMEAD

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlopt

A practical example demonstrating how to use the `NLopt.LN_NELDERMEAD()` optimizer to solve the Rosenbrock function with bounds constraints.

```APIDOC
The Rosenbrock function can be optimized using the `NLopt.LN_NELDERMEAD()` as follows:
```julia
using Optimization
using OptimizationNLopt
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]
f = OptimizationFunction(rosenbrock)
prob = SciMLBase.OptimizationProblem(f, x0, p, lb = [-1.0, -1.0], ub = [1.0, 1.0])
sol = solve(prob, NLopt.LN_NELDERMEAD())
```
```
retcode: Success
u: 2-element Vector{Float64}:
 1.0
 1.0
```
```

--------------------------------

### Install OptimizationNLopt.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlopt

To use the NLopt.jl package, you need to install the OptimizationNLopt package. This is typically done using Julia's package manager.

```julia
import Pkg;
Pkg.add("OptimizationNLopt");
```

--------------------------------

### Install OptimizationLBFGSB.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/lbfgsb

Install the OptimizationLBFGSB package using the Julia package manager.

```julia
using Pkg
Pkg.add("OptimizationLBFGSB")
```

--------------------------------

### Install OptimizationMetaheuristics.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/metaheuristics

Install the OptimizationMetaheuristics package to use metaheuristic algorithms for optimization.

```julia
import Pkg;
Pkg.add("OptimizationMetaheuristics");
```

--------------------------------

### Installation

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/prima

To use the PRIMA.jl package, you need to install it using the Julia package manager.

```APIDOC
## Installation
To use this package, install the OptimizationPRIMA package:
```julia
import Pkg;
Pkg.add("OptimizationPRIMA");
```
```

--------------------------------

### Install OptimizationOptimisers.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/optimisers

Use this command to install the OptimizationOptimisers package, which includes optimizers from Optimisers.jl and the Sophia algorithm.

```julia
import Pkg;
Pkg.add("OptimizationOptimisers");
```

--------------------------------

### Nonlinear Constrained Optimization Setup

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Sets up an optimization problem with a quadratic objective function for nonlinear constrained optimization using IpoptOptimizer. The objective is to minimize x[1]^2 + x[2]^2.

```julia
using Optimization, OptimizationIpopt
using Zygote

# Objective: minimize x[1]^2 + x[2]^2
objective(x, p) = x[1]^2 + x[2]^2

```

--------------------------------

### Updating Initial Values with `reinit!`

Source: https://docs.sciml.ai/Optimization/stable/tutorials/reusage_interface

Shows how to warm-start optimization from different initial points by reinitializing the cache with new `u0` values. This is useful for exploring different starting regions of the solution space.

```julia
# Warm-start optimization from different initial points
u0_values = [[0.0, 0.0], [0.5, 0.5], [1.0, 1.0]]

for u0 in u0_values
    local cache
    cache = Optimization.reinit!(cache; u0 = u0)
    sol = Optimization.solve!(cache)
    println("Starting from ", u0, " converged to ", sol.u)
end
```

--------------------------------

### Install OptimizationEvolutionary.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/evolutionary

Install the OptimizationEvolutionary package using the Julia package manager. This is required to use Evolutionary.jl with Optimization.jl.

```julia
import Pkg;
Pkg.add("OptimizationEvolutionary");
```

--------------------------------

### Unconstrained Minimization Example

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/scipy

Example demonstrating how to solve an unconstrained minimization problem using `ScipyBFGS`.

```APIDOC
using Optimization
using OptimizationSciPy
using ADTypes
using Zygote

rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]

f = OptimizationFunction(rosenbrock, ADTypes.AutoZygote())
prob = OptimizationProblem(f, x0, p)

sol = solve(prob, ScipyBFGS())
@show sol.objective   # ≈ 0 at optimum
```

--------------------------------

### Install OptimizationCMAEvolutionStrategy.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/cmaevolutionstrategy

Install the OptimizationCMAEvolutionStrategy package using the Julia package manager.

```julia
import Pkg;
Pkg.add("OptimizationCMAEvolutionStrategy");
```

--------------------------------

### Portfolio Optimization with IPOPT

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Example demonstrating portfolio optimization using IPOPT, minimizing portfolio variance subject to a target return constraint. Requires `LinearAlgebra` for matrix operations.

```julia
using Optimization, OptimizationIpopt
using Zygote
using LinearAlgebra

# Portfolio optimization: minimize risk subject to return constraint
n_assets = 5
μ = [0.05, 0.10, 0.15, 0.08, 0.12]  # Expected returns
Σ = [0.05 0.01 0.02 0.01 0.00;      # Covariance matrix
     0.01 0.10 0.03 0.02 0.01;
     0.02 0.03 0.15 0.02 0.03;
     0.01 0.02 0.02 0.08 0.02;
     0.00 0.01 0.03 0.02 0.06]

target_return = 0.10

# Objective: minimize portfolio variance
portfolio_risk(w, p) = dot(w, Σ * w)

```

--------------------------------

### Install OptimizationMultistartOptimization.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/multistartoptimization

Install the OptimizationMultistartOptimization package using the Julia package manager. Ensure relevant subpackages for local methods are also loaded.

```julia
import Pkg;
Pkg.add("OptimizationMultistartOptimization");
```

--------------------------------

### Configure Juniper.jl with Ipopt.jl NL Solver

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/mathoptinterface

This example demonstrates configuring Juniper.jl to use Ipopt.jl as its nonlinear solver. It sets specific options for both solvers using nested OptimizerWithAttributes.

```julia
using Optimization, OptimizationMOI, Juniper, Ipopt, ADTypes, ForwardDiff
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
_p = [1.0, 100.0]

f = OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff())
prob = SciMLBase.OptimizationProblem(f, x0, _p)

opt = OptimizationMOI.MOI.OptimizerWithAttributes(Juniper.Optimizer,
    "nl_solver" => OptimizationMOI.MOI.OptimizerWithAttributes(Ipopt.Optimizer,
        "print_level" => 0))
sol = solve(prob, opt)
```

--------------------------------

### Install OptimizationGCMAES.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/gcmaes

Install the OptimizationGCMAES package using Julia's package manager. This is required before using the GCMAES optimizer.

```julia
import Pkg;
Pkg.add("OptimizationGCMAES");
```

--------------------------------

### Define and Setup Neural Network with Sophia

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/sophia

Defines a simple neural network using Lux, sets up its parameters and state, and wraps it in a StatefulLuxLayer. This is a prerequisite for using the Sophia optimizer.

```julia
model = Chain(Dense(1, 32, tanh), Dense(32, 1))
ps, st = Lux.setup(Random.default_rng(), model)
ps_ca = ComponentArray(ps)
smodel = StatefulLuxLayer{true}(model, nothing, st)
```

--------------------------------

### Install Optimization.jl

Source: https://docs.sciml.ai/Optimization/stable

Import the Optimization.jl package using Julia's package manager. Dependencies are typically handled automatically.

```julia
import Pkg
Pkg.add("Optimization")
```

--------------------------------

### Nonlinear Constraints Example

Source: https://docs.sciml.ai/Optimization/stable/API/FAQ

Illustrates a set of nonlinear constraints that can be simplified through symbolic manipulation and tearing.

```julia
0 ~ u1 - sin(u5) * h,
0 ~ u2 - cos(u1),
0 ~ u3 - hypot(u1, u2),
0 ~ u4 - hypot(u2, u3),
```

--------------------------------

### Optimize Rosenbrock Function with NLopt.LN_NELDERMEAD

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nlopt

This example demonstrates optimizing the Rosenbrock function using the NLopt.LN_NELDERMEAD derivative-free algorithm. It includes setting up the optimization problem with bounds and solving it.

```julia
using Optimization
using OptimizationNLopt
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]
f = OptimizationFunction(rosenbrock)
prob = SciMLBase.OptimizationProblem(f, x0, p, lb = [-1.0, -1.0], ub = [1.0, 1.0])
sol = solve(prob, NLopt.LN_NELDERMEAD())
```

```text
retcode: Success
u: 2-element Vector{Float64}:
 1.0
 1.0
```

--------------------------------

### Install PRIMA.jl Package

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/prima

To use the PRIMA.jl package, you need to add it to your Julia environment. This snippet shows the command to install the package using the Pkg manager.

```julia
import Pkg;
Pkg.add("OptimizationPRIMA");
```

--------------------------------

### Setup and Solve Optimization Problem with Minibatching

Source: https://docs.sciml.ai/Optimization/stable/tutorials/minibatch

Configures the optimization problem using `OptimizationFunction` and `OptimizationProblem`, then solves it with the ADAM optimizer. It utilizes `MLUtils.DataLoader` for minibatching and `IterTools.ncycle` to repeat data for a specified number of epochs.

```julia
k = 10
# Pass the data for the batches as separate vectors wrapped in a tuple
train_loader = MLUtils.DataLoader((ode_data, t), batchsize = k)


numEpochs = 300
l1 = loss_adjoint(ps_ca, train_loader.data)[1]

optfun = OptimizationFunction(
    loss_adjoint,
    ADTypes.AutoZygote())
optprob = OptimizationProblem(optfun, ps_ca, train_loader)
using IterTools: ncycle
res1 = solve(
    optprob, Optimisers.ADAM(0.05); callback = callback, epochs = 1000)
```

--------------------------------

### SimpleBFGS Optimization Example

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/simpleoptimization

Demonstrates solving the Rosenbrock problem using the SimpleBFGS optimizer. Requires Optimization.jl and ForwardDiff.jl for automatic differentiation.

```julia
using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = zeros(2)
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleBFGS())
```

--------------------------------

### Local Optimizer Usage

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/prima

Demonstrates how to use the different Powell's algorithms provided by PRIMA.jl with an example problem (Rosenbrock function).

```APIDOC
## Local Optimizer
The five Powell's algorithms of the prima library are provided by the PRIMA.jl package:
`UOBYQA`: (Unconstrained Optimization BY Quadratic Approximations) is for unconstrained optimization, that is Ω = ℝⁿ.
`NEWUOA`: is also for unconstrained optimization. According to M.J.D. Powell, newuoa is superior to uobyqa.
`BOBYQA`: (Bounded Optimization BY Quadratic Approximations) is for simple bound constrained problems, that is Ω = { x ∈ ℝⁿ | xl ≤ x ≤ xu }.
`LINCOA`: (LINearly Constrained Optimization) is for constrained optimization problems with bound constraints, linear equality constraints, and linear inequality constraints.
`COBYLA`: (Constrained Optimization BY Linear Approximations) is for general constrained problems with bound constraints, non-linear constraints, linear equality constraints, and linear inequality constraints.
```julia
using OptimizationBase, OptimizationPRIMA

rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
_p = [1.0, 100.0]

prob = OptimizationProblem(rosenbrock, x0, _p)

sol = solve(prob, UOBYQA(), maxiters = 1000)

sol = solve(prob, NEWUOA(), maxiters = 1000)

sol = solve(prob, BOBYQA(), maxiters = 1000)

sol = solve(prob, LINCOA(), maxiters = 1000)

function con2_c(res, x, p)
    res .= [x[1] + x[2], x[2] * sin(x[1]) - x[1]]
end
optprob = OptimizationFunction(rosenbrock, AutoForwardDiff(), cons = con2_c)
prob = OptimizationProblem(optprob, x0, _p, lcons = [1, -100], ucons = [1, 100])
sol = solve(prob, COBYLA(), maxiters = 1000)
```
```
retcode: Success
u: 2-element Vector{Float64}:
 0.6187959058352076
 0.3812040941647925
```
```

--------------------------------

### Multistart Optimization with EnsembleProblem

Source: https://docs.sciml.ai/Optimization/stable/tutorials/ensemble

Executes multistart optimization using EnsembleProblem with 4 random initial points and BFGS solver. This approach aims to find a significantly lower objective value by exploring multiple starting configurations.

```julia
x0s = [x0, x0 .+ rand(2), x0 .+ rand(2), x0 .+ rand(2)]
function prob_func(prob, i, repeat)
    remake(prob, u0 = x0s[i])
end

ensembleprob = EnsembleProblem(prob; prob_func)
@time sol = solve(ensembleprob, OptimizationOptimJL.BFGS(),
    EnsembleThreads(), trajectories = 4, maxiters = 5)
@show findmin(i -> sol[i].objective, 1:4)[1]
```

```julia
0.0069590832130179536
```

--------------------------------

### SimpleLBFGS Optimization Example

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/simpleoptimization

Demonstrates solving the Rosenbrock problem using the SimpleLBFGS optimizer. Requires Optimization.jl and ForwardDiff.jl for automatic differentiation.

```julia
using SimpleOptimization, Optimization, ForwardDiff

rosenbrock(x, p) = (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2
x0 = zeros(2)
optf = OptimizationFunction(rosenbrock, Optimization.AutoForwardDiff())
prob = OptimizationProblem(optf, x0)
sol = solve(prob, SimpleLBFGS())
```

--------------------------------

### Define and Solve a Mixed Integer Nonlinear Optimization Problem

Source: https://docs.sciml.ai/Optimization/stable/tutorials/linearandinteger

This snippet demonstrates setting up and solving a mixed-integer nonlinear optimization problem. It defines the objective function, constraints, variable bounds, and specifies integer variables. Ensure Juniper and Ipopt are installed and imported.

```julia
using Juniper, Ipopt, ADTypes, Symbolics

v = [10, 20, 12, 23, 42]
w = [12, 45, 12, 22, 21]

objective = (u, p) -> (v = p[1:5]; dot(v, u))

cons = (res, u, p) -> (w = p[6:10]; res .= [sum(w[i] * u[i]^2 for i in 1:5)])

optf = OptimizationFunction(objective, ADTypes.AutoSymbolics(), cons = cons)
optprob = OptimizationProblem(optf,
    zeros(5),
    vcat(v, w);
    sense = SciMLBase.MaxSense,
    lb = zeros(5),
    ub = ones(5),
    lcons = [-Inf],
    ucons = [45.0],
    int = fill(true, 5))

nl_solver = OptimizationMOI.MOI.OptimizerWithAttributes(Ipopt.Optimizer,
    "print_level" => 0)
minlp_solver = OptimizationMOI.MOI.OptimizerWithAttributes(Juniper.Optimizer,
    "nl_solver" => nl_solver)

sol = solve(optprob, minlp_solver)

```

```julia
retcode: Success
u: 5-element Vector{Float64}:
 0.0
 0.0
 0.0
 1.0
 1.0

```

--------------------------------

### Direct Solver Call with Ipopt.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/mathoptinterface

Call an optimizer that supports the MathOptInterface API directly if no specific optimizer options need to be defined. This example uses Ipopt.jl.

```julia
using OptimizationMOI, Ipopt
sol = solve(prob, Ipopt.Optimizer())
```

--------------------------------

### Optimize Rosenbrock Function with QuadDirect

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/quaddirect

Example demonstrating how to use the QuadDirect() solver to optimize the Rosenbrock function. Lower and upper bounds, as well as initial splits, are required for the problem definition.

```julia
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]
f = OptimizationFunction(rosenbrock)
prob = SciMLBase.OptimizationProblem(f, x0, p, lb = [-1.0, -1.0], ub = [1.0, 1.0])
solve(prob, QuadDirect(), splits = ([-0.9, 0, 0.9], [-0.8, 0, 0.8]))
```

--------------------------------

### Solve Rosenbrock Problem with L-BFGS

Source: https://docs.sciml.ai/Optimization/stable/getting_started

Import necessary packages, define the objective function and parameters, create an OptimizationFunction, and solve the problem using the LBFGSB solver. This is a basic example for getting started.

```julia
using OptimizationBase, OptimizationLBFGSB, ADTypes, Zygote
rosenbrock(u, p) = (p[1] - u[1])^2 + p[2] * (u[2] - u[1]^2)^2
u0 = zeros(2)
p = [1.0, 100.0]

optf = OptimizationFunction(rosenbrock, ADTypes.AutoZygote())
prob = OptimizationProblem(optf, u0, p)

sol = solve(prob, OptimizationLBFGSB.LBFGSB())
```

--------------------------------

### Basic Usage

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Demonstrates how to create and use the IpoptOptimizer with Optimization.jl, including default settings and custom options.

```APIDOC
using OptimizationBase, OptimizationIpopt

# Create optimizer with default settings
opt = IpoptOptimizer()

# Or configure Ipopt-specific options
opt = IpoptOptimizer(
    acceptable_tol = 1e-8,
    mu_strategy = "adaptive"
)

# Solve the problem
sol = solve(prob, opt)
```

--------------------------------

### Create and Configure IpoptOptimizer

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Instantiate the IpoptOptimizer with default settings or configure Ipopt-specific options for custom behavior.

```julia
using OptimizationBase, OptimizationIpopt

# Create optimizer with default settings
opt = IpoptOptimizer()

# Or configure Ipopt-specific options
opt = IpoptOptimizer(
    acceptable_tol = 1e-8,
    mu_strategy = "adaptive"
)
```

--------------------------------

### Set Initial Conditions and Parameters

Source: https://docs.sciml.ai/Optimization/stable/tutorials/symbolic

Provide initial values for the unknowns (`u0`) and specific values for the parameters (`p`) to prepare the symbolic system for numerical solving.

```julia
u0 = [x => 1.0
      y => 2.0]
p = [a => 6.0
     b => 7.0]
```

--------------------------------

### Derivative-based Optimizers with Constraints

Source: https://docs.sciml.ai/Optimization/stable/examples/rosenbrock

Demonstrates setting up optimization problems with various constraints, including inequality and equality constraints, and bound constraints on variables. Uses `IPNewton` for solving.

```julia
cons = (res, x, p) -> res .= [x[1]^2 + x[2]^2]
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff(); cons = cons)

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf], ucons = [Inf])
sol = solve(prob, IPNewton()) # Note that -Inf < x[1]^2 + x[2]^2 < Inf is always true

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-5.0], ucons = [10.0])
sol = solve(prob, IPNewton()) # Again, -5.0 < x[1]^2 + x[2]^2 < 10.0

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [-Inf], ucons = [Inf],
    lb = [-500.0, -500.0], ub = [50.0, 50.0])
sol = solve(prob, IPNewton())

prob = SciMLBase.OptimizationProblem(optf, x0, _p, lcons = [0.5], ucons = [0.5],
    lb = [-500.0, -500.0], ub = [50.0, 50.0])
sol = solve(prob, IPNewton())

# Notice now that x[1]^2 + x[2]^2 ≈ 0.5:
res = zeros(1)
cons(res, sol.u, _p)
println(res)
```

```text
┌ Warning: The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.
│         So a `SecondOrder` with AutoForwardDiff() for both inner and outer will be created, this can be suboptimal and not work in some cases so
│         an explicit `SecondOrder` ADtype is recommended.
└ @ OptimizationBase ~/work/Optimization.jl/Optimization.jl/lib/OptimizationBase/src/cache.jl:58
┌ Warning: The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.
│         So a `SecondOrder` with AutoForwardDiff() for both inner and outer will be created, this can be suboptimal and not work in some cases so
│         an explicit `SecondOrder` ADtype is recommended.
└ @ OptimizationBase ~/work/Optimization.jl/Optimization.jl/lib/OptimizationBase/src/cache.jl:58
┌ Warning: The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.
│         So a `SecondOrder` with AutoForwardDiff() for both inner and outer will be created, this can be suboptimal and not work in some cases so
│         an explicit `SecondOrder` ADtype is recommended.
└ @ OptimizationBase ~/work/Optimization.jl/Optimization.jl/lib/OptimizationBase/src/cache.jl:58
┌ Warning: The selected optimization algorithm requires second order derivatives, but `SecondOrder` ADtype was not provided.
│         So a `SecondOrder` with AutoForwardDiff() for both inner and outer will be created, this can be suboptimal and not work in some cases so
│         an explicit `SecondOrder` ADtype is recommended.
└ @ OptimizationBase ~/work/Optimization.jl/Optimization.jl/lib/OptimizationBase/src/cache.jl:58
[0.49999999999999994]
```

--------------------------------

### Set Up Optimization Problem with Ipopt

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Configures the optimization problem using the defined constraints and initial guess for weights. It specifies bounds for weights (no short selling, single asset not exceeding 100%) and constraint bounds.

```julia
optfunc = OptimizationFunction(portfolio_risk, AutoZygote();
                              cons = portfolio_constraints)
w0 = fill(1.0/n_assets, n_assets)

prob = OptimizationProblem(optfunc, w0;
                          lb = zeros(n_assets),     # No short selling
                          ub = ones(n_assets),      # No single asset > 100%
                          lcons = [0.0, 0.0],       # Equality and inequality constraints
                          ucons = [0.0, Inf])
```

--------------------------------

### Callback Example with Plotting

Source: https://docs.sciml.ai/Optimization/stable/API/solve

An example of a callback function that displays the current loss value and optionally plots the prediction against the actual data. This requires the `predict` function to be defined, which in turn calls `solve`.

```julia
function predict(u)
    Array(solve(prob, Tsit5(), p = u))
end

function loss(u, p)
    pred = predict(u)
    sum(abs2, batch .- pred)
end

callback = function (state, l; doplot = false) #callback function to observe training
    display(l)
    # plot current prediction against data
    if doplot
        pred = predict(state.u)
        pl = scatter(t, ode_data[1, :], label = "data")
        scatter!(pl, t, pred[1, :], label = "prediction")
        display(plot(pl))
    end
    return false
end
```

--------------------------------

### Basic Usage of OptimizationODE.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ode

Demonstrates setting up an optimization problem with a manually provided gradient and solving it using the ODEGradientDescent optimizer.

```julia
using OptimizationODE, Optimization, ADTypes, SciMLBase

function f(x, p)
    return sum(abs2, x)
end

function g!(g, x, p)
    @. g = 2 * x
end

x0 = [2.0, -3.0]
p = []

f_manual = OptimizationFunction(f, SciMLBase.NoAD(); grad = g!)
prob_manual = OptimizationProblem(f_manual, x0)

opt = ODEGradientDescent(dt=0.01)
sol = solve(prob_manual, opt; maxiters=50_000)

@show sol.u
@show sol.objective
```

--------------------------------

### Basic Usage of `reinit!`

Source: https://docs.sciml.ai/Optimization/stable/tutorials/reusage_interface

Demonstrates the fundamental usage of `reinit!` to update parameters of an existing optimization cache and solve the problem again. Requires Optimization.jl, OptimizationOptimJL, ADTypes, and ForwardDiff.

```julia
using Optimization, OptimizationOptimJL, ADTypes, ForwardDiff
rosenbrock(u, p) = (p[1] - u[1])^2 + p[2] * (u[2] - u[1]^2)^2
u0 = zeros(2)
p = [1.0, 100.0]

optf = OptimizationFunction(rosenbrock, ADTypes.AutoForwardDiff())
prob = OptimizationProblem(optf, u0, p)

# Initialize cache and solve
cache = Optimization.init(prob, Optim.BFGS())
sol = Optimization.solve!(cache)

# Reinitialize cache with new parameters
cache = Optimization.reinit!(cache; p = [2.0, 50.0])
sol2 = Optimization.solve!(cache)
```

--------------------------------

### Sophia Optimization Result

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/sophia

Example output showing the return code and the optimized parameters (u) after running the Sophia optimizer.

```julia
retcode: Success
u: ComponentVector{Float32}(layer_1 = (weight = Float32[-0.49531996; 2.1828232; … ; 1.6459857; 2.4595056;;], bias = Float32[-0.5366939, 0.35397622, -0.2590761, -0.8706155, -0.6247321, 0.003280748, -0.0031517793, 0.40579414, -0.42528516, 0.2781369  …  -0.67254555, -0.731019, 0.07436459, -0.7001152, -0.76863915, -0.7966375, 0.21679728, 0.03269669, -0.3116036, -0.21531573]), layer_2 = (weight = Float32[-0.23208924 -0.11179227 … 0.032785162 0.23077805], bias = Float32[0.09673637]))
```

--------------------------------

### Optimize Rosenbrock Function with NOMADOpt

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/nomad

Optimize the Rosenbrock function using NOMADOpt. This example demonstrates usage with and without box-constraints.

```julia
using Optimization, OptimizationNOMAD
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]
f = OptimizationFunction(rosenbrock)

prob = OptimizationProblem(f, x0, p)
sol = Optimization.solve(prob, NOMADOpt())

prob = OptimizationProblem(f, x0, p, lb = [-1.0, -1.0], ub = [1.5, 1.5])
sol = Optimization.solve(prob, NOMADOpt())
```

--------------------------------

### Example DomainError in Julia

Source: https://docs.sciml.ai/Optimization/stable/API/FAQ

Demonstrates a DomainError when evaluating a function outside its valid domain, such as taking the logarithm of a negative number.

```julia
julia> log(-1)
ERROR: DomainError with -1.0:
log will only return a complex result if called with a complex argument. Try log(Complex(x)).
```

--------------------------------

### Install OptimizationPyCMA.jl

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/pycma

Add the OptimizationPyCMA package to your Julia environment. This package provides access to the CMA-ES optimizer via the Optimization.jl interface.

```julia
import Pkg
Pkg.add("OptimizationPyCMA")
```

--------------------------------

### Constrained Optimization with COBYLA

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/scipy

Example of using the COBYLA solver from SciPy.jl for constrained optimization problems. This method supports nonlinear constraints.

```julia
using Optimization, OptimizationSciPy

```

--------------------------------

### Create and Solve OptimizationProblem

Source: https://docs.sciml.ai/Optimization/stable/tutorials/symbolic

Complete the symbolic system and create an `OptimizationProblem` with specified initial conditions and parameters. Enable gradient and Hessian calculations for efficient solving with a chosen solver like Newton().

```julia
sys = complete(sys)
prob = OptimizationProblem(sys, u0, p, grad = true, hess = true)
solve(prob, Newton())
```

--------------------------------

### Optimize Rosenbrock with Evolutionary.CMAES

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/evolutionary

Optimize the Rosenbrock function using the CMAES algorithm from Evolutionary.jl. This example demonstrates setting up an OptimizationProblem with bounds and solving it.

```julia
using Optimization, OptimizationEvolutionary
rosenbrock(x, p) = (p[1] - x[1])^2 + p[2] * (x[2] - x[1]^2)^2
x0 = zeros(2)
p = [1.0, 100.0]
f = OptimizationFunction(rosenbrock)
prob = SciMLBase.OptimizationProblem(f, x0, p, lb = [-1.0, -1.0], ub = [1.0, 1.0])
sol = solve(prob, Evolutionary.CMAES(μ = 40, λ = 100))
```

--------------------------------

### Solve Optimization Problem with ScipyCOBYLA

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/scipy

Example of solving an optimization problem with a single non-linear constraint using ScipyCOBYLA. Requires defining the objective function and constraints.

```julia
obj(x, p) = (x[1] + x[2] - 1)^2
cons(res, x, p) = (res .= [x[1]^2 + x[2]^2 - 1.0])
x0 = [0.5, 0.5]
prob = OptimizationProblem(
    OptimizationFunction(obj; cons = cons),
    x0, nothing, lcons = [-1e-6], ucons = [1e-6])  # Small tolerance instead of exact equality

sol = solve(prob, ScipyCOBYLA())
@show sol.u, sol.objective
```

```julia
([0.9999995099640485, 1.1653740143742524e-5], 1.2462829129061485e-10)
```

--------------------------------

### Ipopt Optimizer with Multiple Option Sources

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/ipopt

Demonstrates how options from struct fields, `additional_options`, and `solve` arguments are prioritized. Common interface arguments to `solve` have the highest priority, followed by `additional_options`, and then struct field values.

```julia
opt = IpoptOptimizer(
    acceptable_tol = 1e-6,           # Struct field
    mu_strategy = "adaptive",        # Struct field
    linear_solver = "ma57",          # Struct field (needs HSL)
    print_timing_statistics = "yes", # Struct field
    additional_options = Dict(
        "alpha_for_y" => "primal",   # Not a struct field
        "max_iter" => 500            # Will be overridden by maxiters below
    )
)

sol = solve(prob, IpoptOptimizer();
    maxiters = 1000,  # Overrides max_iter in additional_options
    reltol = 1e-8     # Sets Ipopt's tol
)
```

--------------------------------

### Set Optimizer Options as Keyword Arguments

Source: https://docs.sciml.ai/Optimization/stable/optimization_packages/mathoptinterface

Alternatively, optimizer options can be set as keyword arguments directly in the solve function call. This example uses Ipopt.jl.

```julia
sol = solve(prob, Ipopt.Optimizer(); option_name = option_value, ...)
```

--------------------------------

### Define Variables with Bounds for Short-Term Financing

Source: https://docs.sciml.ai/Optimization/stable/tutorials/linearandinteger

This snippet sets up the necessary using statements and defines variables with their bounds for a short-term financing optimization problem. It requires OptimizationBase, OptimizationMOI, ModelingToolkit, HiGHS, LinearAlgebra, and SciMLBase.

```julia
using OptimizationBase, OptimizationMOI, ModelingToolkit, HiGHS, LinearAlgebra, SciMLBase

# Define variables with bounds

```

--------------------------------

### Solve with Optim.jl's Nelder-Mead Solver

Source: https://docs.sciml.ai/Optimization/stable/getting_started

Use the NelderMead solver from Optim.jl for derivative-free optimization. Ensure Optim.jl is imported.

```julia
using OptimizationOptimJL
sol = solve(prob, Optim.NelderMead())
```

```julia
retcode: Success
u: 2-element Vector{Float64}:
 0.9999634355313174
 0.9999315506115275
```

--------------------------------

### Rosenbrock Function with Adam Optimizer

Source: https://docs.sciml.ai/Optimization/stable/examples/rosenbrock

Solves the Rosenbrock function using the Adam optimizer with reverse-mode automatic differentiation via Zygote.jl. Ensure `OptimizationOptimisers` and `Zygote` are installed.

```julia
import OptimizationOptimisers
optf = SciMLBase.OptimizationFunction(rosenbrock, ADTypes.AutoZygote())
prob = SciMLBase.OptimizationProblem(optf, x0, _p)
sol = solve(prob, OptimizationOptimisers.Adam(0.05), maxiters = 1000, progress = false)
```

--------------------------------

### Define ODE and Model for Minibatching

Source: https://docs.sciml.ai/Optimization/stable/tutorials/minibatch

Sets up the Newton's cooling ODE, a Lux neural network model, and prepares data for minibatching. This includes defining the ODE problem, generating synthetic data, and creating a data loader.

```julia
using Lux, OptimizationBase, OptimizationOptimisers, OrdinaryDiffEq, SciMLSensitivity, MLUtils,
      Random, ComponentArrays, ADTypes, Zygote

function newtons_cooling(du, u, p, t)
    temp = u[1]
    k, temp_m = p
    du[1] = dT = -k * (temp - temp_m)
end

function true_sol(du, u, p, t)
    true_p = [log(2) / 8.0, 100.0]
    newtons_cooling(du, u, true_p, t)
end

model = Chain(Dense(1, 32, tanh), Dense(32, 1))
ps, st = Lux.setup(Random.default_rng(), model)
ps_ca = ComponentArray(ps)
smodel = StatefulLuxLayer{true}(model, nothing, st)

function dudt_(u, p, t)
    smodel(u, p) .* u
end

function callback(state, l) #callback function to observe training
    display(l)
    return false
end

u0 = Float32[200.0]
datasize = 30
tspan = (0.0f0, 1.5f0)

t = range(tspan[1], tspan[2], length = datasize)
true_prob = ODEProblem(true_sol, u0, tspan)
ode_data = Array(solve(true_prob, Tsit5(), saveat = t))

prob = ODEProblem{false}(dudt_, u0, tspan, ps_ca)
```

--------------------------------

### Rosenbrock Function with BlackBoxOptim.jl

Source: https://docs.sciml.ai/Optimization/stable/examples/rosenbrock

Solves the Rosenbrock function with box constraints using BlackBoxOptim.jl. This example showcases a black-box optimization approach suitable for problems without gradient information.

```julia
using OptimizationBBO
prob = SciMLBase.OptimizationProblem(rosenbrock, [0.0, 0.3], _p, lb = [-1.0, 0.2],
    ub = [0.8, 0.43])
sol = solve(prob, BBO_adaptive_de_rand_1_bin()) # -1.0 ≤ x[1] ≤ 0.8, 0.2 ≤ x[2] ≤ 0.43
```
Tokens:
Raw