Optuna

https://optuna.readthedocs.io/en/stable/
Admin
Optuna is an automatic hyperparameter optimization framework designed for machine learning,...
Tokens:379,714
Snippets:2,548
Trust Score:8.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Create MySQL database for Optuna RDBStorage

Source: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html

Creates a MySQL database named 'example' for use as an Optuna RDB storage backend. Run this command on the MySQL server before starting multi-node optimization.

```bash
$ mysql -u username -e "CREATE DATABASE IF NOT EXISTS example"

```

--------------------------------

### Create and optimize a study for copying with optuna.copy_study

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/study/study.html

Create and optimize a study in SQLite storage as the first step of copying it. The example assumes 'example.db' does not already exist; the test setup raises an error if it does.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


study = optuna.create_study(
    study_name="example-study",
    storage="sqlite:///example.db",
)
study.optimize(objective, n_trials=3)
```

--------------------------------

### Multi-objective optimization imports and setup with PyTorch and fvcore

Source: https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/002_multi_objective.html

Imports PyTorch, fvcore, and Optuna, then sets up the device, batch size, and example counts used by the Fashion MNIST multi-objective optimization example. Use this as the starting block for the tutorial's training and FLOPS measurement code.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from fvcore.nn import FlopCountAnalysis

import optuna


DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
DIR = ".."
BATCHSIZE = 128
N_TRAIN_EXAMPLES = BATCHSIZE * 30
N_VALID_EXAMPLES = BATCHSIZE * 30
```

--------------------------------

### Install optunahub via pip

Source: https://optuna.readthedocs.io/en/stable/index.html

Install the OptunaHub package using pip.

```bash
$ pip install optunahub

```

--------------------------------

### Install Optuna via pip

Source: https://optuna.readthedocs.io/en/stable/installation.html

Recommended installation method. Requires Python 3.9 or newer.

```bash
$ pip install optuna

```

--------------------------------

### Download full example code

Source: https://optuna.readthedocs.io/en/stable/reference/visualization/matplotlib/generated/optuna.visualization.matplotlib.optimization_history.html

Link to download the full example code for plot_optimization_history.

```text
Go to the end to download the full example code.
```

--------------------------------

### Ask and Tell

Source: https://optuna.readthedocs.io/en/stable/_downloads/9e17bbf554dfa22e6c92f27c8c7df53d/009_ask_and_tell.ipynb

This example demonstrates the ask-and-tell interface of Optuna, which is useful when you want to control the optimization loop manually. It creates a study with the CMA-ES sampler and evaluates a batch of 10 hyperparameter pairs three times, for a total of 30 trials. The example shows how to use ``study.ask()`` to get a trial and ``study.tell()`` to report the objective value after batched evaluations.

```python
batch_size = 10
study = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())

for _ in range(3):
    # create batch
    trial_numbers = []
    x_batch = []
    y_batch = []
    for _ in range(batch_size):
        trial = study.ask()
        trial_numbers.append(trial.number)
        x_batch.append(trial.suggest_float("x", -10, 10))
        y_batch.append(trial.suggest_float("y", -10, 10))

```

--------------------------------

### Module imports and setup for optuna.trial._trial

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/trial/_trial.html

Imports and module-level setup for the optuna.trial._trial module, including type-checking imports and a deprecation message constant.

```python
from __future__ import annotations

from collections import UserDict
import copy
import math
from typing import Any
from typing import overload
from typing import TYPE_CHECKING

import optuna
from optuna import distributions
from optuna import logging
from optuna import pruners
from optuna._deprecated import deprecated_func
from optuna._warnings import optuna_warn
from optuna.distributions import BaseDistribution
from optuna.distributions import CategoricalChoiceType
from optuna.distributions import CategoricalDistribution
from optuna.distributions import FloatDistribution
from optuna.distributions import IntDistribution
from optuna.study._constrained_optimization import _CONSTRAINTS_KEY
from optuna.study._constrained_optimization import _get_constraints_from_system_attrs
from optuna.trial._base import BaseTrial


if TYPE_CHECKING:
    from collections.abc import Sequence
    import datetime

    from optuna.study import Study
    from optuna.trial import FrozenTrial


_logger = logging.get_logger(__name__)
_suggest_deprecated_msg = "Use suggest_float{args} instead."
```

--------------------------------

### Install optuna-dashboard via pip

Source: https://optuna.readthedocs.io/en/stable/index.html

Install the Optuna Dashboard package using pip. The command is `pip install optuna-dashboard`.

```bash
$ pip install optuna-dashboard

```

--------------------------------

### Example usage of Boto3ArtifactStore with upload_artifact

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/artifacts/_boto3.html

Example from the class docstring showing how to create a Boto3ArtifactStore and use it with upload_artifact in an Optuna objective function. Note that the example uses placeholder ellipses (...) for parts not shown.

```python
import optuna
from optuna.artifacts import upload_artifact
from optuna.artifacts import Boto3ArtifactStore


artifact_store = Boto3ArtifactStore("my-bucket")


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(
        artifact_store=artifact_store,
        file_path=file_path,
        study_or_trial=trial,
    )
    return ...
```

--------------------------------

### Install development version from Git repository

Source: https://optuna.readthedocs.io/en/stable/installation.html

Installs the development version of Optuna from the master branch of the Git repository.

```bash
$ pip install git+https://github.com/optuna/optuna.git

```

--------------------------------

### Install Plotly and nbformat for visualization

Source: https://optuna.readthedocs.io/en/stable/_downloads/a4ea522762978cb29312e091bc523c7a/002_multi_objective.ipynb

Installs Plotly for visualization and nbformat for running the tutorial in Jupyter Notebook. Required before using the visualization functions.

```console
$ pip install plotly
$ pip install nbformat  # Required if you are running this tutorial in Jupyter Notebook.
```

--------------------------------

### Show CLI help with $ optuna --help

Source: https://optuna.readthedocs.io/en/stable/reference/cli.html

Run this command to display the command-line help for Optuna's CLI. The output lists available commands and options. The Command-Line Interface tutorial provides use-cases with examples.

```shell
$ optuna --help
```

--------------------------------

### Get all study names with optuna.study.get_all_study_names

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/study/study.html

Use this example to retrieve all study names from a storage. It creates a study with a SQLite storage, optimizes it, then calls get_all_study_names and asserts the result. The example includes setup and cleanup steps that remove the database file.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db")
study.optimize(objective, n_trials=3)

study_names = optuna.study.get_all_study_names(storage="sqlite:///example.db")
assert len(study_names) == 1

assert study_names[0] == "example-study"
```

--------------------------------

### Example: optuna.get_all_study_summaries usage

Source: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.get_all_study_summaries.html

Shows how to call optuna.get_all_study_summaries with a storage argument. The example imports optuna and defines an objective function, then creates and optimizes a study before retrieving its summary.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return x**2


study = optuna.create_study(storage="sqlite:///example.db")
study.optimize(objective, n_trials=2)

summaries = optuna.get_all_study_summaries(storage="sqlite:///example.db")

```

--------------------------------

### Setup and helper functions for adsorption optimization

Source: https://optuna.readthedocs.io/en/stable/_downloads/6e6a1710a8301512ef0074d3b66ee7fd/012_artifact_tutorial.ipynb

Imports necessary libraries and defines helper functions for energy calculation, slab and molecule creation, and conversion between atoms and JSON strings.

```python
from __future__ import annotations

import io
import logging
import os
import sys
import tempfile

from ase import Atoms
from ase.build import bulk, fcc111, molecule, add_adsorbate
from ase.calculators.emt import EMT
from ase.io import write, read
from ase.optimize import LBFGS
import numpy as np
from optuna.artifacts import FileSystemArtifactStore
from optuna.artifacts import upload_artifact
from optuna.artifacts import download_artifact
from optuna.logging import get_logger
from optuna import create_study
from optuna import Trial


# Add stream handler of stdout to show the messages
get_logger("optuna").addHandler(logging.StreamHandler(sys.stdout))


def get_opt_energy(atoms: Atoms, fmax: float = 0.001) -> float:
    calculator = EMT()
    atoms.set_calculator(calculator)
    opt = LBFGS(atoms, logfile=None)
    opt.run(fmax=fmax)
    return atoms.get_total_energy()


def create_slab() -> tuple[Atoms, float]:
    calculator = EMT()
    bulk_atoms = bulk("Pt", cubic=True)
    bulk_atoms.calc = calculator

    a = np.mean(np.diag(bulk_atoms.cell))
    slab = fcc111("Pt", a=a, size=(4, 4, 4), vacuum=40.0, periodic=True)
    slab.calc = calculator
    E_slab = get_opt_energy(slab, fmax=1e-4)
    return slab, E_slab


def create_mol() -> tuple[Atoms, float]:
    calculator = EMT()
    mol = molecule("CO")
    mol.calc = calculator
    E_mol = get_opt_energy(mol, fmax=1e-4)
    return mol, E_mol


def atoms_to_json(atoms: Atoms) -> str:
    f = io.StringIO()
    write(f, atoms, format="json")
    return f.getvalue()


def json_to_atoms(atoms_str: str) -> Atoms:
    return read(io.StringIO(atoms_str), format="json")


def file_to_atoms(file_path: str) -> Atoms:
    return read(file_path, format="json")
```

--------------------------------

### Get default verbosity level with optuna.logging.get_verbosity

Source: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.logging.get_verbosity.html

Shows how to retrieve the current verbosity level of Optuna's root logger. The example imports optuna and calls get_verbosity() to get the default verbosity level.

```python
import optuna

# The default verbosity level of Optuna is WARNING.
print(optuna.logging.get_verbosity())
```

--------------------------------

### Get all study names with optuna.get_all_study_names

Source: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.get_all_study_names.html

Retrieves all study names stored in a specified storage. The example shows a basic Optuna objective function that suggests a float parameter and returns its value.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return x
```

--------------------------------

### FileSystemArtifactStore example

Source: https://optuna.readthedocs.io/en/stable/reference/artifacts.html

Creates a FileSystemArtifactStore with a local base path and uploads an artifact from within an Optuna objective function. Requires the base directory to exist; the example creates it with os.makedirs.

```python
import os

import optuna
from optuna.artifacts import FileSystemArtifactStore
from optuna.artifacts import upload_artifact


base_path = "./artifacts"
os.makedirs(base_path, exist_ok=True)
artifact_store = FileSystemArtifactStore(base_path=base_path)


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(
        artifact_store=artifact_store,
        file_path=file_path,
        study_or_trial=trial,
    )
    return ...

```

--------------------------------

### Delete a study from SQLite storage with optuna.delete_study

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/study/study.html

Create and optimize a study, then delete it from SQLite storage using optuna.delete_study. The example assumes 'example.db' does not already exist; the test setup raises an error if it does.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


study = optuna.create_study(study_name="example-study", storage="sqlite:///example.db")
study.optimize(objective, n_trials=3)

optuna.delete_study(study_name="example-study", storage="sqlite:///example.db")
```

--------------------------------

### Run distributed example on Machine 1

Source: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html

Runs the distributed example on the first machine. The output shows the study being created in RDB and trials being executed, with the best trial value reported.

```bash
$ python3 distributed_example.py
[I 2025-06-03 14:07:45,306] A new study created in RDB with name: distributed_test
[I 2025-06-03 14:08:45,450] Trial 0 finished with value: 12.694308312865278 and parameters: {'x': -1.5629072837873959}. Best is trial 0 with value: 12.694308312865278.
[I 2025-06-03 14:08:45,450] Trial 2 finished with value: 121.80632032697125 and parameters: {'x': -9.036590067904635}. Best is trial 0 with value: 12.694308312865278.

```

--------------------------------

### Import dependencies and create classification dataset

Source: https://optuna.readthedocs.io/en/stable/_downloads/9e17bbf554dfa22e6c92f27c8c7df53d/009_ask_and_tell.ipynb

Sets up the environment for the classification example by importing NumPy, scikit-learn, and Optuna, then generates a synthetic 10-feature classification dataset and splits it into training and test sets.

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

import optuna


X, y = make_classification(n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
```

--------------------------------

### GCSArtifactStore example with gcloud authentication

Source: https://optuna.readthedocs.io/en/stable/reference/artifacts.html

Creates a GCSArtifactStore for a Google Cloud Storage bucket and uploads an artifact from within an Optuna objective function. Before running, install gcloud and execute 'gcloud auth application-default login' so the Cloud Storage library can find credentials automatically.

```python
import optuna
from optuna.artifacts import GCSArtifactStore, upload_artifact


artifact_store = GCSArtifactStore("my-bucket")


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(
        artifact_store=artifact_store,
        file_path=file_path,
        study_or_trial=trial,
    )
    return ...

```

```bash
gcloud auth application-default login

```

--------------------------------

### Configure logging to file and verify output

Source: https://optuna.readthedocs.io/en/stable/_modules/optuna/logging.html

Use this snippet to set up logging to a file, run an optimization, and then verify the log file contents. The example logs an info message and checks that the file contains the expected lines.

```python
logger.info("Start optimization.")
study.optimize(objective, n_trials=10)

with open("foo.log") as f:
    assert f.readline().startswith("A new study created")
    assert f.readline() == "Start optimization.\n"
```

--------------------------------

### Run gRPC proxy server with MySQL storage

Source: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.storages.run_grpc_proxy_server.html

Example usage of run_grpc_proxy_server. It creates a storage object using get_storage() with a MySQL connection string and starts the server on localhost:13000. Note that get_storage() is recommended over RDBStorage to avoid slowdowns due to lack of in-process caching.

```python
from optuna.storages import run_grpc_proxy_server
from optuna.storages import get_storage

storage = get_storage("mysql+pymysql://<user>:<pass>@<host>/<dbname>[?<options>]")
run_grpc_proxy_server(storage, host="localhost", port=13000)
```

--------------------------------

### Import libraries and define model, training, and evaluation functions

Source: https://optuna.readthedocs.io/en/stable/_downloads/a4ea522762978cb29312e091bc523c7a/002_multi_objective.ipynb

Sets up the necessary imports (torch, torchvision, fvcore, optuna), defines device and data constants, and implements the model definition with hyperparameters, training loop, and evaluation that returns FLOPS and accuracy.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from fvcore.nn import FlopCountAnalysis

import optuna


DEVICE = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
DIR = ".."
BATCHSIZE = 128
N_TRAIN_EXAMPLES = BATCHSIZE * 30
N_VALID_EXAMPLES = BATCHSIZE * 10


def define_model(trial):
    n_layers = trial.suggest_int("n_layers", 1, 3)
    layers = []

    in_features = 28 * 28
    for i in range(n_layers):
        out_features = trial.suggest_int(f"n_units_l{i}", 4, 128)
        layers.append(nn.Linear(in_features, out_features))
        layers.append(nn.ReLU())
        p = trial.suggest_float(f"dropout_{i}", 0.2, 0.5)
        layers.append(nn.Dropout(p))

        in_features = out_features

    layers.append(nn.Linear(in_features, 10))
    layers.append(nn.LogSoftmax(dim=1))

    return nn.Sequential(*layers)


# Defines training and evaluation.
def train_model(model, optimizer, train_loader):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
        optimizer.zero_grad()
        F.nll_loss(model(data), target).backward()
        optimizer.step()


def eval_model(model, valid_loader):
    model.eval()
    correct = 0
    with torch.no_grad():
        for batch_idx, (data, target) in enumerate(valid_loader):
            data, target = data.view(-1, 28 * 28).to(DEVICE), target.to(DEVICE)
            pred = model(data).argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    accuracy = correct / N_VALID_EXAMPLES

    flops = FlopCountAnalysis(model, inputs=(torch.randn(1, 28 * 28).to(DEVICE),)).total()
    return flops, accuracy
```

--------------------------------

### plot_optimization_history example

Source: https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_optimization_history.html

Full example that creates a study, optimizes the objective function, and plots the optimization history. The example is provided as a downloadable Python script.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


study = optuna.create_study()
study.optimize(objective, n_trials=3)

optuna.visualization.plot_optimization_history(study)
```

--------------------------------

### CmaEsSampler Example

Source: https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.CmaEsSampler.html

Example of optimizing a simple quadratic function using CmaEsSampler.

```APIDOC
## CmaEsSampler

### Description
A sampler using cmaes as the backend for Optuna.

### Example
```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -1, 1)
    y = trial.suggest_int("y", -1, 1)
    return x**2 + y


sampler = optuna.samplers.CmaEsSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)
```
```

--------------------------------

### Connect to GrpcStorageProxy and optimize

Source: https://optuna.readthedocs.io/en/stable/_downloads/08c086493809e0245421fbbf4cefdd32/004_distributed.ipynb

Connects to the gRPC proxy storage and runs a study with 50 trials. Requires the proxy server from the previous example to be running. Uses load_if_exists=True to resume an existing study.

```python
import optuna

from optuna.storages import GrpcStorageProxy


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return (x - 2) ** 2


if __name__ == "__main__":
    storage = GrpcStorageProxy(host="localhost", port=13000)
    study = optuna.create_study(
        study_name="grpc_proxy_multinode",
        storage=storage,
        load_if_exists=True,
    )
    study.optimize(objective, n_trials=50)

```

--------------------------------

### QMCSampler Example

Source: https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.QMCSampler.html

Example of optimizing a simple quadratic function using QMCSampler.

```APIDOC
## QMCSampler Example

### Description
Optimize a simple quadratic function by using `QMCSampler`.

### Code
```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -1, 1)
    y = trial.suggest_int("y", -1, 1)
    return x**2 + y


sampler = optuna.samplers.QMCSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=8)
```

### Note
Added in v3.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v3.0.0.
```

--------------------------------

### Run distributed example on Machine 2

Source: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/004_distributed.html

Runs the same distributed example on a second machine. The output shows that the existing study is reused instead of creating a new one, and trials are executed concurrently with the first machine.

```bash
$ python3 distributed_example.py
[I 2025-06-03 14:07:49,318] Using an existing study with name 'distributed_test' instead of creating a new one.
[I 2025-06-03 14:08:49,442] Trial 1 finished with value: 0.21258674253407828 and parameters: {'x': 1.5389287012466746}. Best is trial 31 with value: 9.19159178106083e-05.
[I 2025-06-03 14:09:49,480] Trial 3 finished with value: 0.24343413718999274 and parameters: {'x': 2.493390451052706}. Best is trial 31 with value: 9.19159178106083e-05.

```

--------------------------------

### FrozenTrial usage example

Source: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.FrozenTrial.html

Example of using FrozenTrial with Optuna study optimization.

```python
import optuna


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return x**2


study = optuna.create_study()
study.optimize(objective, n_trials=3)

best_trial = study.best_trial
print(best_trial)

```

--------------------------------

### Boto3ArtifactStore example

Source: https://optuna.readthedocs.io/en/stable/reference/artifacts.html

Creates a Boto3ArtifactStore for an S3-compatible bucket and uploads an artifact from within an Optuna objective function. The client is created automatically unless one is provided.

```python
import optuna
from optuna.artifacts import upload_artifact
from optuna.artifacts import Boto3ArtifactStore


artifact_store = Boto3ArtifactStore("my-bucket")


def objective(trial: optuna.Trial) -> float:
    ... = trial.suggest_float("x", -10, 10)
    file_path = generate_example(...)
    upload_artifact(
        artifact_store=artifact_store,
        file_path=file_path,
        study_or_trial=trial,
    )
    return ...

```
Tokens:
Raw