Oumi: Open Universal Machine Intelligence

https://www.oumi.ai/docs/en/latest
Admin
Oumi is an open-source platform designed for ML engineers and researchers to train, fine-tune,...
Tokens:1,432,742
Snippets:8,589
Trust Score:8.6
Update:11 hours ago
Show doc for...
Tokens:
Raw
### Run training with quickstart config

Source: https://www.oumi.ai/docs/en/latest/get_started/quickstart.html

Command to start training using the quickstart recipe. Uses the `oumi train` CLI with the config path.

```bash
oumi train -c configs/recipes/smollm/sft/135m/quickstart_train.yaml
```

--------------------------------

### Install Oumi and run training and inference

Source: https://www.oumi.ai/docs/en/latest/index.html

Install Oumi with GPU support (or CPU-only with `pip install oumi`), then train a model using the quickstart config and run interactive inference.

```bash
# Install with GPU support (or use `pip install oumi` for CPU-only)
pip install oumi[gpu]

# Train a model
oumi train -c configs/recipes/smollm/sft/135m/quickstart_train.yaml

# Run inference
oumi infer -c configs/recipes/smollm/inference/135m_infer.yaml --interactive
```

--------------------------------

### Example: Using LocalCloud to run a job

Source: https://www.oumi.ai/docs/en/latest/api/oumi.launcher.clouds.html

Shows how to instantiate LocalCloud, create a JobResources with cloud='local', and a JobConfig with a run command, then call up_cluster to start the job. Requires necessary credentials and configurations for the cloud platform.

```python
>>> from oumi.launcher import JobConfig, JobResources
>>> from oumi.launcher.clouds import LocalCloud
>>> local_cloud = LocalCloud()
>>> job_resources = JobResources(cloud="local")
>>> job_config = JobConfig(
...     name="my_job", resources=job_resources, run="python train.py"
... )
>>> job_status = local_cloud.up_cluster(job_config, name="my_cluster")
```

--------------------------------

### Serve docs locally

Source: https://www.oumi.ai/docs/en/latest/development/docs_guide.html

Build the latest docs and open them in Chrome. Local URL localhost:8000/get_started/quickstart.html corresponds to https://oumi.ai/docs/en/latest/get_started/quickstart.html. Cancel with ctrl+c.

```bash
# Switch to your oumi directory, wherever that may be, then run
make docs-serve

# To cancel: ctrl+c
```

--------------------------------

### Install evaluation dependencies in setup script

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/evaluate/leaderboards.md

YAML configuration for a setup script that installs Oumi's evaluation packages on the remote machine before the job starts. This is executed during cluster creation.

```yaml
setup: |
  pip install oumi[evaluation]
```

--------------------------------

### Build docs for first time

Source: https://www.oumi.ai/docs/en/latest/development/docs_guide.html

Run this command the first time you build the docs locally.

```bash
make docs-rebuild
```

--------------------------------

### Install Oumi with dev dependencies

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/train/environments/notebooks.md

Installs Oumi OSS with development dependencies, which includes Jupyter. Use this option to get everything at once.

```bash
# Install Oumi OSS with development dependencies (includes Jupyter)
pip install oumi[dev]
```

--------------------------------

### Set up GCP service account credentials

Source: https://www.oumi.ai/docs/en/latest/get_started/quickstart.html

Export the path to the service account key JSON file, activate the service account, and set the GCP project. Replace `/path/to/key.json` and `<YOUR_PROJECT>` with your actual values.

```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud config set project <YOUR_PROJECT>
```

--------------------------------

### Launch a training job with oumi launch

Source: https://www.oumi.ai/docs/en/latest/_sources/get_started/quickstart.md

Launch a training job using the quickstart GCP config file.

```yaml
configs/recipes/smollm/sft/135m/quickstart_gcp_job.yaml
```

```bash
oumi launch up -c configs/recipes/smollm/sft/135m/quickstart_gcp_job.yaml
```

--------------------------------

### Clone and install sphinx-term fork

Source: https://www.oumi.ai/docs/en/latest/development/docs_guide.html

One-time setup: clone the local fork of sphinx-term and install it in the same Conda env used for Oumi OSS development.

```bash
git clone https://github.com/oumi-ai/sphinx-term.git
cd sphinx-term
git submodule update --init --recursive
pip install -e .
```

--------------------------------

### Cross-rank statistics output example

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/train/monitoring.md

Example output from cross-rank statistics in a 4-GPU training setup, showing GPU temperature and timer metrics with mean, standard deviation, min, and max values.

```text
Cross-Rank Statistics:
GPU Temperature:
    max:
        mean: 75.2C   # Average of max temperatures across ranks
        std_dev: 2.1C # Variation in max temperatures
        min: 72.0C    # Lowest max temperature among ranks
        max: 78.0C    # Highest max temperature among ranks
    mean:
        mean: 71.5C   # Average of mean temperatures
        std_dev: 1.8C # Variation in mean temperatures
        min: 69.0C    # Lowest mean temperature
        max: 74.0C    # Highest mean temperature

Timers:
    model_forward:
        mean:
            mean: 0.096s    # Average forward pass time across ranks
            std_dev: 0.008s # Variation in forward pass times
            min: 0.085s     # Fastest rank
            max: 0.105s     # Slowest rank
        max:
            mean: 0.125s    # Average of max forward times
            std_dev: 0.010s # Variation in max times
            min: 0.115s     # Lowest max time
            max: 0.142s     # Highest max time
    preprocessing:
        mean:
            mean: 0.016s    # Average preprocessing time
            std_dev: 0.003s # Variation in preprocessing
            min: 0.012s     # Fastest rank
            max: 0.020s     # Slowest rank
```

--------------------------------

### Train SmolLM-135M with quickstart recipe (Bash)

Source: https://www.oumi.ai/docs/en/latest/user_guides/train/train.html

Runs supervised fine-tuning on SmolLM-135M using the quickstart_train.yaml config. Downloads the model, loads the yahma/alpaca-cleaned dataset, uses the TRL_SFT trainer, and saves output to config.output_dir.

```bash
# Train a small model (SmolLM-135M)

oumi train -c configs/recipes/smollm/sft/135m/quickstart_train.yaml
```

--------------------------------

### SkyClient.submit

Source: https://www.oumi.ai/docs/en/latest/api/oumi.launcher.clients.html

Submit a cluster launch without waiting for the job to start. Pass the returned request ID to wait() to get the job or to cancel_request() to cancel the launch.

```APIDOC
## SkyClient.submit(job: JobConfig, cluster_name: str | None = None, **kwargs) -> str

### Description
Submit a cluster launch without waiting for the job to start. Pass the returned request ID to wait() to get the job or to cancel_request() to cancel the launch.

### Method
POST

### Endpoint
N/A (SDK method)

### Parameters
- **job** (JobConfig) - Required - Job to run on the cluster.
- **cluster_name** (str | None) - Optional - Name of the cluster to create.
- **kwargs** - Optional - Launch options, including idle_minutes_to_autostop and down.

### Returns
ID of the launch request on the Sky Pilot server.

```

--------------------------------

### TurnStatsAnalyzer example

Source: https://www.oumi.ai/docs/en/latest/api/oumi.analyze.analyzers.html

Shows how to instantiate TurnStatsAnalyzer, create a Conversation with user and assistant messages, and analyze it to get turn statistics. The output prints the number of turns.

```python
>>> from oumi.analyze.analyzers.turn_stats import TurnStatsAnalyzer
>>> from oumi.core.types.conversation import Conversation, Message, Role
>>>
>>> analyzer = TurnStatsAnalyzer()
>>> conversation = Conversation(messages=[
...     Message(role=Role.USER, content="What is Python?"),
...     Message(
...         role=Role.ASSISTANT,
...         content="Python is a programming language.",
...     ),
... ])
>>> result = analyzer.analyze(conversation)
>>> print(f"Turns: {result.num_turns}")
Turns: 2
```

--------------------------------

### Start basic vLLM server

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/infer/inference_engines.md

Launch a vLLM server for development and testing with a specified model and port.

```bash
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --port 6864
```

--------------------------------

### Start Training with build_trainer

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/train/environments/notebooks.md

Builds a trainer from the loaded configuration and starts the training process.

```python
# Start training
trainer = build_trainer(config)
trainer.train()
```

--------------------------------

### Multi-modal Inference Setup with VLLMInferenceEngine (Python)

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/infer/infer.md

Configures model parameters for a multi-modal model (LLaVA) and initializes the VLLMInferenceEngine. This is the first part of the multi-modal inference example.

```python
from oumi.inference import VLLMInferenceEngine
from oumi.core.configs import InferenceConfig, InferenceEngineType, ModelParams, GenerationParams
from oumi.core.types.conversation import Conversation, ContentItem, Message, Role, Type

model_params = ModelParams(
    model_name="llava-hf/llava-1.5-7b-hf",
    model_max_length=1024,
    chat_template="llava",
)

engine = VLLMInferenceEngine(model_params)

```

--------------------------------

### Get full request headers

Source: https://www.oumi.ai/docs/en/latest/_modules/oumi/inference/remote_inference_engine.html

Builds the complete header set by starting with the accept-encoding header and updating with the result of _get_auth_headers. This method is final and should not be overridden; override _get_auth_headers instead.

```python
    @final
    def _get_request_headers(
        self, remote_params: RemoteParams | None
    ) -> dict[str, str]:
        """Returns the full header set. Override `_get_auth_headers` instead."""
        headers = {_ACCEPT_ENCODING_KEY: _ACCEPT_ENCODING}
        headers.update(self._get_auth_headers(remote_params))
        return headers
```

--------------------------------

### Launch a job with Launcher, JobConfig, and JobResources

Source: https://www.oumi.ai/docs/en/latest/api/oumi.launcher.html

Shows how to create a Launcher instance, define job resources and configuration, and launch a job with the up method. Requires cloud credentials and configurations for the intended platform.

```python
>>> from oumi.launcher import Launcher, JobConfig, JobResources
>>> launcher = Launcher()
>>> job_resources = JobResources(cloud="local")
>>> job_config = JobConfig(
...     name="my_job", resources=job_resources, run="python train.py"
... )
>>> job_status = launcher.up(job_config, cluster_name="my_cluster")
```

--------------------------------

### _get_api_key method

Source: https://www.oumi.ai/docs/en/latest/_modules/oumi/inference/gcp_inference_engine.html

Gets the authentication token for GCP. Uses service account credentials from api_key if provided, otherwise uses default credentials. Requires google-auth, with a RuntimeError if not installed.

```python
    @override
    def _get_api_key(self, remote_params: RemoteParams) -> str:
        """Gets the authentication token for GCP."""
        try:
            from google.auth import default  # pyright: ignore[reportMissingImports]
            from google.auth.transport.requests import (  # pyright: ignore[reportMissingImports]
                Request,
            )
            from google.oauth2 import (  # pyright: ignore[reportMissingImports]
                service_account,
            )
        except ModuleNotFoundError:
            raise RuntimeError(
                "Google-auth is not installed. "
                "Please install oumi with GCP extra:`pip install oumi[gcp]`, "
                "or install google-auth with `pip install google-auth`."
            )

        if remote_params.api_key:
            credentials = service_account.Credentials.from_service_account_file(
                filename=remote_params.api_key,
                scopes=["https://www.googleapis.com/auth/cloud-platform"],
            )
        else:
            credentials, _ = default(
                scopes=["https://www.googleapis.com/auth/cloud-platform"]
            )
        credentials.refresh(Request())  # type: ignore
        return credentials.token  # type: ignore
```

--------------------------------

### Verify installation with oumi --help

Source: https://www.oumi.ai/docs/en/latest/get_started/installation.html

Run `oumi --help` to verify that Oumi OSS is installed correctly. This should print the help message.

```bash
oumi --help
```

--------------------------------

### Quick Start: Using RuleBasedJudge

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/judge/judge.md

Instantiate RuleBasedJudge with a config and judge a list of input dicts. The output objects expose field_values and field_scores; the example prints judgment and score for each input.

```python
from oumi.judges.rule_based_judge import RuleBasedJudge

judge = RuleBasedJudge(judge_config="oumi://configs/projects/judges/rule_based/regex_match_phone.yaml")

outputs = judge.judge([
    {"response": "Call me at 555-1234."},
    {"response": "Send an email."},
])

for out in outputs:
    print(out.field_values["judgment"], out.field_scores["judgment"])
# True 1.0
# False 0.0
```

--------------------------------

### Build and serve documentation with Sphinx

Source: https://www.oumi.ai/docs/en/latest/development/style_guide.html

Use these make commands to rebuild the documentation from scratch or serve the existing HTML.

```bash
make docs-rebuild # cleanup everything, rebuild apidocs, rebuild html
make docs-serve   # serve existing html
```

--------------------------------

### Oumi Inference Workflow Example

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/infer/infer.md

Shows the five-step workflow of initializing an engine, preparing input, configuring inference, running inference, and processing the output.

```python
# 1. Initialize engine
engine = VLLMInferenceEngine(model_params)

# 2. Prepare input
conversation = Conversation(messages=[...])

# 3. Configure inference
config = InferenceConfig(...)

# 4. Run inference
result = engine.infer([conversation], config)

# 5. Process output
response = result[0].messages[-1].content
```

--------------------------------

### Using TelemetryManager.capture_operation

Source: https://www.oumi.ai/docs/en/latest/api/oumi.telemetry.html

Example from the TelemetryManager class docstring showing how to get the singleton instance and use capture_operation as a context manager to automatically capture timing, success/failure, and exceptions for the wrapped code.

```python
telemetry = TelemetryManager.get_instance()
with telemetry.capture_operation("my-event"):
    do_work()  # Timing, success/failure, and exceptions captured automatically
```

--------------------------------

### LengthAnalyzer example: compute token counts

Source: https://www.oumi.ai/docs/en/latest/api/oumi.analyze.analyzers.html

Shows how to use LengthAnalyzer.from_config with a tokenizer name, create a Conversation, and analyze it to get total token count. The output is 'Total tokens: 12'.

```python
>>> from oumi.analyze.analyzers.length import LengthAnalyzer
>>> from oumi.core.types.conversation import Conversation, Message, Role
>>>
>>> analyzer = LengthAnalyzer.from_config({"tokenizer_name": "cl100k_base"})
>>> conversation = Conversation(messages=[
...     Message(role=Role.USER, content="Hello, how are you?"),
...     Message(role=Role.ASSISTANT, content="I'm doing well, thanks!"),
... ])
>>> result = analyzer.analyze(conversation)
>>> print(f"Total tokens: {result.total_tokens}")
Total tokens: 12
```

--------------------------------

### Guided decoding with Pydantic schema and OpenAIInferenceEngine

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/infer/common_workflows.md

Use GuidedDecodingParams with a Pydantic model schema to constrain the model to output valid JSON. The example sets temperature to 0 for deterministic output and validates the response with model_validate_json.

```python
from pydantic import BaseModel
from oumi.inference import OpenAIInferenceEngine
from oumi.core.configs import (
    ModelParams,
    RemoteParams,
    GenerationParams,
    GuidedDecodingParams,
)


# Define output schema using Pydantic
class ProductInfo(BaseModel):
    name: str
    price: float
    features: list[str]
    color: str


config = InferenceConfig(
    model=ModelParams(model_name="gpt-4o-mini"),

    generation=GenerationParams(
        max_new_tokens=512,
        temperature=0,  # Use deterministic output for structured data
        guided_decoding=GuidedDecodingParams(json=ProductInfo.model_json_schema()),
    ),
)

# Configure engine for JSON output
engine = OpenAIInferenceEngine(
    model_params=config.model,
    remote_params=config.remote_params,
)

# Extract and validate structured data
text = (
    "I'd like to tell you about the new iPhone 15. It's priced at $999 and comes with "
    "some amazing features including the A17 Pro chip, USB-C connectivity, and a premium "
    "Titanium design."
)
conversation = Conversation(
    messages=[
        Message(
            role=Role.USER, content=f"Extract product information as JSON from: {text}"
        )
    ]
)
result = engine.infer([conversation], inference_config=config)
product = ProductInfo.model_validate_json(result[0].messages[-1].content)
```

--------------------------------

### Set up Oumi environment

Source: https://www.oumi.ai/docs/en/latest/development/dev_setup.html

Creates a new Conda env, installs relevant packages, and installs pre-commit.

```bash
make setup
```

--------------------------------

### Start TensorBoard

Source: https://www.oumi.ai/docs/en/latest/user_guides/train/environments/local.html

Run this command to start TensorBoard, pointing it to the output directory where training logs are stored.

```bash
# Start TensorBoard
tensorboard --logdir oumi_output_dir
```

--------------------------------

### Basic chat interaction using VLLMInferenceEngine in Python

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/infer/infer.md

Initialize a VLLM inference engine with a small free model, create a conversation, and get a response. The example uses `HuggingFaceTB/SmolLM2-135M-Instruct` and prints the assistant's reply.

```python
from oumi.inference import VLLMInferenceEngine
from oumi.core.configs import InferenceConfig, ModelParams
from oumi.core.types.conversation import Conversation, Message, Role

# Initialize with a small, free model
engine = VLLMInferenceEngine(
    ModelParams(
        model_name="HuggingFaceTB/SmolLM2-135M-Instruct",
        model_kwargs={"device_map": "auto"}
    )
)

# Create a conversation
conversation = Conversation(
    messages=[Message(role=Role.USER, content="What is Oumi OSS?")]
)

# Get response
result = engine.infer([conversation], InferenceConfig())
print(result[0].messages[-1].content)
```

--------------------------------

### Quickstart training config (quickstart_train.yaml)

Source: https://www.oumi.ai/docs/en/latest/get_started/quickstart.html

YAML configuration for fine-tuning SmolLM 135M Instruct using the TRL_SFT trainer. Sets model, data, and training hyperparameters for a quickstart run.

```yaml
# FFT config for SmolLM 135M Instruct.
#
# Usage:
#   oumi train -c configs/recipes/smollm/sft/135m/quickstart_train.yaml
#
# See Also:
#   - Documentation: https://oumi.ai/docs/en/latest/user_guides/train/train.html
#   - Config class: oumi.core.configs.TrainingConfig
#   - Config source: https://github.com/oumi-ai/oumi/blob/main/src/oumi/core/configs/training_config.py
#   - Other training configs: configs/**/*train.yaml

model:
  model_name: "HuggingFaceTB/SmolLM2-135M-Instruct"
  model_max_length: 2048
  torch_dtype_str: "bfloat16"
  attn_implementation: "sdpa"
  load_pretrained_weights: True
  trust_remote_code: True

data:
  train:
    datasets:
      - dataset_name: "yahma/alpaca-cleaned"

training:
  trainer_type: TRL_SFT
  save_final_model: True
  save_steps: 100
  max_steps: 10
  per_device_train_batch_size: 4
  gradient_accumulation_steps: 4

  ddp_find_unused_parameters: False
  optimizer: "adamw_torch"
  learning_rate: 2.0e-05
  compile: False

  dataloader_num_workers: "auto"
  dataloader_prefetch_factor: 32

  seed: 192847
  use_deterministic: True

  logging_steps: 5
  log_model_summary: False
  empty_device_cache_steps: 50
  run_name: "smollm135m.fft.quickstart"
  output_dir: "output/smollm135m.fft"
  include_performance_metrics: True

```

--------------------------------

### Transform GRPO example to Conversation

Source: https://www.oumi.ai/docs/en/latest/_modules/oumi/core/datasets/base_grpo_dataset.html

This method transforms a sample into a Conversation object. It calls _transform_grpo_example to get a dictionary with prompt and completion keys, then builds a messages list with user and assistant roles, and returns a Conversation.from_dict result.

```python
        """
        # Contains prompt and completion.
        example_dict = self._transform_grpo_example(sample)
        conversation_dict = {
            "messages": [
                {
                    "content": example_dict[_PROMPT_KEY],
                    "role": "user",
                },
                {
                    "content": example_dict[_COMPLETION_KEY],
                    "role": "assistant",
                },
            ],
        }
        return Conversation.from_dict(conversation_dict)
```

--------------------------------

### Sample GCP Job Configuration (hello_world_gcp_job.yaml)

Source: https://www.oumi.ai/docs/en/latest/_sources/user_guides/launch/deploy.md

Example YAML configuration for a hello world job on GCP. Used as the sample job for this tutorial.

```yaml
configs/examples/misc/hello_world_gcp_job.yaml
```
Tokens:
Raw