Overbrainer

https://docs.rs/overbrainer
Admin
Overbrainer is a tool designed to distill a large parent LLM into a smaller child model through...
Tokens:1,154,922
Snippets:7,432
Trust Score:9.1
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Build Example from question and messages

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Constructs an Example from a question and its messages, including metadata. This is part of the answers pipeline where the example is built after pushing the reply message.

```rust
messages.push(reply);
Example {
    id: question.id,
    topic: question.topic,
    subtopic: question.subtopic,
    messages,
    meta,
}
```

--------------------------------

### Build Example from question and messages

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html

Constructs an Example from a question and its messages, setting reasoning_content to None and pushing the reply before building the Example struct.

```rust
    reasoning_content: None,
});
messages.push(reply);
Example {
    id: question.id,
    topic: question.topic,
    subtopic: question.subtopic,
    messages,
    meta,
}
```

--------------------------------

### Example construction and dataset helper

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search=std%3A%3Avec

Defines the example function that builds an Example with a given subtopic and index, and the dataset function that creates 30 examples across subtopics 'a' and 'b'.

```rust
fn example(subtopic: &str, n: usize) -> Example {
    Example {
        id: Id::of(&[subtopic, &n.to_string()]),
        topic: "t".into(),
        subtopic: subtopic.into(),
        messages: Vec::new(),
        meta: Meta {
            model: "m".into(),
            input_tokens: 0,
            output_tokens: 0,
            finish_reason: FinishReason::Stop,
            reasoning_kind: ReasoningKind::Raw,
            excluded: None,
        },
    }
}

fn dataset() -> Vec<Example> {
    (0..20)
        .map(|n| example("a", n))
        .chain((0..10).map(|n| example("b", n)))
        .collect()
}
```

--------------------------------

### start

Source: https://docs.rs/overbrainer/latest/overbrainer/runs/fn.start.html

Prepares the files of the run record, copies them to the target, and starts the job. The record is saved with the job as `Running`.

```APIDOC
## Function `start`

### Description
Prepares the files of the run `record` (from `create`), copies them to the target and starts the job. The record is saved with the job, as `Running`. The `Running` status event is left to `watch`, which reports it on its first look at the job.

### Signature
```rust
pub async fn start<E: Executor, T: Trainer>(
    ctx: &RunCtx<'_, E>,
    trainer: &T,
    launch: Launch<'_>,
    record: RunRecord,
) -> Result<RunRecord, RunError>
```

### Parameters
- `ctx` (`&RunCtx<'_, E>`) - The run context.
- `trainer` (`&T`) - The trainer instance.
- `launch` (`Launch<'_>`) - The launch configuration.
- `record` (`RunRecord`) - The run record to start.

### Returns
- `Result<RunRecord, RunError>` - The updated run record on success, or a `RunError` on failure.

### Errors
Returns a `RunError` when a file cannot be prepared, copied, or the job cannot start; the record is then saved as `Failed` with the reason (a failure to save it is only logged, and the original error returned). When the job started but its record cannot be saved, nothing could find the job again: it is cancelled, on a best effort basis, and the save error is returned.
```

--------------------------------

### Record example usage and stats

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html

Records usage and updates stage statistics for an example, publishing an ItemDone event with the example ID and usage.

```rust
fn record<C: LlmClient>(
    ctx: &Ctx<'_>,
    parent: &RoleClient<C>,
    example: &Example,
    stats: &mut StageStats,
) {
    let usage = crate::llm::Usage {
        input_tokens: example.meta.input_tokens,
        output_tokens: example.meta.output_tokens,
    };
    stats.add_usage(usage, parent.price.as_ref());
    if example.meta.excluded.is_some() {
        stats.excluded += 1;
    } else {
        stats.done += 1;
    }
    ctx.bus.publish(Event::ItemDone {
        stage: Stage::Answers,
        id: example.id.to_string(),
        usage: Some(usage),
    });
}
```

--------------------------------

### Get Runpod target from settings

Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/runpod_train.rs.html?search=

Gets the Runpod target a run was started on from settings. Errors if the target is no longer a runpod target in overbrainer.toml.

```rust
fn target_of(settings: &Settings, record: &RunRecord) -> anyhow::Result<RunpodTarget> {
    settings
        .targets
        .get(&record.target)
        .and_then(RunpodTarget::from_target)
        .with_context(|| {
            format!(
                "target `{}` of run {} is no longer a runpod target in overbrainer.toml",
                record.target, record.id
            )
        })
}
```

--------------------------------

### Command::Init

Source: https://docs.rs/overbrainer/latest/overbrainer/cli/enum.Command.html?search=

Creates an example project with overbrainer.toml, .env.example, prompts/, and .gitignore. The target directory defaults to the project directory.

```APIDOC
## Command::Init

### Description
Creates an example project (overbrainer.toml, .env.example, prompts/, .gitignore).

### Variant
`Init { dir: Option<PathBuf> }`

### Fields
- **dir** (Option<PathBuf>) - Optional - Target directory. Defaults to the project directory (`-C`).

```

--------------------------------

### Start answers pipeline from dataset view

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html

Test setup that starts the answers pipeline from the dataset view by pressing 'r', Down, and Enter. This is the beginning of the test for a failed quiet load waiting before the next one.

```rust
    let mut app = dataset_app();
    let effects = press(
        &mut app,
        &[KeyCode::Char('r'), KeyCode::Down, KeyCode::Enter],
    );

```

--------------------------------

### Build example from completion

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search=

Builds an Example from a question, system prompt, and completion. Converts raw reasoning to summary if configured, constructs metadata, extracts the assistant reply, and assembles messages with optional system prompt and the user question.

```rust
fn example<C: LlmClient>(
    ctx: &Ctx<'_>,
    parent: &RoleClient<C>,
    question: Question,
    system: String,
    mut completion: Completion,
) -> Example {
    if completion.reasoning.kind == ReasoningKind::Raw && hides_raw_reasoning(ctx, parent) {
        completion.reasoning.kind = ReasoningKind::Summary;
    }
    let mut meta = Meta {
        model: parent.model.model.clone(),
        input_tokens: completion.usage.input_tokens,
        output_tokens: completion.usage.output_tokens,
        finish_reason: completion.finish,
        reasoning_kind: completion.reasoning.kind,
        excluded: None,
    };
    let (reply, excluded) = assistant(completion, parent.model.reasoning);
    meta.excluded = excluded;
    let mut messages = Vec::with_capacity(3);
    if ctx.settings.pipeline.include_system_prompt {
        messages.push(Message {
            role: Role::System,
            content: system,
            reasoning_content: None,
        });
    }
    messages.push(Message {
        role: Role::User,
        content: question.text,
        reasoning_content: None,
    });
}
```

--------------------------------

### Grid and per_topic helpers and test: every topic gets its share of small subtopics

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search=std%3A%3Avec

Defines the grid helper to create examples across topics and subtopics, the per_topic helper to count examples per topic, and tests that stratify distributes eval examples proportionally across topics.

```rust
/// `topics` topics of `subtopics` subtopics of `size` examples each.
fn grid(topics: &[&str], subtopics: usize, size: usize) -> Vec<Example> {
    let mut examples = Vec::new();
    for topic in topics {
        for subtopic in 0..subtopics {
            for n in 0..size {
                let mut example = example(&format!("{topic}{subtopic}"), n);
                example.topic = (*topic).to_string();
                examples.push(example);
            }
        }
    }
    examples
}

fn per_topic(eval: &[Example]) -> BTreeMap<String, usize> {
    let mut counts = BTreeMap::new();
    for example in eval {
        *counts.entry(example.topic.clone()).or_default() += 1;
    }
    counts
}

#[test]
fn every_topic_gets_its_share_of_small_subtopics() {
    let (_, eval) = stratify(grid(&["a", "b", "c", "d"], 10, 5), 0.1, 42);
    assert_eq!(
        per_topic(&eval),
        BTreeMap::from([
            ("a".to_string(), 5),
            ("b".to_string(), 5),
            ("c".to_string(), 5),
            ("d".to_string(), 5),
        ])
    );
    let (_, eval) = stratify(grid(&["a", "b", "c"], 30, 3), 0.1, 42);
    assert_eq!(
        per_topic(&eval),
        BTreeMap::from([
            ("a".to_string(), 9),
            ("b".to_string(), 9),
            ("c".to_string(), 9),
        ])
    );
}
```

--------------------------------

### Setup app with runs and metrics

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html?search=

Creates a temporary project with two runs (one running, one succeeded), writes a metrics file for the succeeded run, and presses '3' to select the runs view. Returns the temp dir and app.

```rust
fn runs_app() -> Result<(tempfile::TempDir, App), Box<dyn std::error::Error>> {
    let (dir, mut app) = project_app()?;
    let runs = crate::runs::Runs::new(dir.path());
    for (id, state) in [
        (FIRST, crate::runs::RunState::Running),
        (SECOND, crate::runs::RunState::Succeeded),
    ] {
        runs.save(&crate::tui::snapshots::run(id, "homelab", state))?;
    }
    std::fs::write(
        runs.run_dir(SECOND)?.join(crate::train::METRICS_FILE),
        "{\"event\": \"log\", \"time\": 2, \"step\": 1, \"loss\": 1.5}\n",
    )?;
    keys(&mut app, &[KeyCode::Char('3')]);
    Ok((dir, app))
}
```

--------------------------------

### Get session context in end method

Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/runpod_train.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Gets the session context at the start of the `end` method, which ends the pod of a run whose job ended and prints its outcome. The outcome is printed even when the pod could not be ended.

```rust
let ctx = self.session.ctx();
```

--------------------------------

### Create Example with Meta

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html

Constructs an Example with a unique id derived from subtopic and index, a fixed topic, empty messages, and a Meta with zero token counts, FinishReason::Stop, ReasoningKind::Raw, and excluded set to None.

```rust
Example {
    id: Id::of(&[subtopic, &n.to_string()]),
    topic: "t".into(),
    subtopic: subtopic.into(),
    messages: Vec::new(),
    meta: Meta {
        model: "m".into(),
        input_tokens: 0,
        output_tokens: 0,
        finish_reason: FinishReason::Stop,
        reasoning_kind: ReasoningKind::Raw,
        excluded: None,
    },
}
```

--------------------------------

### Test job ended before first poll (setup)

Source: https://docs.rs/overbrainer/latest/src/overbrainer/runs/train.rs.html

Starts a test for a job that ended before the first poll, setting up the project, runs, and event bus.

```rust
#[tokio::test]
async fn a_job_that_ended_before_the_first_poll_reports_only_its_end()
-> Result<(), Box<dyn std::error::Error>> {
    let project = tempfile::tempdir()?;
    let runs = Runs::new(project.path());
    let bus = EventBus::new();
```

--------------------------------

### Prepare training start plan and list Runpod prices

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=std%3A%3Avec

Defines the StartPlan and RunpodPlan structs, the Prices type alias, and the prepare and list_prices functions used by the TUI before a training run starts. prepare loads settings and data files, builds a StartPlan with warnings, and list_prices fetches GPU prices with a 10-second timeout.

```rust
//! What `t` shows before a training run starts: the target, the model, the data,
//! and for Runpod the GPU types with their list prices and the most `max_hours`
//! can cost.

use std::path::Path;
use std::time::Duration;

use serde::de::IgnoredAny;

use crate::config::{Adapter, EnvSource, Runtime, Target};
use crate::dataset::{DataFiles, read};
use crate::train::reasoning_template_warning;

/// Total time the list prices may take; the dialog never waits for them.
pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10);

/// What a training run started now would use.
#[derive(Debug, Clone, PartialEq)]
pub(super) struct StartPlan {
    /// `training.target`.
    pub(super) target: String,
    /// What kind of target it is, for example `runpod, Secure Cloud`.
    pub(super) kind: String,
    /// Base model, adapter, epochs and learning rate.
    pub(super) model: String,
    /// Examples in `data/train.jsonl`.
    pub(super) train: usize,
    /// Examples in `data/eval.jsonl`.
    pub(super) eval: usize,
    /// The pod of a Runpod target.
    pub(super) runpod: Option<RunpodPlan>,
    /// What the flow would warn about.
    pub(super) warnings: Vec<String>,
}

/// The pod a Runpod run would ask for.
#[derive(Debug, Clone, PartialEq)]
pub(super) struct RunpodPlan {
    /// GPU types, tried in order.
    pub(super) gpu_types: Vec<String>,
    /// GPUs per pod.
    pub(super) gpu_count: u32,
    /// When the watchdog deletes the pod.
    pub(super) max_hours: f64,
}

/// The list price of each GPU type, per GPU and hour, when it could be read.
pub(super) type Prices = Vec<(String, Option<f64>)>;

/// What a run started now in the project in `dir` would use, from its settings
/// (read with `env`) and data files only.
///
/// # Errors
///
/// Returns why no run can start: the settings cannot be loaded, there is no
/// `[training]` section, its target is unknown, or the data cannot be read.
pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result<StartPlan, String> {
    let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?;
    let training = settings
        .training
        .as_ref()
        .ok_or("no [training] section in overbrainer.toml")?;
    let target = settings
        .targets
        .get(&training.target)
        .ok_or_else(|| format!("unknown target `{}`", training.target))?;
    let files = DataFiles::new(dir);
    let count = |path: &Path| {
        read::<IgnoredAny>(path)
            .map(|lines| lines.len())
            .map_err(|error| format!("{:#}", anyhow::Error::from(error)))
    };
    let adapter = match training.adapter {
        Adapter::Lora => "lora",
        Adapter::Qlora => "qlora",
        Adapter::Full => "full",
    };
    let mut warnings: Vec<String> = reasoning_template_warning(training).into_iter().collect();
    if training.hub_model_id.is_some() && settings.hf_token.is_none() {
        warnings.push(
            "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail"
                .to_string(),
        );
    }
    Ok(StartPlan {
        target: training.target.clone(),
        kind: kind(target),
        model: format!(
            "{}, {adapter}, {} epochs, lr {:e}",
            training.base_model, training.epochs, training.learning_rate
        ),
        train: count(&files.train)?,
        eval: count(&files.eval)?,
        runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan {
            gpu_types: spec.gpu_types,
            gpu_count: spec.gpu_count,
            max_hours: spec.max_hours,
        }),
        warnings,
    })
}

/// The kind of `target`, never its host nor any key.
fn kind(target: &Target) -> String {
    let runtime = |runtime: &Runtime| match runtime {
        Runtime::Docker => "docker",
        Runtime::Native => "native",
    };
    match target {
        Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)),
        Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)),
        Target::Runpod { .. } => "runpod, Secure Cloud".to_string(),
    }
}

/// The list price of each of `gpu_types` on the Runpod account of the project
/// in `dir` (its settings read with `env`), best effort: a type whose price
/// cannot be read has none, and the whole lookup gives up after
/// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs.
pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec<String>) -> Prices {
    list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await
}
```

--------------------------------

### start.rs module: prepare, list_prices, and supporting types

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=u32+-%3E+bool

Defines the StartPlan and RunpodPlan structs, the Prices type alias, the prepare function that validates configuration and data files, the kind helper, and the async list_prices function with a timeout. Used to show what a training run would use before starting.

```rust
1//! What `t` shows before a training run starts: the target, the model, the data,
2//! and for Runpod the GPU types with their list prices and the most `max_hours`
3//! can cost.
4
5use std::path::Path;
6use std::time::Duration;
7
8use serde::de::IgnoredAny;
9
10use crate::config::{Adapter, EnvSource, Runtime, Target};
11use crate::dataset::{DataFiles, read};
12use crate::train::reasoning_template_warning;
13
14/// Total time the list prices may take; the dialog never waits for them.
15pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10);
16
17/// What a training run started now would use.
18#[derive(Debug, Clone, PartialEq)]
19pub(super) struct StartPlan {
20    /// `training.target`.
21    pub(super) target: String,
22    /// What kind of target it is, for example `runpod, Secure Cloud`.
23    pub(super) kind: String,
24    /// Base model, adapter, epochs and learning rate.
25    pub(super) model: String,
26    /// Examples in `data/train.jsonl`.
27    pub(super) train: usize,
28    /// Examples in `data/eval.jsonl`.
29    pub(super) eval: usize,
30    /// The pod of a Runpod target.
31    pub(super) runpod: Option<RunpodPlan>,
32    /// What the flow would warn about.
33    pub(super) warnings: Vec<String>,
34}
35
36/// The pod a Runpod run would ask for.
37#[derive(Debug, Clone, PartialEq)]
38pub(super) struct RunpodPlan {
39    /// GPU types, tried in order.
40    pub(super) gpu_types: Vec<String>,
41    /// GPUs per pod.
42    pub(super) gpu_count: u32,
43    /// When the watchdog deletes the pod.
44    pub(super) max_hours: f64,
45}
46
47/// The list price of each GPU type, per GPU and hour, when it could be read.
48pub(super) type Prices = Vec<(String, Option<f64>)>;
49
50/// What a run started now in the project in `dir` would use, from its settings
51/// (read with `env`) and data files only.
52///
53/// # Errors
54///
/// Returns why no run can start: the settings cannot be loaded, there is no
/// `[training]` section, its target is unknown, or the data cannot be read.
55pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result<StartPlan, String> {
56    let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?;
57    let training = settings
58        .training
59        .as_ref()
60        .ok_or("no [training] section in overbrainer.toml")?;
61    let target = settings
62        .targets
63        .get(&training.target)
64        .ok_or_else(|| format!("unknown target `{}`", training.target))?;
65    let files = DataFiles::new(dir);
66    let count = |path: &Path| {
67        read::<IgnoredAny>(path)
68            .map(|lines| lines.len())
69            .map_err(|error| format!("{:#}", anyhow::Error::from(error)))
70    };
71    let adapter = match training.adapter {
72        Adapter::Lora => "lora",
73        Adapter::Qlora => "qlora",
74        Adapter::Full => "full",
75    };
76    let mut warnings: Vec<String> = reasoning_template_warning(training).into_iter().collect();
77    if training.hub_model_id.is_some() && settings.hf_token.is_none() {
78        warnings.push(
79            "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail"
80                .to_string(),
81        );
82    }
83    Ok(StartPlan {
84        target: training.target.clone(),
85        kind: kind(target),
86        model: format!(
87            "{}, {adapter}, {} epochs, lr {:e}",
88            training.base_model, training.epochs, training.learning_rate
89        ),
90        train: count(&files.train)?,
91        eval: count(&files.eval)?,
92        runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan {
93            gpu_types: spec.gpu_types,
94            gpu_count: spec.gpu_count,
95            max_hours: spec.max_hours,
96        }),
97        warnings,
98    })
99}
100
101/// The kind of `target`, never its host nor any key.
102fn kind(target: &Target) -> String {
103    let runtime = |runtime: &Runtime| match runtime {
104        Runtime::Docker => "docker",
105        Runtime::Native => "native",
106    };
107    match target {
108        Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)),
109        Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)),
110        Target::Runpod { .. } => "runpod, Secure Cloud".to_string(),
111    }
112}
113
114/// The list price of each of `gpu_types` on the Runpod account of the project
115/// in `dir` (its settings read with `env`), best effort: a type whose price
116/// cannot be read has none, and the whole lookup gives up after
117/// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs.
118pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec<String>) -> Prices {
119    list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await
120}
```

--------------------------------

### terminal.rs: terminal setup, restore, and Screen struct

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/terminal.rs.html?search=

The full source of the terminal.rs module, covering terminal setup, restoration, and the Screen struct. It installs a panic hook that restores the terminal only for the owning thread, uses a RESTORED flag to avoid double restoration, and never drops the Terminal to avoid panics on closed stderr.

```rust
1//! The real terminal: setting it up, restoring it on every exit path, reading its
2//! input, and handing it to the editor and back.
3
4use std::io::{self, Write};
5use std::mem::ManuallyDrop;
6use std::panic::{self, PanicHookInfo};
7use std::sync::Arc;
8use std::sync::atomic::{AtomicBool, Ordering};
9use std::thread;
10use std::time::Duration;
11
12use crossterm::cursor::{Hide, Show};
13use crossterm::event::{self, Event, EventStream};
14use crossterm::execute;
15use crossterm::terminal::{
16    EnterAlternateScreen, LeaveAlternateScreen, disable_raw_mode, enable_raw_mode,
17};
18use futures::StreamExt;
19use ratatui::backend::CrosstermBackend;
20use ratatui::{DefaultTerminal, Terminal};
21use tokio::sync::mpsc::UnboundedSender;
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;

/// A panic hook, shared so the guard can put it back.
type Hook = dyn Fn(&PanicHookInfo<'_>) + Send + Sync + 'static;

/// Restores the terminal when dropped, and meanwhile owns the panic hook.
///
/// Built before the terminal is set up, so a setup that fails halfway (raw mode
/// on, alternate screen not entered) is undone too, as is an error returned from
/// the loop.
///
/// The hook restores the terminal, then runs the previous hook, only for a panic
/// on the thread that owns the terminal: that panic ends the TUI. A panic on any
/// other thread is a background task's, which tokio catches and the TUI survives;
/// the hook only logs it at ERROR, into the Logs view, since the previous hook
/// would print over the alternate screen.
pub(super) struct TerminalGuard {
    previous: Arc<Hook>,
}

impl TerminalGuard {
    /// Installs the hook for the current thread, which will own the terminal.
    pub(super) fn enter() -> Self {
        let previous: Arc<Hook> = Arc::from(panic::take_hook());
        let owner = thread::current().id();
        let chained = Arc::clone(&previous);
        panic::set_hook(Box::new(move |info| {
            if thread::current().id() == owner {
                restore();
                chained(info);
            } else {
                tracing::error!("a background task panicked: {info}");
            }
        }));
        Self { previous }
    }
}

impl Drop for TerminalGuard {
    fn drop(&mut self) {
        restore();
        // Hooks cannot be changed while this thread panics; the process ends then.
        if !thread::panicking() {
            drop(panic::take_hook());
            let previous = Arc::clone(&self.previous);
            panic::set_hook(Box::new(move |info| previous(info)));
        }
    }
}

/// Enables raw mode and enters the alternate screen, like `ratatui::try_init`
/// without the panic hook it installs (the guard's hook replaces it).
///
/// The terminal is never dropped. ratatui's `Drop for Terminal` shows a hidden
/// cursor again and, when that fails on a closed terminal, reports it with
/// `eprintln!`, which panics on a dead stderr and aborts the process during an
/// unwind. [`restore`] shows the cursor instead, from the guard and the panic
/// hook, and the terminal's memory is left to the end of the process.
///
/// # Errors
///
/// Returns an error when the terminal cannot be set up.
pub(super) fn init() -> io::Result<ManuallyDrop<DefaultTerminal>> {
    RESTORED.store(false, Ordering::SeqCst);
    enable_raw_mode()?;
    execute!(io::stdout(), EnterAlternateScreen)?;
    Terminal::new(CrosstermBackend::new(io::stdout())).map(ManuallyDrop::new)
}

/// Set while [`Screen::suspend`] has put the terminal back in its normal mode
/// (the editor has it, or the TUI waits for its work once its loop ended):
/// [`restore`] then has nothing to undo.
static RESTORED: AtomicBool = AtomicBool::new(false);

/// Leaves the alternate screen and raw mode, and shows the cursor, unless
/// [`Screen::suspend`] already did. A failure is written to stderr, ignoring a
/// failed write: the terminal may be gone.
fn restore() {
    if RESTORED.load(Ordering::SeqCst) {
        return;
    }
    if let Err(error) = ratatui::try_restore() {
        writeln!(io::stderr(), "cannot restore the terminal: {error}").ok();
    }
    execute!(io::stdout(), Show).ok();
}

/// The screen of the real terminal, with the task reading its input.
///
/// While the editor has the terminal, no task reads it and the terminal is in
/// its normal mode; the guard's panic hook still restores it then, which leaving
/// the alternate screen and raw mode a second time does no harm to.
pub(super) struct Screen {
    reader: Option<InputTask>,
    events: UnboundedSender<io::Result<Event>>,
}

impl Screen {
```

--------------------------------

### Save answered question example

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search=

Builds an Example from the question, system prompt, and completion, records it in stats, and appends it to the output Appender. Returns a PipelineError if appending fails.

```rust
fn save<C: LlmClient>(
    ctx: &Ctx<'_>,
    parent: &RoleClient<C>,
    out: &mut Appender,
    stats: &mut StageStats,
    (question, system, completion): (Question, String, Completion),
) -> Result<(), PipelineError> {
    let example = example(ctx, parent, question, system, completion);
    record(ctx, parent, &example, stats);
    out.append(&example)?;
    Ok(())
}
```

--------------------------------

### tests module: test setup for split.rs

Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search=

Test module that imports the parent module and dataset types (FinishReason, Id, Meta, ReasoningKind) and defines a helper to create Example instances for testing.

```rust
#[cfg(test)]
mod tests {
    use super::*;
    use crate::dataset::{FinishReason, Id, Meta, ReasoningKind};

    fn example(subtopic: &str, n: usize) -> Example {
```

--------------------------------

### Validate secret names with invalid and valid examples

Source: https://docs.rs/overbrainer/latest/src/overbrainer/exec/ssh.rs.html

Tests that secret names containing invalid characters (empty, starting with digit, hyphen, space, semicolon, or shell expansion) are rejected, while a valid name like '_A1' is accepted.

```rust
for name in ["", "1ABC", "A-B", "A B", "A;rm -rf /", "$(x)"] {
    assert!(
        matches!(launcher(&with_name(name)), Err(ExecError::InvalidSecret(_))),
        "{name:?} was accepted"
    );
}
assert!(launcher(&with_name("_A1")).is_ok());
```

--------------------------------

### metric_line formats TrainMetric as detailed training log

Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/progress.rs.html?search=std%3A%3Avec

Builds a human-readable string with step, epoch, and logged metric values (e.g., loss, lr, grad_norm). The example shows a step fraction and epoch, but the implementation is incomplete in the provided snippet (starts building parts).

```rust
/// Step, epoch and the logged values of `metric`, for example
/// `step 12/40 (30%), epoch 0.90, loss 1.2345, lr 2.00e-4, grad_norm 0.812`.
#[must_use]
pub fn metric_line(metric: &TrainMetric) -> String {
    let mut parts = vec![match metric.max_steps.filter(|max| *max > 0) {
```

--------------------------------

### fn spawn(&self, job: &JobCommand) -> impl Future<Output = Result<JobId, ExecError>> + Send

Source: https://docs.rs/overbrainer/latest/overbrainer/exec/trait.Executor.html?search=

Starts a job in the background and returns its handle. The job continues running even if the process exits or loses connection. Environment setup depends on the target type.

```APIDOC
## fn spawn(&self, job: &JobCommand) -> impl Future<Output = Result<JobId, ExecError>> + Send

### Description
Starts `job` in the background and returns its handle. The job keeps running when this process exits or loses its connection to the target.

The environment the job starts with depends on the target. A local job inherits overbrainer’s own environment, minus every variable whose name starts with `OVERBRAINER_` or `VAULT_` (see `LocalExecutor`). A job over SSH gets whatever the remote non-interactive `sh -c` gives it, which is the SSH server’s environment for that user: no login shell and no interactive startup file are read (see `SshExecutor`). The job’s secrets are then set on top of that, on its environment only. A `native` target whose `axolotl` relies on the caller’s shell setup, such as a conda activation or a `PATH` entry from a profile, can therefore behave differently on each.

### Method
spawn

### Signature
`fn spawn(&self, job: &JobCommand) -> impl Future<Output = Result<JobId, ExecError>> + Send`

### Parameters
#### Parameters
- **job** (&JobCommand) - Required - The job command to start.

### Returns
- **JobId** - The handle to the started job.

### Errors
Returns `ExecError::InvalidSecret`, naming the variable only, when a secret’s name is not an environment variable name or its value holds a line break or a NUL byte; nothing is started then. Returns another `ExecError` when the job cannot be started.

```

--------------------------------

### Prepare training run plan and list Runpod prices in start.rs

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E

Defines the data structures and functions that prepare a training run plan before it starts. It loads settings, validates the target, counts data files, and builds warnings; the async list_prices function fetches GPU prices with a 10-second timeout.

```rust
1//! What `t` shows before a training run starts: the target, the model, the data,
2//! and for Runpod the GPU types with their list prices and the most `max_hours`
3//! can cost.
4
5use std::path::Path;
6use std::time::Duration;
7
8use serde::de::IgnoredAny;
9
10use crate::config::{Adapter, EnvSource, Runtime, Target};
11use crate::dataset::{DataFiles, read};
12use crate::train::reasoning_template_warning;
13
14/// Total time the list prices may take; the dialog never waits for them.
15pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10);
16
/// What a training run started now would use.
17#[derive(Debug, Clone, PartialEq)]
18pub(super) struct StartPlan {
19    /// `training.target`.
20    pub(super) target: String,
21    /// What kind of target it is, for example `runpod, Secure Cloud`.
22    pub(super) kind: String,
23    /// Base model, adapter, epochs and learning rate.
24    pub(super) model: String,
25    /// Examples in `data/train.jsonl`.
26    pub(super) train: usize,
27    /// Examples in `data/eval.jsonl`.
28    pub(super) eval: usize,
29    /// The pod of a Runpod target.
30    pub(super) runpod: Option<RunpodPlan>,
31    /// What the flow would warn about.
32    pub(super) warnings: Vec<String>,
33}
34
/// The pod a Runpod run would ask for.
35#[derive(Debug, Clone, PartialEq)]
36pub(super) struct RunpodPlan {
37    /// GPU types, tried in order.
38    pub(super) gpu_types: Vec<String>,
39    /// GPUs per pod.
40    pub(super) gpu_count: u32,
41    /// When the watchdog deletes the pod.
42    pub(super) max_hours: f64,
43}
44
/// The list price of each GPU type, per GPU and hour, when it could be read.
45pub(super) type Prices = Vec<(String, Option<f64>)>;
46
/// What a run started now in the project in `dir` would use, from its settings
/// (read with `env`) and data files only.
47///
48/// # Errors
49///
/// Returns why no run can start: the settings cannot be loaded, there is no
/// `[training]` section, its target is unknown, or the data cannot be read.
50pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result<StartPlan, String> {
51    let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?;
52    let training = settings
53        .training
54        .as_ref()
55        .ok_or("no [training] section in overbrainer.toml")?;
56    let target = settings
57        .targets
58        .get(&training.target)
59        .ok_or_else(|| format!("unknown target `{}`", training.target))?;
60    let files = DataFiles::new(dir);
61    let count = |path: &Path| {
62        read::<IgnoredAny>(path)
63            .map(|lines| lines.len())
64            .map_err(|error| format!("{:#}", anyhow::Error::from(error)))
65    };
66    let adapter = match training.adapter {
67        Adapter::Lora => "lora",
68        Adapter::Qlora => "qlora",
69        Adapter::Full => "full",
70    };
71    let mut warnings: Vec<String> = reasoning_template_warning(training).into_iter().collect();
72    if training.hub_model_id.is_some() && settings.hf_token.is_none() {
73        warnings.push(
74            "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail"
75                .to_string(),
76        );
77    }
78    Ok(StartPlan {
79        target: training.target.clone(),
80        kind: kind(target),
81        model: format!(
82            "{}, {adapter}, {} epochs, lr {:e}",
83            training.base_model, training.epochs, training.learning_rate
84        ),
85        train: count(&files.train)?,
86        eval: count(&files.eval)?,
87        runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan {
88            gpu_types: spec.gpu_types,
89            gpu_count: spec.gpu_count,
90            max_hours: spec.max_hours,
91        }),
92        warnings,
93    })
94}
95
/// The kind of `target`, never its host nor any key.
96fn kind(target: &Target) -> String {
97    let runtime = |runtime: &Runtime| match runtime {
98        Runtime::Docker => "docker",
99        Runtime::Native => "native",
100    };
101    match target {
102        Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)),
103        Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)),
104        Target::Runpod { .. } => "runpod, Secure Cloud".to_string(),
105    }
106}
107
/// The list price of each of `gpu_types` on the Runpod account of the project
/// in `dir` (its settings read with `env`), best effort: a type whose price
/// cannot be read has none, and the whole lookup gives up after
/// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs.
108pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec<String>) -> Prices {
109    list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await
110}
```

--------------------------------

### Test c on a starting run cancels it once its job started

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html?search=

Test that pressing 'c' on a starting run cancels it once its job started.

```rust
#[test]
fn c_on_a_starting_run_cancels_it_once_its_job_started() {
    let mut app = app();
    starting(&mut app, TaskId(6));
    assert_eq!(app.cancel_run(FIRST), [], "never during the start");
    assert_eq!(
        status(&app),
        Some("run 20260921-133200-a1b2 is starting: it is cancelled once its job started")
    );
}
```

--------------------------------

### Abandon a single start task with status messages

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/follow.rs.html?search=

Abandons a single start task if it is still starting and not already detached, with warnings if the job started or info messages otherwise.

```rust
pub(super) fn abandon_start(&mut self, id: TaskId) -> Vec<Effect> {
    let Some(follow) = self.training.tasks.get(&id) else {
        return Vec::new();
    };
    let run = follow.run();
    if !follow.starting() {
        self.say(
            Severity::Warn,
            format!(
                "{run}: its job started, so it was not abandoned and its pod keeps billing; \
                 press c to cancel it"
            ),
        );
        return Vec::new();
    }
    if follow.detach == Detach::Done {
        self.say(Severity::Info, format!("{run} is being abandoned"));
        return Vec::new();
    }
    self.say(Severity::Info, format!("abandoning {run}"));
    self.abandon(&[id])
}
```

--------------------------------

### start

Source: https://docs.rs/overbrainer/latest/src/overbrainer/runs/train.rs.html

Starts a training run by preparing and uploading the run directory, launching the job through the executor, and recording the run as Running or Failed. Returns the updated RunRecord or a RunError.

```APIDOC
## start

### Description
Starts a training run by preparing and uploading the run directory, launching the job through the executor, and recording the run as Running or Failed. Returns the updated RunRecord or a RunError.

### Method
async fn start

### Signature
pub async fn start<E: Executor, T: Trainer>(
    ctx: &RunCtx<'_, E>,
    trainer: &T,
    launch: Launch<'_>,
    mut record: RunRecord,
) -> Result<RunRecord, RunError>

### Parameters
- **ctx** (RunCtx) - Required - The run context providing executor and storage.
- **trainer** (Trainer) - Required - The trainer instance used to prepare the run.
- **launch** (Launch) - Required - Launch configuration including runtime and secrets.
- **record** (RunRecord) - Required - The run record to be updated and saved.

### Returns
- **Ok(RunRecord)** - The updated run record with job ID and state set to Running.
- **Err(RunError)** - If the job cannot be prepared, copied, or started; the record is saved as Failed with the error message.

### Errors
Returns a RunError when a file cannot be prepared, copied, or the job cannot start; the record is then saved as Failed with the reason. When the job started but its record cannot be saved, the job is cancelled on a best effort basis and the save error is returned.
```

--------------------------------

### Helper to create app with provisioning run

Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/event_loop.rs.html?search=std%3A%3Avec

Helper that returns an App for the project in `dir`, with a Runpod run still provisioning as task 4.

```rust
use crate::tui::training::{Follow, Job};
let mut app = app();
app.project.dir = dir.to_path_buf();
app.training
    .tasks
    .insert(TaskId(4), Follow::new(Job::Start { runpod: true }, RUN));
app
```

--------------------------------

### start

Source: https://docs.rs/overbrainer/latest/overbrainer/runs/fn.start.html?search=u32+-%3E+bool

Prepares the files of the run record, copies them to the target, and starts the job. The record is saved with the job as `Running`; the `Running` status event is left to `watch`.

```APIDOC
## Function `start`

### Description
Prepares the files of the run `record` (from `create`), copies them to the target and starts the job. The record is saved with the job, as `Running`. The `Running` status event is left to `watch`, which reports it on its first look at the job.

### Signature
```rust
pub async fn start<E: Executor, T: Trainer>(
    ctx: &RunCtx<'_, E>,
    trainer: &T,
    launch: Launch<'_>,
    record: RunRecord,
) -> Result<RunRecord, RunError>
```

### Parameters
- `ctx` (`&RunCtx<'_, E>`) - Required - The run context.
- `trainer` (`&T`) - Required - The trainer instance.
- `launch` (`Launch<'_>`) - Required - The launch configuration.
- `record` (`RunRecord`) - Required - The run record to prepare and start.

### Returns
- `Result<RunRecord, RunError>` - On success, returns the updated `RunRecord`; on failure, returns a `RunError`.

### Errors
Returns a `RunError` when a file cannot be prepared, copied, or the job cannot start; the record is then saved as `Failed` with the reason (a failure to save it is only logged, and the original error returned). When the job started but its record cannot be saved, nothing could find the job again: it is cancelled, on a best effort basis, and the save error is returned.
```
Tokens:
Raw