### Build Example from question and messages Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E Constructs an Example from a question and its messages, including metadata. This is part of the answers pipeline where the example is built after pushing the reply message. ```rust messages.push(reply); Example { id: question.id, topic: question.topic, subtopic: question.subtopic, messages, meta, } ``` -------------------------------- ### Build Example from question and messages Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html Constructs an Example from a question and its messages, setting reasoning_content to None and pushing the reply before building the Example struct. ```rust reasoning_content: None, }); messages.push(reply); Example { id: question.id, topic: question.topic, subtopic: question.subtopic, messages, meta, } ``` -------------------------------- ### Example construction and dataset helper Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search=std%3A%3Avec Defines the example function that builds an Example with a given subtopic and index, and the dataset function that creates 30 examples across subtopics 'a' and 'b'. ```rust fn example(subtopic: &str, n: usize) -> Example { Example { id: Id::of(&[subtopic, &n.to_string()]), topic: "t".into(), subtopic: subtopic.into(), messages: Vec::new(), meta: Meta { model: "m".into(), input_tokens: 0, output_tokens: 0, finish_reason: FinishReason::Stop, reasoning_kind: ReasoningKind::Raw, excluded: None, }, } } fn dataset() -> Vec { (0..20) .map(|n| example("a", n)) .chain((0..10).map(|n| example("b", n))) .collect() } ``` -------------------------------- ### start Source: https://docs.rs/overbrainer/latest/overbrainer/runs/fn.start.html Prepares the files of the run record, copies them to the target, and starts the job. The record is saved with the job as `Running`. ```APIDOC ## Function `start` ### Description Prepares the files of the run `record` (from `create`), copies them to the target and starts the job. The record is saved with the job, as `Running`. The `Running` status event is left to `watch`, which reports it on its first look at the job. ### Signature ```rust pub async fn start( ctx: &RunCtx<'_, E>, trainer: &T, launch: Launch<'_>, record: RunRecord, ) -> Result ``` ### Parameters - `ctx` (`&RunCtx<'_, E>`) - The run context. - `trainer` (`&T`) - The trainer instance. - `launch` (`Launch<'_>`) - The launch configuration. - `record` (`RunRecord`) - The run record to start. ### Returns - `Result` - The updated run record on success, or a `RunError` on failure. ### Errors Returns a `RunError` when a file cannot be prepared, copied, or the job cannot start; the record is then saved as `Failed` with the reason (a failure to save it is only logged, and the original error returned). When the job started but its record cannot be saved, nothing could find the job again: it is cancelled, on a best effort basis, and the save error is returned. ``` -------------------------------- ### Record example usage and stats Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html Records usage and updates stage statistics for an example, publishing an ItemDone event with the example ID and usage. ```rust fn record( ctx: &Ctx<'_>, parent: &RoleClient, example: &Example, stats: &mut StageStats, ) { let usage = crate::llm::Usage { input_tokens: example.meta.input_tokens, output_tokens: example.meta.output_tokens, }; stats.add_usage(usage, parent.price.as_ref()); if example.meta.excluded.is_some() { stats.excluded += 1; } else { stats.done += 1; } ctx.bus.publish(Event::ItemDone { stage: Stage::Answers, id: example.id.to_string(), usage: Some(usage), }); } ``` -------------------------------- ### Get Runpod target from settings Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/runpod_train.rs.html?search= Gets the Runpod target a run was started on from settings. Errors if the target is no longer a runpod target in overbrainer.toml. ```rust fn target_of(settings: &Settings, record: &RunRecord) -> anyhow::Result { settings .targets .get(&record.target) .and_then(RunpodTarget::from_target) .with_context(|| { format!( "target `{}` of run {} is no longer a runpod target in overbrainer.toml", record.target, record.id ) }) } ``` -------------------------------- ### Command::Init Source: https://docs.rs/overbrainer/latest/overbrainer/cli/enum.Command.html?search= Creates an example project with overbrainer.toml, .env.example, prompts/, and .gitignore. The target directory defaults to the project directory. ```APIDOC ## Command::Init ### Description Creates an example project (overbrainer.toml, .env.example, prompts/, .gitignore). ### Variant `Init { dir: Option }` ### Fields - **dir** (Option) - Optional - Target directory. Defaults to the project directory (`-C`). ``` -------------------------------- ### Start answers pipeline from dataset view Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html Test setup that starts the answers pipeline from the dataset view by pressing 'r', Down, and Enter. This is the beginning of the test for a failed quiet load waiting before the next one. ```rust let mut app = dataset_app(); let effects = press( &mut app, &[KeyCode::Char('r'), KeyCode::Down, KeyCode::Enter], ); ``` -------------------------------- ### Build example from completion Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search= Builds an Example from a question, system prompt, and completion. Converts raw reasoning to summary if configured, constructs metadata, extracts the assistant reply, and assembles messages with optional system prompt and the user question. ```rust fn example( ctx: &Ctx<'_>, parent: &RoleClient, question: Question, system: String, mut completion: Completion, ) -> Example { if completion.reasoning.kind == ReasoningKind::Raw && hides_raw_reasoning(ctx, parent) { completion.reasoning.kind = ReasoningKind::Summary; } let mut meta = Meta { model: parent.model.model.clone(), input_tokens: completion.usage.input_tokens, output_tokens: completion.usage.output_tokens, finish_reason: completion.finish, reasoning_kind: completion.reasoning.kind, excluded: None, }; let (reply, excluded) = assistant(completion, parent.model.reasoning); meta.excluded = excluded; let mut messages = Vec::with_capacity(3); if ctx.settings.pipeline.include_system_prompt { messages.push(Message { role: Role::System, content: system, reasoning_content: None, }); } messages.push(Message { role: Role::User, content: question.text, reasoning_content: None, }); } ``` -------------------------------- ### Grid and per_topic helpers and test: every topic gets its share of small subtopics Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search=std%3A%3Avec Defines the grid helper to create examples across topics and subtopics, the per_topic helper to count examples per topic, and tests that stratify distributes eval examples proportionally across topics. ```rust /// `topics` topics of `subtopics` subtopics of `size` examples each. fn grid(topics: &[&str], subtopics: usize, size: usize) -> Vec { let mut examples = Vec::new(); for topic in topics { for subtopic in 0..subtopics { for n in 0..size { let mut example = example(&format!("{topic}{subtopic}"), n); example.topic = (*topic).to_string(); examples.push(example); } } } examples } fn per_topic(eval: &[Example]) -> BTreeMap { let mut counts = BTreeMap::new(); for example in eval { *counts.entry(example.topic.clone()).or_default() += 1; } counts } #[test] fn every_topic_gets_its_share_of_small_subtopics() { let (_, eval) = stratify(grid(&["a", "b", "c", "d"], 10, 5), 0.1, 42); assert_eq!( per_topic(&eval), BTreeMap::from([ ("a".to_string(), 5), ("b".to_string(), 5), ("c".to_string(), 5), ("d".to_string(), 5), ]) ); let (_, eval) = stratify(grid(&["a", "b", "c"], 30, 3), 0.1, 42); assert_eq!( per_topic(&eval), BTreeMap::from([ ("a".to_string(), 9), ("b".to_string(), 9), ("c".to_string(), 9), ]) ); } ``` -------------------------------- ### Setup app with runs and metrics Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html?search= Creates a temporary project with two runs (one running, one succeeded), writes a metrics file for the succeeded run, and presses '3' to select the runs view. Returns the temp dir and app. ```rust fn runs_app() -> Result<(tempfile::TempDir, App), Box> { let (dir, mut app) = project_app()?; let runs = crate::runs::Runs::new(dir.path()); for (id, state) in [ (FIRST, crate::runs::RunState::Running), (SECOND, crate::runs::RunState::Succeeded), ] { runs.save(&crate::tui::snapshots::run(id, "homelab", state))?; } std::fs::write( runs.run_dir(SECOND)?.join(crate::train::METRICS_FILE), "{\"event\": \"log\", \"time\": 2, \"step\": 1, \"loss\": 1.5}\n", )?; keys(&mut app, &[KeyCode::Char('3')]); Ok((dir, app)) } ``` -------------------------------- ### Get session context in end method Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/runpod_train.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E Gets the session context at the start of the `end` method, which ends the pod of a run whose job ended and prints its outcome. The outcome is printed even when the pod could not be ended. ```rust let ctx = self.session.ctx(); ``` -------------------------------- ### Create Example with Meta Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html Constructs an Example with a unique id derived from subtopic and index, a fixed topic, empty messages, and a Meta with zero token counts, FinishReason::Stop, ReasoningKind::Raw, and excluded set to None. ```rust Example { id: Id::of(&[subtopic, &n.to_string()]), topic: "t".into(), subtopic: subtopic.into(), messages: Vec::new(), meta: Meta { model: "m".into(), input_tokens: 0, output_tokens: 0, finish_reason: FinishReason::Stop, reasoning_kind: ReasoningKind::Raw, excluded: None, }, } ``` -------------------------------- ### Test job ended before first poll (setup) Source: https://docs.rs/overbrainer/latest/src/overbrainer/runs/train.rs.html Starts a test for a job that ended before the first poll, setting up the project, runs, and event bus. ```rust #[tokio::test] async fn a_job_that_ended_before_the_first_poll_reports_only_its_end() -> Result<(), Box> { let project = tempfile::tempdir()?; let runs = Runs::new(project.path()); let bus = EventBus::new(); ``` -------------------------------- ### Prepare training start plan and list Runpod prices Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=std%3A%3Avec Defines the StartPlan and RunpodPlan structs, the Prices type alias, and the prepare and list_prices functions used by the TUI before a training run starts. prepare loads settings and data files, builds a StartPlan with warnings, and list_prices fetches GPU prices with a 10-second timeout. ```rust //! What `t` shows before a training run starts: the target, the model, the data, //! and for Runpod the GPU types with their list prices and the most `max_hours` //! can cost. use std::path::Path; use std::time::Duration; use serde::de::IgnoredAny; use crate::config::{Adapter, EnvSource, Runtime, Target}; use crate::dataset::{DataFiles, read}; use crate::train::reasoning_template_warning; /// Total time the list prices may take; the dialog never waits for them. pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10); /// What a training run started now would use. #[derive(Debug, Clone, PartialEq)] pub(super) struct StartPlan { /// `training.target`. pub(super) target: String, /// What kind of target it is, for example `runpod, Secure Cloud`. pub(super) kind: String, /// Base model, adapter, epochs and learning rate. pub(super) model: String, /// Examples in `data/train.jsonl`. pub(super) train: usize, /// Examples in `data/eval.jsonl`. pub(super) eval: usize, /// The pod of a Runpod target. pub(super) runpod: Option, /// What the flow would warn about. pub(super) warnings: Vec, } /// The pod a Runpod run would ask for. #[derive(Debug, Clone, PartialEq)] pub(super) struct RunpodPlan { /// GPU types, tried in order. pub(super) gpu_types: Vec, /// GPUs per pod. pub(super) gpu_count: u32, /// When the watchdog deletes the pod. pub(super) max_hours: f64, } /// The list price of each GPU type, per GPU and hour, when it could be read. pub(super) type Prices = Vec<(String, Option)>; /// What a run started now in the project in `dir` would use, from its settings /// (read with `env`) and data files only. /// /// # Errors /// /// Returns why no run can start: the settings cannot be loaded, there is no /// `[training]` section, its target is unknown, or the data cannot be read. pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result { let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?; let training = settings .training .as_ref() .ok_or("no [training] section in overbrainer.toml")?; let target = settings .targets .get(&training.target) .ok_or_else(|| format!("unknown target `{}`", training.target))?; let files = DataFiles::new(dir); let count = |path: &Path| { read::(path) .map(|lines| lines.len()) .map_err(|error| format!("{:#}", anyhow::Error::from(error))) }; let adapter = match training.adapter { Adapter::Lora => "lora", Adapter::Qlora => "qlora", Adapter::Full => "full", }; let mut warnings: Vec = reasoning_template_warning(training).into_iter().collect(); if training.hub_model_id.is_some() && settings.hf_token.is_none() { warnings.push( "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail" .to_string(), ); } Ok(StartPlan { target: training.target.clone(), kind: kind(target), model: format!( "{}, {adapter}, {} epochs, lr {:e}", training.base_model, training.epochs, training.learning_rate ), train: count(&files.train)?, eval: count(&files.eval)?, runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan { gpu_types: spec.gpu_types, gpu_count: spec.gpu_count, max_hours: spec.max_hours, }), warnings, }) } /// The kind of `target`, never its host nor any key. fn kind(target: &Target) -> String { let runtime = |runtime: &Runtime| match runtime { Runtime::Docker => "docker", Runtime::Native => "native", }; match target { Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)), Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)), Target::Runpod { .. } => "runpod, Secure Cloud".to_string(), } } /// The list price of each of `gpu_types` on the Runpod account of the project /// in `dir` (its settings read with `env`), best effort: a type whose price /// cannot be read has none, and the whole lookup gives up after /// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs. pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec) -> Prices { list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await } ``` -------------------------------- ### start.rs module: prepare, list_prices, and supporting types Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=u32+-%3E+bool Defines the StartPlan and RunpodPlan structs, the Prices type alias, the prepare function that validates configuration and data files, the kind helper, and the async list_prices function with a timeout. Used to show what a training run would use before starting. ```rust 1//! What `t` shows before a training run starts: the target, the model, the data, 2//! and for Runpod the GPU types with their list prices and the most `max_hours` 3//! can cost. 4 5use std::path::Path; 6use std::time::Duration; 7 8use serde::de::IgnoredAny; 9 10use crate::config::{Adapter, EnvSource, Runtime, Target}; 11use crate::dataset::{DataFiles, read}; 12use crate::train::reasoning_template_warning; 13 14/// Total time the list prices may take; the dialog never waits for them. 15pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10); 16 17/// What a training run started now would use. 18#[derive(Debug, Clone, PartialEq)] 19pub(super) struct StartPlan { 20 /// `training.target`. 21 pub(super) target: String, 22 /// What kind of target it is, for example `runpod, Secure Cloud`. 23 pub(super) kind: String, 24 /// Base model, adapter, epochs and learning rate. 25 pub(super) model: String, 26 /// Examples in `data/train.jsonl`. 27 pub(super) train: usize, 28 /// Examples in `data/eval.jsonl`. 29 pub(super) eval: usize, 30 /// The pod of a Runpod target. 31 pub(super) runpod: Option, 32 /// What the flow would warn about. 33 pub(super) warnings: Vec, 34} 35 36/// The pod a Runpod run would ask for. 37#[derive(Debug, Clone, PartialEq)] 38pub(super) struct RunpodPlan { 39 /// GPU types, tried in order. 40 pub(super) gpu_types: Vec, 41 /// GPUs per pod. 42 pub(super) gpu_count: u32, 43 /// When the watchdog deletes the pod. 44 pub(super) max_hours: f64, 45} 46 47/// The list price of each GPU type, per GPU and hour, when it could be read. 48pub(super) type Prices = Vec<(String, Option)>; 49 50/// What a run started now in the project in `dir` would use, from its settings 51/// (read with `env`) and data files only. 52/// 53/// # Errors 54/// /// Returns why no run can start: the settings cannot be loaded, there is no /// `[training]` section, its target is unknown, or the data cannot be read. 55pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result { 56 let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?; 57 let training = settings 58 .training 59 .as_ref() 60 .ok_or("no [training] section in overbrainer.toml")?; 61 let target = settings 62 .targets 63 .get(&training.target) 64 .ok_or_else(|| format!("unknown target `{}`", training.target))?; 65 let files = DataFiles::new(dir); 66 let count = |path: &Path| { 67 read::(path) 68 .map(|lines| lines.len()) 69 .map_err(|error| format!("{:#}", anyhow::Error::from(error))) 70 }; 71 let adapter = match training.adapter { 72 Adapter::Lora => "lora", 73 Adapter::Qlora => "qlora", 74 Adapter::Full => "full", 75 }; 76 let mut warnings: Vec = reasoning_template_warning(training).into_iter().collect(); 77 if training.hub_model_id.is_some() && settings.hf_token.is_none() { 78 warnings.push( 79 "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail" 80 .to_string(), 81 ); 82 } 83 Ok(StartPlan { 84 target: training.target.clone(), 85 kind: kind(target), 86 model: format!( 87 "{}, {adapter}, {} epochs, lr {:e}", 88 training.base_model, training.epochs, training.learning_rate 89 ), 90 train: count(&files.train)?, 91 eval: count(&files.eval)?, 92 runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan { 93 gpu_types: spec.gpu_types, 94 gpu_count: spec.gpu_count, 95 max_hours: spec.max_hours, 96 }), 97 warnings, 98 }) 99} 100 101/// The kind of `target`, never its host nor any key. 102fn kind(target: &Target) -> String { 103 let runtime = |runtime: &Runtime| match runtime { 104 Runtime::Docker => "docker", 105 Runtime::Native => "native", 106 }; 107 match target { 108 Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)), 109 Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)), 110 Target::Runpod { .. } => "runpod, Secure Cloud".to_string(), 111 } 112} 113 114/// The list price of each of `gpu_types` on the Runpod account of the project 115/// in `dir` (its settings read with `env`), best effort: a type whose price 116/// cannot be read has none, and the whole lookup gives up after 117/// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs. 118pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec) -> Prices { 119 list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await 120} ``` -------------------------------- ### terminal.rs: terminal setup, restore, and Screen struct Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/terminal.rs.html?search= The full source of the terminal.rs module, covering terminal setup, restoration, and the Screen struct. It installs a panic hook that restores the terminal only for the owning thread, uses a RESTORED flag to avoid double restoration, and never drops the Terminal to avoid panics on closed stderr. ```rust 1//! The real terminal: setting it up, restoring it on every exit path, reading its 2//! input, and handing it to the editor and back. 3 4use std::io::{self, Write}; 5use std::mem::ManuallyDrop; 6use std::panic::{self, PanicHookInfo}; 7use std::sync::Arc; 8use std::sync::atomic::{AtomicBool, Ordering}; 9use std::thread; 10use std::time::Duration; 11 12use crossterm::cursor::{Hide, Show}; 13use crossterm::event::{self, Event, EventStream}; 14use crossterm::execute; 15use crossterm::terminal::{ 16 EnterAlternateScreen, LeaveAlternateScreen, disable_raw_mode, enable_raw_mode, 17}; 18use futures::StreamExt; 19use ratatui::backend::CrosstermBackend; 20use ratatui::{DefaultTerminal, Terminal}; 21use tokio::sync::mpsc::UnboundedSender; use tokio::task::JoinHandle; use tokio_util::sync::CancellationToken; /// A panic hook, shared so the guard can put it back. type Hook = dyn Fn(&PanicHookInfo<'_>) + Send + Sync + 'static; /// Restores the terminal when dropped, and meanwhile owns the panic hook. /// /// Built before the terminal is set up, so a setup that fails halfway (raw mode /// on, alternate screen not entered) is undone too, as is an error returned from /// the loop. /// /// The hook restores the terminal, then runs the previous hook, only for a panic /// on the thread that owns the terminal: that panic ends the TUI. A panic on any /// other thread is a background task's, which tokio catches and the TUI survives; /// the hook only logs it at ERROR, into the Logs view, since the previous hook /// would print over the alternate screen. pub(super) struct TerminalGuard { previous: Arc, } impl TerminalGuard { /// Installs the hook for the current thread, which will own the terminal. pub(super) fn enter() -> Self { let previous: Arc = Arc::from(panic::take_hook()); let owner = thread::current().id(); let chained = Arc::clone(&previous); panic::set_hook(Box::new(move |info| { if thread::current().id() == owner { restore(); chained(info); } else { tracing::error!("a background task panicked: {info}"); } })); Self { previous } } } impl Drop for TerminalGuard { fn drop(&mut self) { restore(); // Hooks cannot be changed while this thread panics; the process ends then. if !thread::panicking() { drop(panic::take_hook()); let previous = Arc::clone(&self.previous); panic::set_hook(Box::new(move |info| previous(info))); } } } /// Enables raw mode and enters the alternate screen, like `ratatui::try_init` /// without the panic hook it installs (the guard's hook replaces it). /// /// The terminal is never dropped. ratatui's `Drop for Terminal` shows a hidden /// cursor again and, when that fails on a closed terminal, reports it with /// `eprintln!`, which panics on a dead stderr and aborts the process during an /// unwind. [`restore`] shows the cursor instead, from the guard and the panic /// hook, and the terminal's memory is left to the end of the process. /// /// # Errors /// /// Returns an error when the terminal cannot be set up. pub(super) fn init() -> io::Result> { RESTORED.store(false, Ordering::SeqCst); enable_raw_mode()?; execute!(io::stdout(), EnterAlternateScreen)?; Terminal::new(CrosstermBackend::new(io::stdout())).map(ManuallyDrop::new) } /// Set while [`Screen::suspend`] has put the terminal back in its normal mode /// (the editor has it, or the TUI waits for its work once its loop ended): /// [`restore`] then has nothing to undo. static RESTORED: AtomicBool = AtomicBool::new(false); /// Leaves the alternate screen and raw mode, and shows the cursor, unless /// [`Screen::suspend`] already did. A failure is written to stderr, ignoring a /// failed write: the terminal may be gone. fn restore() { if RESTORED.load(Ordering::SeqCst) { return; } if let Err(error) = ratatui::try_restore() { writeln!(io::stderr(), "cannot restore the terminal: {error}").ok(); } execute!(io::stdout(), Show).ok(); } /// The screen of the real terminal, with the task reading its input. /// /// While the editor has the terminal, no task reads it and the terminal is in /// its normal mode; the guard's panic hook still restores it then, which leaving /// the alternate screen and raw mode a second time does no harm to. pub(super) struct Screen { reader: Option, events: UnboundedSender>, } impl Screen { ``` -------------------------------- ### Save answered question example Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/answers.rs.html?search= Builds an Example from the question, system prompt, and completion, records it in stats, and appends it to the output Appender. Returns a PipelineError if appending fails. ```rust fn save( ctx: &Ctx<'_>, parent: &RoleClient, out: &mut Appender, stats: &mut StageStats, (question, system, completion): (Question, String, Completion), ) -> Result<(), PipelineError> { let example = example(ctx, parent, question, system, completion); record(ctx, parent, &example, stats); out.append(&example)?; Ok(()) } ``` -------------------------------- ### tests module: test setup for split.rs Source: https://docs.rs/overbrainer/latest/src/overbrainer/pipeline/split.rs.html?search= Test module that imports the parent module and dataset types (FinishReason, Id, Meta, ReasoningKind) and defines a helper to create Example instances for testing. ```rust #[cfg(test)] mod tests { use super::*; use crate::dataset::{FinishReason, Id, Meta, ReasoningKind}; fn example(subtopic: &str, n: usize) -> Example { ``` -------------------------------- ### Validate secret names with invalid and valid examples Source: https://docs.rs/overbrainer/latest/src/overbrainer/exec/ssh.rs.html Tests that secret names containing invalid characters (empty, starting with digit, hyphen, space, semicolon, or shell expansion) are rejected, while a valid name like '_A1' is accepted. ```rust for name in ["", "1ABC", "A-B", "A B", "A;rm -rf /", "$(x)"] { assert!( matches!(launcher(&with_name(name)), Err(ExecError::InvalidSecret(_))), "{name:?} was accepted" ); } assert!(launcher(&with_name("_A1")).is_ok()); ``` -------------------------------- ### metric_line formats TrainMetric as detailed training log Source: https://docs.rs/overbrainer/latest/src/overbrainer/cli/progress.rs.html?search=std%3A%3Avec Builds a human-readable string with step, epoch, and logged metric values (e.g., loss, lr, grad_norm). The example shows a step fraction and epoch, but the implementation is incomplete in the provided snippet (starts building parts). ```rust /// Step, epoch and the logged values of `metric`, for example /// `step 12/40 (30%), epoch 0.90, loss 1.2345, lr 2.00e-4, grad_norm 0.812`. #[must_use] pub fn metric_line(metric: &TrainMetric) -> String { let mut parts = vec![match metric.max_steps.filter(|max| *max > 0) { ``` -------------------------------- ### fn spawn(&self, job: &JobCommand) -> impl Future> + Send Source: https://docs.rs/overbrainer/latest/overbrainer/exec/trait.Executor.html?search= Starts a job in the background and returns its handle. The job continues running even if the process exits or loses connection. Environment setup depends on the target type. ```APIDOC ## fn spawn(&self, job: &JobCommand) -> impl Future> + Send ### Description Starts `job` in the background and returns its handle. The job keeps running when this process exits or loses its connection to the target. The environment the job starts with depends on the target. A local job inherits overbrainer’s own environment, minus every variable whose name starts with `OVERBRAINER_` or `VAULT_` (see `LocalExecutor`). A job over SSH gets whatever the remote non-interactive `sh -c` gives it, which is the SSH server’s environment for that user: no login shell and no interactive startup file are read (see `SshExecutor`). The job’s secrets are then set on top of that, on its environment only. A `native` target whose `axolotl` relies on the caller’s shell setup, such as a conda activation or a `PATH` entry from a profile, can therefore behave differently on each. ### Method spawn ### Signature `fn spawn(&self, job: &JobCommand) -> impl Future> + Send` ### Parameters #### Parameters - **job** (&JobCommand) - Required - The job command to start. ### Returns - **JobId** - The handle to the started job. ### Errors Returns `ExecError::InvalidSecret`, naming the variable only, when a secret’s name is not an environment variable name or its value holds a line break or a NUL byte; nothing is started then. Returns another `ExecError` when the job cannot be started. ``` -------------------------------- ### Prepare training run plan and list Runpod prices in start.rs Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/start.rs.html?search=Option%3CT%3E%2C+%28T+-%3E+U%29+-%3E+Option%3CU%3E Defines the data structures and functions that prepare a training run plan before it starts. It loads settings, validates the target, counts data files, and builds warnings; the async list_prices function fetches GPU prices with a 10-second timeout. ```rust 1//! What `t` shows before a training run starts: the target, the model, the data, 2//! and for Runpod the GPU types with their list prices and the most `max_hours` 3//! can cost. 4 5use std::path::Path; 6use std::time::Duration; 7 8use serde::de::IgnoredAny; 9 10use crate::config::{Adapter, EnvSource, Runtime, Target}; 11use crate::dataset::{DataFiles, read}; 12use crate::train::reasoning_template_warning; 13 14/// Total time the list prices may take; the dialog never waits for them. 15pub(super) const PRICES_TIMEOUT: Duration = Duration::from_secs(10); 16 /// What a training run started now would use. 17#[derive(Debug, Clone, PartialEq)] 18pub(super) struct StartPlan { 19 /// `training.target`. 20 pub(super) target: String, 21 /// What kind of target it is, for example `runpod, Secure Cloud`. 22 pub(super) kind: String, 23 /// Base model, adapter, epochs and learning rate. 24 pub(super) model: String, 25 /// Examples in `data/train.jsonl`. 26 pub(super) train: usize, 27 /// Examples in `data/eval.jsonl`. 28 pub(super) eval: usize, 29 /// The pod of a Runpod target. 30 pub(super) runpod: Option, 31 /// What the flow would warn about. 32 pub(super) warnings: Vec, 33} 34 /// The pod a Runpod run would ask for. 35#[derive(Debug, Clone, PartialEq)] 36pub(super) struct RunpodPlan { 37 /// GPU types, tried in order. 38 pub(super) gpu_types: Vec, 39 /// GPUs per pod. 40 pub(super) gpu_count: u32, 41 /// When the watchdog deletes the pod. 42 pub(super) max_hours: f64, 43} 44 /// The list price of each GPU type, per GPU and hour, when it could be read. 45pub(super) type Prices = Vec<(String, Option)>; 46 /// What a run started now in the project in `dir` would use, from its settings /// (read with `env`) and data files only. 47/// 48/// # Errors 49/// /// Returns why no run can start: the settings cannot be loaded, there is no /// `[training]` section, its target is unknown, or the data cannot be read. 50pub(super) fn prepare(dir: &Path, env: EnvSource) -> Result { 51 let settings = crate::config::load(dir, env).map_err(|error| format!("{error:#}"))?; 52 let training = settings 53 .training 54 .as_ref() 55 .ok_or("no [training] section in overbrainer.toml")?; 56 let target = settings 57 .targets 58 .get(&training.target) 59 .ok_or_else(|| format!("unknown target `{}`", training.target))?; 60 let files = DataFiles::new(dir); 61 let count = |path: &Path| { 62 read::(path) 63 .map(|lines| lines.len()) 64 .map_err(|error| format!("{:#}", anyhow::Error::from(error))) 65 }; 66 let adapter = match training.adapter { 67 Adapter::Lora => "lora", 68 Adapter::Qlora => "qlora", 69 Adapter::Full => "full", 70 }; 71 let mut warnings: Vec = reasoning_template_warning(training).into_iter().collect(); 72 if training.hub_model_id.is_some() && settings.hf_token.is_none() { 73 warnings.push( 74 "training.hub_model_id is set but OVERBRAINER_HF_TOKEN is not: the push will fail" 75 .to_string(), 76 ); 77 } 78 Ok(StartPlan { 79 target: training.target.clone(), 80 kind: kind(target), 81 model: format!( 82 "{}, {adapter}, {} epochs, lr {:e}", 83 training.base_model, training.epochs, training.learning_rate 84 ), 85 train: count(&files.train)?, 86 eval: count(&files.eval)?, 87 runpod: crate::runpod::RunpodTarget::from_target(target).map(|spec| RunpodPlan { 88 gpu_types: spec.gpu_types, 89 gpu_count: spec.gpu_count, 90 max_hours: spec.max_hours, 91 }), 92 warnings, 93 }) 94} 95 /// The kind of `target`, never its host nor any key. 96fn kind(target: &Target) -> String { 97 let runtime = |runtime: &Runtime| match runtime { 98 Runtime::Docker => "docker", 99 Runtime::Native => "native", 100 }; 101 match target { 102 Target::Local { runtime: r, .. } => format!("local, {}", runtime(r)), 103 Target::Ssh { runtime: r, .. } => format!("ssh, {}", runtime(r)), 104 Target::Runpod { .. } => "runpod, Secure Cloud".to_string(), 105 } 106} 107 /// The list price of each of `gpu_types` on the Runpod account of the project /// in `dir` (its settings read with `env`), best effort: a type whose price /// cannot be read has none, and the whole lookup gives up after /// [`PRICES_TIMEOUT`]. Only the client's fixed messages reach the logs. 108pub(super) async fn list_prices(dir: &Path, env: EnvSource, gpu_types: Vec) -> Prices { 109 list_prices_within(dir, env, gpu_types, PRICES_TIMEOUT).await 110} ``` -------------------------------- ### Test c on a starting run cancels it once its job started Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/app.rs.html?search= Test that pressing 'c' on a starting run cancels it once its job started. ```rust #[test] fn c_on_a_starting_run_cancels_it_once_its_job_started() { let mut app = app(); starting(&mut app, TaskId(6)); assert_eq!(app.cancel_run(FIRST), [], "never during the start"); assert_eq!( status(&app), Some("run 20260921-133200-a1b2 is starting: it is cancelled once its job started") ); } ``` -------------------------------- ### Abandon a single start task with status messages Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/follow.rs.html?search= Abandons a single start task if it is still starting and not already detached, with warnings if the job started or info messages otherwise. ```rust pub(super) fn abandon_start(&mut self, id: TaskId) -> Vec { let Some(follow) = self.training.tasks.get(&id) else { return Vec::new(); }; let run = follow.run(); if !follow.starting() { self.say( Severity::Warn, format!( "{run}: its job started, so it was not abandoned and its pod keeps billing; \ press c to cancel it" ), ); return Vec::new(); } if follow.detach == Detach::Done { self.say(Severity::Info, format!("{run} is being abandoned")); return Vec::new(); } self.say(Severity::Info, format!("abandoning {run}")); self.abandon(&[id]) } ``` -------------------------------- ### start Source: https://docs.rs/overbrainer/latest/src/overbrainer/runs/train.rs.html Starts a training run by preparing and uploading the run directory, launching the job through the executor, and recording the run as Running or Failed. Returns the updated RunRecord or a RunError. ```APIDOC ## start ### Description Starts a training run by preparing and uploading the run directory, launching the job through the executor, and recording the run as Running or Failed. Returns the updated RunRecord or a RunError. ### Method async fn start ### Signature pub async fn start( ctx: &RunCtx<'_, E>, trainer: &T, launch: Launch<'_>, mut record: RunRecord, ) -> Result ### Parameters - **ctx** (RunCtx) - Required - The run context providing executor and storage. - **trainer** (Trainer) - Required - The trainer instance used to prepare the run. - **launch** (Launch) - Required - Launch configuration including runtime and secrets. - **record** (RunRecord) - Required - The run record to be updated and saved. ### Returns - **Ok(RunRecord)** - The updated run record with job ID and state set to Running. - **Err(RunError)** - If the job cannot be prepared, copied, or started; the record is saved as Failed with the error message. ### Errors Returns a RunError when a file cannot be prepared, copied, or the job cannot start; the record is then saved as Failed with the reason. When the job started but its record cannot be saved, the job is cancelled on a best effort basis and the save error is returned. ``` -------------------------------- ### Helper to create app with provisioning run Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/event_loop.rs.html?search=std%3A%3Avec Helper that returns an App for the project in `dir`, with a Runpod run still provisioning as task 4. ```rust use crate::tui::training::{Follow, Job}; let mut app = app(); app.project.dir = dir.to_path_buf(); app.training .tasks .insert(TaskId(4), Follow::new(Job::Start { runpod: true }, RUN)); app ``` -------------------------------- ### start Source: https://docs.rs/overbrainer/latest/overbrainer/runs/fn.start.html?search=u32+-%3E+bool Prepares the files of the run record, copies them to the target, and starts the job. The record is saved with the job as `Running`; the `Running` status event is left to `watch`. ```APIDOC ## Function `start` ### Description Prepares the files of the run `record` (from `create`), copies them to the target and starts the job. The record is saved with the job, as `Running`. The `Running` status event is left to `watch`, which reports it on its first look at the job. ### Signature ```rust pub async fn start( ctx: &RunCtx<'_, E>, trainer: &T, launch: Launch<'_>, record: RunRecord, ) -> Result ``` ### Parameters - `ctx` (`&RunCtx<'_, E>`) - Required - The run context. - `trainer` (`&T`) - Required - The trainer instance. - `launch` (`Launch<'_>`) - Required - The launch configuration. - `record` (`RunRecord`) - Required - The run record to prepare and start. ### Returns - `Result` - On success, returns the updated `RunRecord`; on failure, returns a `RunError`. ### Errors Returns a `RunError` when a file cannot be prepared, copied, or the job cannot start; the record is then saved as `Failed` with the reason (a failure to save it is only logged, and the original error returned). When the job started but its record cannot be saved, nothing could find the job again: it is cancelled, on a best effort basis, and the save error is returned. ``` -------------------------------- ### Start Training Run Source: https://docs.rs/overbrainer/latest/src/overbrainer/tui/follow.rs.html Starts a training run on the specified target, unless a start is refused, and returns the resulting effects. ```rust if self.start_refused() { return Vec::new(); } let runpod = plan.runpod.is_some(); let effects = self.train(Job::Start { runpod }, ""); self.say(Severity::Info, format!("starting a run on {}", plan.target)); effects ```