### gflow Quick Start Commands

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

Essential commands to get started with gflow, including starting the daemon, checking GPU availability, submitting a GPU job, and monitoring job queues.

```bash
# Start the daemon (if not already running)
gflowd up

# See availability + current allocations
ginfo

# Submit a GPU job
gbatch --gpus 1 python train.py

# Track jobs and allocations
gqueue -s Running,Queued -f JOBID,NAME,ST,NODES,NODELIST(REASON)
```

--------------------------------

### Quick Start gflow Usage

Source: https://github.com/andpuqing/gflow/blob/main/README.md

Demonstrates the basic workflow of initializing, starting, submitting a job, checking status, and stopping the gflow scheduler.

```bash
gflowd init
gflowd up
gbatch --gpus 1 --name demo bash -lc 'echo "hello from gflow"; sleep 30'
gqueue
gjob show <job_id>
gflowd down
```

--------------------------------

### Start MCP Server for AI Agents

Source: https://context7.com/andpuqing/gflow/llms.txt

Instructions to start the gflow MCP server and integrate it with AI agents like Claude Code and Codex. Configuration examples for different agents are provided.

```bash
# Start MCP server (used by agent CLIs as stdio subprocess)
gflow mcp serve

# Claude Code integration
claude mcp add --scope user gflow -- gflow mcp serve
claude mcp list
claude mcp get gflow

# Codex integration
codex mcp add gflow -- gflow mcp serve
codex mcp list

# Or configure Codex via ~/.codex/config.toml:
# [mcp_servers.gflow]
# command = "gflow"
# args = ["mcp", "serve"]

# OpenCode integration via opencode.json:
# {
#   "mcp": {
#     "gflow": {
#       "type": "local",
#       "command": ["gflow", "mcp", "serve"],
#       "enabled": true
#     }
#   }
# }

# Verify MCP is working
gflowd status
ginfo
gflow mcp serve --help
```

--------------------------------

### gflow CLI Project Usage Examples

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of using gflow client commands with the --project flag to associate jobs with specific projects.

```bash
gbatch --project ml-research python train.py
gqueue --project ml-research
gqueue --format JOBID,NAME,PROJECT,ST,TIME
```

--------------------------------

### Install gflow Nightly Build

Source: https://github.com/andpuqing/gflow/blob/main/README.md

Installs the latest nightly build of 'runqd' from the TestPyPI repository, useful for testing pre-release versions.

```bash
pip install --index-url https://test.pypi.org/simple/ runqd
```

--------------------------------

### Install gflow with Cargo

Source: https://github.com/andpuqing/gflow/blob/main/README.md

Installs the gflow binary directly from source using the Cargo package manager for Rust.

```bash
cargo install gflow
```

--------------------------------

### gflowd CLI Configuration Examples

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of using gflowd and other gflow client commands with the --config flag to specify a custom configuration file path.

```bash
gflowd --config <path> up
ginfo --config <path>
gbatch --config <path> --gpus 1 python train.py
```

--------------------------------

### Install gflow with Python Tooling

Source: https://github.com/andpuqing/gflow/blob/main/README.md

Installs the 'runqd' package, which is the Python distribution of gflow, using common Python package managers.

```bash
uv tool install runqd
# or
pipx install runqd
# or
pip install runqd
```

--------------------------------

### gstats Basic Usage and Examples

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gstats-reference.md

Demonstrates the fundamental usage of the gstats command for retrieving scheduler statistics. Includes examples for current user stats, time-based filtering, specific user stats, and different output formats like JSON and CSV.

```bash
gstats
gstats --since 7d
gstats --user alice
gstats --all-users --output json
gstats --since today --output csv
```

--------------------------------

### gqueue Dependency Tree Example Output

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gqueue-reference.md

Provides an example of the output format when using the dependency tree view (`-t` option), showing job relationships and states.

```text
JOBID  NAME   ST  TIME      NODES  NODELIST(REASON)
1      prep   CD  00:02:15  0      -
├─2    train  R   00:10:03  1      0
└─3    eval   PD  -         0      (WaitingForDependency)
```

--------------------------------

### Submit Jobs via CLI

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gbatch-reference.md

Examples of using gbatch to submit jobs with various resource constraints, scheduling priorities, and dependency requirements.

```bash
gbatch --gpus 1 python train.py
gbatch --time 2:00:00 python train.py
gbatch --memory 8G python train.py
gbatch --gpu-memory 20G --shared --gpus 1 python train.py
gbatch --priority 50 python urgent.py
gbatch --depends-on 123 --no-auto-cancel python next.py
gbatch --array 1-10 python task.py --i '$GFLOW_ARRAY_TASK_ID'
```

--------------------------------

### Troubleshooting Dependent Jobs Not Starting

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/job-dependencies.md

Offers commands to diagnose why a dependent job might not be starting. This involves checking the overall job status, the status of the parent job, and using 'ginfo' for more details.

```bash
gqueue -t
gqueue -j <parent_job_id> -f JOBID,ST
ginfo
```

--------------------------------

### Quick Start Job Dependencies with Gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/job-dependencies.md

Demonstrates the basic usage of 'gbatch' to submit jobs with sequential dependencies. The 'train.py' job depends on 'preprocess.py', and 'evaluate.py' depends on 'train.py'.

```bash
gbatch --time 10 python preprocess.py
gbatch --depends-on @ --gpus 1 --time 4:00:00 python train.py
gbatch --depends-on @ --time 10 python evaluate.py
```

--------------------------------

### Complete ML Workflow Example with gflow

Source: https://context7.com/andpuqing/gflow/llms.txt

Demonstrates a typical machine learning workflow using gflow, including submitting jobs with dependencies, specifying GPU requirements, monitoring progress, and handling job failures.

```bash
# Start the scheduler
gflowd up

# Submit a preprocessing job (no GPU needed)
gbatch --time 10 --name preprocess python preprocess.py
# Output: Submitted batch job 1

# Submit training job depending on preprocessing
gbatch --depends-on @ --gpus 1 --time 4:00:00 --name train python train.py
# Output: Submitted batch job 2

# Submit evaluation depending on training
gbatch --depends-on @ --gpus 1 --time 30 --name evaluate python evaluate.py
# Output: Submitted batch job 3

# View dependency tree
gqueue -t
# Output:
# JOBID  NAME        ST  TIME      NODES  NODELIST(REASON)
# 1      preprocess  R   00:02:15  0      -
# └─2    train       PD  -         1      (WaitingForDependency)
#   └─3  evaluate    PD  -         1      (WaitingForDependency)

# Monitor job progress
gqueue -w

# Check logs of running job
gjob log 1 --last 20

# Attach to running job (Ctrl-B D to detach)
gjob attach 1

# If training fails, redo with more time and cascade to dependents
gjob redo 2 --time 8:00:00 --cascade
```

--------------------------------

### gflow Configuration Options (TOML)

Source: https://context7.com/andpuqing/gflow/llms.txt

Example TOML configuration file for gflow, demonstrating settings for the daemon, GPU allocation, project tracking, and notifications (webhooks and email).

```toml
# ~/.config/gflow/gflow.toml

[daemon]
host = "localhost"
port = 59000
gpus = [0, 1, 2, 3]  # Restrict to specific GPUs
gpu_allocation_strategy = "sequential"  # or "random"
gpu_poll_interval_secs = 10  # How often to check for external GPU usage

# Timezone for reservations and display
timezone = "America/Los_Angeles"

# Project tracking
[projects]
known_projects = ["ml-research", "cv-team", "nlp-project"]
require_project = false  # Set true to require project on all jobs

# Webhook notifications
[[notifications.webhooks]]
url = "https://api.example.com/gflow/events"
events = ["job_completed", "job_failed", "job_timeout"]
filter_users = ["alice", "bob"]
headers = { Authorization = "Bearer token123" }
timeout_secs = 10
max_retries = 3

# Email notifications
[[notifications.emails]]
smtp_url = "smtps://user:pass@smtp.example.com:465"
from = "gflow <noreply@example.com>"
to = ["alice@example.com", "ml-oncall@example.com"]
events = ["job_failed", "job_timeout"]
subject_prefix = "[gflow-prod]"
timeout_secs = 10
max_retries = 3
```

--------------------------------

### Manage gflow Daemon

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Commands to initialize, start, check status, and stop the gflow daemon process.

```bash
gflowd init
gflowd up
gflowd status
gflowd down
gflowd restart
```

--------------------------------

### Start gflow MCP Server

Source: https://github.com/andpuqing/gflow/blob/main/README.md

Starts gflow in MCP (Multi-Client Protocol) mode, allowing it to act as a local server for AI tools that support this protocol.

```bash
gflow mcp serve
```

--------------------------------

### Monitor GPU Status in Real-time

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/ginfo-reference.md

Examples of executing ginfo directly or using standard Linux utilities like watch to monitor GPU allocation changes over time.

```bash
ginfo
watch -n 2 ginfo
```

--------------------------------

### Initialize gflowd Scheduler

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/ai-integration/mcp-and-skills.md

Commands to ensure the local gflowd scheduler is running and accessible. 'gflowd up' starts the scheduler, 'gflowd status' checks its health, and 'ginfo' provides system information. gflowd must be started before gflow mcp serve.

```bash
gflowd up
gflowd status
ginfo
```

--------------------------------

### gflow TOML Configuration Example

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

A minimal TOML configuration file for gflowd, specifying the host and port for the daemon. It also includes commented-out options for GPU configuration.

```toml
[daemon]
host = "localhost"
port = 59000
# gpus = [0, 2]
# gpu_allocation_strategy = "sequential" # or "random"
# gpu_poll_interval_secs = 10
```

--------------------------------

### Create GPU Reservation with gctl

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gctl-reference.md

Creates a GPU reservation for a specified user, with options to reserve by GPU count (dynamic allocation) or by specific GPU indices. The command supports precise scheduling with start times and durations, accepting ISO8601 or custom formats for time inputs. Time inputs must align with `:00` or `:30` intervals, and durations must be multiples of 30 minutes.

```bash
gctl reserve create --user alice --gpus 2 --start '2026-01-28 14:00' --duration 2h
gctl reserve create --user alice --gpu-spec 0,2 --start '2026-01-28 14:00' --duration 2h
gctl reserve create --user bob --gpu-spec 0-3 --start '2026-01-28 16:00' --duration 1h
```

--------------------------------

### Multiple Job Dependencies (AND/OR) with Gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/job-dependencies.md

Illustrates submitting jobs that depend on multiple parent jobs. '--depends-on-all' requires all specified parent jobs to succeed, while '--depends-on-any' allows the job to start if any one parent succeeds. Shorthands like '@' can also be used.

```bash
# AND: all parents must finish successfully
gbatch --depends-on-all 101,102,103 python merge.py

# OR: any one parent finishing successfully is enough
gbatch --depends-on-any 201,202,203 python process_first_success.py

# Shorthands also work in lists
gbatch --depends-on-all @,@~1,@~2 python process_recent_jobs.py
```

--------------------------------

### Create GPU Reservations with gctl

Source: https://context7.com/andpuqing/gflow/llms.txt

Reserve GPUs for specific users or for specific time periods using gctl reserve create. Reservations can be based on user, duration, start time, and specific GPU indices or specs.

```bash
gctl reserve create --user alice --gpus 2 --start '2026-01-28 14:00' --duration 2h
gctl reserve create --user alice --gpu-spec 0,2 --start '2026-01-28 14:00' --duration 2h
```

--------------------------------

### Configure Advanced Job Options

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/job-submission.md

Examples of common job configurations including shared GPU mode, priority, conda environments, and complex dependency chains.

```bash
# Shared GPU mode
gbatch --gpus 1 --shared --gpu-memory 20G python train.py

# Dependencies
gbatch --depends-on <job_id|@|@~N> python next.py
gbatch --depends-on-all 1,2,3 python merge.py

# Dry run
gbatch --dry-run --gpus 1 python train.py
```

--------------------------------

### Customizing gqueue Output Format

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gqueue-reference.md

Explains how to specify a custom output format for gqueue by selecting desired fields. Includes examples of default and custom formats.

```text
JOBID,NAME,ST,TIME,NODES,NODELIST(REASON)
```

```bash
gqueue -f JOBID,NAME,PROJECT,ST,TIMELIMIT,MEMORY,NODELIST(REASON)
```

--------------------------------

### List and Manage GPU Reservations with gctl

Source: https://context7.com/andpuqing/gflow/llms.txt

View and manage existing GPU reservations using gctl reserve list, get, and cancel commands. Filters can be applied for active reservations, specific users, or time ranges.

```bash
gctl reserve list
gctl reserve list --active
gctl reserve list --user alice --status active
gctl reserve list --timeline --range 48h
gctl reserve get abc123
gctl reserve cancel abc123
```

--------------------------------

### Override gflow Configuration with Environment Variables

Source: https://context7.com/andpuqing/gflow/llms.txt

Examples of using environment variables to override gflow configuration settings, particularly for daemon host, port, GPUs, and GPU allocation strategy.

```bash
# Environment variables (override config)
export GFLOW_DAEMON__HOST=localhost
export GFLOW_DAEMON__PORT=59000
export GFLOW_DAEMON__GPUS=0,2
export GFLOW_DAEMON__GPU_ALLOCATION_STRATEGY=random
export GFLOW_DAEMON__GPU_POLL_INTERVAL_SECS=3
```

--------------------------------

### gflowd Environment Variable Configuration

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of setting gflow daemon configuration using environment variables. Nested keys are separated by double underscores.

```bash
export GFLOW_DAEMON__HOST=localhost
export GFLOW_DAEMON__PORT=59000
export GFLOW_DAEMON__GPUS=0,2
export GFLOW_DAEMON__GPU_ALLOCATION_STRATEGY=random
export GFLOW_DAEMON__GPU_POLL_INTERVAL_SECS=3
```

--------------------------------

### Basic NVIDIA GPU Check

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

A quick check to ensure NVIDIA GPUs are detected and the gflow daemon is running. This involves running `nvidia-smi`, starting the `gflowd` daemon, and then checking `ginfo`.

```bash
nvidia-smi
gflowd up
ginfo
```

--------------------------------

### View Session Statistics with gstats

Source: https://context7.com/andpuqing/gflow/llms.txt

This command displays statistics for the current gflow session. It requires the gflow CLI to be installed and accessible in the system's PATH. The output provides insights into ongoing and completed jobs.

```bash
gstats --since today
```

--------------------------------

### Troubleshooting: Job Not Getting GPU

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

If a job is not being allocated a GPU, these commands can help diagnose the issue by checking overall GPU status, job queue details, and specific GPU restrictions.

```bash
ginfo
gqueue -j <job_id> -f JOBID,ST,NODES,NODELIST(REASON)
gctl show-gpus
```

--------------------------------

### Get GPU Reservation Details with gctl

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gctl-reference.md

Fetches and displays detailed information about a specific GPU reservation using its unique reservation ID. This command is helpful for inspecting the configuration and status of individual reservations.

```bash
gctl reserve get <reservation_id>
```

--------------------------------

### Webhook Notification Payload Example (JSON)

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/notifications.md

An example JSON payload structure for webhook notifications, illustrating the fields included for different event types. Fields like 'job' and 'scheduler' may vary based on the specific event.

```json
{
  "event": "job_completed",
  "timestamp": "2026-02-04T12:30:45Z",
  "job": { "id": 42, "user": "alice", "state": "Finished" },
  "scheduler": { "host": "gpu-server-01", "version": "0.4.11" }
}
```

--------------------------------

### Submit Jobs with gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/job-submission.md

Demonstrates how to submit jobs using direct command-line arguments. This includes setting resources like GPUs, time limits, project codes, and notification settings.

```bash
gbatch python train.py
gbatch --gpus 1 --time 2:00:00 --name train-resnet python train.py
gbatch --project ml-research python train.py
gbatch --notify-email alice@example.com --notify-on job_failed,job_timeout python train.py
```

--------------------------------

### Check gflow MCP Server Help

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/ai-integration/mcp-and-skills.md

Displays the help information for the 'gflow mcp serve' command. This is useful for verifying that the MCP subcommand is available and understanding its options.

```bash
gflow mcp serve --help
```

--------------------------------

### gctl Reserve Command Timezone Override

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Example of overriding the timezone for a specific gctl reserve command.

```bash
gctl reserve create --user alice --gpus 2 --start "2026-02-01 14:00" --duration "2h" --timezone "UTC"
```

--------------------------------

### Manage Job Dependencies

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Configure job execution order using single or multiple dependencies, including shorthand references for recent jobs.

```bash
gbatch --depends-on <job_id|@|@~N> python next.py
gbatch --depends-on-all 1,2,3 python merge.py
gbatch --depends-on-any 4,5 python fallback.py
```

--------------------------------

### GET /gstats/completion

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gstats-reference.md

Generates shell completion scripts for the gstats CLI tool to improve user experience in terminal environments.

```APIDOC
## GET /gstats/completion

### Description
Generates shell completion scripts for supported shells to enable auto-completion for gstats commands.

### Method
GET

### Endpoint
gstats completion <shell>

### Parameters
#### Path Parameters
- **shell** (string) - Required - The target shell (bash, zsh, or fish)

### Request Example
`gstats completion bash`

### Response
#### Success Response (200)
- **script** (string) - The shell completion script content
```

--------------------------------

### GET /gstats

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gstats-reference.md

Retrieves scheduler usage statistics based on user filters, time windows, and output format preferences.

```APIDOC
## GET /gstats

### Description
Retrieves scheduler usage statistics for the current user or all users. Supports filtering by time window and exporting data in various formats.

### Method
GET

### Endpoint
gstats

### Parameters
#### Query Parameters
- **--user** (string) - Optional - Filter by specific username (default: current user)
- **--all-users** (boolean) - Optional - Aggregate statistics across all users
- **--since** (string) - Optional - Time window (e.g., 1h, 7d, 30d, today, or ISO timestamp)
- **--output** (string) - Optional - Format of the output: table, json, or csv (default: table)

### Request Example
`gstats --user alice --since 7d --output json`

### Response
#### Success Response (200)
- **total_jobs** (integer) - Total number of jobs processed
- **avg_wait_secs** (float) - Average time jobs spent in queue
- **total_gpu_hours** (float) - Total GPU usage in hours
- **success_rate** (float) - Percentage of successfully completed jobs

#### Response Example
{
  "total_jobs": 150,
  "avg_wait_secs": 45.5,
  "total_gpu_hours": 12.2,
  "success_rate": 0.98
}
```

--------------------------------

### gflowd CLI GPU Poll Interval Override

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of overriding the GPU poll interval using gflowd CLI flags.

```bash
gflowd up --gpu-poll-interval-secs 3
gflowd reload --gpu-poll-interval-secs 1
```

--------------------------------

### gflowd CLI GPU Selection Override

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of overriding GPU selection using gflowd CLI flags, supporting individual indices and ranges.

```bash
gflowd up --gpus 0,2
gflowd restart --gpus 0-3
```

--------------------------------

### Submit Batch Jobs with gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Submit scripts or commands to the queue with specific resource requirements like GPUs, memory, and environment settings.

```bash
gbatch python train.py --epochs 100
gbatch train.sh
gbatch --gpus 1 --time 2:00:00 --name train-resnet python train.py
gbatch --gpus 1 --shared --gpu-memory 20G python train.py
gbatch --conda-env myenv python script.py
```

--------------------------------

### gflow GPU Selection Configuration

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

TOML configuration to restrict which physical GPUs the gflow scheduler can allocate. Example specifies GPUs with indices 0 and 2.

```toml
[daemon]
gpus = [0, 2]
```

--------------------------------

### Inspect and Monitor Jobs

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Commands to view GPU availability, monitor active or completed jobs, and visualize job dependency trees.

```bash
ginfo
gqueue
gqueue -a
gqueue -P ml-research
gqueue -f JOBID,NAME,ST,TIME,NODES,NODELIST(REASON)
gqueue -t
```

--------------------------------

### gflowd CLI GPU Allocation Strategy Override

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/configuration.md

Examples of overriding the GPU allocation strategy using gflowd CLI flags, which take precedence over the config file.

```bash
gflowd up --gpu-allocation-strategy random
gflowd restart --gpu-allocation-strategy sequential
```

--------------------------------

### Basic gqueue Usage

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gqueue-reference.md

Demonstrates the fundamental ways to invoke the gqueue command, including listing active jobs and all jobs.

```bash
gqueue                               # active jobs (Queued, Hold, Running)
gqueue -a                            # all jobs including completed
```

--------------------------------

### Runtime Resource Control

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Manage GPU allocation restrictions, concurrency limits, and create resource reservations.

```bash
gctl set-gpus 0,2
gctl set-limit <job_or_group_id> 2
gctl reserve create --user alice --gpus 2 --start '2026-01-28 14:00' --duration 2h
```

--------------------------------

### Overriding GPU Allocation Strategy on Daemon Startup

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

The GPU allocation strategy can also be overridden directly when starting the gflow daemon using the `--gpu-allocation-strategy` flag.

```bash
gflowd up --gpu-allocation-strategy random
```

--------------------------------

### Monitor System GPU Information Continuously

Source: https://context7.com/andpuqing/gflow/llms.txt

Use the 'watch' command in conjunction with 'ginfo' to continuously monitor GPU status at a specified interval (e.g., every 2 seconds).

```bash
watch -n 2 ginfo
```

--------------------------------

### Manage gflowd daemon lifecycle

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gflowd-reference.md

Commands to control the daemon state, including starting, stopping, reloading, and restarting. These commands support GPU-specific overrides for scheduling.

```bash
gflowd up
gflowd up --gpus 0,2 --gpu-allocation-strategy random
gflowd reload
gflowd restart --gpus 0-3
gflowd status
gflowd down
```

--------------------------------

### Parameter Sweeps and Cartesian Products

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gbatch-reference.md

Use the --param flag to perform parameter sweeps, creating a cartesian product of inputs for job execution.

```bash
gbatch --param lr=0.001,0.01 --param bs=32,64 python train.py --lr {lr} --batch-size {bs}
gbatch --param-file params.csv --name-template 'run_{id}' python train.py --id {id}
```

--------------------------------

### Configure Jobs with Script Directives

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gbatch-reference.md

Demonstrates how to embed gflow configuration options directly into a shell script using GFLOW comments. These directives are parsed by gbatch upon submission.

```bash
#!/bin/bash
# GFLOW --gpus=1
# GFLOW --shared
# GFLOW --time=2:00:00
# GFLOW --memory=4G
# GFLOW --gpu-memory=20G
# GFLOW --priority=20
# GFLOW --conda-env=myenv
# GFLOW --depends-on=123
# GFLOW --project=ml-research
# GFLOW --notify-email=alice@example.com
# GFLOW --notify-on=job_failed,job_timeout
```

--------------------------------

### Monitor Job States with gqueue

Source: https://context7.com/andpuqing/gflow/llms.txt

Commands to check job states (Queued, Running, Failed, etc.) and view reasons for pending jobs. Displays job ID, name, status, and node list with reasons.

```bash
# Job states:
# PD (Queued)    - Waiting to run (pending dependencies or resources)
# H  (Hold)      - On hold by user request
# R  (Running)   - Currently executing
# CD (Finished)  - Completed successfully
# F  (Failed)    - Terminated with error
# CA (Cancelled) - Cancelled by user or system
# TO (Timeout)   - Exceeded time limit

# Check job state and reason
gqueue -f JOBID,NAME,ST,NODELIST(REASON)
# Output:
# JOBID  NAME     ST  NODELIST(REASON)
# 42     train    PD  (WaitingForDependency)
# 43     eval     PD  (WaitingForGpu)
# 44     report   H   (JobHeldUser)
```

--------------------------------

### Display GPU and Scheduler Information

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/ginfo-reference.md

Basic usage of the ginfo command to retrieve current GPU allocation status. It supports shell completion and custom configuration paths.

```bash
ginfo
ginfo completion <shell>
```

--------------------------------

### Inspecting GPU Status with ginfo

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

The `ginfo` command provides a snapshot of GPU availability, current allocations, and job assignments on the system. It helps understand which GPUs are idle or in use.

```bash
ginfo
```

--------------------------------

### gqueue Views and Refresh Options

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gqueue-reference.md

Illustrates how to view jobs in a dependency tree or grouped by state, and how to enable auto-refreshing the job list.

```bash
gqueue -t                            # dependency tree view
gqueue -g                            # group by state
gqueue -w                            # auto-refresh every 2s
gqueue -w --interval 5               # auto-refresh every 5s
```

--------------------------------

### Troubleshoot Timeouts

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/time-limits.md

Provides commands to resubmit timed-out jobs with updated limits and verify configuration settings to ensure timeouts are behaving as expected.

```bash
# Resubmit a job with a new time limit
gjob redo <job_id> --time 4:00:00

# Verify time format (30 seconds vs 30 minutes)
gbatch --time 0:30 sleep 1000
gbatch --time 30 sleep 1000

# Check system info and specific job limits
ginfo
gqueue -j <job_id> -f JOBID,ST,TIMELIMIT
```

--------------------------------

### Manage Claude Code MCP Servers

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/ai-integration/mcp-and-skills.md

Commands to list and retrieve details of configured MCP servers in Claude Code. 'claude mcp list' shows all configured servers, and 'claude mcp get gflow' displays the configuration for the gflow server.

```bash
claude mcp list
claude mcp get gflow
```

--------------------------------

### Submit Job with Project Label

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

This bash command demonstrates how a user submits a job using gbatch, specifying the project label 'ml-research'. This is required when the gflowd configuration enforces project labels for job submissions, aiding in organization and tracking.

```bash
gbatch --project ml-research python train.py

```

--------------------------------

### Submit Jobs with gbatch

Source: https://context7.com/andpuqing/gflow/llms.txt

The `gbatch` command submits jobs to the gflow scheduler, analogous to Slurm's `sbatch`. It supports direct commands, script files, and numerous options for resource allocation (GPUs, memory), time limits, job naming, priorities, conda environments, project tracking, notifications, and complex dependency management (AND/OR modes). It also facilitates job arrays, parameter sweeps, and dry runs.

```bash
# Submit a simple command
gbatch python train.py --epochs 100

# Submit with GPU, time limit, and job name
gbatch --gpus 1 --time 2:00:00 --name train-resnet python train.py

# Submit with multiple GPUs
gbatch --gpus 2 python multi_gpu_train.py

# Shared GPU mode (multiple jobs on one GPU with VRAM limits)
gbatch --gpus 1 --shared --gpu-memory 20G python train.py

# Set priority (0-255, higher runs first)
gbatch --priority 50 python urgent.py

# Use a specific conda environment
gbatch --conda-env myenv python script.py

# Add project code for tracking
gbatch --project ml-research python train.py

# Enable email notifications for specific events
gbatch --notify-email alice@example.com --notify-on job_failed,job_timeout python train.py

# Submit with dependencies (@ = most recent job, @~N = Nth most recent)
gbatch --depends-on @ python next_step.py
gbatch --depends-on @~1 python after_previous.py

# Multiple dependencies - AND mode (all must complete)
gbatch --depends-on-all 101,102,103 python merge.py

# Multiple dependencies - OR mode (any one completes)
gbatch --depends-on-any 201,202,203 python process_first_success.py

# Disable auto-cancel when dependency fails
gbatch --depends-on 123 --no-auto-cancel python next.py

# Job arrays for batch processing
gbatch --array 1-10 python process.py --task '$GFLOW_ARRAY_TASK_ID'

# Parameter sweeps (cartesian product)
gbatch --param lr=0.001,0.01 --param bs=32,64 python train.py --lr {lr} --batch-size {bs}

# Parameter sweep from CSV file with max concurrency
gbatch --param-file params.csv --max-concurrent 2 --name-template 'run_{id}' python train.py --id {id}

# Preview submission without actually submitting
gbatch --dry-run --gpus 1 python train.py

# Submit a script with embedded directives
cat > train.sh << 'EOF'
#!/bin/bash
# GFLOW --gpus=1
# GFLOW --time=2:00:00
# GFLOW --memory=8G
# GFLOW --gpu-memory=20G
# GFLOW --priority=20
# GFLOW --conda-env=myenv
# GFLOW --project=ml-research

python train.py
EOF
chmod +x train.sh
gbatch train.sh
```

--------------------------------

### Configure gflowd Daemon Host and Port

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

This TOML configuration snippet sets the host and port for the gflowd daemon. It's crucial for multi-user setups to bind to 'localhost' for security unless a specific network boundary is established. The daemon's state and logs are stored in the home directory of the user running gflowd.

```toml
[daemon]
host = "localhost"
port = 59000

```

--------------------------------

### Debug and Manage Jobs

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/tips.md

Commands for viewing queue status, safe cancellation, and cascading job retries.

```bash
gqueue -t
gcancel --dry-run <job_id>
gcancel <job_id>
gjob redo <job_id> --cascade
```

--------------------------------

### Display Scheduler Usage Statistics with gstats

Source: https://context7.com/andpuqing/gflow/llms.txt

The gstats command provides comprehensive statistics on scheduler usage, including job counts, wait times, runtimes, GPU-hours, and success rates for the current user.

```bash
gstats
```

--------------------------------

### Initialize gflowd configuration

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gflowd-reference.md

Initializes the gflow daemon configuration file. Supports interactive mode or non-interactive mode using the --yes flag to accept defaults.

```bash
gflowd init
gflowd init --yes
gflowd init --force --gpus 0,2 --port 59000
```

--------------------------------

### View All User Jobs and Statistics

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

These bash commands provide administrator views of job queues and statistics across all users. 'gqueue -u all' lists all jobs, 'gstats --user alice' shows statistics for a specific user, and 'gctl reserve list --timeline' displays a timeline of reservations.

```bash
gqueue -u all
gstats --user alice
gctl reserve list --timeline --range 48h

```

--------------------------------

### Set Job Time Limits

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/time-limits.md

Demonstrates how to define execution time limits for Gflow jobs using command-line flags with gbatch or via shell script directives. Command-line flags take precedence over script directives.

```bash
gbatch --time 2:00:00 python train.py
```

```bash
#!/bin/bash
# GFLOW --time 2:00:00

python train.py
```

--------------------------------

### Monitor Jobs with gqueue

Source: https://context7.com/andpuqing/gflow/llms.txt

The `gqueue` command lists and monitors jobs in the gflow scheduler, offering various filtering, formatting, and viewing options for real-time visibility. Users can filter jobs by state, ID, user, project, and time window, as well as view only jobs with active tmux sessions.

```bash
# Show active jobs (Queued, Hold, Running)
gqueue

# Show all jobs including completed
gqueue -a

# Show only completed jobs
gqueue -c

# Filter by job state
gqueue -s Running,Queued

# Filter by specific job IDs
gqueue -j 12,13,14

# Filter by user (use 'all' for all users)
gqueue -u alice
gqueue -u all

# Filter by project code
gqueue -P ml-research

# Filter jobs since a time window
gqueue --since 1h
gqueue --since today
gqueue --since yesterday
gqueue --since 7d

# Only jobs with active tmux sessions
gqueue -T
```

--------------------------------

### View gqueue Dependency Tree

Source: https://context7.com/andpuqing/gflow/llms.txt

The gqueue -t command displays a hierarchical dependency tree of jobs, showing their states (CD, R, PD) and relationships. This helps visualize job dependencies and their current status.

```bash
gqueue -t
```

--------------------------------

### Show Detailed Job Information with gjob

Source: https://context7.com/andpuqing/gflow/llms.txt

The gjob show command displays detailed information for specified jobs or job ranges. It is useful for inspecting the configuration and status of individual or multiple jobs.

```bash
gjob show 42
gjob show 10-15
```

--------------------------------

### Submitting GPU Jobs with gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

The `gbatch` command is used to submit jobs that require GPU resources. You can specify the number of GPUs needed using the `--gpus` flag. gflow then assigns physical GPU indices.

```bash
gbatch --gpus 1 python train.py
gbatch --gpus 2 python multi_gpu_train.py
```

--------------------------------

### Submit Job with Project Label (User View)

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

This bash command demonstrates a regular user submitting a job with a project label, consistent with administrator-configured policies. It ensures jobs are correctly categorized and tracked within the shared environment.

```bash
gbatch --project ml-research python train.py
gqueue --project ml-research

```

--------------------------------

### Submitting Shared GPU Jobs with gbatch

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/gpu-management.md

For scenarios where multiple jobs can share a single physical GPU, use the `--shared` flag with `gbatch`. This requires specifying a per-GPU VRAM limit using `--gpu-memory`.

```bash
gbatch --gpus 1 --shared --gpu-memory 20G python train.py
```

--------------------------------

### Execute Parameter Sweeps

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/tips.md

Perform cartesian product parameter sweeps using --param or load parameters from a CSV file. Supports ranges and concurrency limits.

```bash
gbatch --dry-run \
  --param lr=0.001,0.01 \
  --param bs=32,64 \
  --name-template 'lr{lr}_bs{bs}' \
  python train.py --lr {lr} --batch-size {bs}

gbatch --param-file params.csv --name-template 'run_{id}' python train.py --id {id}
gbatch --param lr=0.001,0.01 --max-concurrent 1 python train.py --lr {lr}
```

--------------------------------

### Aggregate Statistics with gstats

Source: https://context7.com/andpuqing/gflow/llms.txt

Commands to aggregate statistics across all users or output in specific formats like JSON or CSV. CSV output includes headers for metric and value.

```bash
# Aggregate across all users
gstats --all-users

# Output as JSON
gstats --output json

# Output as CSV (for scripts)
gstats --since today --output csv
```

--------------------------------

### Preview Job Cancellation with gcancel --dry-run

Source: https://context7.com/andpuqing/gflow/llms.txt

Use gcancel --dry-run to preview which jobs would be cancelled, including any dependent jobs that would be automatically affected. This helps in understanding the impact of a cancellation.

```bash
gcancel --dry-run 42
```

--------------------------------

### Job Control and Maintenance

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/quick-reference.md

Commands to cancel, hold, release, or update existing jobs and view their logs.

```bash
gcancel <job_id>
gjob hold <job_id>
gjob release <job_id>
gjob log <job_id> --last 50
gjob update <job_id> --gpus 2 --time-limit 4:00:00
```

--------------------------------

### Filtering gqueue Jobs by State, ID, User, and Project

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gqueue-reference.md

Shows how to filter the job list based on specific criteria such as job state, job IDs, user, and project code.

```bash
gqueue -s Running,Queued             # filter by state
gqueue -j 12,13,14                   # filter by job IDs (comma-separated)
gqueue -u alice                      # filter by user (default: current user; use 'all' for all users)
gqueue -P ml-research                # filter by project code
```

--------------------------------

### View Current User's Jobs and Statistics

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

These bash commands show the default per-user views for regular users. 'gqueue' lists the current user's jobs, and 'gstats' provides statistics for the current user. These commands are essential for users to monitor their own submitted tasks.

```bash
gqueue
gstats

```

--------------------------------

### Inspect Job Time Limits

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/time-limits.md

Commands to view the current time limits and status of submitted jobs using the gqueue and gjob utilities.

```bash
gqueue -f JOBID,NAME,ST,TIME,TIMELIMIT
gjob show <job_id>
```

--------------------------------

### Control GPU Resources

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/tips.md

Restrict available GPUs for gflow or create reservations to block resources for specific users and time windows.

```bash
gctl set-gpus 0,2
gctl show-gpus
gctl set-gpus all
gctl reserve create --user alice --gpus 2 --start '2026-01-28 14:00' --duration 2h
gctl reserve list --active
gctl reserve list --timeline --range 48h
gctl reserve cancel <reservation_id>
```

--------------------------------

### List GPU Reservations with gctl

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gctl-reference.md

Retrieves a list of existing GPU reservations. Supports filtering by status (e.g., 'active'), user, and timeline view with a specified time range. This command is useful for auditing and managing scheduled GPU resources.

```bash
gctl reserve list
gctl reserve list --active
gctl reserve list --user alice --status active
gctl reserve list --timeline --range 48h
```

--------------------------------

### Manage gflow Daemon Lifecycle

Source: https://context7.com/andpuqing/gflow/llms.txt

The `gflowd` command manages the gflow daemon, including initialization, startup, status checks, and shutdown. It runs as a daemon in a tmux session and exposes a REST API for client commands. Options include interactive or non-interactive initialization, specifying GPU restrictions, and adjusting polling intervals.

```bash
# Initialize configuration (interactive)
gflowd init

# Initialize with defaults non-interactively
gflowd init --yes

# Start the daemon
gflowd up

# Start with specific GPU restrictions and random allocation
gflowd up --gpus 0,2 --gpu-allocation-strategy random

# Start with faster GPU occupancy polling (default: 10 seconds)
gflowd up --gpu-poll-interval-secs 3

# Check daemon status
gflowd status
# Output: gflowd is running (PID 12345)

# Reload configuration without downtime
gflowd reload

# Restart daemon with new GPU settings
gflowd restart --gpus 0-3

# Stop the daemon
gflowd down

# Generate shell completions
gflowd completion bash >> ~/.bashrc
gflowd completion zsh >> ~/.zshrc
```

--------------------------------

### Control GPU Allocation with gctl

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/multi-user.md

These bash commands illustrate administrator usage of gctl to manage GPU resources. 'gctl set-gpus' restricts the number of GPUs available for general use, while 'gctl reserve create' allows scheduling specific GPU allocations for users at defined times. 'gctl reserve list' shows active reservations.

```bash
gctl set-gpus 0-3
gctl reserve create --user alice --gpus 2 --start '2026-01-28 14:00' --duration 2h
gctl reserve list --active

```

--------------------------------

### Generate shell completion scripts

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/reference/gflowd-reference.md

Generates auto-completion scripts for various shells to improve CLI usability.

```bash
gflowd completion bash
gflowd completion zsh
gflowd completion fish
```

--------------------------------

### Manage Job Dependencies with Shortcuts

Source: https://github.com/andpuqing/gflow/blob/main/docs/src/user-guide/tips.md

Use the '@' symbol to reference recent job submissions as dependencies. This allows for chaining jobs without manually tracking IDs.

```bash
gbatch --time 10 python preprocess.py
gbatch --depends-on @ --gpus 1 --time 4:00:00 python train.py
gbatch --depends-on @ --time 10 python evaluate.py
gbatch --depends-on-all @,@~1,@~2 python merge.py
```

--------------------------------

### Customize gqueue Output Format

Source: https://context7.com/andpuqing/gflow/llms.txt

The gqueue command allows customization of output fields to display specific job information. Users can select fields like JOBID, NAME, ST, TIME, NODES, NODELIST(REASON), PROJECT, TIMELIMIT, and MEMORY.

```bash
gqueue -f JOBID,NAME,ST,TIME,NODES,NODELIST(REASON)
gqueue -f JOBID,NAME,PROJECT,ST,TIMELIMIT,MEMORY,NODELIST(REASON)
```

--------------------------------

### Group gqueue Jobs by State

Source: https://context7.com/andpuqing/gflow/llms.txt

Use the gqueue -g command to group and display jobs based on their current state (e.g., running, queued, failed). This provides a consolidated view of job distribution across different statuses.

```bash
gqueue -g
```