### Bhatti CLI User Setup Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v5.md

An example of running the `bhatti setup` command, which prompts for API endpoint and key, saves the configuration, and tests the connection. This is the initial setup for a user.

```bash
$ bhatti setup
API endpoint [https://api.bhatti.sh]:
API key: ****
Saved to ~/.bhatti/config.yaml
Testing connection... ✓ connected (user: alice, 2 sandboxes)
```

--------------------------------

### Computer Tier init.sh Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

An example init.sh script for the computer tier, starting KasmVNC, dbus-daemon, pulseaudio, and XFCE.

```bash
Xkasmvnc :99 -geometry 1280x720 -websocketPort 6080 ... &
for i in $(seq 1 30); do xdpyinfo && break; sleep 0.1; done
dbus-daemon --system --fork
pulseaudio --start
startxfce4 &
echo "DISPLAY=:99" > /run/bhatti/env
```

--------------------------------

### Computer Tier init.sh Example (Simplified)

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Shell script to start KasmVNC, dbus-daemon, pulseaudio, and XFCE.

```bash
Xkasmvnc :99 -geometry 1280x720 &
dbus-daemon --system --fork
pulseaudio --start
startxfce4 &
```

--------------------------------

### Install and Start Systemd Service Directly

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-install-ux.md

This script snippet directly installs the systemd unit file and prompts the user to start the bhatti service. It handles both interactive and non-interactive (CI/piped) installations.

```bash
cp "$DATA_DIR/bhatti.service" /etc/systemd/system/bhatti.service
systemctl daemon-reload

# Only prompt in interactive mode — skip in CI/piped installs
if command -v systemctl >/dev/null 2>&1 && [ -t 0 ] && [ "${BHATTI_NO_PROMPT:-}" != "1" ]; then
    echo ""
    printf "  Start bhatti now? [Y/n]: "
    read -r start_choice < /dev/tty 2>/dev/null || start_choice="y"
    case "${start_choice:-y}" in
        n|N|no|NO) echo "  Skipped. Start later with: sudo systemctl enable --now bhatti" ;; 
        *) systemctl enable --now bhatti; success "bhatti service started" 
    esac
else
    info "Start with: sudo systemctl enable --now bhatti"
fi
```

--------------------------------

### Setup Bhatti CLI with API Key

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/WEBSITE.md

After installing the CLI, this command sets up the client by pasting the provided API key. It configures the CLI to connect to your Bhatti server or the demo endpoint.

```bash
bhatti setup   # paste the API key
```

--------------------------------

### Docker Tier init.sh Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Shell script to start the Docker daemon, poll for its socket, and set permissions.

```bash
dockerd > /var/log/dockerd.log 2>&1 &
for i in $(seq 1 100); do
    [ -S /var/run/docker.sock ] && break; sleep 0.1
done
```

--------------------------------

### Install and Enable Bhatti Systemd Service

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v3.md

Installs the Bhatti systemd service file, reloads the systemd daemon, enables the service to start on boot, and starts the service immediately. Includes a post-stop hook to clean up TUN/TAP interfaces.

```bash
echo "==> Installing systemd service..."
cat > /etc/systemd/system/bhatti.service << 'EOF'
[Unit]
Description=Bhatti Sandbox Infrastructure
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/bhatti serve
WorkingDirectory=/var/lib/bhatti
Environment=HOME=/root
Restart=always
RestartSec=5
ExecStopPost=/bin/sh -c 'ip -o link show type tun | grep tap | cut -d: -f2 | xargs -r -n1 ip link del'
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable bhatti
systemctl start bhatti
```

--------------------------------

### Example .service file structure

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-systemd-rc.md

This is an example of a systemd service file format. It shows the typical sections [Unit], [Service], and [Install] with common directives used for defining and managing services.

```ini
[Unit]
Description=OpenBSD Secure Shell server

[Service]
Type=simple
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D
RuntimeDirectory=sshd
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

--------------------------------

### Install Packages with Systemd

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Shows successful package installation using `bhatti exec` when Systemd is managing services, ensuring services start automatically.

```bash
bhatti exec dev -- sudo apt-get install -y openssh-server
# ✅ Installs, resolved manages DNS, sshd starts automatically

bhatti exec dev -- sudo apt-get install -y postgresql
# ✅ Installs, systemctl starts postgresql, pg_isready works

bhatti exec dev -- sudo apt-get install -y nginx
# ✅ Installs, systemctl starts nginx, curl localhost works

bhatti exec dev -- sudo apt-get install -y redis-server
# ✅ Installs, systemctl starts redis, redis-cli ping works
```

--------------------------------

### Run Init Script as Session

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v2.md

Initializes a new session with a given script and user. It sets up a pseudo-terminal (PTY) and a background reader to capture the script's output and manage its lifecycle. Use this to execute setup scripts when a sandbox starts.

```go
func runInitSession(script, user string) {
    sess := newSession([]string{"sh", "-c", script}, true, 0)
    master, slave := openPTY()
    sess.Master = master
    sess.ID = "init" // well-known session ID

    cmd := exec.Command("sh", "-c", script)
    cmd.Env = buildEnv(nil)
    cmd.Stdin = slave
    cmd.Stdout = slave
    cmd.Stderr = slave
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Setsid: true, Setctty: true, Ctty: 0,
        Credential: &syscall.Credential{Uid: 1000, Gid: 1000},
    }
    cmd.Dir = "/workspace"
    cmd.Start()
    slave.Close()
    sess.Cmd = cmd

    // Background reader: PTY master → scrollback
    go func() {
        buf := make([]byte, 4096)
        for {
            n, err := master.Read(buf)
            if n > 0 {
                sess.Scrollback.Write(buf[:n])
                sess.mu.Lock()
                if sess.Attached != nil {
                    proto.WriteFrame(sess.Attached, proto.STDOUT, buf[:n])
                }
                sess.mu.Unlock()
            }
            if err != nil {
                exitCode := exitCodeFromErr(cmd.Wait())
                sess.mu.Lock()
                sess.ExitCode = &exitCode
                sess.mu.Unlock()
                return
            }
        }
    }()
}
```

--------------------------------

### Bhatti Demo CLI Installation

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/WEBSITE.md

This command installs the Bhatti CLI for trying out the demo service. It's a unified installer that detects the platform.

```bash
curl -fsSL https://bhatti.sh/cli | sh
```

--------------------------------

### Minimal Tier Script: Base Ubuntu Setup

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Installs essential packages, configures locales, and creates a sudo user within the chroot environment. Cleans up apt cache afterwards.

```bash
#!/bin/bash
# Minimal tier: bare Ubuntu + lohar dependencies.
# Called by build-tier.sh with $MOUNT, $ARCH, $AGENT set.
set -euo pipefail

chroot "$MOUNT" /bin/bash -c '
set -eu
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y --no-install-recommends \
    iproute2 ca-certificates sudo curl locales

# Locale
sed -i "/en_US.UTF-8/s/^# //g" /etc/locale.gen
locale-gen

# Create lohar user (bash is the default shell — comes with minbase)
useradd -m -s /bin/bash -G sudo lohar
echo "lohar ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

apt-get clean
rm -rf /var/lib/apt/lists/*
'
```

--------------------------------

### Kernel Boot Args Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v5.md

An example of the kernel 'ip=' parameter format, showing the user's gateway IP.

```text
ip=10.0.1.2::10.0.1.1:255.255.255.0::eth0:off:1.1.1.1:8.8.8.8:
```

--------------------------------

### Install PLAN-v7 with Systemd

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Install PLAN-v7 using the install script with systemd. Use the --image flag to select the desired tier (minimal, browser, or docker).

```bash
sudo ./scripts/install.sh --systemd                    # minimal (default)
sudo ./scripts/install.sh --systemd --image browser    # browser tier
sudo ./scripts/install.sh --systemd --image docker     # docker tier
```

--------------------------------

### Display usage information with --help flag

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-install-ux.md

Provides a help message detailing available flags, environment variables, and usage examples for the installation script.

```bash
usage() {
    cat <<EOF
Usage: install.sh [flags]

Flags:
  --tier <name>       Tier for fresh install (minimal, browser, docker, computer)
  --tiers <list|all>  Additional tiers to install on update (comma-separated or "all")
  --mode <cli|server> Skip install type prompt
  --force             Skip major version upgrade confirmation
  --quiet             Suppress output (exit code only, for CI)
  --verbose           Enable debug output (set -x)
  -h, --help          Show this help

Environment variables (equivalent, for piped installs):
  BHATTI_TIER, BHATTI_TIERS, BHATTI_MODE, BHATTI_FORCE=1

Examples:
  curl -fsSL bhatti.sh/install | bash                             # CLI (auto-detected)
  curl -fsSL bhatti.sh/install | sudo bash                        # server (prompted)
  curl -fsSL bhatti.sh/install | sudo bash -s -- --tiers all      # flags via pipe
  sudo ./scripts/install.sh --tier computer                       # server, computer tier
  sudo ./scripts/install.sh --tiers all                           # update + pull all tiers
EOF
}
```

--------------------------------

### Browser Tier init.sh Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Shell script to start headless Chrome and poll for the Chrome DevTools Protocol (CDP) endpoint.

```bash
"$HEADLESS_SHELL" --remote-debugging-port=9222 &
for i in $(seq 1 50); do
    curl -sf http://127.0.0.1:9222/json/version && break; sleep 0.1
done
```

--------------------------------

### Install postgresql and check readiness

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/EXPERIMENT-systemd-boot.md

Installs the postgresql package and checks if the database is ready to accept connections.

```bash
bhatti exec "$name" -- sudo apt-get install -y postgresql 2>&1 | tail -5
bhatti exec "$name" -- sudo pg_isready 2>/dev/null || echo "pg not ready"
```

--------------------------------

### Install Debootstrap and QEMU

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Installs debootstrap for rootfs creation and optionally qemu-user-static for cross-architecture emulation on the build runner.

```bash
sudo apt-get update
sudo apt-get install -y debootstrap
${{ matrix.arch == 'arm64' && 'sudo apt-get install -y qemu-user-static' || '' }}
```

--------------------------------

### Instrumenting Sandbox Creation Phases

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-create-performance.md

Logs the start and end of various stages within the sandbox creation process, including rootfs operations, IP allocation, network setup, and Firecracker configuration.

```go
// 1. Copy rootfs
    phase("rootfs_copy_start")
    if err = copyRootfs(baseImage, rootfsPath); err != nil { ... }
    phase("rootfs_copy_done")

    // 1b. Inject lohar
    phase("lohar_inject_start")
    if err = injectLoharIntoRootfs(rootfsPath, e.cfg.DataDir); err != nil { ... }
    phase("lohar_inject_done")

    // 1c. Resize
    if spec.DiskSizeMB > 0 {
        phase("resize_start")
        exec.Command("e2fsck", "-f", "-y", rootfsPath).Run()
        phase("e2fsck_done")
        exec.Command("truncate", "-s", fmt.Sprintf("%dM", spec.DiskSizeMB), rootfsPath).Run()
        phase("truncate_done")
        exec.Command("resize2fs", rootfsPath).Run()
        phase("resize2fs_done")
    }

    // 2. Allocate CID + paths
    phase("alloc_cid")

    // 3. Network: ensure bridge, allocate IP, create TAP
    phase("network_start")
    if err = ensureUserBridge(userNet); err != nil { ... }
    phase("bridge_done")
    guestIP, err := userNet.Pool.Allocate()
    phase("ip_alloc_done")
    tapName, err = createTapDevice(id, userNet.BridgeName)
    phase("tap_done")

    // 4. Config drive
    phase("config_drive_start")
    if err = createConfigDrive(configDrivePath, SandboxConfig{...}); err != nil { ... }
    phase("config_drive_done")

    // 5. Resolve paths for jailer
    phase("resolve_paths")

    // 6. Start FC (jailed)
    phase("fc_start_begin")
    fcProc, err := e.startFC(socketPath, startFCOpts{...})
    phase("fc_start_done")  // includes jailer setup + socket ready wait

    // 7. FC API configuration
    phase("fc_api_start")
    fcPut(ctx, client, "/logger", ...)
    phase("fc_api_logger")
    fcPut(ctx, client, "/metrics", ...)
    phase("fc_api_metrics")
    fcPut(ctx, client, "/boot-source", ...)
    phase("fc_api_boot_source")
    fcPut(ctx, client, "/drives/rootfs", ...)
    phase("fc_api_drive_rootfs")
    fcPut(ctx, client, "/machine-config", ...)
    phase("fc_api_machine_config")
    fcPut(ctx, client, "/entropy", ...)
    phase("fc_api_entropy")
    fcPut(ctx, client, "/balloon", ...)
    phase("fc_api_balloon")
    fcPut(ctx, client, "/vsock", ...)
    phase("fc_api_vsock")
    fcPut(ctx, client, "/network-interfaces/eth0", ...)
    phase("fc_api_network")
    fcPut(ctx, client, "/drives/config", ...)
    phase("fc_api_drive_config")
    // + volume drives
    phase("fc_api_volumes_done")

    // 8. Boot
    phase("instance_start")
    fcPut(ctx, client, "/actions", `{"action_type":"InstanceStart"}`)
    phase("instance_started")

    // 9. Wait for agent
    phase("wait_ready_start")
    agentClient.WaitReady(ctx, 30*time.Second)
    phase("wait_ready_done")

    // 10. Store VM
    phase("store_vm")
    e.mu.Lock()
    e.vms[id] = vm
    e.mu.Unlock()
    phase("create_complete")
```

--------------------------------

### Install Script Command

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/WEBSITE.md

Command to download and execute the Bhatti installation script. This is the primary call to action for self-hosting.

```bash
curl -fsSL bhatti.sh/i | sh
```

--------------------------------

### Install Script Change: Download Pre-built vs. Build Everything

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Illustrates the shift in the install script from a comprehensive build process to downloading pre-built artifacts. This change significantly reduces installation time.

```bash
# Old (10+ min):
debootstrap ... && chroot ... && apt-get install ...
```

--------------------------------

### Interactive bhatti Setup

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/cli-reference.md

Initiates an interactive setup process for bhatti, prompting the user for the API endpoint URL and token.

```bash
bhatti setup
```

--------------------------------

### Install redis-server and check responsiveness

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/EXPERIMENT-systemd-boot.md

Installs the redis-server package and checks if the Redis service is responding to PING commands.

```bash
bhatti exec "$name" -- sudo apt-get install -y redis-server 2>&1 | tail -5
bhatti exec "$name" -- redis-cli ping 2>/dev/null || echo "redis not responding"
```

--------------------------------

### Create a bhatti Sandbox with Options

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/cli-reference.md

Demonstrates creating a bhatti sandbox with specified name, CPU, and memory limits. Also shows examples for setting environment variables and running an initialization script.

```bash
bhatti create --name dev --cpus 2 --memory 1024
bhatti create --name worker --env API_KEY=sk-abc,NODE_ENV=prod
bhatti create --name builder --init "npm install && npm run build"
```

--------------------------------

### CLI-Only Install for Remote Server

Source: https://github.com/sahil-shubham/bhatti/blob/main/README.md

Installs only the Bhatti CLI to control a remote Bhatti server. Requires setup with server URL and token.

```bash
curl -fsSL bhatti.sh/install | bash
bhatti setup --url https://your-server:8080 --token bht_...
# or:
bhatti setup    # interactive
```

--------------------------------

### Bhatti Docker Usage Examples

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Demonstrates how to create a Docker-enabled sandbox, execute commands within it, and manage Docker Compose services.

```bash
bhatti create --name ci --image docker --memory 2048
bhatti exec ci -- docker run --rm postgres:16 postgres --version
bhatti exec ci -- docker compose up -d
```

--------------------------------

### Install and Use SSHFS in Bhatti Sandbox

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-fuse-support.md

Example of installing SSHFS within a Bhatti sandbox and then using it to mount a remote directory. This demonstrates FUSE functionality.

```bash
bhatti exec dev -- sudo apt-get install -y sshfs
bhatti exec dev -- sshfs user@server:/data /mnt/remote
```

--------------------------------

### Set up Workspace Directory and Guest Agent

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN.md

Creates a workspace directory and copies the Bhatti agent executable into the sandbox, setting appropriate permissions.

```bash
# Create workspace directory
mkdir -p "$MOUNT/workspace"
chown 1000:1000 "$MOUNT/workspace"

# Install guest agent as /usr/local/bin/bhatti-agent
cp "$AGENT" "$MOUNT/usr/local/bin/bhatti-agent"
chmod 755 "$MOUNT/usr/local/bin/bhatti-agent"
```

--------------------------------

### Running Multiple Daemons with --init

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

This snippet demonstrates how to manage multiple daemon processes, including dependencies, using the `--init` flag. It includes commands to start PostgreSQL and Redis, waits for PostgreSQL to be ready, and then starts the application server.

```bash
bhatti create --name stack --keep-hot --init \
  "postgres -D /var/lib/postgresql/data & \
  redis-server --daemonize yes \
  # Wait for deps \
  until pg_isready; do sleep 0.5; done \
  cd /workspace && node server.js"

```

--------------------------------

### Verify bhatti v0.5.14 Installation

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/MIGRATION-v0.5.14-btrfs.md

Checks the installed bhatti version and verifies that all sandboxes have recovered successfully after the upgrade. Also includes a smoke test to start, execute a command in, and stop a sandbox.

```bash
# Check version
bhatti version
# Expected: bhatti v0.5.14

# Check all 10 sandboxes recovered
journalctl -u bhatti -n 50 --no-pager | grep -E 'recovered|recovery'
# Expected: "recovery complete" with count=10

# Check rory is accessible (as kowshik or admin)
bhatti list
# rory should appear with status=stopped

# Quick smoke test: start and exec on a sandbox
bhatti start sandbox-167074
bhatti exec sandbox-167074 -- echo hello
bhatti stop sandbox-167074
```

--------------------------------

### Create Sandbox with Init Script

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Examples of creating sandboxes with custom init scripts using the `bhatti create` command.

```bash
bhatti create --name api --init "cd /workspace && node server.js"
bhatti create --name agent --init "hermes gateway" --keep-hot
```

--------------------------------

### Get bhatti Version

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/cli-reference.md

Retrieves the installed bhatti CLI version and the API endpoint it is configured to use.

```bash
bhatti version
# → bhatti v0.1.0
# → api: https://api.bhatti.sh
```

--------------------------------

### Compile and Run a Simple FUSE Filesystem

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-fuse-support.md

Installs dependencies, compiles a basic FUSE 'hello world' filesystem, mounts it, reads from it, and then unmounts it.

```bash
# Install build deps (gcc + fuse3 headers)
bhatti exec test -- sudo apt-get update -qq
bhatti exec test -- sudo apt-get install -y --no-install-recommends gcc libfuse3-dev

# Write the FUSE hello world
bhatti exec test -- sh -c 'cat > /tmp/hellofs.c << "EOF"
#define FUSE_USE_VERSION 30
#include <fuse3/fuse.h>
#include <string.h>
#include <errno.h>

static const char *hello = "fuse works!\n";
static const size_t hello_len = 12;

static int fs_getattr(const char *path, struct stat *st, struct fuse_file_info *fi) {
    (void)fi;
    memset(st, 0, sizeof(*st));
    if (strcmp(path, "/") == 0)     { st->st_mode = S_IFDIR | 0755; st->st_nlink = 2; return 0; }
    if (strcmp(path, "/hello") == 0) { st->st_mode = S_IFREG | 0444; st->st_nlink = 1; st->st_size = hello_len; return 0; }
    return -ENOENT;
}
static int fs_readdir(const char *path, void *buf, fuse_fill_dir_t f, off_t o, struct fuse_file_info *fi, enum fuse_readdir_flags fl) {
    (void)o; (void)fi; (void)fl;
    f(buf, ".", NULL, 0, 0); f(buf, "..", NULL, 0, 0); f(buf, "hello", NULL, 0, 0); return 0;
}
static int fs_read(const char *path, char *buf, size_t size, off_t off, struct fuse_file_info *fi) {
    (void)fi;
    if (strcmp(path, "/hello") != 0) return -ENOENT;
    if ((size_t)off >= hello_len) return 0;
    if (off + size > hello_len) size = hello_len - off;
    memcpy(buf, hello + off, size);
    return size;
}
static struct fuse_operations ops = { .getattr = fs_getattr, .readdir = fs_readdir, .read = fs_read };
int main(int argc, char *argv[]) { return fuse_main(argc, argv, &ops, NULL); }
EOF"

# Compile, mount, verify, unmount
bhatti exec test -- gcc -Wall /tmp/hellofs.c -o /tmp/hellofs $(pkg-config --cflags --libs fuse3)
bhatti exec test -- mkdir -p /home/lohar/mnt
bhatti exec test -- /tmp/hellofs /home/lohar/mnt -f &   # foreground, backgrounded
sleep 1
bhatti exec test -- cat /home/lohar/mnt/hello
# fuse works!
bhatti exec test -- fusermount3 -u /home/lohar/mnt
```

--------------------------------

### Actionable Error Message Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-cli-ux.md

Provides specific recovery steps when a sandbox is not running, guiding the user on how to resolve the issue.

```bash
Error: sandbox "dev" is not running

  Resume it first:
    bhatti start dev
```

--------------------------------

### Filesystem Setup for btrfs Loopback

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-reliability.md

Steps to set up btrfs on a loopback image for the /var/lib/bhatti directory. This involves creating a btrfs image, stopping the bhatti service, rsyncing data, mounting the new filesystem, and updating fstab.

```bash
# Create btrfs image (pre-allocated to avoid ENOSPC)
fallocate -l 500G /var/lib/bhatti-btrfs.img   # or whatever size
mkfs.btrfs -f /var/lib/bhatti-btrfs.img

# Stop bhatti, rsync data, mount, restart
systemctl stop bhatti
mkdir -p /mnt/bhatti-new
mount -o loop,noatime,compress=zstd:1 /var/lib/bhatti-btrfs.img /mnt/bhatti-new
rsync -aHAX --sparse /var/lib/bhatti/ /mnt/bhatti-new/
umount /mnt/bhatti-new
mv /var/lib/bhatti /var/lib/bhatti-ext4-backup
mkdir -p /var/lib/bhatti
mount -o loop,noatime,compress=zstd:1 /var/lib/bhatti-btrfs.img /var/lib/bhatti
echo '/var/lib/bhatti-btrfs.img /var/lib/bhatti btrfs loop,noatime,compress=zstd:1 0 0' >> /etc/fstab
systemctl start bhatti
```

--------------------------------

### Btrfs Setup Commands

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-create-performance.md

Commands to set up a 20GB btrfs filesystem with compression for testing. Includes creating the image file, formatting it, and mounting it.

```bash
sudo fallocate -l 20G /var/lib/bhatti-btrfs.img
sudo mkfs.btrfs -f /var/lib/bhatti-btrfs.img
sudo mount -o loop,noatime,compress=zstd:1 /var/lib/bhatti-btrfs.img /var/lib/bhatti
```

--------------------------------

### Lifecycle Logging Examples

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v5.md

Provides examples of structured logging for key lifecycle events such as sandbox creation and destruction, as well as authentication failures. These logs include relevant identifiers and context.

```go
// Sandbox created
slog.Info("sandbox.created",
    "sandbox_id", sb.ID, "name", sb.Name,
    "user", user.Name, "cpus", spec.CPUs, "memory_mb", spec.MemoryMB)

// Sandbox destroyed
slog.Info("sandbox.destroyed", "sandbox_id", sb.ID, "name", sb.Name, "user", user.Name)

// Auth failure
slog.Warn("auth.failed", "ip", r.RemoteAddr, "reason", "invalid api key")
```

--------------------------------

### Configure s6-overlay Service

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Example of s6-overlay service configuration, defining service types, run commands, and dependencies. This setup is common for containerized services.

```shell
/etc/s6-overlay/s6-rc.d/
  lohar/
    type: longrun
    run: exec /usr/local/bin/lohar --agent
    dependencies.d/
      base
  dockerd/
    type: longrun
    run: exec dockerd
    dependencies.d/
      lohar
```

--------------------------------

### JSON Output Example - List Command

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/CLI_MIGRATION_PLAN.md

Demonstrates using the --json flag with the 'list' command to get raw API response data.

```bash
bhatti list              # table for humans
bhatti list --json       # JSON for agents
```

--------------------------------

### CLI Command for Creating Sandbox with Volume

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v6.md

This example shows how to create a sandbox and attach a volume to it using the bhatti CLI. It demonstrates the format for specifying the volume name, mount point, and read-only option.

```bash
bhatti create --name dev --volume workspace:/workspace
bhatti create --name dev --volume datasets:/data:ro   # read-only
```

--------------------------------

### Timeline of Create and Guest Boot Events

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-create-performance.md

Illustrates the sequence of events from the host's create start to the guest's TCP readiness and the impact of SYN retransmissions.

```text
  0ms     Host: create() starts
 14ms     Host: rootfs copy done (btrfs reflink)
106ms     Host: lohar inject done
154ms     Host: config drive done
175ms     Host: FC jailer started, socket ready
206ms     Host: InstanceStart API returns
206ms     Host: WaitReady starts → TCP SYN sent to guest IP
243ms     Guest: kernel first instruction (measured via btime)
324ms     Guest: lohar tcp_listen (kernel 81ms + lohar 4ms)
          ... SYN was already sent at 206ms, dropped (guest not listening) ...
          ... Linux default SYN retransmit timeout: 1 second ...
~1206ms   Host: kernel retransmits SYN
~1211ms   Guest: SYN-ACK → handshake completes
1229ms    Host: WaitReady done, create complete
```

--------------------------------

### Start Firecracker Process

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/SNAPSHOT-RELIABILITY-TRACE.md

Initiates a new Firecracker process. In jailed mode, this involves setup like removing old jail directories, creating new ones, hard-linking files, starting the jailer, and waiting for the API socket. Failures in hard-linking or jailer startup are handled with cleanup and error reporting.

```go
fcProc, err := e.startFC(newSocketPath, startFCOpts{...})
```

--------------------------------

### Snapshot and Restore Test Setup

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/EXPERIMENT-systemd-boot.md

Sets up a microVM for snapshot and restore testing. It creates an instance, starts a long-running process, and records initial state before stopping the VM for snapshotting.

```bash
# For each image (A and B):
name="snap-test"
bhatti create --name "$name" --cpus 1 --memory 2048 [--image systemd-minimal]

# Start a long-running process
bhatti exec "$name" -- sh -c 'while true; do date >> /tmp/ticker.log; sleep 1; done &'
bhatti exec "$name" -- sh -c 'echo "pre-snapshot" > /tmp/state.txt'

# Verify process is running
bhatti exec "$name" -- ps aux | grep ticker
bhatti exec "$name" -- wc -l /tmp/ticker.log

# Stop (snapshot to disk)
bhatti stop "$name"
sleep 5
```

--------------------------------

### Create Sandbox Directory and Setup Cleanup

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/SNAPSHOT-RELIABILITY-TRACE.md

Creates the sandbox directory and sets up a deferred function for cleanup in case of errors. Handles errors during directory creation gracefully.

```go
id := generateID()
sandboxDir := filepath.Join(e.cfg.DataDir, "sandboxes", id)
os.MkdirAll(sandboxDir, 0700)
defer func() {
    if err != nil {
        killFC, destroyTAP, releaseIP, RemoveAll(sandboxDir)
    }
}()
```

--------------------------------

### New Sidebar Structure

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-docs-rewrite.md

This is a proposed new sidebar structure for the documentation, organizing content into logical sections like Getting Started, Sandboxes, Managing, Under the Hood, Reference, and Contributing.

```text
Getting Started
  Quickstart
  Self-Hosting
  Concepts

Sandboxes
  Create & Destroy
  Run Commands
  Interactive Shell
  Web Shell
  Files
  Preview URLs

Managing
  Users & Auth
  Secrets
  Volumes
  Images & Tiers
  Templates

Updating & Uninstalling              ← new, from our install work

Under the Hood                       ← new section
  Architecture Overview
  Lohar: PID 1 Inside Every VM
  Thermal States & Snapshots
  Networking: Bridges, TAP, and ip=
  The Wire Protocol
  Design Decisions

Reference
  CLI Reference
  API Reference
  Configuration
  Wire Protocol Frames              ← the byte-level spec, split from narrative

Contributing
  Development Setup                  ← new, currently missing
  Testing
  Building the Kernel
  Adding a Tier                      ← new, from tiers.md in the repo
```

--------------------------------

### Error Handling for Executing on Stopped Sandbox

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-cli-ux.md

Checks that the CLI provides actionable error messages when attempting to execute a command on a stopped sandbox, guiding the user to start it first.

```go
func TestCLIErrorExecOnStopped(t *testing.T)
    // Create → stop → exec. Verify stderr contains:
    //   sandbox "<name>" is not running
    //   Resume it first:
    //     bhatti start <name>
```

--------------------------------

### Create a new sandbox with Bhatti

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v8.md

Use the `create` command to provision a new sandbox. Options include specifying names, resources, environment variables, init scripts, custom images, templates, and persistent volumes.

```go
var createCmd = &cobra.Command{
    Use:   "create [flags]",
    Short: "Create a new sandbox",
    Long: `Create a new sandbox VM. Each sandbox is an isolated Linux environment
with its own kernel, filesystem, and network.`,
    Example: `  # Basic sandbox
  bhatti create --name dev

  # Custom resources
  bhatti create --name ml --cpus 4 --memory 4096

  # With environment variables and init script
  bhatti create --name api --env API_KEY=sk-abc --init "npm install"

  # From a custom image
  bhatti create --name py --image python-3.12

  # From a template
  bhatti create --name exp --template ml-env

  # With a persistent volume
  bhatti create --name work --volume workspace:/workspace`,
}
```

--------------------------------

### Install Firecracker Binary

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/pi-setup.md

Downloads, extracts, and installs the Firecracker and jailer binaries to /usr/local/bin. Verifies the installation by checking the version.

```bash
VERSION=1.15.0
ARCH=aarch64

curl -fsSL \
  "https://github.com/firecracker-microvm/firecracker/releases/download/v${VERSION}/firecracker-v${VERSION}-${ARCH}.tgz" \
  | tar xz

sudo mv release-v${VERSION}-${ARCH}/firecracker-v${VERSION}-${ARCH} /usr/local/bin/firecracker
sudo mv release-v${VERSION}-${ARCH}/jailer-v${VERSION}-${ARCH} /usr/local/bin/jailer
sudo chmod +x /usr/local/bin/firecracker /usr/local/bin/jailer
rm -rf release-v${VERSION}-${ARCH}

# Verify
firecracker --version
```

--------------------------------

### CLI Usage Instructions

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v3.md

Prints the usage instructions for the Bhatti CLI, detailing available commands, their arguments, and environment variables for configuration.

```go
func printUsage() {
    fmt.Fprintf(os.Stderr, `Usage: bhatti <command> [args]

Commands:
  serve                         Start the bhatti daemon
  create [flags]                Create a new sandbox
  list                          List sandboxes
  destroy <id|name>             Destroy a sandbox
  exec <id|name> -- CMD...      Execute a command
  shell <id|name>               Open an interactive shell
  ps <id|name>                  List sessions in a sandbox
  file read|write|ls <id> PATH  File operations
  secret set|list|delete        Manage secrets

Environment:
  BHATTI_URL     API endpoint (default: http://localhost:8080)
  BHATTI_TOKEN   Auth token (default: from ~/.bhatti/config.yaml)
`)
}

```

--------------------------------

### Installing Packages Without Recommends

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/EXPERIMENT-systemd-predictions.md

This command installs a package while excluding any 'Recommended' dependencies. This can be useful to prevent the installation of systemd-resolved.

```bash
apt-get install --no-install-recommends openssh-server
```

--------------------------------

### Bhatti CLI Usage Example

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/WEBSITE.md

Demonstrates common commands for interacting with the Bhatti CLI, including creating, executing commands within, and shelling into a sandbox.

```bash
$ bhatti create --name dev --cpus 2 --memory 1024
Created sandbox "dev" (a1b2c3d4) in 3.4s

$ bhatti exec dev -- uname -a
Linux dev 6.1.155 #1 SMP aarch64 GNU/Linux

$ bhatti exec dev -- node --version
v22.16.0

$ bhatti shell dev
lohar@dev:/workspace$
```

--------------------------------

### Install Build Dependencies

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Installs essential build dependencies for kernel compilation on Ubuntu. Includes conditional installation of aarch64 cross-compiler.

```bash
sudo apt-get update
sudo apt-get install -y build-essential flex bison libelf-dev libssl-dev bc
${{ matrix.arch == 'aarch64' && 'sudo apt-get install -y gcc-aarch64-linux-gnu' || '' }}
```

--------------------------------

### Install Packages with Lohar (Current)

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-sandbox-guardrails.md

Demonstrates package installation using `bhatti exec` with Lohar as PID 1. Note potential failures when packages rely on systemd services.

```bash
bhatti exec dev -- sudo apt-get install -y python3 python3-pip
# ✅ Works (no systemd deps)

bhatti exec dev -- sudo apt-get install -y openssh-server
# ❌ Installs systemd-resolved → DNS breaks → VM bricked

bhatti exec dev -- sudo apt-get install -y postgresql
# ⚠️ Installs but postgresql doesn't start (postinst calls systemctl)
# User must manually: sudo -u postgres pg_ctlcluster 16 main start

bhatti exec dev -- sudo apt-get install -y nginx
# ⚠️ Installs but nginx doesn't start
# User must manually: sudo nginx

bhatti exec dev -- sudo apt-get install -y redis-server
# ⚠️ Installs but redis doesn't start
# User must manually: sudo redis-server --daemonize yes
```

--------------------------------

### Run Install Script with Environment Variables

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-install-ux.md

The install script only accepts BHATTI_ env vars when run directly. Use this for CI or piped installs.

```bash
sudo BHATTI_TIERS=all ./scripts/install.sh
```

--------------------------------

### Benchmark Execution and Comparison

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v4.md

Demonstrates how to run Go benchmarks and compare results using `benchstat`. This is useful for performance analysis and tracking changes over time.

```bash
go test -bench=. -benchmem -count=5 ./pkg/agent/proto/ > old.txt
# ... make changes ...
go test -bench=. -benchmem -count=5 ./pkg/agent/proto/ > new.txt
benchstat old.txt new.txt
```

--------------------------------

### Verify Firecracker Installation

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN.md

Checks the installed Firecracker version.

```bash
firecracker --version
```

--------------------------------

### Guest Boot Epoch vs Host Create Start

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/INVESTIGATION-create-performance.md

Compares the boot time of the guest kernel with the start time of the host's create process to determine the guest boot delay.

```text
Guest btime (from /proc/stat):  epoch 1777194284.441s
Host create start:              epoch 1777194284.198s
Delta: guest booted 243ms after host create started
```

--------------------------------

### CLI Install Script

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v5.md

A bash script for installing the Bhatti CLI. It automatically detects the OS and architecture, fetches the latest version from GitHub releases if not specified, and installs the binary to `/usr/local/bin`.

```bash
#!/bin/bash
set -euo pipefail
VERSION="${BHATTI_VERSION:-latest}"
REPO="sahil-shubham/bhatti"
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
case "$ARCH" in
    x86_64)        ARCH="amd64" ;; 
    aarch64|arm64) ARCH="arm64" ;; 
    *) echo "unsupported: $ARCH" >&2; exit 1 ;; 
esac
if [ "$VERSION" = "latest" ]; then
    VERSION=$(curl -fsSL "https://api.github.com/repos/$REPO/releases/latest" \
        | grep tag_name | cut -d'"' -f4)
fi
URL="https://github.com/$REPO/releases/download/${VERSION}/bhatti-${OS}-${ARCH}"
echo "Installing bhatti $VERSION ($OS/$ARCH)..."
curl -fsSL "$URL" -o /tmp/bhatti && chmod +x /tmp/bhatti
INSTALL_DIR="/usr/local/bin"
if [ -w "$INSTALL_DIR" ]; then mv /tmp/bhatti "$INSTALL_DIR/bhatti"
else sudo mv /tmp/bhatti "$INSTALL_DIR/bhatti"; fi
echo "bhatti $VERSION installed"
```

--------------------------------

### Start enabled services at boot

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-systemd-rc.md

Initiates the process of starting services that are enabled to auto-start on boot. This function scans the 'multi-user.target.wants' directory and starts each service, mimicking systemd's behavior.

```go
// Start enabled services (reads the wants directory, starts each)
startEnabledServices()

```

--------------------------------

### Minimal Tier Script: Workspace and DNS Setup

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-v7.md

Creates a workspace directory for the user and configures DNS resolution within the chroot.

```bash
# Workspace
mkdir -p "$MOUNT/workspace"
chown 1000:1000 "$MOUNT/workspace"

# DNS
cat > "$MOUNT/etc/resolv.conf" << 'EOF'
nameserver 1.1.1.1
nameserver 8.8.8.8
EOF
```

--------------------------------

### Install Jailer Binary

Source: https://github.com/sahil-shubham/bhatti/blob/main/docs/archive/PLAN-jailer.md

Installs the jailer binary to the system and makes it executable.

```bash
cp jailer-v1.14.0-x86_64 /usr/local/bin/jailer
chmod +x /usr/local/bin/jailer
```