### Run Basic Task Workflow Example

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/examples/taskworkflow.md

Execute the taskworkflow example without any networking setup. Requires super user privileges.

```bash
sudo /path/to/firecracker-containerd/examples/taskworkflow
```

--------------------------------

### Install CNI Dependencies

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Run this make target to install the required CNI plugins and dependencies for network setup.

```bash
$ sudo make demo-network
```

--------------------------------

### Build and Install Firecracker-Containerd

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Builds firecracker-containerd, its shim, and agent, then installs them to /usr/local/bin. Assumes components are installed for example configurations.

```bash
make
make install
make firecracker
make install-firecracker
```

--------------------------------

### Run Remote Snapshotter Example

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/examples/cmd/remote-snapshotter/README.md

Launch a container within a Firecracker microVM using the remote snapshotter. This command starts the example client with a specified image.

```bash
./remote-snapshotter ghcr.io/firecracker-microvm/firecracker-containerd/amazonlinux:latest-esgz
```

--------------------------------

### Remote Snapshotter Setup (stargz / lazy-loading)

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Sets up lazy image loading using a demux-snapshotter and stargz-snapshotter. This involves building a FUSE-capable kernel, configuring containerd proxy plugins, and starting host daemons.

```bash
# 1. Build a FUSE-capable kernel and stargz rootfs
make kernel install-kernel
make image-stargz install-stargz-rootfs

# 2. Add proxy plugin to /etc/firecracker-containerd/config.toml
cat >> /etc/firecracker-containerd/config.toml <<'EOF'
[proxy_plugins]
  [proxy_plugins.proxy]
    type    = "snapshot"
    address = "/var/lib/demux-snapshotter/snapshotter.sock"
EOF

# 3. Configure /etc/demux-snapshotter/config.toml
sudo tee /etc/demux-snapshotter/config.toml <<'EOF'
[snapshotter.listener]
  type    = "unix"
  address = "/var/lib/demux-snapshotter/snapshotter.sock"

[snapshotter.proxy.address.resolver]
  type    = "http"
  address = "http://127.0.0.1:10001"

[snapshotter.metrics]
  enable = false

[debug]
  logLevel = "debug"
EOF

# 4. Start the three required host daemons
sudo firecracker-containerd --config /etc/firecracker-containerd/config.toml &
sudo demux-snapshotter &
sudo http-address-resolver &

# 5. Pull a container using the proxy snapshotter (image must be in estargz format)
sudo firecracker-ctr \
  --address /run/firecracker-containerd/containerd.sock \
  images pull \
  --snapshotter proxy \
  ghcr.io/containerd/stargz-snapshotter/ubuntu:20.04-esgz
```

--------------------------------

### Run Task Workflow Example with Networking

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/examples/taskworkflow.md

Execute the taskworkflow example with networking configured. This requires a pre-configured tap device (e.g., tap0) and specifies the container's IP address and gateway. Note: This example may not work if more than one container is run simultaneously on the host.

```bash
sudo /path/to/firecracker-containerd/examples/taskworkflow -ip 172.16.0.2/24 \
    -gw 172.16.0.1
```

--------------------------------

### Build Remote Snapshotter

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/examples/cmd/remote-snapshotter/README.md

Build the remote snapshotter example client. This command is used to compile the necessary tools for the example.

```bash
make remote-snapshotter
```

--------------------------------

### Example Docker Credentials JSON

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docker-credential-mmds/README.md

Example JSON structure for 'docker-credentials' to be placed in MMDS, mapping hostnames to username and password.

```json
{
	"docker-credentials": {
		"public.ecr.aws": {
			"username": "123456789012",
			"password": "access_key"
		},
		"docker.io": {
			"username": "user",
			"password": "pass"
		}
	}
}
```

--------------------------------

### Install Dependencies on Debian

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Installs necessary build tools, Go, Docker CE, and device-mapper for Debian-based systems. Ensure you are in your home directory before running.

```bash
#!/bin/bash

cd ~

# Install git, Go 1.23, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb http://ftp.debian.org/debian bullseye-backports main" | \
  sudo tee /etc/apt/sources.list.d/bullseye-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.23 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg

# Debian's Go 1.23 package installs "go" command under /usr/lib/go-1.23/bin
export PATH=/usr/lib/go-1.23/bin:$PATH

cd ~

# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key finger docker@docker.com | grep '9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools -
sudo usermod -aG docker $(whoami)

# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup

```

--------------------------------

### Setup Demo Network

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Installs CNI configuration to create a bridge, a veth pair, and an adapter for connecting the tap to Firecracker. This is required for remote snapshotter support.

```bash
make demo-network
```

--------------------------------

### Setup Thinpool with dmsetup

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

This bash script sets up a devicemapper thin pool using loop devices. Note: This method is not recommended for production due to performance limitations.

```bash
#!/bin/bash

# Sets up a devicemapper thin pool with loop devices in
# /var/lib/firecracker-containerd/snapshotter/devmapper

set -ex

DIR=/var/lib/firecracker-containerd/snapshotter/devmapper
POOL=fc-dev-thinpool

if [[ ! -f "${DIR}/data" ]]; then
touch "${DIR}/data"
truncate -s 100G "${DIR}/data"
fi

if [[ ! -f "${DIR}/metadata" ]]; then
touch "${DIR}/metadata"
truncate -s 2G "${DIR}/metadata"
fi

DATADEV="$(losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
DATADEV="$(losetup --find --show ${DIR}/data)"
fi

METADEV="$(losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
METADEV="$(losetup --find --show ${DIR}/metadata)"
fi

SECTORSIZE=512
DATASIZE="$(blockdev --getsize64 -q ${DATADEV})"
LENGTH_SECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}")
DATA_BLOCK_SIZE=128 # see https://www.kernel.org/doc/Documentation/device-mapper/thin-provisioning.txt
LOW_WATER_MARK=32768 # picked arbitrarily
THINP_TABLE="0 ${LENGTH_SECTORS} thin-pool ${METADEV} ${DATADEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK} 1 skip_block_zeroing"
echo "${THINP_TABLE}"

if ! $(dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
dmsetup create "${POOL}" --table "${THINP_TABLE}"
fi

```

--------------------------------

### Containerd Starting Shim Binary

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/shim-design.md

This code snippet illustrates how containerd initiates the shim binary for a container task. The shim is responsible for either starting a new shim process or returning an address to an existing one.

```go
func (m *Manager) startShim(ctx context.Context, id string, opts *options.Options) (_shim.Shim, error) {
	// ... omitted for brevity ...
	shim, err := m.shimFactory(ctx, id, opts)
	if err != nil {
		return nil, err
	}

	return shim, nil
}
```

--------------------------------

### HTTP Address Resolver Agent Request Example

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/README.md

Example HTTP GET request to the address resolver agent to query the network address for a remote snapshotter.

```http
GET /address?namespace="cbfad871-0862-4dd6-ae7a-52e9b1c16ede"

```

--------------------------------

### Shim Start Routine Implementation

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/shim-design.md

This Go code defines the shim start routine for firecracker-containerd, which determines how shim processes are managed when a new task is initiated. It handles the logic for deciding whether to reuse an existing shim or start a new one.

```go
func (s *service) ShimStart(ctx context.Context, req *types.ShimStartRequest) (*types.ShimStartResponse, error) {
	// ... existing code ...

	// If the shim is already running, we should return the existing shim.
	// This is the case where we have a 1-to-1 mapping between shim and task.
	// This is also the case where we have a 1-to-many mapping between shim and task.
	// In both cases, we are returning the existing shim.
	// The only difference is that in the 1-to-1 case, the shim is guaranteed to be
	// running, whereas in the 1-to-many case, the shim might not be running yet.
	// We need to handle both cases.
	shim, err := s.getShim(ctx, req.ContainerID)
	if err != nil {
		// If the shim is not running, we need to start it.
		// This is the case where we have a 1-to-1 mapping between shim and task.
		// This is also the case where we have a 1-to-many mapping between shim and task.
		// In both cases, we are starting a new shim.
		// The only difference is that in the 1-to-1 case, the shim is guaranteed to be
		// running, whereas in the 1-to-many case, the shim might not be running yet.
		// We need to handle both cases.
		resp, err := s.shimStart(ctx, req.ContainerID, req.Runtime, req.RuntimeVersion, req.Options)
		if err != nil {
			return nil, err
		}
		return resp, nil
	}

	// If the shim is already running, we should return the existing shim.
	// This is the case where we have a 1-to-1 mapping between shim and task.
	// This is also the case where we have a 1-to-many mapping between shim and task.
	// In both cases, we are returning the existing shim.
	// The only difference is that in the 1-to-1 case, the shim is guaranteed to be
	// running, whereas in the 1-to-many case, the shim might not be running yet.
	// We need to handle both cases.
	return &types.ShimStartResponse{ID: shim.ID()}, nil
}

```

--------------------------------

### Build and Install firecracker-containerd

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Downloads, builds, and installs firecracker-containerd, its agent, runc, a root filesystem, and related CNI dependencies. Also downloads the kernel and configures directories for firecracker-containerd.

```bash
#!/bin/bash
cd ~

# Check out firecracker-containerd and build it.  This includes:
# * firecracker-containerd runtime, a containerd v2 runtime
# * firecracker-containerd agent, an inside-VM component
# * runc, to run containers inside the VM
# * a Debian-based root filesystem configured as read-only with a read-write
#   overlay
# * firecracker-containerd, an alternative containerd binary that includes the
#   firecracker VM lifecycle plugin and API
# * tc-redirect-tap and other CNI dependencies that enable VMs to start with
#   access to networks available on the host
git clone https://github.com/firecracker-microvm/firecracker-containerd.git
cd firecracker-containerd
sg docker -c 'make all image firecracker'
sudo make install install-firecracker demo-network

cd ~

# Download kernel
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin

# Configure our firecracker-containerd binary to use our new snapshotter and
# separate storage from the default containerd binary
sudo mkdir -p /etc/firecracker-containerd
sudo mkdir -p /var/lib/firecracker-containerd/containerd

```

--------------------------------

### Build Firecracker Containerd Project

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Clone the repository with submodules and build all components using 'make all'. Install binaries and verify KVM and kernel requirements.

```bash
# Clone with submodules (includes Firecracker VMM and runc)
git clone --recurse-submodules https://github.com/firecracker-microvm/firecracker-containerd
cd firecracker-containerd

# Build all binaries: runtime shim, agent, containerd binary, firecracker, runc, rootfs image
make all image firecracker

# Install binaries to /usr/local/bin
sudo make install install-firecracker demo-network

# Verify KVM and kernel requirements
[ "$(uname) $(uname -m)" = "Linux x86_64" ] || echo "ERROR: not Linux x86_64"
[ -r /dev/kvm ] && [ -w /dev/kvm ] || echo "ERROR: /dev/kvm inaccessible"
(( $(uname -r | cut -d. -f1)*1000 + $(uname -r | cut -d. -f2) >= 4014 )) || echo "ERROR: kernel too old"
```

--------------------------------

### Install Dependencies on RPM-based Systems (Amazon Linux 2)

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Installs necessary build tools, Go, Docker CE, and device-mapper for RPM-based systems like Amazon Linux 2. Ensure you are in your home directory before running.

```bash
#!/bin/bash

cd ~

# Install git, make, curl
sudo yum -y update
sudo yum -y install \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg \
  gcc


# Amazon Linux 2 packages can sometimes be dated, so let's install using
# the Go installer. The installer will handle any path changes and just
# need to source environment variables afterwards for the existing shell session.
curl -LO https://get.golang.org/$(uname)/go_installer && \
  chmod +x go_installer && \
  ./go_installer -version 1.23 && \
  rm go_installer && \
  source .bash_profile

cd ~

# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
sudo yum -y update
sudo amazon-linux-extras install -y  docker
sudo usermod -aG docker $(whoami)

sudo yum -y install device-mapper

```

--------------------------------

### Start http-address-resolver Daemon

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Launch the http-address-resolver daemon. This service maps containerd namespaces to VM vsock addresses, which is necessary for routing requests to the correct remote snapshotter.

```bash
snapshotter/http-address-resolver
```

--------------------------------

### HTTP Address Resolver Agent Response Example

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/README.md

Example JSON response from the address resolver agent, providing network details for a remote snapshotter.

```json
{
  "network": "unix",
  "address": "/var/lib/firecracker-containerd/shim-base/default#cbfad871-0862-4dd6-ae7a-52e9b1c16ede/firecracker.vsock",
  "snapshotter_port": "10000",
  "metrics_port": "10001",
  "labels": {
    "namespace": "cbfad871-0862-4dd6-ae7a-52e9b1c16ede"
  }
}

```

--------------------------------

### Start Firecracker Containerd Service

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

This command starts the Firecracker containerd service. Ensure the PATH includes the firecracker-containerd binary and specify the configuration file path.

```bash
$ sudo PATH=$PATH /usr/local/bin/firecracker-containerd \
  --config /etc/firecracker-containerd/config.toml
```

--------------------------------

### Start firecracker-containerd

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Starts the firecracker-containerd service in the foreground. Use this command in a separate terminal.

```bash
sudo firecracker-containerd --config /etc/firecracker-containerd/config.toml
```

--------------------------------

### Setup device mapper thin pool

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Configures a device mapper thin pool for block-device snapshotting. This involves creating data and metadata files, setting up loop devices, and creating the dmsetup table.

```bash
sudo mkdir -p /var/lib/firecracker-containerd/snapshotter/devmapper
cd /var/lib/firecracker-containerd/snapshotter/devmapper
DIR=/var/lib/firecracker-containerd/snapshotter/devmapper
POOL=fc-dev-thinpool

if [[ ! -f "${DIR}/data" ]]; then
    sudo touch "${DIR}/data"
    sudo truncate -s 100G "${DIR}/data"
fi

if [[ ! -f "${DIR}/metadata" ]]; then
    sudo touch "${DIR}/metadata"
    sudo truncate -s 2G "${DIR}/metadata"
fi

DATADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/data)"
if [[ -z "${DATADEV}" ]]; then
    DATADEV="$(sudo losetup --find --show ${DIR}/data)"
fi

METADEV="$(sudo losetup --output NAME --noheadings --associated ${DIR}/metadata)"
if [[ -z "${METADEV}" ]]; then
    METADEV="$(sudo losetup --find --show ${DIR}/metadata)"
fi

SECTORSIZE=512
DATASIZE="$(sudo blockdev --getsize64 -q ${DATADEV})"
LENGTH_SECTORS=$(bc <<< "${DATASIZE}/${SECTORSIZE}")
DATA_BLOCK_SIZE=128
LOW_WATER_MARK=32768
THINP_TABLE="0 ${LENGTH_SECTORS} thin-pool ${METADEV} ${DATADEV} ${DATA_BLOCK_SIZE} ${LOW_WATER_MARK} 1 skip_block_zeroing"
echo "${THINP_TABLE}"

if ! $(sudo dmsetup reload "${POOL}" --table "${THINP_TABLE}"); then
    sudo dmsetup create "${POOL}" --table "${THINP_TABLE}"
fi

cd ~
```

--------------------------------

### Start Firecracker Containerd Daemon

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Start the firecracker-containerd daemon in the foreground, pointing to the TOML configuration file. Ensure the shim base directory exists.

```bash
# Ensure the shim base directory exists
sudo mkdir -p /var/lib/firecracker-containerd

# Start the daemon (foreground)
sudo firecracker-containerd --config /etc/firecracker-containerd/config.toml

# Expected output (abbreviated):
time="..." level=info msg="starting containerd" revision="..." version="v1.7.x"
time="..." level=info msg="loading plugin" type=io.containerd.snapshotter.v1 id=devmapper
```

--------------------------------

### Build Firecracker VM Image Natively

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/tools/image-builder/README.md

Execute this command from the 'tools/image-builder' directory to build the VM image without Docker. Ensure 'agent' and 'runc' binaries are pre-built and placed in the correct locations, and that 'debootstrap' and 'mksquashfs' are installed.

```bash
make rootfs.img
```

--------------------------------

### Create demux-snapshotter Runtime Directory

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Ensure the directory for the demux-snapshotter's Unix socket exists. This is a prerequisite for the demux-snapshotter to start and function correctly.

```bash
mkdir -p /var/lib/demux-snapshotter
```

--------------------------------

### Configure firecracker-containerd Binary

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Use this TOML configuration for the firecracker-containerd binary. Ensure the snapshotter configuration matches your setup.

```toml
version = 2
disabled_plugins = ["io.containerd.grpc.v1.cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"

[debug]
  level = "debug"

```

--------------------------------

### Build Firecracker VM Image with Docker

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/tools/image-builder/README.md

Run this command from the root of the firecracker-containerd package to build the VM image using Docker. This requires Docker to be installed and runnable without sudo.

```bash
make image
```

--------------------------------

### Complete End-to-End Workflow with Firecracker Containerd Go API

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

This Go program demonstrates the full lifecycle of creating a Firecracker VM, pulling an Nginx image, creating and starting a container within the VM, waiting for its execution, and then cleaning up all resources. It requires connecting to both containerd and the firecracker-control service.

```go
package main

import (
    "context"
    "fmt"
    "log"
    "syscall"
    "time"

    "github.com/containerd/containerd"
    "github.com/containerd/containerd/cio"
    "github.com/containerd/containerd/namespaces"
    "github.com/containerd/containerd/oci"

    fcclient "github.com/firecracker-microvm/firecracker-containerd/firecracker-control/client"
    "github.com/firecracker-microvm/firecracker-containerd/proto"
    "github.com/firecracker-microvm/firecracker-containerd/runtime/firecrackeroci"
)

const (
    containerdSock = "/run/firecracker-containerd/containerd.sock"
    ttrpcSock      = containerdSock + ".ttrpc"
)

func run() error {
    // 1. Connect to containerd and the fc-control service
    cdClient, err := containerd.New(containerdSock)
    if err != nil {
        return fmt.Errorf("containerd client: %w", err)
    }
    defer cdClient.Close()

    fcClient, err := fcclient.New(ttrpcSock)
    if err != nil {
        return fmt.Errorf("fc client: %w", err)
    }
    defer fcClient.Close()

    ctx := namespaces.WithNamespace(context.Background(), "fc-example")

    // 2. Pull image
    image, err := cdClient.Pull(ctx, "docker.io/library/nginx:alpine",
        containerd.WithPullUnpack,
        containerd.WithPullSnapshotter("devmapper"),
    )
    if err != nil {
        return fmt.Errorf("pull image: %w", err)
    }
    log.Printf("Pulled: %s", image.Name())

    // 3. Create VM
    vmID := "demo-vm"
    if _, err := fcClient.CreateVM(ctx, &proto.CreateVMRequest{ 
        VMID:       vmID,
        MachineCfg: &proto.FirecrackerMachineConfiguration{VcpuCount: 2, MemSizeMib: 2048},
        ExitAfterAllTasksDeleted: true,
    }); err != nil {
        return fmt.Errorf("CreateVM: %w", err)
    }
    defer fcClient.StopVM(ctx, &proto.StopVMRequest{VMID: vmID, TimeoutSeconds: 10})
    log.Printf("VM %s created", vmID)

    // 4. Create container pinned to the VM
    ctr, err := cdClient.NewContainer(ctx, "demo-nginx",
        containerd.WithSnapshotter("devmapper"),
        containerd.WithNewSnapshot("demo-nginx-snap", image),
        containerd.WithNewSpec(
            oci.WithImageConfig(image),
            firecrackeroci.WithVMID(vmID),
            firecrackeroci.WithVMNetwork,
        ),
        containerd.WithRuntime("aws.firecracker", nil),
    )
    if err != nil {
        return fmt.Errorf("NewContainer: %w", err)
    }
    defer ctr.Delete(ctx, containerd.WithSnapshotCleanup)

    // 5. Create and start task
    task, err := ctr.NewTask(ctx, cio.NewCreator(cio.WithStdio))
    if err != nil {
        return fmt.Errorf("NewTask: %w", err)
    }
    defer task.Delete(ctx)

    exitC, err := task.Wait(ctx)
    if err != nil {
        return fmt.Errorf("task.Wait: %w", err)
    }

    if err := task.Start(ctx); err != nil {
        return fmt.Errorf("task.Start: %w", err)
    }
    log.Printf("Task %s started (PID %d)", task.ID(), task.Pid())

    // 6. Let it run, then terminate
    time.Sleep(5 * time.Second)
    if err := task.Kill(ctx, syscall.SIGTERM); err != nil {
        return fmt.Errorf("task.Kill: %w", err)
    }

    status := <-exitC
    code, _, err := status.Result()
    if err != nil {
        return fmt.Errorf("exit status: %w", err)
    }
    log.Printf("Task exited with code %d", code)
    return nil
}

func main() {
    if err := run(); err != nil {
        log.Fatal(err)
    }
}

```

--------------------------------

### Configure Kernel Arguments for Firecracker VM

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/tools/image-builder/README.md

A complete set of kernel arguments for starting the agent at VM startup and using the root filesystem as a reusable 'lower layer' for an overlay-based filesystem. These can be set via the 'kernel_args' setting in '/etc/containerd/firecracker-runtime.json'.

```bash
ro console=ttyS0 noapic reboot=k panic=1 pci=off nomodules systemd.journald.forward_to_console systemd.unit=firecracker.target init=/sbin/overlay-init
```

--------------------------------

### Configure VM-Local Rootfs with firecrackeroci.WithVMLocalImageConfig

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Use `WithVMLocalImageConfig` when the container rootfs is inside the VM, for example, with remote snapshotters. This method configures environment variables, command, and working directory from the image manifest without accessing the host's rootfs.

```go
image, _ := client.Pull(ctx, "docker.io/library/redis:alpine",
    containerd.WithPullSnapshotter("proxy"))   // proxy = demux-snapshotter

container, err := client.NewContainer(
    ctx,
    "my-redis",
    containerd.WithSnapshotter("proxy"),
    containerd.WithNewSnapshot("my-redis-snap", image),
    containerd.WithNewSpec(
        // WithVMLocalImageConfig avoids accessing the rootfs on the host;
        // the agent inside the VM resolves UID/GID mappings and inspects rootfs.
        firecrackeroci.WithVMLocalImageConfig(image),
        firecrackeroci.WithVMID("my-vm-1"),
    ),
    containerd.WithRuntime("aws.firecracker", nil),
)
if err != nil {
    log.Fatalf("NewContainer: %v", err)
}
```

--------------------------------

### Configure User and Root Limits in limits.conf

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/scaling.md

Set 'nofile', 'nproc', and 'stack' limits for both normal users and root in /etc/security/limits.conf to accommodate a large number of microVMs. Ensure these values are sufficiently high for your scaling needs.

```bash
* soft nofile 1000000
* hard nofile 1000000
root soft nofile 1000000
root hard nofile 1000000
* soft nproc 4000000
* hard nproc 4000000
root soft nproc 4000000
root hard nproc 4000000
* soft stack 65536
* hard stack 65536
root soft stack 65536
root hard stack 65536
```

--------------------------------

### Build Snapshotter and Resolver

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/README.md

Build the demux-snapshotter and http-address-resolver binaries using make.

```bash
make

```

--------------------------------

### Build docker-credential-mmds

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docker-credential-mmds/README.md

Build the docker-credential-mmds binary using make.

```bash
make
```

--------------------------------

### Build Components with GO111MODULE=on

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Builds the components when the repository is cloned inside GOPATH and using Go version 1.11.x. Explicitly sets GO111MODULE environment variable.

```bash
GO111MODULE=on make all
```

--------------------------------

### Build Firecracker Binary

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Builds the Firecracker binary from the cloned repository. Ensure you have the necessary build tools and dependencies installed.

```bash
make firecracker
```

--------------------------------

### Download Firecracker Kernel

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Downloads the necessary kernel binary for Firecracker using curl. Ensure you have curl installed and a stable internet connection.

```bash
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/x86_64/kernels/vmlinux.bin
```

--------------------------------

### List Block Devices with lsblk

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/design-approaches.md

This command lists block devices within a VM. It's used in the Agent's initialization phase to identify and match block devices with drive IDs based on their MAJOR:MINOR numbers.

```bash
lsblk
```

--------------------------------

### Build Root Filesystem Image

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/getting-started.md

Builds a Debian-based root filesystem image using the included image builder. This process requires Docker and the user to be in the 'docker' group. The image is then copied to the runtime directory.

```bash
make image
sudo mkdir -p /var/lib/firecracker-containerd/runtime
sudo cp tools/image-builder/rootfs.img /var/lib/firecracker-containerd/runtime/default-rootfs.img
```

--------------------------------

### Start demux-snapshotter Daemon

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Execute the demux-snapshotter binary. This daemon is responsible for multiplexing snapshotter requests from firecracker-containerd to the appropriate remote snapshotter within a VM.

```bash
snapshotter/demux-snapshotter
```

--------------------------------

### Start Firecracker-Containerd Daemon

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Run the firecracker-containerd daemon with its configuration file. This is one of the three essential host-side daemons required for remote snapshotter functionality.

```bash
firecracker-containerd --config /etc/firecracker-containerd/config.toml
```

--------------------------------

### firecrackeroci.WithVMID

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Annotate a container's OCI spec with the `aws.firecracker.vm.id` annotation so the runtime shim routes the container into a specific, pre-created microVM instead of starting a new one.

```APIDOC
## firecrackeroci.WithVMID — Assign a Container to a Pre-created VM

### Description
Annotate a container's OCI spec with the `aws.firecracker.vm.id` annotation so the runtime shim routes the container into a specific, pre-created microVM instead of starting a new one.

### Method Signature
```go
firecrackeroci.WithVMID(vmID string)
```

### Parameters
- **vmID** (string) - Required - The ID of the pre-created VM to assign the container to.

### Usage Example
```go
import (
    "github.com/containerd/containerd"
    "github.com/containerd/containerd/oci"
    "github.com/containerd/containerd/namespaces"
    "github.com/firecracker-microvm/firecracker-containerd/runtime/firecrackeroci"
)

client, _ := containerd.New("/run/firecracker-containerd/containerd.sock")
ctx := namespaces.WithNamespace(context.Background(), "default")

image, _ := client.Pull(ctx, "docker.io/library/nginx:alpine",
    containerd.WithPullSnapshotter("devmapper"))

container, err := client.NewContainer(
    ctx,
    "my-nginx",
    containerd.WithSnapshotter("devmapper"),
    containerd.WithNewSnapshot("my-nginx-snap", image),
    containerd.WithNewSpec(
        oci.WithImageConfig(image),
        firecrackeroci.WithVMID("my-vm-1"),     // pin to existing VM
        firecrackeroci.WithVMNetwork,            // give the container host-network inside the VM
    ),
    containerd.WithRuntime("aws.firecracker", nil),
)
if err != nil {
    log.Fatalf("NewContainer: %v", err)
}
defer container.Delete(ctx, containerd.WithSnapshotCleanup)
log.Printf("Container %s created inside VM my-vm-1", container.ID())
```
```

--------------------------------

### Run Integration Tests

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/README.md

Execute integration tests for the snapshotter package.

```bash
make integ-test

```

--------------------------------

### Run Unit Tests

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/snapshotter/README.md

Execute unit tests for the snapshotter package.

```bash
make test

```

--------------------------------

### Configure demux-snapshotter Listener and Resolver

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md

Configure the demux-snapshotter to listen on a Unix socket and resolve remote snapshotter addresses via HTTP. This setup is crucial for routing snapshotter requests to the correct VM.

```toml
[snapshotter.listener]
  type = "unix"
  address = "/var/lib/demux-snapshotter/snapshotter.sock"

[snapshotter.proxy.address.resolver]
  type = "http"
  address = "http://127.0.0.1:10001"

[snapshotter.metrics]
  enable = false

[debug]
  logLevel = "debug"
```

--------------------------------

### Configure firecracker-containerd

Source: https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/quickstart.md

Sets up the base directory and configuration file for firecracker-containerd. Ensure the pool_name matches the device mapper pool name.

```bash
sudo mkdir -p /var/lib/firecracker-containerd
sudo tee /etc/firecracker-containerd/config.toml <<EOF
version = 2
disabled_plugins = ["io.containerd.grpc.v1.cri"]
root = "/var/lib/firecracker-containerd/containerd"
state = "/run/firecracker-containerd"
[grpc]
  address = "/run/firecracker-containerd/containerd.sock"
[plugins]
  [plugins."io.containerd.snapshotter.v1.devmapper"]
    pool_name = "fc-dev-thinpool"
    base_image_size = "10GB"
    root_path = "/var/lib/firecracker-containerd/snapshotter/devmapper"

[debug]
  level = "debug"
EOF
```

--------------------------------

### Get Container Rootfs Path in VM

Source: https://context7.com/firecracker-microvm/firecracker-containerd/llms.txt

Computes the canonical path to a container's rootfs directory inside the microVM given a task ID. This helper is used internally by the agent and can be utilized for custom tooling.

```go
import (
    "fmt"
    client "github.com/firecracker-microvm/firecracker-containerd/client"
)

// Given a task ID, compute the path to the container's rootfs inside the VM.
taskID := "my-task-abc123"
path := client.RootFSPathInVM(taskID)
fmt.Println(path)
// Output: /var/lib/firecracker-containerd/bundle/my-task-abc123/rootfs
```