### Full Infrastructure Setup with Hetzner Cloud

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

End-to-end workflow for setting up the entire infrastructure from scratch. This includes creating a Hetzner Cloud project, provisioning a runner-manager server, configuring firewall rules, and installing GitLab Runner with the plugin.

```sh
# 1. Create a dedicated Hetzner Cloud project context
hcloud context create gitlab-ci
# Token: <paste API token>
# Context gitlab-ci created and activated

# 2. Upload your SSH public key
hcloud ssh-key create --name dev --public-key-from-file ~/.ssh/id_ed25519.pub
# SSH key 22155019 created

# 3. Create the runner-manager server
hcloud server create \
  --name runner-manager \
  --image debian-12 \
  --type cpx22 \
  --location hel1 \
  --ssh-key dev \
  --label runner=
# Server 55574479 created  IPv4: 203.0.113.1

# 4. Create and apply a firewall (ICMP + SSH only)
hcloud firewall create --name runner --rules-file <(echo '[ 
  {"description":"allow icmp","direction":"in","source_ips":["0.0.0.0/0","::/0"],"protocol":"icmp"},
  {"description":"allow ssh","direction":"in","source_ips":["0.0.0.0/0","::/0"],"protocol":"tcp","port":"22"}
]')
hcloud firewall apply-to-resource runner --type label_selector --label-selector runner
```

--------------------------------

### Install and Start Fleeting Plugin for GitLab Runner

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Installs the Fleeting plugin for GitLab Runner and restarts the gitlab-runner service. Ensure the /etc/gitlab-runner/config.toml file is correctly configured before running these commands.

```bash
gitlab-runner fleeting install
systemctl restart gitlab-runner
```

--------------------------------

### Install Fleeting Plugin Hetzner

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Installs the plugin binary using the gitlab-runner command. Ensure the runner is restarted after installation to activate the plugin.

```sh
# Install the plugin binary declared in config.toml
gitlab-runner fleeting install

# Expected output:
# runner: 11Qjxy-Gi, plugin: hetznercloud/fleeting-plugin-hetzner:latest,
# path: /root/.config/fleeting/plugins/registry.gitlab.com/hetznercloud/
#        fleeting-plugin-hetzner/0.6.0/plugin

# Start / restart the runner to activate the plugin
systemctl restart gitlab-runner

# Verify the runner and plugin are running
systemctl status --output=cat --no-pager gitlab-runner
# Expected log lines:
# level=info msg="plugin initialized" version=v0.6.0
# level=info msg="increasing instances" amount=2
# level=info msg="instance is ready" took=1m5s
```

--------------------------------

### Start Development Environment

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Bring up the development environment using Docker Compose. This will create Hetzner Cloud servers and incur costs. Ensure HCLOUD_TOKEN and RUNNER_TOKEN are configured.

```bash
make -C dev up
```

--------------------------------

### Install GitLab Runner

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Execute these commands to install GitLab Runner on your system. Refer to the official GitLab Runner installation documentation for more details.

```bash
sudo apt-get update
sudo apt-get install -y gitlab-runner
```

--------------------------------

### Install gitlab-runner Package

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Install the gitlab-runner package using apt after adding the repository. This command installs the runner and its necessary dependencies like git.

```sh
apt install gitlab-runner
```

--------------------------------

### Install GitLab Runner on Hetzner Cloud

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Installs gitlab-runner on a Hetzner Cloud server. This involves SSHing into the runner-manager, downloading the GitLab Runner repository script, and installing the gitlab-runner package.

```bash
hcloud server ssh runner-manager
curl -sSL "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
apt install -y gitlab-runner
```

--------------------------------

### Install Fleeting Plugin for GitLab Runner

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/quickstart.md

This command installs the fleeting plugin required for GitLab runner fleet scaling. Refer to the GitLab documentation for detailed installation instructions.

```sh
gitlab-runner fleeting install
```

--------------------------------

### Minimal config.toml Setup for Hetzner Autoscaler

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Configures GitLab Runner with the docker-autoscaler executor and the Hetzner plugin. The `instance_ready_command` ensures cloud-init completes before jobs are scheduled.

```toml
concurrent = 20  # max_instances * capacity_per_instance

[[runners]]
name    = "hetzner-docker-autoscaler"
url     = "https://gitlab.com"
token   = "$RUNNER_TOKEN"
executor = "docker-autoscaler"

[runners.docker]
image = "busybox:latest"

[runners.autoscaler]
plugin = "hetznercloud/fleeting-plugin-hetzner:latest"

update_interval              = "1m"
update_interval_when_expecting = "5s"
capacity_per_instance        = 4
max_instances                = 5
max_use_count                = 0

# cloud-init exit code 2 = success with recoverable warnings
instance_ready_command = "cloud-init status --wait || test $? -eq 2"

[runners.autoscaler.plugin_config]
name          = "hetzner-docker-autoscaler"
token         = "$HCLOUD_TOKEN"
location      = "hel1"
server_type   = "cpx42"
image         = "debian-12"
private_networks = []

user_data = """#cloud-config
package_update: true
package_upgrade: true

apt:
  sources:
    docker.list:
      source: deb [signed-by=$KEY_FILE] https://download.docker.com/linux/debian $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - ca-certificates
  - docker-ce

swap:
  filename: /var/swap.bin
  size: auto
  maxsize: 4294967296  # 4 GB
"""

[runners.autoscaler.connector_config]
# Required when private_networks is empty — connect via public IP
use_external_addr = true

```

--------------------------------

### Add GitLab Runner APT Repository

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Add the GitLab Runner apt package repository to your system before installing the package. This ensures you get the latest stable version.

```sh
curl -sSL "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
```

--------------------------------

### Verify GitLab Runner Fleeting Plugin Installation

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

This output confirms that the fleeting plugin has been initialized and is ready to be used by GitLab Runner.

```text
Runtime platform                                    arch=amd64 os=linux pid=2524 revision=12030cf4 version=17.5.3
runner: 11Qjxy-Gi, plugin: hetznercloud/fleeting-plugin-hetzner:latest, path: /root/.config/fleeting/plugins/registry.gitlab.com/hetznercloud/fleeting-plugin-hetzner/0.6.0/plugin
```

--------------------------------

### Volume Attachment Configuration

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Attach an additional Hetzner Volume for extra storage. The minimum volume size is 10 GB. Use cloud-init `bootcmd` to format and mount the volume before Docker starts. Set `volume_size = 0` to disable attachment.

```toml
[runners.autoscaler.plugin_config]
name        = "runner-with-volume"
token       = "$HCLOUD_TOKEN"
location    = "fsn1"
server_type = "cpx42"
image       = "debian-12"
volume_size = 200   # GB — attached as /dev/disk/by-id/scsi-SHC_Volume_*

user_data = "#cloud-config
package_update: true
package_upgrade: true

apt:
  sources:
    docker.list:
      source: deb https://download.docker.com/linux/debian $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - docker-ce

bootcmd:
  # Format and mount the attached Volume before Docker starts
  - mkfs.ext4 -F -m 0 /dev/disk/by-id/scsi-SHC_Volume_*
  - mount /dev/disk/by-id/scsi-SHC_Volume_* /mnt
"
```

--------------------------------

### Restart GitLab Runner Service

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Restart the `gitlab-runner` service to apply the new configuration and start using the autoscaler plugin.

```sh
systemctl restart gitlab-runner
```

--------------------------------

### Prometheus Alert Rule for GitLab Runner Worker Processing Errors

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

This alert rule triggers if worker processing errors, such as failures in starting the fleeting plugin, occur more than zero times over a 5-minute window. It is useful for diagnosing issues with job execution.

```yaml
groups:
  - name: GitLab Runner
    rules:
      - alert: GitLabRunnerWorkerProcessingErrors
        annotations:
          summary: GitLab Runner worker processing error rate is more than 0 for the past 5m.
        expr: |
          increase(
            gitlab_runner_worker_processing_failures_total{job="gitlab-runner", failure_type="other"}[1m]
          ) > 0
        for: 5m
```

--------------------------------

### Format and Mount Volume with Cloud-Init

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/volumes.md

Use `bootcmd` in your `cloud-init` configuration to format the attached volume with ext4 and mount it to `/mnt`. This assumes a single volume is attached.

```diff
 // ...
 [runners.autoscaler.plugin_config]
 name = "runner-docker-autoscaler0"
 token = "<your-hetzner-cloud-token>"

 location = "fsn1"
 server_type = "cpx42"
 image = "debian-12"
 volume_size = 200

 user_data = """#cloud-config
 package_update: true
 package_upgrade: true

 apt:
   sources:
     docker.list:
       source: deb https://download.docker.com/linux/debian $RELEASE stable
       keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

 packages:
   - docker-ce

+bootcmd:
+  - mkfs.ext4 -F -m 0 /dev/disk/by-id/scsi-SHC_Volume_*
+  - mount /dev/disk/by-id/scsi-SHC_Volume_* /mnt
 "" " 
```

--------------------------------

### Build the Plugin Binary

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Compile the Fleeting Plugin Hetzner into an executable binary. Ensure your Go version is up-to-date.

```bash
make build
```

--------------------------------

### Build Plugin Binary (Specific Directory)

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Build the plugin binary from the cmd/fleeting-plugin-hetzner directory. This is useful for testing the plugin directly.

```bash
cd cmd/fleeting-plugin-hetzner
go build
```

--------------------------------

### Instance Ready Command for Cloud Init

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/reference/configuration.md

Use this command with Cloud Init to ensure instances are ready before scheduling jobs. It checks the cloud-init status and exits with code 2 if it fails.

```bash
cloud-init status --wait || test $? -eq 2
```

--------------------------------

### Build Fleeting Plugin Hetzner from Source

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Build the plugin binary locally for development or testing. Go 1.25+ is required. The resulting binary can be referenced directly in `config.toml`.

```sh
# Build the plugin
make build

# Run unit tests
make test

# Run integration tests (requires a real Hetzner Cloud token)
export HCLOUD_TOKEN="your-hcloud-token"
make test  # integration tests are automatically included when HCLOUD_TOKEN is set

# Build manually for local runner testing
cd cmd/fleeting-plugin-hetzner
go build

# Point config.toml at the local binary:
#   plugin = "/home/user/fleeting-plugin-hetzner/cmd/fleeting-plugin-hetzner/fleeting-plugin-hetzner"

# Start the full dev environment (creates real Hetzner servers — incurs costs)
export HCLOUD_TOKEN="your-hcloud-token"
export RUNNER_TOKEN="your-runner-token"
make -C dev up

# Check dev environment logs
docker compose logs

# Tear down dev environment and all Hetzner resources
make -C dev down
```

--------------------------------

### Verify Hetzner Resources

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Lists all resources in Hetzner Cloud, including the runner-manager and dynamically created runner instances. This command helps verify that the plugin is provisioning resources as expected.

```bash
hcloud all list
```

--------------------------------

### Run All Pre-commit Hooks

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Execute all pre-commit hooks to ensure code quality and consistency before committing.

```bash
pre-commit run --all
```

--------------------------------

### Run All Tests

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Execute all unit and integration tests for the plugin. Ensure the HCLOUD_TOKEN environment variable is set for integration tests.

```bash
make test
```

--------------------------------

### Hetzner Cloud Plugin Configuration Parameters

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Defines the essential parameters for creating Hetzner Cloud servers via the plugin. Required fields include name, token, location, server_type, and image.

```toml
[runners.autoscaler.plugin_config]
# --- Required ---
name        = "my-runner-group"   # Prefix for created server names
token       = "$HCLOUD_TOKEN"     # Hetzner Cloud API token (or HCLOUD_TOKEN env var)
location    = "fsn1"              # hcloud location list
server_type = "cpx42"            # Single type; or list for fallback:
```

--------------------------------

### Create and Apply Firewall Rules

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Create a Hetzner Cloud Firewall to enhance security by allowing only ICMP and SSH access. Apply this firewall to servers matching the 'runner' label selector.

```sh
hcloud firewall create --name runner --rules-file <(echo '[{
  "description": "allow icmp from everywhere",
  "direction": "in",
  "source_ips": ["0.0.0.0/0", "::/0"],
  "protocol": "icmp"
},
{
  "description": "allow ssh from everywhere",
  "direction": "in",
  "source_ips": ["0.0.0.0/0", "::/0"],
  "protocol": "tcp",
  "port": "22"
}]')
```

```sh
hcloud firewall apply-to-resource runner --type label_selector --label-selector runner
```

--------------------------------

### Create Runner Manager Instance

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Create a Hetzner Cloud server instance to act as the GitLab Runner Manager. This instance will handle scaling, job execution, and log forwarding for your CI jobs.

```sh
hcloud server create --name runner-manager --image debian-12 --type cpx22 --location hel1 --ssh-key dev --label runner=
```

--------------------------------

### Connect to Runner Manager Instance

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Use this command to SSH into the runner-manager server you created. Ensure you have the necessary SSH keys configured.

```bash
hcloud server ssh runner-manager
```

--------------------------------

### Runner Manager Server Information

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

This output displays system information and the login prompt for the runner-manager server. It confirms successful connection and provides the root prompt.

```text
Linux runner-manager 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64

The programs included with the Debian GNU/Linux system are free software; 
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@runner-manager:~#
```

--------------------------------

### Check Development Environment Logs

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

View the logs for the development environment to check its health and status.

```bash
docker compose logs
```

--------------------------------

### Autoscaling Policies Configuration

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Define time-based idle capacity rules using cron expressions. Multiple policies are evaluated in order, with the last matching policy taking precedence. Use `idle_count = 0` as a default and add specific policies for business hours to maintain warm instances.

```toml
# Default: no idle instances outside business hours
[[runners.autoscaler.policy]]
periods   = ["* * * * *"]        # every minute — always matches
timezone  = "Europe/Berlin"
idle_count = 0
idle_time  = "50m"

# Business hours Mon–Fri 07:00–19:00: keep 8 idle job slots warm
[[runners.autoscaler.policy]]
periods   = ["* 7-19 * * 1-5"]
timezone  = "Europe/Berlin"
idle_count = 8                   # job slots, not instance count
idle_time  = "50m"
```

--------------------------------

### Public IP Pool Configuration

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Use pre-allocated Hetzner Primary IPs for new instances, ensuring predictable public addresses. Enable `public_ip_pool_enabled = true` and filter IPs using `public_ip_pool_selector`.

```toml
[runners.autoscaler.plugin_config]
name                   = "runner-static-ips"
token                  = "$HCLOUD_TOKEN"
location               = "fsn1"
server_type            = "cpx42"
image                  = "debian-12"
public_ip_pool_enabled  = true
public_ip_pool_selector = "pool=ci-runners"  # Only use Primary IPs with this label

[runners.autoscaler.connector_config]
use_external_addr = true

# Pre-create the Primary IPs in your Hetzner project and label them:
#   hcloud primary-ip create --name ci-ip-1 --type ipv4 --datacenter fsn1-dc14
#   hcloud primary-ip add-label ci-ip-1 pool=ci-runners
```

--------------------------------

### Configure GitLab Runner with Hetzner Autoscaler

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Replace the content of `/etc/gitlab-runner/config.toml` with this configuration to enable the Hetzner Cloud autoscaler plugin. Ensure you update the runner URL, runner token, and Hetzner Cloud token.

```toml
concurrent = 10

log_level = "info"
log_format = "text"

[[runners]]
name = "hetzner-docker-autoscaler"
url = "https://gitlab.com" # TODO: Change me with the GitLab instance URL for the runner
token = "$RUNNER_TOKEN" # TODO: Change me with the runner authentication token

executor = "docker-autoscaler"

[runners.docker]
image = "alpine:latest"

[runners.autoscaler]
plugin = "hetznercloud/fleeting-plugin-hetzner:latest"

capacity_per_instance = 4
max_instances = 5
max_use_count = 0

instance_ready_command = "cloud-init status --wait || test $? -eq 2"

[runners.autoscaler.plugin_config]
name = "runner-docker-autoscaler"
token = "$HCLOUD_TOKEN" # TODO: Change me with the Hetzner Cloud authentication token

location = "hel1"
server_type = "cpx22"
image = "debian-12"

user_data = """#cloud-config
package_update: true
package_upgrade: true

apt:
  sources:
    docker.list:
      source: deb [signed-by=$KEY_FILE] https://download.docker.com/linux/debian $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - ca-certificates
  - docker-ce

swap:
  filename: /var/swap.bin
  size: auto
  maxsize: 4294967296 # 4GB
"""

[runners.autoscaler.connector_config]
use_external_addr = true

[[runners.autoscaler.policy]]
periods = ["* * * * *"]
timezone = "Europe/Berlin" # TODO: Change me with your timezone
idle_count = 8
idle_time = "1h"

```

--------------------------------

### GitLab Runner Configuration for Hetzner Autoscaler

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/quickstart.md

This TOML configuration sets up a GitLab runner to use the Hetzner Cloud autoscaler plugin. Ensure you replace placeholder tokens and adjust settings like location and server type for your environment.

```toml
concurrent = 20 # max_instances * capacity_per_instance

check_interval = 0

[[runners]]
name = "hetzner-docker-autoscaler"
url = "https://gitlab.com" # TODO: Change me with the GitLab instance URL for the runner
token = "$RUNNER_TOKEN" # TODO: Change me with the runner authentication token

executor = "docker-autoscaler"

[runners.docker]
image = "busybox:latest"

[runners.autoscaler]
plugin = "hetznercloud/fleeting-plugin-hetzner:latest"

update_interval = "1m"
update_interval_when_expecting = "5s"

capacity_per_instance = 4
max_instances = 5
max_use_count = 0

# cloud-init>=23.4 returns an exit code 2 when the setup succeeded but some recoverable errors occurred.
# See https://cloudinit.readthedocs.io/en/latest/explanation/return_codes.html
instance_ready_command = "cloud-init status --wait || test $? -eq 2"

[runners.autoscaler.plugin_config]
name = "hetzner-docker-autoscaler"
token = "$HCLOUD_TOKEN" # TODO: Change me with the Hetzner Cloud authentication token

location = "hel1"
server_type = "cpx42"
image = "debian-12"
private_networks = []

user_data = """#cloud-config
package_update: true
package_upgrade: true

apt:
  sources:
    docker.list:
      source: deb [signed-by=$KEY_FILE] https://download.docker.com/linux/debian $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

packages:
  - ca-certificates
  - docker-ce

swap:
  filename: /var/swap.bin
  size: auto
  maxsize: 4294967296 # 4GB
"""

[runners.autoscaler.connector_config]
# without private network, the instances are only reachable through their public addresses.
use_external_addr = true

[[runners.autoscaler.policy]]
periods = ["* * * * *"]
timezone = "Europe/Berlin"
idle_count = 0
idle_time = "50m"

[[runners.autoscaler.policy]]
periods = ["* 7-19 * * 1-5"]
timezone = "Europe/Berlin"
# idle_count refers to the number of jobs, not the number of instances.
idle_count = 8
idle_time = "50m"

```

--------------------------------

### Monitor GitLab Runner Logs

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Follows the logs for the gitlab-runner service to monitor instance creation and readiness. Useful for debugging and observing autoscaling behavior.

```bash
journalctl --output=cat -f -u gitlab-runner
```

--------------------------------

### Configure gitlab-runner for S3 Shared Cache

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/shared-cache.md

Configure the gitlab-runner TOML file to enable and set up the S3 shared cache. Ensure you replace placeholder values with your actual S3 credentials and bucket information.

```toml
[runners.cache]
Type = "s3"
Shared = true

[runners.cache.s3]
ServerAddress = "fsn1.your-objectstorage.com"
BucketName = "gitlab-ci-cache-7d2f6722"
AccessKey = "LR3IWEWNCFHV46133962"
SecretKey = "URyhbHBO6bMAoNmRJTf0TmTeXCngCMBGnjbvIp9g"

```

--------------------------------

### Create Hetzner Cloud Context

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Create a new context for the `hcloud` CLI, named after your Hetzner Cloud project. This context will be used by the fleeting plugin to communicate with the Hetzner Cloud API.

```sh
hcloud context create gitlab-ci
```

--------------------------------

### Configure Volume Size in Plugin

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/volumes.md

Define the `volume_size` in your autoscaler plugin configuration to specify the desired storage capacity for the attached volume.

```diff
 // ...
 [runners.autoscaler.plugin_config]
 name = "runner-docker-autoscaler0"
 token = "<your-hetzner-cloud-token>"

 location = "fsn1"
 server_type = "cpx42"
 image = "debian-12"
+volume_size = 200

 user_data = """#cloud-config
 package_update: true
 package_upgrade: true

 apt:
   sources:
     docker.list:
       source: deb https://download.docker.com/linux/debian $RELEASE stable
       keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

 packages:
   - docker-ce
 "" " 
```

--------------------------------

### Tear Down Development Environment

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/README.md

Shut down and remove all resources created by the development environment. Use this when you are finished to avoid ongoing costs.

```bash
make -C dev down
```

--------------------------------

### Configure Shared Cache with Hetzner Object Storage

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Use this configuration to enable an S3-compatible shared cache across all auto-scaled instances. Ensure the BucketName has a random suffix to avoid collisions.

```toml
[runners.cache]
Type   = "s3"
Shared = true

[runners.cache.s3]
ServerAddress = "fsn1.your-objectstorage.com"      # Hetzner Object Storage endpoint
BucketName    = "gitlab-ci-cache-7d2f6722"          # Add a random suffix to avoid collisions
AccessKey     = "LR3IWEWNCFHV46133962"
SecretKey     = "URyhbHBO6bMAoNmRJTf0TmTeXCngCMBGnjbvIp9g"
```

--------------------------------

### Private Network Configuration

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Connect runner instances to Hetzner private networks for reachability without public IPs. Set `use_external_addr = false` to route traffic through the private network address.

```toml
[runners.autoscaler.plugin_config]
name             = "runner-private-net"
token            = "$HCLOUD_TOKEN"
location         = "fsn1"
server_type      = "cpx42"
image            = "debian-12"
private_networks = ["my-ci-network"]   # Hetzner network name or ID
public_ipv4_disabled = true            # Optional: disable public IPs entirely
public_ipv6_disabled = true

[runners.autoscaler.connector_config]
use_external_addr = false   # Use the private network IP to reach instances
os       = "linux"
protocol = "ssh"
```

--------------------------------

### Configure Prometheus to Scrape GitLab Runner Metrics

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

Add this Prometheus scrape configuration to collect metrics from your GitLab Runner. Ensure the `targets` match your GitLab Runner host and port.

```yaml
scrape_configs:
  - job_name: gitlab-runner
    static_configs:
      - targets:
          - my-gitlab-runner-host:9252
```

--------------------------------

### Upload Public SSH Key

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Upload your public SSH key to Hetzner Cloud. This allows you to connect to your instances using SSH without password-based authentication.

```sh
hcloud ssh-key create --name dev --public-key-from-file ~/.ssh/id_ed25519.pub
```

--------------------------------

### Check GitLab Runner Service Status

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/tutorials/setup-gitlab-ci-test-infrastructure-from-scratch.md

Verify that the `gitlab-runner` service is active and running, and observe its logs for any scaling activities or errors.

```sh
systemctl status --output=cat --no-pager gitlab-runner
```

--------------------------------

### Prometheus Alert Rule for GitLab Runner Job Queue Duration

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

This alert rule triggers if the average job queue duration in GitLab Runner exceeds 5 seconds over a 5-minute period. It helps identify potential bottlenecks in job scheduling and processing.

```yaml
groups:
  - name: GitLab Runner
    rules:
      - alert: GitLabRunnerJobQueueDuration
        annotations:
          summary: GitLab Runner job queue duration is more than 5 seconds for the past 5m.
        expr: |
          sum (
            increase(gitlab_runner_job_queue_duration_seconds_sum{job="gitlab-runner"}[1m])
            /
            increase(gitlab_runner_job_queue_duration_seconds_count{job="gitlab-runner"}[1m])
          ) by (job) > 5
        for: 5m
```

--------------------------------

### Configure Prometheus for GitLab Runner Metrics

Source: https://context7.com/hetznercloud/fleeting-plugin-hetzner/llms.txt

Configure Prometheus to scrape GitLab Runner metrics and set up alerting rules for runner failures and job queue saturation. The metrics endpoint is exposed at `:9252`.

```yaml
# prometheus.yml — scrape the runner metrics endpoint
scrape_configs:
  - job_name: gitlab-runner
    static_configs:
      - targets:
          - my-gitlab-runner-host:9252

# alerting_rules.yml — fire when the runner is unreachable or has errors
groups:
  - name: GitLab Runner
    rules:
      - alert: NoMetrics
        annotations:
          summary: No metrics scraped for the past 1m.
        expr: >
          absent(up{job="gitlab-runner"})
          or up{job="gitlab-runner"} == 0
        for: 1m

      - alert: GitLabRunnerErrors
        annotations:
          summary: GitLab Runner error rate > 0 for 5m.
        expr: >
          increase(
            gitlab_runner_errors_total{job="gitlab-runner", level=~"error|fatal|panic"}[1m]
          ) > 0
        for: 5m

      - alert: GitLabRunnerWorkerProcessingErrors
        annotations:
          summary: Worker processing failures (e.g. plugin crash) for 5m.
        expr: >
          increase(
            gitlab_runner_worker_processing_failures_total{job="gitlab-runner", failure_type="other"}[1m]
          ) > 0
        for: 5m

      - alert: GitLabRunnerJobQueueDuration
        annotations:
          summary: Average job queue duration > 5s for 5m.
        expr: >
          sum(
            increase(gitlab_runner_job_queue_duration_seconds_sum{job="gitlab-runner"}[1m])
            /
            increase(gitlab_runner_job_queue_duration_seconds_count{job="gitlab-runner"}[1m])
          ) by (job) > 5
        for: 5m
```

--------------------------------

### Prometheus Alert Rule for GitLab Runner Errors

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

This alert rule triggers when the rate of errors or fatal issues reported by the GitLab Runner exceeds zero over a 5-minute period. It helps detect runtime problems within the runner.

```yaml
groups:
  - name: GitLab Runner
    rules:
      - alert: GitLabRunnerErrors
        annotations:
          summary: GitLab Runner error rate is more than 0 for the past 5m.
        expr: |
          increase(
            gitlab_runner_errors_total{job="gitlab-runner", level=~"error|fatal|panic"}[1m]
          ) > 0
        for: 5m
```

--------------------------------

### Prometheus Alert Rule for No GitLab Runner Metrics

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

This alert rule triggers if Prometheus fails to scrape metrics from the gitlab-runner job for a specified duration. It helps identify connectivity or configuration issues.

```yaml
groups:
  - name: GitLab Runner
    rules:
      - alert: NoMetrics
        annotations:
          summary: No metrics were scraped for the past 1m.
        expr: |
          absent(up{job="gitlab-runner"})
          or
          up{job="gitlab-runner"} == 0
        for: 1m
```

--------------------------------

### Prometheus Alert Rule for GitLab API Request Errors

Source: https://github.com/hetznercloud/fleeting-plugin-hetzner/blob/main/docs/guides/monitoring.md

This alert rule triggers when the rate of GitLab API request errors (status codes 4xx or 5xx) exceeds zero over a 5-minute period. It helps identify problems with communication between the runner and the GitLab API.

```yaml
groups:
  - name: GitLab Runner
    rules:
      - alert: GitLabAPIRequestErrors
        annotations:
          summary: GitLab API request error rate is more than 0 for the past 5m.
        expr: |
          increase(
            gitlab_runner_api_request_statuses_total{job="gitlab-runner", status=~"(4|5).."}[1m]
          ) > 0
        for: 5m
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.