### Terragrunt Plan Output Example (Disk Size Change)

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

This output indicates that Terraform plans to replace the Cloud SQL instance due to a disk size change. Ensure `sql_instance_disk_size` in `config.yaml` is sufficiently large to avoid data loss.

```text
STDOUT [infra] tofu: -/+ resource "google_sql_database_instance" "concourse" {
STDOUT [infra] tofu:       ~ settings {
STDOUT [infra] tofu:           ~ disk_size                    = 44 -> 38 # forces replacement
(...)
STDOUT [infra] tofu: Plan: 1 to add, 1 to change, 1 to destroy.
```

--------------------------------

### Deploy Automatic Certificate Regeneration

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

Navigate to the Terragrunt module directory and apply the configuration to deploy the Kubernetes CronJob. This command initiates the infrastructure setup.

```bash
terragrunt apply --config=cert_regen.hcl
```

--------------------------------

### Launch Interactive CredHub CLI Session

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Starts an interactive CredHub CLI session within the Kubernetes cluster for managing secrets, including pipeline credentials and certificates. Requires kubectl to be configured for the target cluster.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/start-credhub-cli.sh

# Prerequisites: kubectl configured for target cluster
gcloud container clusters get-credentials wg-ci --zone europe-west3-a

# Start interactive CredHub CLI pod
cd terragrunt/concourse-wg-ci
../scripts/concourse/start-credhub-cli.sh

# Inside the pod, interact with CredHub:
# Set a password secret
credhub set -t password -n '/concourse/main/s3_access_password' -w 'supersecret'

# Generate a random password
credhub generate -t password -n '/concourse/main/s3_access_password' -l 128 -S

# Set a certificate
credhub set -t certificate -n '/concourse/main/my_cert' \
  -c "$(cat cert.pem)" \
  -p "$(cat key.pem)" \
  -r "$(cat ca.pem)"

# List all secrets for a team
credhub find -p '/concourse/main'

# Get a secret value
credhub get -n '/concourse/main/s3_access_password'
```

--------------------------------

### Kubernetes CronJob Resource Definition

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

This is an example of the Kubernetes CronJob resource that will be created by Terraform. It defines the schedule and job template for certificate regeneration.

```hcl
resource "kubernetes_cron_job_v1" "automatic_certificate_regeneration"
(...)
```

--------------------------------

### Apply CredHub Manifest to Kubernetes

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/terraform-modules/concourse/app/files/config/credhub/_ytt_lib/credhub/kubernetes/README.md

Interpolates the CredHub manifest using ytt and then applies it to the Kubernetes cluster. Ensure ytt is installed and configured.

```bash
ytt -f kubernetes/ | kubectl apply -f -
```

--------------------------------

### CredHub Connection Error Log

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/end_to_end_testing.md

Example error output indicating a failure to interpolate task configuration due to connection issues with CredHub.

```text
failed to interpolate task config:
Get "https://credhub.concourse.svc.cluster.local:9000/api/v1/data?name-like=%2Fconcourse%2Fmain%2Fe2e-test%2Fcredhub-cli-tf-81a16e": 
dial tcp 10.108.6.95:9000: connect: connection refused (after 5 retries)
```

--------------------------------

### Get GKE Cluster Credentials

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Obtain credentials for a specific GKE cluster to configure kubectl for access. Replace 'wg-ci' and 'europe-west3-a' with your cluster name and zone.

```bash
gcloud container clusters get-credentials wg-ci --zone europe-west3-a
```

--------------------------------

### CredHub Certificate Regeneration Output

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

This is an example of the output you should see when a certificate is successfully regenerated by CredHub. It includes the certificate ID, name, type, and creation timestamp.

```text
id: 68875a90-c1b7-4391-a2af-bd3a8f33ce47
name: /concourse/main/cert_1
type: certificate
value: <redacted>
version_created_at: "2024-05-07T12:23:43Z"
(...)
```

--------------------------------

### Authenticate and Set GCP Project

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Log in to your GCP account and set the active project. Ensure you have the necessary permissions.

```bash
gcloud auth login && gcloud auth application-default login
gcloud config set project <your project name>
```

--------------------------------

### Authenticate with Google Cloud

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/end_to_end_testing.md

Required initial step to authenticate the local environment with gcloud.

```bash
gcloud auth login && gcloud auth application-default login
```

--------------------------------

### Run Terragrunt Apply

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

Navigate to the directory containing your config.yaml file and run 'terragrunt apply' to apply the infrastructure changes.

```bash
# cd to folder with config.yaml

terragrunt apply
```

--------------------------------

### Manage Concourse Web Pod Lifecycle

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Commands to scale the Concourse web deployment replicas.

```bash
kubectl -n concourse scale deployment concourse-web --replicas=0
```

```bash
kubectl -n concourse scale deployment concourse-web --replicas=1
```

--------------------------------

### Create On-Demand SQL Instance Backup

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Execute a script to create a backup of the Concourse SQL instance. This is a crucial step before applying upgrades.

```bash
../scripts/concourse/create-sql-backup.sh
```

--------------------------------

### Navigate to files directory

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/developer_notes.md

Change directory to the files folder to prepare for vendir operations.

```bash
cd ./files
```

--------------------------------

### Handle CredHub Backup and Import

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Commands to copy the backup file to a pod and import data into CredHub.

```bash
kubectl -n default cp credhub_backup.json credhub-cli-<id>:/go/credhub_backup.json
```

```bash
credhub import -j -f credhub_backup.json
```

--------------------------------

### Execute Disaster Recovery Restore

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Run the recovery script to restore infrastructure and application state. Requires an existing GKE cluster and Cloud SQL backup.

```bash
cd terragrunt/concourse-wg-ci
../scripts/concourse/dr-restore.sh
```

--------------------------------

### Apply Terragrunt Infrastructure

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Deploys the entire infrastructure stack defined in the configuration.

```sh
terragrunt run-all apply
```

--------------------------------

### Execute Automated DR Restore

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Run the restoration script from the configuration directory.

```bash
cd <folder witgh config.yaml>
../scripts/concourse/dr-restore.sh
```

--------------------------------

### Login to New Concourse Version

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Log in to the upgraded Concourse instance using the `fly` CLI. Replace `<target name>` with your specific target.

```bash
fly login -t <target name>
```

--------------------------------

### Configure Kubectl Access for GKE

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Commands to list clusters, fetch credentials, and verify cluster status for debugging or manual operations.

```bash
# List available clusters
gcloud container clusters list
# NAME    LOCATION        MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
# wg-ci   europe-west3-a  1.34.0          34.159.31.85  e2-standard-4  1.34.0           3          RUNNING

# Get credentials for kubectl
gcloud container clusters get-credentials wg-ci --zone europe-west3-a --project app-runtime-interfaces-wg
# Fetching cluster endpoint and auth data.
# kubeconfig entry generated for wg-ci.

# Verify context
kubectl config current-context
# gke_app-runtime-interfaces-wg_europe-west3-a_wg-ci

# Check Concourse pods
kubectl get pods -n concourse
# NAME                               READY   STATUS    RESTARTS   AGE
# concourse-web-7d8f9c7d8-x2j4k      1/1     Running   0          2d
# concourse-worker-0                 1/1     Running   0          2d
# credhub-5f8d9c7d8-k2l3m            1/1     Running   0          2d
# uaa-6f9d8c7d8-n3m4p                1/1     Running   0          2d

# Check CredHub certificate expiry
kubectl get secret -n concourse credhub-root-ca
# If older than 1 year, regenerate by deleting and restarting secretgen-controller
```

--------------------------------

### Taint and Re-provision Backend

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Commands to force re-provisioning of the backend when Carvel kapp fails to apply changes.

```bash
cd ./backend
terragrunt taint carvel_kapp.concourse_backend
terragrunt plan
terragrunt apply
```

--------------------------------

### Sync vendir configuration

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/developer_notes.md

Synchronizes dependencies defined in vendir.yml.

```bash
vendir sync
```

--------------------------------

### Deploy Concourse CI using Helm with Custom Values

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Uses the Helm Terraform provider to deploy Concourse CI. Custom values are provided via a YAML file and `set` arguments, including external URL, load balancer IP, GitHub authentication, and build log retention. Ensure the specified values file exists and variables are defined.

```hcl
data "helm_template" "concourse" {
  name       = "concourse"
  repository = "https://concourse-charts.storage.googleapis.com/"
  chart      = "concourse"
  version    = var.concourse_helm_version
  values     = [file("files/${var.gke_workers_pool_machine_type}.yml")]

  set = concat([
    {
      name  = "concourse.web.externalUrl"
      value = "https://${var.load_balancer_dns}"
    },
    {
      name  = "web.service.api.loadBalancerIP"
      value = var.load_balancer_ip
    },
    {
      name  = "concourse.web.auth.mainTeam.github.team"
      value = var.concourse_github_mainTeam
    },
    {
      name  = "concourse.web.auth.mainTeam.localUser"
      value = ""  # Remove local users for security
    },
    {
      name  = "concourse.web.containerPlacementStrategy"
      value = var.concourse_container_placement_strategy
    },
    {
      name  = "concourse.worker.runtime"
      value = "containerd"
    }
  ], var.concourse_max_days_to_retain_build_logs != null ? [
    {
      name  = "concourse.web.maxDaysToRetainBuildLogs"
      value = var.concourse_max_days_to_retain_build_logs
    }
  ] : [])
}

resource "carvel_kapp" "concourse_app" {
  app          = "concourse-app"
  namespace    = "concourse"
  config_yaml  = data.carvel_ytt.concourse_app.result
  diff_changes = true
  depends_on   = [kubernetes_secret_v1.github_oauth, carvel_kapp.credhub_uaa]
}
```

--------------------------------

### Workaround for gke-gcloud-auth-plugin with asdf

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Copies the GKE auth plugin to the asdf shims directory if the gcloud CLI fails to locate it.

```bash
cp ~/.asdf/installs/gcloud/415.0.0/bin/gke-gcloud-auth-plugin ~/.asdf/shims
```

--------------------------------

### List GKE Clusters

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Use this command to list available Google Kubernetes Engine clusters in your project.

```bash
gcloud container clusters list
```

--------------------------------

### Plan and Apply Concourse Stack Deployment (Rollback)

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Re-deploy the Concourse stack after restoring the SQL instance during a roll-back. This ensures a consistent state.

```bash
terragrunt run --all plan
terragrunt run --all apply
```

--------------------------------

### Re-run DR Restore

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Command to resume the restoration process after backend re-provisioning.

```bash
cd ..
../scripts/concourse/dr_restore.sh
```

--------------------------------

### Manage Concourse CLI and Kubernetes Context

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Commands for logging into the Concourse CLI and updating the kubectl context.

```bash
fly -t wg-ci-test login -c https://concourse-test.app-runtime-interfaces.ci.cloudfoundry.org
```

```bash
gcloud container clusters get-credentials wg-ci[-test] --region us-east1-b
```

--------------------------------

### Login to Existing Fly Target

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/terragrunt/concourse-wg-ci-test/e2e_test/README.md

If the target is already added, use this command to log in to the specified target.

```bash
fly login -t wg-ci-test
```

--------------------------------

### Configure Repository Runner YAML

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Defines the configuration parameters for GitHub repository runners, including replica counts and resource limits.

```yaml
github_repos:
  - name: app-autoscaler-release
    owner: cloudfoundry
    hpa_scaleup_trigger_duration: "5m"
    hpa_scaledown_delay_seconds: 300
    runners_min_replicas: 0
    runners_max_replicas: 10
    runnerset_resource_request_cpu: "500m"
    runnerset_resource_request_mem: "1Gi"
    runnerset_resource_limits_cpu: "2000m"
    runnerset_resource_limits_mem: "4Gi"
    var_lib_docker_size: "50Gi"

tf_modules:
  github_arc: "../../terraform-modules/actions_runner_controller/team"
```

--------------------------------

### Apply cgroupv2 Configuration to GKE Node Pools

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Update GKE node pools to use Linux cgroupv2. This requires a system configuration file and applies the setting to specified node pools.

```yaml
linuxConfig:
  cgroupMode: 'CGROUP_MODE_V2'
```

```shell
gcloud container node-pools update default-pool --system-config-from-file=./cgroupv2.yaml --region europe-west3-a --cluster wg-ci
```

```shell
gcloud container node-pools update concourse-workers --system-config-from-file=./cgroupv2.yaml --region europe-west3-a --cluster wg-ci
```

--------------------------------

### Execute GitHub OAuth Secret Creation

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Runs the script to create GitHub OAuth secrets in GCP Secret Manager from the configuration directory.

```bash
cd <folder with config.yaml>
../scripts/create-github-oauth-gcp.sh
```

--------------------------------

### Execute End-to-End Tests for wg-ci-test Cluster

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Run end-to-end tests for the `wg-ci-test` cluster after the Concourse upgrade. This verifies the deployment and functionality.

```bash
cd e2e_test
terragrunt apply --source-update
cd ..
```

--------------------------------

### Destroy Project Infrastructure

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Steps to remove encryption keys from state and delete the project resources.

```shell
cd <folder with config.yaml>/dr_create

terragrunt state rm google_secret_manager_secret_version.credhub_encryption_key
terragrunt state rm google_secret_manager_secret.credhub_encryption_key
```

```shell
gcloud secrets delete <gke_name>-credhub-encryption-key --project=<your project name>
```

```shell
terragrunt run-all destroy
```

--------------------------------

### GitHub Actions Runner Controller Infrastructure Configuration

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Configuration file for deploying GitHub Actions Runner Controller (ARC) infrastructure. It specifies project, region, zone, GCS bucket, GKE cluster details, GitHub token, Let's Encrypt settings, and DNS configuration. Ensure all referenced variables and external resources are correctly set up.

```yaml
project: app-runtime-interfaces-wg
region: europe-west3
zone: europe-west3-a

gcs_bucket: terraform-wg-ci
gcs_prefix: github-arc-infra

gke_name: wg-ci

# GitHub token for runner registration
arc_github_access_token_name: app-autoscaler-ci-bot-actions-controller-token

# Let's Encrypt configuration
arc_letsencrypt_notifications_email: notifications@example.com
arc_letsencrypt_staging: false
arc_letsencrypt_production: true

# DNS for webhook server
dns_zone: app-runtime-interfaces
dns_domain: ci.cloudfoundry.org

# Helm chart versions
cert_manager_helm_version: "v1.20.0"
arc_helm_version: "0.23.7"

tf_modules:
  github_arc: "../..//terraform-modules/actions_runner_controller/infra"
```

--------------------------------

### Set Current Kubectl Context

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Verify the current context set for kubectl after obtaining credentials.

```bash
kubectl config current-context
```

--------------------------------

### Navigate to Concourse Cluster Directory

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/secrets_rotation.md

Change directory to the location of your concourse cluster configuration.

```bash
cd terragrunt/concourse-wg-ci-test
```

--------------------------------

### Configure Concourse Deployment Settings

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

YAML configuration settings for region, database version, and GKE control plane version.

```yaml
region: us-east1
zone: us-east1-b
secondary_zone: us-east1-c
```

```yaml
database_version: "POSTGRES_16"
```

```yaml
gke_controlplane_version: "1.31"
```

--------------------------------

### Execute Terraform and Terragrunt Operations

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Commands to plan and apply infrastructure changes using Terragrunt.

```bash
terragrunt run-all plan
```

```bash
terragrunt run-all apply
```

--------------------------------

### View CronJobs in Kubernetes

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

Use `kubectl` to list all CronJobs in the specified namespace. This helps verify that the certificate regeneration CronJob has been successfully deployed.

```bash
$ kubectl -n concourse get cronjobs
NAME                       SCHEDULE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
certificate-regeneration   @monthly   False     0        <none>          50m
```

--------------------------------

### Configure Google Cloud SQL Database Instance for Concourse

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Defines a Google Cloud SQL instance for Concourse CI, specifying database version, region, backup configurations, and maintenance windows. Ensure 'var.db_terraform_deletion_protection' and 'var.db_engine_level_deletion_protection' are set appropriately to manage deletion protection.

```hcl
resource "google_sql_database_instance" "concourse" {
  database_version    = var.database_version
  name                = var.sql_instance_name
  project             = var.project
  region              = var.region
  deletion_protection = var.db_terraform_deletion_protection

  settings {
    activation_policy = "ALWAYS"
    availability_type = "REGIONAL"

    backup_configuration {
      location = var.sql_instance_backup_location
      backup_retention_settings {
        retained_backups = "7"
        retention_unit   = "COUNT"
      }
      enabled                        = "true"
      point_in_time_recovery_enabled = "true"
      start_time                     = "00:00"
      transaction_log_retention_days = "7"
    }

    deletion_protection_enabled = var.db_engine_level_deletion_protection
    disk_autoresize             = "true"
    disk_size                   = var.sql_instance_disk_size
    disk_type                   = "PD_SSD"
    edition                     = "ENTERPRISE"

    ip_configuration {
      ipv4_enabled = "true"
    }

    location_preference {
      zone           = var.zone
      secondary_zone = var.sql_instance_secondary_zone
    }

    maintenance_window {
      day          = 7  # Sunday
      hour         = 0  # 0:00 - 1:00
      update_track = "stable"
    }

    tier = var.sql_instance_tier
  }
}
```

--------------------------------

### Terraform GitHub Repository Webhook Configuration

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

This configuration for github_repository_webhook uses a bypass to dynamically set the repository path. It iterates over a list of repositories to create webhooks in a scalable manner, overriding the default provider configuration.

```hcl
resource "github_repository_webhook" "github_webhook" {
    for_each = { for repo in var.github_repos: repo.name => repo }

    repository = "../${each.value.owner}/${each.value.name}"
    configuration {
      ...
    }
    ...
}
```

--------------------------------

### Create HorizontalRunnerAutoscaler Terraform Resource

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Creates a Kubernetes namespace and a HorizontalRunnerAutoscaler resource for webhook-based scaling of GitHub Actions runners.

```hcl
# terraform-modules/actions_runner_controller/team/github_runnerset_autoscale.tf
resource "kubernetes_namespace" "github_actions_runners" {
  metadata {
    name = "${var.team_name}-actions-runners"
  }
}

resource "kubectl_manifest" "github_repo_runners_hpa" {
  for_each = { for repo in var.github_repos : repo.name => repo }

  yaml_body = <<EOT
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: "${each.value.owner}--${each.value.name}"
  namespace: "${kubernetes_namespace.github_actions_runners.metadata[0].name}"
spec:
  minReplicas: ${each.value.runners_min_replicas}
  maxReplicas: ${each.value.runners_max_replicas}
  scaleTargetRef:
    kind: RunnerSet
    name: "${each.value.owner}--${each.value.name}"
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}
        duration: "${each.value.hpa_scaleup_trigger_duration}"
  scaleDownDelaySecondsAfterScaleOut: ${each.value.hpa_scaledown_delay_seconds}
EOT

  depends_on = [google_container_node_pool.team_github_arc]
}
```

--------------------------------

### Deploy Full Concourse Stack with Terragrunt

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Orchestrates the deployment of the entire Concourse infrastructure stack, including infra, backend, dr_create, and app modules. Ensure prerequisites like gcloud authentication and Terragrunt are met.

```bash
# Prerequisites: Install required tools via asdf or nix
# asdf plugin install via ./asdf-plugin-install.sh

# Authenticate to GCP
gcloud auth login && gcloud auth application-default login
gcloud config set project 'app-runtime-interfaces-wg'

# Navigate to your deployment folder
cd terragrunt/concourse-wg-ci

# Deploy full stack (infra, backend, dr_create, app)
terragrunt run-all apply

# Expected output:
# Group 1: infra (VPC, GKE, CloudSQL)
# Group 2: backend (namespaces, secretgen, sql-proxy)
# Group 3: dr_create (CredHub encryption key backup)
# Group 4: app (Concourse, CredHub, UAA)
```

--------------------------------

### Configure Certificates for Regeneration

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

Define the comma-separated list of certificate names to be regenerated in the `config.yaml` file. Ensure the paths are correct for your CredHub environment.

```yaml
certificates_to_regenerate: "/concourse/main/cert_1,/concourse/main/cert_2"
```

--------------------------------

### Manually Trigger CronJob for Testing

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

To test the CronJob's functionality, you can manually create a job from the CronJob definition. This allows for immediate execution and log checking.

```bash
kubectl -n concourse create job --from=cronjob/certificate-regeneration cert-regen-job
# wait a few seconds
kubectl -n concourse get pods # search pod "cert-regen-job-<xyz>"
kubectl -n concourse logs cert-regen-job-<xyz>
```

--------------------------------

### Login to Concourse via Fly

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/end_to_end_testing.md

Configures the fly CLI to connect to a specific Concourse instance.

```bash
fly login -t <gke_name> -c https://<your concourse url> -n <your team default:main>
```

--------------------------------

### Checkout Renovate Pull Request for Fly CLI Version

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Switch to the branch that updates the `fly` CLI version in the `.tool-versions` file. This ensures compatibility with the new Concourse version.

```bash
git checkout origin/renovate/concourse-concourse-8.x
```

--------------------------------

### Copy CredHub backup from pod

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Transfers the exported CredHub backup file from the CLI pod to the local machine.

```bash
kubectl -n default cp credhub-cli-<id>:/go/credhub_backup.json credhub_backup.json
```

--------------------------------

### Execute Terragrunt Operations

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Commands to plan and apply infrastructure changes for a specific component.

```shell
cd terragrunt/concourse-<gke_name>/concourse/app
terragrunt plan
terragrunt apply
```

--------------------------------

### Execute PostgreSQL Secrets Rotation Script

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/secrets_rotation.md

Run the bash script to initiate the secrets rotation process for PostgreSQL. This script reads configuration from config.yaml and manages secret updates and application restarts.

```bash
../scritps/concourse/secret_rotation_postgresql.sh
```

--------------------------------

### Dump Concourse database

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Creates a SQL dump of the Concourse database using pg_dump via the Cloud SQL Auth Proxy.

```bash
pg_dump "postgresql://concourse@localhost:5432/concourse" > "concourse_backup.sql"
```

--------------------------------

### Checkout Renovate Pull Request for Concourse Version

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Switch to the specific branch created by Renovate for the Concourse version upgrade. This ensures you are working with the intended changes.

```bash
git pull
git checkout origin/renovate/concourse-20.x
```

--------------------------------

### Delete Test Job

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

After testing, it is recommended to delete the manually created job to clean up Kubernetes resources.

```bash
kubectl -n concourse delete job cert-regen-job
```

--------------------------------

### Set GitHub Token Environment Variable

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

Before running terragrunt, set the GITHUB_TOKEN environment variable in your terminal. Missing this will cause errors when creating webhooks.

```bash
export GITHUB_TOKEN="ghp..."
```

--------------------------------

### Team-Specific GitHub Actions Runner Configuration

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Configuration for team-specific GitHub Actions Runner Controller (ARC) infrastructure. It defines node pool settings, autoscaling parameters, and webhook server details for a given team. Ensure the GKE cluster and necessary permissions are pre-configured.

```yaml
# Team config.yaml for ARC runners
project: app-runtime-interfaces-wg
region: europe-west3
zone: europe-west3-a

gcs_bucket: terraform-wg-ci
team_name: autoscaler
gke_name: wg-ci

arc_webhook_server_domain: arc-webhook.app-runtime-interfaces.ci.cloudfoundry.org
arc_webhook_server_token_name: arc-webhook-server-token

# Node pool configuration
gke_arc_node_pool_machine_type: e2-standard-4
gke_arc_node_pool_disk_size_gb: 100
gke_arc_node_pool_count: 0
gke_arc_node_pool_autoscaling_max: 10
gke_arc_node_pool_ssd_count: 0
gke_arc_runner_storage_type: pd-balanced
```

--------------------------------

### Generate ARC Webhook Server Token

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

Executes the script to generate and store a random hex token in GCP Secret Manager for webhook authentication.

```bash
# cd to folder with config.yaml ie `github-arc-infra-wg-ci-test`

../scritps/actions-controller/create-arc-webhook-server-token-gcp.sh
```

--------------------------------

### Perform Disaster Recovery Restore

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Automates the disaster recovery process for the Concourse deployment by restoring from Cloud SQL backups and GCP Secret Manager encryption keys. Requires the CredHub encryption key to be pre-existing in GCP Secret Manager.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/dr-restore.sh

# Prerequisites:
# 1. CredHub encryption key exists in GCP Secret Manager (created by dr_create module)

```

--------------------------------

### Generate GitHub Access Token Secret

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

Executes the script to create a GitHub access token in GCP Secret Manager for runner registration.

```bash
# Generate oauth token for your github account.

# Provide the token in `token=` variable in the script

# cd to folder with config.yaml ie `github-arc-infra-wg-ci-test`

../scritps/actions-controller/create-github-access-token-gcp.sh
```

--------------------------------

### Concourse Deployment Configuration

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Central YAML file defining parameters for a Concourse deployment, including GCP settings, GKE configuration, and database details. This file is referenced by the Terragrunt configuration.

```yaml
# config.yaml - Complete Concourse deployment configuration
project: app-runtime-interfaces-wg
region: us-east1
zone: us-east1-b
secondary_zone: us-east1-c

# Terraform state storage
gcs_bucket: terraform-wg-ci-test
gcs_prefix: concourse

# DNS configuration for Concourse URL: https://<dns_record>.<dns_zone>.<dns_domain>
dns_record: concourse-test
dns_zone: app-runtime-interfaces
dns_domain: ci.cloudfoundry.org

# GKE cluster name (used as prefix for all resources)
gke_name: wg-ci-test

```

--------------------------------

### Include CA in Regeneration Process

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

To include the Certificate Authority (CA) in the regeneration process, add its CredHub path to the beginning of the `certificates_to_regenerate` list. This ensures the CA is regenerated first, followed by certificates signed by it.

```yaml
certificates_to_regenerate: "/concourse/main/my_CA,/concourse/main/cert_1,/concourse/main/cert_2"
```

--------------------------------

### Plan Terragrunt changes with source update

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/developer_notes.md

Updates the terragrunt cache folders when terraform source module code changes.

```sh
terragrunt run --all plan --source-update
```

--------------------------------

### Configure Cloud SQL Deletion Protection

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Enables deletion protection for a Google Cloud SQL instance via Terraform.

```hcl
resource "google_sql_database_instance" "concourse" {

  # This option prevents Terraform from deleting an instance
  deletion_protection = true
```

```hcl
  settings {
    deletion_protection_enabled = "true"
  }
```

--------------------------------

### Retrieve GCP Secret Manager secrets

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Use this command to fetch the latest version of a specific secret from GCP Secret Manager.

```bash
gcloud secrets versions access latest --secret=<secret-name>
```

--------------------------------

### Rotate PostgreSQL Secrets

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Automates password rotation for Concourse, CredHub, and UAA databases. Requires interactive confirmation.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/secret_rotation_postgresql.sh

# Navigate to deployment folder
cd terragrunt/concourse-wg-ci

# Execute rotation (interactive confirmation required)
../scripts/concourse/secret_rotation_postgresql.sh
```

--------------------------------

### Troubleshoot CredHub Key Mismatch

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Remove the existing encryption key secret to allow for re-initialization during the recovery process.

```bash
kubectl delete secret -n concourse credhub-encryption-key
```

--------------------------------

### Perform Database Maintenance

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

SQL commands to drop, recreate, and restore the Concourse database.

```sql
psql -h 127.0.0.1 -p 5432 -U concourse -d postgres
DROP DATABASE concourse;
CREATE DATABASE concourse;
```

```sql
psql -h 127.0.0.1 -p 5432 -U concourse -d concourse -f concourse_backup.sql
```

--------------------------------

### Terraform GitHub Provider Bypass for Owner

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/actions-runner-controller/README.md

This workaround in providers.tf configures the GitHub provider with a local path as the owner. This allows for scalable creation of webhooks for multiple repositories per team by overriding the provider configuration.

```hcl
provider "github" {
  owner = "./"
}
```

--------------------------------

### Apply Concourse Version Upgrade with Terragrunt

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Apply the planned changes to upgrade the Concourse version. This command also updates the Terraform source.

```bash
terragrunt run --all apply --source-update
```

--------------------------------

### Add Fly Target

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/terragrunt/concourse-wg-ci-test/e2e_test/README.md

Use this command to add a new target to your fly CLI, specifying the Concourse instance URL and team.

```bash
fly login -t wg-ci-test -c https://concourse-test.app-runtime-interfaces.ci.cloudfoundry.org/ -n main
```

--------------------------------

### Create GitHub OAuth Credentials in GCP Secret Manager

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Generates the necessary GitHub OAuth credentials in GCP Secret Manager for Concourse GitHub authentication. This script must be run before deploying the Concourse application module.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/create-github-oauth-gcp.sh

# 1. Create GitHub OAuth App at https://github.com/settings/developers
#    - Homepage URL: https://concourse-test.app-runtime-interfaces.ci.cloudfoundry.org
#    - Callback URL: https://concourse-test.app-runtime-interfaces.ci.cloudfoundry.org/sky/issuer/callback

# 2. Edit script with your OAuth credentials
id="your_github_oauth_client_id"
secret="your_github_oauth_client_secret"

# 3. Run from folder containing config.yaml
cd terragrunt/concourse-wg-ci
../scripts/concourse/create-github-oauth-gcp.sh

# Script creates GCP secret: <gke_name>-concourse-github-oauth
# Format stored:
# id: <client_id>
# secret: <client_secret>
```

--------------------------------

### Login to CredHub

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/terraform-modules/concourse/app/files/config/credhub/_ytt_lib/credhub/kubernetes/README.md

Logs into the CredHub instance using the cluster IP, UAA credentials, and CA certificates. The CA certificates are retrieved from Kubernetes secrets.

```bash
credhub login -s https://<CLUSTER_IP>:9000 -u credhub -p password --ca-cert="$(kubectl get secret server-ca -o json | jq -r .data.certificate | base64 -D)" --ca-cert="$(kubectl get secret uaa-ca -o json | jq -r .data.certificate | base64 -D)"
```

--------------------------------

### Concourse Infrastructure Terragrunt Configuration

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Configures remote state for GCS and defines inputs for the Concourse infrastructure module. Ensure 'config.yaml' is present and correctly formatted.

```hcl
# terragrunt/concourse-wg-ci/infra/terragrunt.hcl
locals {
  config = yamldecode(file("../config.yaml"))
}

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket                    = "${local.config.gcs_bucket}"
    prefix                    = "${local.config.gcs_prefix}/concourse-infra"
    project                   = "${local.config.project}"
    location                  = "${local.config.region}"
    enable_bucket_policy_only = true
  }
}

terraform {
  source = local.config.tf_modules.infra
}

inputs = {
  project                           = local.config.project
  region                            = local.config.region
  zone                              = local.config.zone
  gke_name                          = local.config.gke_name
  gke_deletion_protection           = local.config.gke_deletion_protection
  gke_controlplane_version          = local.config.gke_controlplane_version
  gke_cluster_ipv4_cidr             = local.config.gke_cluster_ipv4_cidr
  gke_services_ipv4_cidr_block      = local.config.gke_services_ipv4_cidr_block
  gke_master_ipv4_cidr_block        = local.config.gke_master_ipv4_cidr_block
  gke_subnet_cidr                   = local.config.gke_subnet_cidr
  gke_default_pool_machine_type     = local.config.gke_default_pool_machine_type
  gke_default_pool_node_count       = local.config.gke_default_pool_node_count
  gke_default_pool_autoscaling_max  = local.config.gke_default_pool_autoscaling_max
  gke_workers_pool_machine_type     = local.config.gke_workers_pool_machine_type
  gke_workers_pool_node_count       = local.config.gke_workers_pool_node_count
  gke_workers_pool_autoscaling_max  = local.config.gke_workers_pool_autoscaling_max
  gke_cloud_nat_min_ports_per_vm    = local.config.gke_cloud_nat_min_ports_per_vm
  gke_http_load_balancing_disabled  = local.config.gke_http_load_balancing_disabled
  database_version                  = local.config.database_version
  db_terraform_deletion_protection  = local.config.db_terraform_deletion_protection
  sql_instance_name                 = "${local.config.gke_name}-concourse"
  sql_instance_tier                 = local.config.sql_instance_tier
  sql_instance_disk_size            = local.config.sql_instance_disk_size
  sql_instance_backup_location      = local.config.sql_instance_backup_location
  sql_instance_secondary_zone       = local.config.secondary_zone
  dns_record                        = local.config.dns_record
  dns_zone                          = local.config.dns_zone
  dns_domain                        = local.config.dns_domain
  dns_name                          = "${local.config.dns_zone}-${local.config.dns_domain}."
  wg_ci_human_account_permissions   = local.config.wg_ci_human_account_permissions
  github_secret_name                = "${local.config.gke_name}-concourse-github-oauth"
}

```

--------------------------------

### Retrieve Concourse PostgreSQL password

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Extracts and decodes the base64-encoded PostgreSQL password from the Kubernetes secret.

```bash
kubectl -n concourse get secret concourse-postgresql-password -o yaml | yq -r .data.password | base64 -d
```

--------------------------------

### Deploy Concourse E2E Pipeline

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Terraform configuration to deploy an E2E test pipeline to Concourse, including secret management and execution commands.

```hcl
# terraform-modules/concourse/e2e_test/e2e_pipeline.tf
resource "concourse_pipeline" "e2e-test" {
  team_name     = var.fly_team
  pipeline_name = var.pipeline
  is_exposed    = var.pipeline_exposed
  is_paused     = false

  pipeline_config = templatefile("pipeline.yml", {
    pipeline_job              = var.pipeline_job
    credhub-test-secret-name  = "${var.credhub-test-secret-prefix}-${random_id.credhub_cli.hex}"
    credhub-test-secret-value = "${var.credhub-test-secret-prefix}-${random_id.credhub_cli.hex}-value"
    credhub-test-secret-path  = var.credhub-test-secret-path
  })
  pipeline_config_format = "yaml"

  depends_on = [kubernetes_job.credhub_cli]
}

# Run e2e tests
# cd terragrunt/concourse-wg-ci-test/e2e_test
# terragrunt apply
# fly -t ari-test trigger-job -j e2e-pipeline/e2e-job --watch
```

--------------------------------

### Destroy Automatic Certificate Regeneration

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/certificate_regeneration.md

To remove the deployed CronJob and associated resources, navigate to the Terragrunt module directory and execute the destroy command with the same configuration file.

```bash
terragrunt destroy --config=cert_regen.hcl
```

--------------------------------

### Configure GKE Cluster

Source: https://context7.com/cloudfoundry/app-runtime-interfaces-infrastructure/llms.txt

Terraform resource definition for a private GKE cluster with Workload Identity and Config Connector enabled.

```hcl
# terraform-modules/concourse/infra/gke_cluster.tf
resource "google_container_cluster" "wg_ci" {
  provider                 = google-beta
  name                     = var.gke_name
  location                 = var.zone
  project                  = var.project
  initial_node_count       = "1"
  remove_default_node_pool = true
  min_master_version       = var.gke_controlplane_version
  deletion_protection      = var.gke_deletion_protection

  release_channel {
    channel = "STABLE"
  }

  workload_identity_config {
    workload_pool = "${var.project}.svc.id.goog"
  }

  logging_config {
    enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
  }

  ip_allocation_policy {
    cluster_ipv4_cidr_block  = var.gke_cluster_ipv4_cidr
    services_ipv4_cidr_block = var.gke_services_ipv4_cidr_block
  }

  private_cluster_config {
    enable_private_endpoint = "false"
    enable_private_nodes    = "true"
    master_ipv4_cidr_block  = var.gke_master_ipv4_cidr_block
  }

  network    = google_compute_network.vpc.name
  subnetwork = google_compute_subnetwork.subnet.name
  networking_mode = "VPC_NATIVE"

  addons_config {
    config_connector_config {
      enabled = "true"
    }
    gce_persistent_disk_csi_driver_config {
      enabled = "true"
    }
    horizontal_pod_autoscaling {
      disabled = "true"
    }
    http_load_balancing {
      disabled = var.gke_http_load_balancing_disabled
    }
  }

  enable_shielded_nodes = "true"
}
```

--------------------------------

### Export CredHub data

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/region_change.md

Exports all CredHub data to a JSON file for migration purposes.

```bash
credhub export --output-file=credhub_backup.json
```

--------------------------------

### Manage CredHub CA Secrets

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/README.md

Commands to inspect, delete, and recreate the CredHub root CA secret and associated controllers.

```shell
kubectl get secret -n concourse credhub-root-ca
```

```shell
kubectl delete secret -n concourse credhub-root-ca
```

```shell
kubectl scale deploy -n secretgen-controller secretgen-controller --replicas=0
kubectl scale deploy -n secretgen-controller secretgen-controller --replicas=1
kubectl wait deployment -n secretgen-controller secretgen-controller --for=jsonpath='{.spec.replicas}'=1 --timeout=30s
```

```shell
kubectl get secret -n concourse credhub-root-ca
```

```shell
kubectl delete pods --namespace='concourse' --selector='app=credhub'
kubectl delete pods --namespace='concourse' --selector='release=concourse'
```

--------------------------------

### Terraform Credhub Encryption Key Error

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Error message indicating the Credhub encryption key is missing from Google Secret Manager.

```hcl
│ Error: Error retrieving available secret manager secret versions: googleapi: Error 404: Secret [projects/899763165748/secrets/wg-ci-test-credhub-encryption-key] not found or has no versions.
│
│   with data.google_secret_manager_secret_version.credhub_encryption_key,
│   on credhub_dr_check.tf line 2, in data "google_secret_manager_secret_version" "credhub_encryption_key":
│    2: data "google_secret_manager_secret_version" "credhub_encryption_key" {
```

--------------------------------

### Terraform Encryption Key Mismatch Error

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Error message indicating a mismatch between GCP Secret Manager and Kubernetes secrets.

```hcl
│ Error: Call to unknown function
│ 
│   on .terraform/modules/assertion_encryption_key_identical/.tf line 6, in locals:
│    6:   content = var.condition ? "" : SEE_ABOVE_ERROR_MESSAGE(true ? null : "ERROR: ${var.error_message}")
│     ├────────────────
│     │ var.error_message is "*** Encryption keys in GCP Secret Manager and kubernetes secrets do not match ***"
│ 
│ There is no function named "SEE_ABOVE_ERROR_MESSAGE".
```

--------------------------------

### Delete Concourse Deployments

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/concourse_minor_version_upgrade.md

Remove Concourse worker and web deployments from the Kubernetes namespace. This is part of the roll-back procedure.

```bash
kubectl -n concourse delete deployment concourse-worker
kubectl -n concourse delete deployment concourse-web
```

--------------------------------

### Kubernetes Secret Conflict Error

Source: https://github.com/cloudfoundry/app-runtime-interfaces-infrastructure/blob/main/docs/concourse/disaster_recovery.md

Error message occurring when the Credhub encryption key secret already exists in Kubernetes during restoration.

```hcl
╷
│ Error: secrets "credhub-encryption-key" already exists
│
│   with kubernetes_secret_v1.credhub_encryption_key,
│   on credhub_restore.tf line 6, in resource "kubernetes_secret_v1" "credhub_encryption_key":
│    6: resource "kubernetes_secret_v1" "credhub_encryption_key" {
│
╵
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.