### Install Tool Versions with asdf

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Installs all required CLI tools based on the .tool-versions file using the asdf version manager. Ensures consistent tooling across development environments by adding plugins and installing specified versions.

```bash
#!/usr/bin/env bash
# asdf-plugin-install.sh
set -euo pipefail

if [ ! $(which asdf) ]; then
    echo "ERROR: asdf not found, install from: https://asdf-vm.com/guide/getting-started.html"
    exit 1
fi

if [ ! -s .tool-versions ]; then
    echo "ERROR: .tools-versions not found in root directory"
    exit 1
fi

# Add required plugins from .tool-versions
echo ">> Add required plugins"
for p in $(cat .tool-versions | awk '{ print $1 }'); do
    asdf plugin add "${p}" || true
done

# Install all tools at specified versions
echo ">> Install all plugins"
asdf install

# Verify installations
echo ">> Show installed versions"
asdf current

# .tool-versions contents:
# concourse 7.13.1
# gcloud 542.0.0
# helm 3.19.0
# kubectl 1.34.1
# terraform 1.13.3
# terragrunt 0.89.3
```

--------------------------------

### Start CredHub CLI Session in Kubernetes

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Launches an interactive CredHub CLI session within a Kubernetes cluster. It automatically fetches CredHub credentials from Kubernetes secrets and uses a specified container image.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/start-credhub-cli.sh
set -eu

# Example CredHub commands:
#   credhub set -t password -n '/concourse/main/s3_access_password' -w 'supersecret'
#   credhub generate -t password -n '/concourse/main/s3_access_password' -l 128 -S

credhub_server="https://credhub.concourse.svc.cluster.local:9000"
credhub_ca_cert="$(kubectl --namespace concourse get secret credhub-root-ca -o json | jq -r .data.certificate | base64 --decode)"
credhub_client="credhub_admin_client"
credhub_secret="$(kubectl --namespace concourse get secret credhub-admin-client-credentials -o json | jq -r .data.password | base64 --decode)"

kubectl run credhub-cli-$(openssl rand -hex 4) \
        --rm -i -t \
        --restart=Never \
        --image=cloudfoundry/cf-deployment-concourse-tasks:v20.3.0 \
        --env="CREDHUB_SERVER=$credhub_server" \
        --env="CREDHUB_CA_CERT=$credhub_ca_cert" \
        --env="CREDHUB_CLIENT=$credhub_client" \
        --env="CREDHUB_SECRET=$credhub_secret" \
        -- /bin/bash
```

--------------------------------

### Identify CredHub certificate expiration error

Source: https://github.com/cloudfoundry/app-runtime-deployments-infrastructure/blob/main/README.md

Example of the error message displayed in Concourse when the CredHub certificate has expired.

```text
Get "https://credhub.concourse.svc.cluster.local:9000/api/v1/data?name-like=%2Fconcourse%2Fmain%2Fcf-deployment%2Fard_wg_gitbot_ssh_key": x509: certificate has expired or is not yet valid
```

--------------------------------

### Create Cloud SQL Backup

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Creates a manual Cloud SQL backup for a specified instance after verifying the configuration file. Requires yq and gcloud CLI tools.

```bash
set -euo pipefail

if [ ! -s "./config.yaml" ]; then
    echo "ERROR: Please 'cd' to your folder with config.yaml and run this script again."
    exit 1
fi

project="$(yq .project config.yaml)"
sql_instance="$(yq .gke_name config.yaml)-concourse"
epoch_date="$(date +%s)"

gcloud_format=( --format="table[box](windowStartTime, id, backupKind, status, type, description)" )

# Confirm before creating backup
read -p "About to create CloudSQL backup for instance $sql_instance. Confirm with 'yes': " -r
if [[ ! "$REPLY" == "yes" ]]; then
    echo "Canceling"
    exit 1
fi

# Show current backups
echo ">> Current backups list"
gcloud sql backups list -i "${sql_instance}" --project "${project}" "${gcloud_format[@]}"

# Create new backup
echo ">> Creating backup with description scripted-${epoch_date}"
gcloud sql backups create --instance="${sql_instance}" --description scripted-"${epoch_date}" --project "$project"

# Verify backup created
echo ">> New backups list"
gcloud sql backups list -i "${sql_instance}" --project "$project" "${gcloud_format[@]}"

echo ">> FINISHED | Note: please delete ON_DEMAND backup manually"
```

--------------------------------

### Configure Backend Module with Terragrunt

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Sets up database connections and Config Connector permissions. Depends on the infrastructure module for existing cluster resources.

```hcl
# terragrunt/concourse-wg-ci/backend/terragrunt.hcl
dependencies {
  paths = ["../infra"]
}

locals {
  config = yamldecode(file("../config.yaml"))
}

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "${local.config.gcs_bucket}"
    prefix         = "${local.config.gcs_prefix}/concourse-backend"
    project        = "${local.config.project}"
    location       = "${local.config.region}"
    enable_bucket_policy_only = false
  }
}

terraform {
  source = local.config.tf_modules.backend
}

inputs = {
  project = local.config.project
  region  = local.config.region
  zone    = local.config.zone
  gke_name = local.config.gke_name
  sql_instance_name = "${local.config.gke_name}-concourse"
  wg_ci_cnrm_service_account_permissions = local.config.wg_ci_cnrm_service_account_permissions
}
```

--------------------------------

### Configure Infrastructure Module with Terragrunt

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Defines core GCP resources including GKE, Cloud SQL, and networking. Requires a local config.yaml file for environment-specific variables.

```hcl
# terragrunt/concourse-wg-ci/infra/terragrunt.hcl
locals {
  config = yamldecode(file("../config.yaml"))
}

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "${local.config.gcs_bucket}"
    prefix         = "${local.config.gcs_prefix}/concourse-infra"
    project        = "${local.config.project}"
    location       = "${local.config.region}"
    enable_bucket_policy_only = true
  }
}

terraform {
  source = local.config.tf_modules.infra
}

inputs = {
  project = local.config.project
  region  = local.config.region
  zone    = local.config.zone

  # GKE Configuration
  gke_name = local.config.gke_name
  gke_deletion_protection = local.config.gke_deletion_protection
  gke_controlplane_version = local.config.gke_controlplane_version
  gke_cluster_ipv4_cidr = local.config.gke_cluster_ipv4_cidr
  gke_services_ipv4_cidr_block = local.config.gke_services_ipv4_cidr_block

  # Worker Pool Configuration
  gke_workers_pool_machine_type = local.config.gke_workers_pool_machine_type
  gke_workers_pool_node_count = local.config.gke_workers_pool_node_count

  # Cloud SQL Configuration
  database_version = local.config.database_version
  sql_instance_name = "${local.config.gke_name}-concourse"
  sql_instance_tier = local.config.sql_instance_tier

  # DNS Configuration
  dns_record = local.config.dns_record
  dns_zone = local.config.dns_zone
  dns_domain = local.config.dns_domain
}
```

--------------------------------

### Central Configuration File for Environment Settings

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Defines all environment-specific settings for GCP project, GKE cluster, database, and Concourse. This YAML file serves as the single source of truth for configuration.

```yaml
# terragrunt/concourse-wg-ci/config.yaml
project: cf-deployments-wg
region: us-east1
zone: us-east1-b
secondary_zone: us-east1-c

# GCS State Backend
gcs_bucket: terraform-ard-wg-ci
gcs_prefix: concourse

# DNS Configuration
dns_record: concourse
dns_zone: wg-ard
dns_domain: ci.cloudfoundry.org

# GKE Cluster
gke_name: wg-ci
gke_deletion_protection: true
gke_controlplane_version: "1.32.6-gke.1013000"
gke_cluster_ipv4_cidr: 10.104.0.0/14
gke_services_ipv4_cidr_block: 10.108.0.0/20
gke_subnet_cidr: 10.10.0.0/24

# Default Pool
gke_default_pool_machine_type: e2-standard-4
gke_default_pool_node_count: 1
gke_default_pool_autoscaling_max: 3

# Worker Pool (scale to 16 for major stemcell releases)
gke_workers_pool_machine_type: n2-standard-4
gke_workers_pool_node_count: 6
gke_workers_pool_autoscaling_max: 6
gke_workers_pool_ssd_count: 1
gke_workers_min_memory: 4Gi
gke_workers_max_memory: 12Gi

# Cloud SQL
database_version: "POSTGRES_16"
sql_instance_tier: db-custom-1-4096
sql_instance_disk_size: 38

# Concourse
concourse_helm_version: "18.1.1"
concourse_github_mainTeam: "cloudfoundry:wg-app-runtime-deployments"
concourse_max_days_to_retain_build_logs: 30
concourse_max_build_logs_to_retain: 200

# Terraform Module Sources
tf_modules:
  infra: "git@github.com:cloudfoundry/app-runtime-interfaces-infrastructure.git//terraform-modules/concourse/infra?ref=1.12.0"
  backend: "git@github.com:cloudfoundry/app-runtime-interfaces-infrastructure.git//terraform-modules/concourse/backend?ref=1.12.0"
  app: "git@github.com:cloudfoundry/app-runtime-interfaces-infrastructure.git//terraform-modules/concourse/app?ref=1.12.0"

```

--------------------------------

### Execute Disaster Recovery Restore

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Orchestrates a full system recovery by applying Terragrunt modules in a specific sequence. Ensures infrastructure, backend, CredHub, and application components are restored correctly.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/dr-restore.sh
set -euo pipefail

if [ ! -s "./config.yaml" ]; then
    echo "ERROR: Please 'cd' to your folder with config.yaml and run this script again."
    exit 1
fi

echo ">> Executing DR restore. You will be asked to confirm changes to apply."

echo "[1/5] Terragrunt apply for infra only"
( cd ./infra && terragrunt apply --terragrunt-source-update )

echo "[2/5] Terragrunt apply for backend only"
( cd ./backend && terragrunt apply --terragrunt-source-update )

echo "[3/5] Carvel might not learn new state during recovery - retrigger"
( cd ./backend && terragrunt apply --terragrunt-source-update )

echo "[4/5] Restore CredHub encryption key and populate SQL user passwords from secretgen"
( cd ./dr_restore && terragrunt apply --terragrunt-config=credhub_sql_passwords.hcl --terragrunt-source-update )

echo "[5/5] Terragrunt apply for app only"
( cd ./app && terragrunt apply --terragrunt-source-update )

echo "-- DR recovery completed"
```

--------------------------------

### Recreate CredHub certificates

Source: https://github.com/cloudfoundry/app-runtime-deployments-infrastructure/blob/main/README.md

Commands to authenticate with GCP, access the cluster, and restart the CredHub pod to trigger certificate recreation.

```bash
gcloud auth login
gcloud config set project cf-deployments-wg
gcloud container clusters get-credentials wg-ci --region us-east1-b
kubectl cluster-info
kubectl -n concourse get pods # search credhub pod
kubectl -n concourse delete pod credhub-abc-xyz
```

--------------------------------

### Create GitHub OAuth Secret in GCP

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Automates the creation of a GCP Secret Manager entry for Concourse GitHub OAuth credentials. Ensure your GitHub OAuth app credentials and config.yaml are present before execution.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/create-github-oauth-gcp.sh
set -euo pipefail

# Enter your GitHub OAuth credentials
id="your_github_oauth_client_id"
secret="your_github_oauth_client_secret"

if [ -z "${id}" ] || [ -z "${secret}" ]; then
  echo "ERROR: Please enter your credentials on the top of this script"
  exit 1
fi

if [ ! -s "./config.yaml" ]; then
    echo "ERROR: Please 'cd' to your folder with config.yaml and run this script again."
    exit 1
fi

secret_id="$(yq .gke_name config.yaml)-concourse-github-oauth"
secret_region="$(yq .region config.yaml)"
project="$(yq .project config.yaml)"

# Create the secret in GCP Secret Manager
gcloud secrets create "${secret_id}" \
 --replication-policy=\"user-managed\" \
 --locations="${secret_region}" \
 --project="${project}"

# Add the secret version with OAuth credentials
printf "id: %s\nsecret: %s" ${id} ${secret} | \
gcloud secrets versions add "${secret_id}" --data-file=- --project="${project}"
```

--------------------------------

### Rotate PostgreSQL Secrets

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Rotates database passwords by deleting Kubernetes secrets, triggering secret regeneration, and syncing changes to Cloud SQL. Includes safety checks for non-test environments.

```bash
#!/usr/bin/env bash
# terragrunt/scripts/concourse/secret_rotation_postgresql.sh
set -euo pipefail

if [ ! -s "./config.yaml" ]; then
    echo "ERROR: Please 'cd' to your folder with config.yaml and run this script again."
    exit 1
fi

tg_secret_rotation_params=( --terragrunt-config=rotate.hcl --terragrunt-source-update --auto-approve)

project="$(yq .project config.yaml)"
zone="$(yq .zone config.yaml)"
gke_name="$(yq .gke_name config.yaml)"

secrets=(concourse-postgresql-password credhub-postgresql-password uaa-postgresql-password)
deployments=(concourse-web concourse-worker credhub uaa)

# Safety check for non-test environments
if [[ ! "$gke_name" =~ .*test*. ]]; then
  read -p "Detected a non-test environment. Confirm with 'yes': " -r
  if [[ ! "$REPLY" == "yes" ]]; then
      echo "Canceling"
      exit 1
  fi
fi

# Get kubectl credentials
gcloud container clusters get-credentials ${gke_name} --zone ${zone} --project ${project}

# Delete existing secrets
echo ">> Deleting existing postgresql secrets"
for secret in "${secrets[@]}"; do
  kubectl delete secret -n concourse $secret || true
done

# Restart secretgen controller to regenerate secrets
echo ">> Restarting secretgen controller"
kubectl scale deploy -n secretgen-controller secretgen-controller --replicas=0
kubectl scale deploy -n secretgen-controller secretgen-controller --replicas=1
kubectl wait deployment -n secretgen-controller secretgen-controller --for=jsonpath='{.spec.replicas}'=1 --timeout=30s

# Wait for secrets to be recreated
for secret in "${secrets[@]}"; do
  while ! kubectl get secret -n concourse $secret; do
    echo "Waiting for secret: $secret"
    sleep 5
  done
done

# Sync to Cloud SQL
echo ">> Apply terragrunt to sync kubernetes secrets with CloudSQL Users"
( cd ./secret_rotation_postgresql && terragrunt apply "${tg_secret_rotation_params[@]}" )

# Restart deployments
for deployment in "${deployments[@]}"; do
  kubectl scale deploy -n concourse "${deployment}" --replicas=0
done

for deployment in "${deployments[@]}"; do
  kubectl scale deploy -n concourse "${deployment}" --replicas=1
done

kubectl -n concourse wait --for=condition=available --timeout=200s \
  deployment/concourse-web deployment/concourse-worker deployment/credhub deployment/uaa
```

--------------------------------

### Terragrunt Configuration for App Deployment

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Defines dependencies, local configuration, and Terraform module inputs for deploying the Concourse CI application. It reads configuration from a local YAML file and depends on infra and backend modules.

```hcl
dependencies {
  paths = ["../infra", "../backend"]
}

dependency "infra" {
  config_path = "../infra"
}

locals {
  config = yamldecode(file("../config.yaml"))
}

terraform {
  source = local.config.tf_modules.app
}

inputs = {
  project = local.config.project
  region  = local.config.region
  zone    = local.config.zone

  concourse_helm_version = local.config.concourse_helm_version
  gke_name = local.config.gke_name

  # Worker configuration
  gke_workers_pool_machine_type = local.config.gke_workers_pool_machine_type
  gke_workers_pool_node_count = local.config.gke_workers_pool_node_count
  gke_workers_min_memory = local.config.gke_workers_min_memory
  gke_workers_max_memory = local.config.gke_workers_max_memory

  # Load balancer from infra module
  load_balancer_ip = dependency.infra.outputs.load_balancer_ip
  load_balancer_dns = dependency.infra.outputs.load_balancer_dns

  # GitHub OAuth
  concourse_github_mainTeam = local.config.concourse_github_mainTeam
  concourse_container_placement_strategy = local.config.concourse_container_placement_strategy
  concourse_max_days_to_retain_build_logs = local.config.concourse_max_days_to_retain_build_logs
  concourse_max_build_logs_to_retain = local.config.concourse_max_build_logs_to_retain
}
```

--------------------------------

### Troubleshoot Expired CredHub Certificates

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Provides steps to troubleshoot and resolve x509 certificate errors in Concourse jobs caused by expired CredHub certificates. It includes authentication, cluster configuration, and pod deletion for regeneration.

```bash
# Troubleshoot expired CredHub certificates
# Error: x509: certificate has expired or is not yet valid

# Authenticate and configure kubectl
gcloud auth login
gcloud config set project cf-deployments-wg
gcloud container clusters get-credentials wg-ci --region us-east1-b
kubectl cluster-info

# Find and delete the CredHub pod to regenerate certificates
kubectl -n concourse get pods | grep credhub
kubectl -n concourse delete pod credhub-abc-xyz  # Replace with actual pod name
```

--------------------------------

### Configure Automatic CredHub Certificate Regeneration

Source: https://context7.com/cloudfoundry/app-runtime-deployments-infrastructure/llms.txt

Configures automatic monthly regeneration of CredHub certificates using Terragrunt and Terraform. It targets specific certificate paths for Cloud Foundry environment load balancers and uses a GCS backend for remote state.

```hcl
# terragrunt/concourse-wg-ci/automatic_certificate_regeneration/cert_regen.hcl
locals {
  config = yamldecode(file("../config.yaml"))
}

remote_state {
  backend = "gcs"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "${local.config.gcs_bucket}"
    prefix         = "${local.config.gcs_prefix}/automatic-certificate-regeneration"
    project        = "${local.config.project}"
    location       = "${local.config.region}"
    enable_bucket_policy_only = false
  }
}

terraform {
  source = local.config.tf_modules.automatic_certificate_regeneration
}

inputs = {
  project = local.config.project
  region  = local.config.region
  zone    = local.config.zone
  gke_name = local.config.gke_name

  # Certificates to automatically regenerate monthly
  certificates_to_regenerate = local.config.certificates_to_regenerate
  # Example: "/concourse/main/luna_lb,/concourse/main/hermione_lb,/concourse/main/snape_lb"
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.