### Clone Vault mTLS Kafka Example Source: https://developer.hashicorp.com/validated-patterns/vault/vault-securing-kafka Download the example code repository for the Vault mTLS Kafka pattern. This repository contains all necessary configurations and scripts. ```bash $ git clone https://github.com/hashicorp-validated-designs/vault-mtls-kafka ``` -------------------------------- ### Sample Ansible Playbook for Image Provisioning Source: https://developer.hashicorp.com/validated-patterns/packer/integrate-hcp-packer-with-ansible-automation-platform A basic Ansible playbook to install the Apache HTTP server on the image being built. ```yaml --- - name: 'Provision Image' hosts: default become: true tasks: - name: install Apache package: name: 'httpd' state: present ``` -------------------------------- ### Chef Recipe: Install Vault Agent and Authenticate Source: https://developer.hashicorp.com/validated-patterns/vault/vault-secrets-queried-with-chef This Chef recipe installs the Vault Agent and configures it to authenticate with Vault. It then uses the authenticated token to retrieve a secret. ```ruby #install Vault Agent and create template file include_recipe "vault-chef::vault-agent" # Execute the Vault agent command to authenticate and exit after auth execute 'vault agent' do command 'vault agent -config=/tmp/vault-agent.hcl -exit-after-auth' action :run notifies :read, 'vault_secret[secret]', :immediately # Notify the vault_secret block end # Use Chef's built in secret method, connect to Vault, and write to a file file "/tmp/vault-secret" do content lazy { secret(name: "secret/data/#{node['vault']['secret-key']}", service: :hashi_vault, config: { vault_addr: node['vault']['address'], namespace: node['vault']['namespace'], auth_method: :token, token: File.read("/tmp/vault-token-via-agent") })[:data].to_s } end # Before Chef 17.5, the secret method did not exist. Here is a custom resource that can be used # vault_secret calls the custom resource in resources/vault_secret.rb # Read the KV secret from Vault using the token generated by the agent vault_secret 'secret' do vault_address node['vault']['address'] vault_namespace node['vault']['namespace'] token lazy { # Lazy evaluation ensures the token file is read at execution time File.exist?("/tmp/vault-token-via-agent") ? File.read("/tmp/vault-token-via-agent") : "Didn't get the token from the agent!" } secret_key node['vault']['secret-key'] action :nothing # Triggered only by notification from the execute block end # Use the secret in your recipe ruby_block 'print_secret' do block do secret_value = node.run_state['vault_secret'] puts "\n\nThe secret value is: #{secret_value}" end end ``` ```ruby #install Vault Agent and create template file include_recipe "vault-chef::vault-agent" # Execute the Vault agent command to authenticate and exit after auth execute 'vault agent' do command 'vault agent -config=/tmp/vault-agent.hcl -exit-after-auth' action :run notifies :read, 'vault_secret[secret]', :immediately # Notify the vault_secret block end ``` -------------------------------- ### Clone Root Policies Setup Repository Source: https://developer.hashicorp.com/validated-patterns/vault/manage-vault-with-terraform Clone your specific repository for setting up root policies. This is a crucial step before applying standard policies to the Vault root namespace. ```bash $ git clone YOUR_ROOT_POLICIES_REPOSITORY ``` -------------------------------- ### Backstage JWT Log Example Source: https://developer.hashicorp.com/validated-patterns/vault/integrate-vault-with-backstage-and-okta An example of a Backstage log entry containing an idToken, which is the JWT payload from Okta. ```log [backend]: idToken: 'eyJraWQiOiJOSXVYaU81dkV4LT...' ``` -------------------------------- ### Initialize Terraform with Provider Upgrade Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-and-refactor-terraform-modules Run `terraform init -upgrade` to install the new module version and update provider dependencies. Review the changes to `.terraform.lock.hcl` before committing. ```bash $ terraform init -upgrade Initializing the backend... Upgrading modules... Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.0.0 for s3_bucket... - s3_bucket in .terraform/modules/s3_bucket Initializing provider plugins... - Finding hashicorp/aws versions matching ">= 3.75.0, 3.75.0"... - Installing hashicorp/aws v3.75.0... - Installed hashicorp/aws v3.75.0 (signed by HashiCorp) Terraform has made some changes to the provider dependency selections recorded in the .terraform.lock.hcl file. Review those changes and commit them to your version control system if they represent changes you intended to make. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` ```bash $ terraform init -upgrade Initializing the backend... Upgrading modules... Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.0.0 for s3_bucket... - s3_bucket in .terraform/modules/s3_bucket Initializing provider plugins... - Finding hashicorp/aws versions matching ">= 3.75.0, 3.75.0"... - Installing hashicorp/aws v3.75.0... - Installed hashicorp/aws v3.75.0 (signed by HashiCorp) Terraform has made some changes to the provider dependency selections recorded in the .terraform.lock.hcl file. Review those changes and commit them to your version control system if they represent changes you intended to make. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` -------------------------------- ### Initialize Terraform Configuration Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-terraform-provider Run this command to download and install the specified AWS provider version and create a lock file. This ensures consistent provider versions across runs. ```bash $ terraform init Initializing the backend... Initializing provider plugins... - Finding hashicorp/aws versions matching "3.74.3"... - Installing hashicorp/aws v3.74.3... - Installed hashicorp/aws v3.74.3 (signed by HashiCorp) Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` -------------------------------- ### Extract and Install teectl Source: https://developer.hashicorp.com/validated-patterns/nomad/nomad-workload-modernization-with-traefik Extract the downloaded tarball, move the teectl binary to /usr/local/bin, and set appropriate permissions for execution. This makes the tool available system-wide. ```bash $ tar -xzf teectl_v2.11.6_linux_amd64.tar.gz teectl && sudo mv teectl /usr/local/bin/teectl && sudo chown root: /usr/local/bin/teectl && sudo chmod +x /usr/local/bin/teectl ``` ```bash $ tar -xzf teectl_v2.11.6_linux_amd64.tar.gz teectl && sudo mv teectl /usr/local/bin/teectl && sudo chown root: /usr/local/bin/teectl && sudo chmod +x /usr/local/bin/teectl ``` -------------------------------- ### Example Pull Request Description for Terraform Module Upgrade Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-and-refactor-terraform-modules This is an example pull request description detailing the upgrade of an AWS S3 bucket module and the AWS provider. It explains the reasons for the changes, including a bug fix, and requests a review. ```markdown This PR upgrades the AWS S3 bucket module version from v2.15.0 to v3.0.0. As part of the upgrade, the AWS provider major version was also upgraded to meet the updated module requirements. Normally, I wouldn't introduce this much change when upgrading a module, but I've found a [bug](https://github.com/hashicorp/terraform-provider-aws/issues/28701) in this provider version that required bumping major versions. Please review for accuracy and adherence to best practices. ``` -------------------------------- ### Run Vault Agent and Launch Kafka Source: https://developer.hashicorp.com/validated-patterns/vault/vault-securing-kafka Start Vault Agent to obtain certificates and launch the Kafka broker and topics. Vault Agent will manage certificate renewal and Kafka restarts. ```bash $ make agent_kafka_and_topics ``` -------------------------------- ### Vault Benchmark Results Source: https://developer.hashicorp.com/validated-patterns/vault/vault-migrate-storage-raft Example output from the vault-benchmark tool, displaying performance metrics for different operations. This table helps in analyzing the throughput, latency, and success ratio of the benchmarked Vault cluster. ```text | op | count | rate | throughput | mean | 95th% | 99th% | successRatio | |----------------------|-------|-----------|------------|-------------|-------------|--------------|--------------| | approle_logins | 2619 | 87.350307 | 87.195473 | 61.307741ms | 78.387069ms | 94.788229ms | 100.00% | | static_secret_writes | 2536 | 84.530378 | 84.369422 | 55.059506ms | 82.072336ms | 109.441173ms | 100.00% | ``` ```text | op | count | rate | throughput | mean | 95th% | 99th% | successRatio | |----------------------|-------|-----------|------------|-------------|-------------|--------------|--------------| | approle_logins | 2619 | 87.350307 | 87.195473 | 61.307741ms | 78.387069ms | 94.788229ms | 100.00% | | static_secret_writes | 2536 | 84.530378 | 84.369422 | 55.059506ms | 82.072336ms | 109.441173ms | 100.00% | ``` -------------------------------- ### Terraform Codebase Structure Overview Source: https://developer.hashicorp.com/validated-patterns/terraform/terraform-vault-backed-dynamic-credentials-aws This overview details the directory structure of the repository, highlighting modules for provisioning prerequisites, onboarding TFE/HCP Terraform projects and workspaces, and examples for provider configurations. ```plaintext ├── module-provisioning-examples │ ├── 1. prerequisites │ ├── 2. aws-account-onboarding │ └── 3. tfe-project-workspace-onboarding ├── modules │ ├── terraform-aws-vault-dynamic-secrets-prereqs │ ├── terraform-tfe-vault-dynamic-creds-onboarding │ ├── terraform-tfe-vault-dynamic-creds-variables │ ├── terraform-vault-aws-account-onboarding │ └── terraform-vault-jwt-tfe-identity ├── provider-usage-examples │ ├── multi-aws-account-provider-configs │ └── single-aws-account-provider-configs ``` ```plaintext ├── module-provisioning-examples │ ├── 1. prerequisites │ ├── 2. aws-account-onboarding │ └── 3. tfe-project-workspace-onboarding ├── modules │ ├── terraform-aws-vault-dynamic-secrets-prereqs │ ├── terraform-tfe-vault-dynamic-creds-onboarding │ ├── terraform-tfe-vault-dynamic-creds-variables │ ├── terraform-vault-aws-account-onboarding │ └── terraform-vault-jwt-tfe-identity ├── provider-usage-examples │ ├── multi-aws-account-provider-configs │ └── single-aws-account-provider-configs ``` -------------------------------- ### Initialize Terraform with Upgrade Flag Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-and-refactor-terraform-modules Command to re-initialize Terraform configuration, forcing an upgrade of modules and providers to their latest compatible versions. This example shows a potential error related to provider version constraints. ```bash $ terraform init -upgrade Initializing the backend... Upgrading modules... Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.0.0 for s3_bucket... - s3_bucket in .terraform/modules/s3_bucket Initializing provider plugins... - Finding hashicorp/aws versions matching "3.69.0, >= 3.75.0"... ⌂ │ Error: Failed to query available provider packages │ │ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints 3.69.0, >= 3.75.0 │ │ To see which modules are currently depending on hashicorp/aws and what versions are specified, run the following command: │ terraform providers ⌃ ``` ```bash $ terraform init -upgrade Initializing the backend... Upgrading modules... Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.0.0 for s3_bucket... - s3_bucket in .terraform/modules/s3_bucket Initializing provider plugins... - Finding hashicorp/aws versions matching "3.69.0, >= 3.75.0"... ⌂ │ Error: Failed to query available provider packages │ │ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints 3.69.0, >= 3.75.0 │ │ To see which modules are currently depending on hashicorp/aws and what versions are specified, run the following command: │ terraform providers ⌃ ``` -------------------------------- ### GitHub Actions Workflow to Get and Inspect JWT Source: https://developer.hashicorp.com/validated-patterns/vault/retrieve-vault-secrets-from-github-actions This workflow demonstrates how to obtain a JWT from the GitHub OIDC provider and then log its base64 encoded value. This is useful for debugging and understanding the token's structure during initial setup. ```yaml name: inspect-jwt on: push: branches: - main paths-ignore: - 'terraform/**' - '*.md' jobs: token-job: permissions: contents: 'read' id-token: 'write' runs-on: self-hosted name: Run Step 1 steps: - name: Get JWT id: auth-token run: | export TOKEN=$(curl -sSL -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL") echo $TOKEN | jq .value | base64 ``` ```yaml name: inspect-jwt on: push: branches: - main paths-ignore: - 'terraform/**' - '*.md' jobs: token-job: permissions: contents: 'read' id-token: 'write' runs-on: self-hosted name: Run Step 1 steps: - name: Get JWT id: auth-token run: | export TOKEN=$(curl -sSL -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL") echo $TOKEN | jq .value | base64 ``` -------------------------------- ### Initialize Terraform Configuration Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-and-refactor-terraform-modules Run 'terraform init' to download provider plugins and modules. This command sets up your working directory for Terraform operations. ```bash $ terraform init Initializing the backend... Initializing modules... Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 2.15.0 for s3_bucket... - s3_bucket in .terraform/modules/s3_bucket Initializing provider plugins... - Finding hashicorp/aws versions matching "3.69.0, ~> 3.69"... - Installing hashicorp/aws v3.69.0... - Installed hashicorp/aws v3.69.0 (signed by HashiCorp) Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ``` -------------------------------- ### Verify Vault Secrets Operator Installation Source: https://developer.hashicorp.com/validated-patterns/vault/vault-kubernetes-auth Check the status of pods and services in the Vault Secrets Operator system namespace to confirm a successful installation. ```bash kubectl get pods,svc -n vault-secrets-operator-system ``` -------------------------------- ### Configure Backstage with Static Key Store Source: https://developer.hashicorp.com/validated-patterns/vault/integrate-vault-with-backstage-and-okta Add the paths to your generated public and private key files in the `app-config.yaml` file to configure Backstage's static key store. The first key declared will be used for signing. ```yaml auth: keyStore: provider: static static: keys: # Must be declared at least once and the first one will be used for signing - keyId: primary publicKeyFile: /path/to/public.key privateKeyFile: /path/to/private.key ``` ```yaml auth: keyStore: provider: static static: keys: # Must be declared at least once and the first one will be used for signing - keyId: primary publicKeyFile: /path/to/public.key privateKeyFile: /path/to/private.key ``` -------------------------------- ### Monitor Vault Secrets Operator Installation Status Source: https://developer.hashicorp.com/validated-patterns/vault/vault-pki-with-vso-and-openshift Monitor the ClusterServiceVersion (CSV) resource for a 'Succeeded' status to confirm installation completion. Also, check that the VSO pods are running. ```bash oc get csv -n openshift-operators oc get pods -n openshift-operators ``` -------------------------------- ### Create Vault Agent User and Directories Source: https://developer.hashicorp.com/validated-patterns/vault/vault-agent-approle Sets up the necessary user and directories for the Vault Agent. Ensures correct ownership and permissions for secure operation. ```bash #!/bin/bash # Create vault-agent user if it does not exist id vault-agent &>/dev/null || useradd --system --home /opt/vault-agent --shell /bin/false vault-agent # Create required directories with proper permissions for DIR in /opt/vault-agent /opt/vault-agent/auth /opt/vault-agent/config /opt/vault-agent/secrets /opt/vault-agent/templates /var/log/vault-agent; do mkdir -p "$DIR" chown vault-agent:vault-agent "$DIR" chmod 0755 "$DIR" done ``` ```bash #!/bin/bash # Create vault-agent user if it does not exist id vault-agent &>/dev/null || useradd --system --home /opt/vault-agent --shell /bin/false vault-agent # Create required directories with proper permissions for DIR in /opt/vault-agent /opt/vault-agent/auth /opt/vault-agent/config /opt/vault-agent/secrets /opt/vault-agent/templates /var/log/vault-agent; do mkdir -p "$DIR" chown vault-agent:vault-agent "$DIR" chmod 0755 "$DIR" done ``` -------------------------------- ### Install Vault Secrets Operator with Helm Source: https://developer.hashicorp.com/validated-patterns/vault/vault-kubernetes-auth Installs the Vault Secrets Operator using Helm, applying the custom configuration values. This makes Vault secrets available as Kubernetes secrets. ```bash $ helm install vault-secrets-operator hashicorp/vault-secrets-operator \ --namespace vault-secrets-operator-system \ --create-namespace \ --values vault-operator-values.yaml ``` -------------------------------- ### Approve Vault Secrets Operator Install Plan Source: https://developer.hashicorp.com/validated-patterns/vault/vault-pki-with-vso-and-openshift Manually approve the installation plan to deploy VSO. List pending plans, describe the relevant one for VSO, and then patch it to set 'approved' to true. ```bash oc get installplans -n openshift-operators oc describe installplan install- -n openshift-operators oc patch installplan install- -n openshift-operators \ --type merge --patch '{"spec":{"approved":true}}' ``` -------------------------------- ### Run Vault Benchmark Source: https://developer.hashicorp.com/validated-patterns/vault/vault-migrate-storage-raft Execute the vault-benchmark binary with a specified configuration file to establish a performance baseline for Vault clusters. ```bash $ vault-benchmark run -config=config_cluster_b.hcl ``` -------------------------------- ### Example Output Structure of Vault AWS Roles Source: https://developer.hashicorp.com/validated-patterns/terraform/terraform-vault-backed-dynamic-credentials-aws This example shows the expected structure of the `vault_aws_roles_01` output, detailing the mapping between Terraform-friendly role names and their corresponding AWS IAM role ARNs and Vault role identifiers. ```json vault_aws_roles_01 = { "123456789012_TFC_VAULT_BACKED_AWS_RUN_VAULT_ROLE_DB_ADMIN" = { "aws_run_role_arn" = "arn:aws:iam::123456789012:role/vault-service-role" "tag_suffix" = "123456789012_DB_ADMIN" "vault_role" = "123456789012-dynamodb-admin" } "123456789012_TFC_VAULT_BACKED_AWS_RUN_VAULT_ROLE_EC2_PROV" = { "aws_run_role_arn" = "arn:aws:iam::123456789012:role/vault-service-role" "tag_suffix" = "123456789012_EC2_PROV" "vault_role" = "123456789012-ec2-instance-provisioner" } "123456789012_TFC_VAULT_BACKED_AWS_RUN_VAULT_ROLE_S3_ADMIN" = { "aws_run_role_arn" = "arn:aws:iam::123456789012:role/vault-service-role" "tag_suffix" = "123456789012_S3_ADMIN" "vault_role" = "123456789012-s3-full-access" } } ``` -------------------------------- ### Deploy Application and Stream Logs Source: https://developer.hashicorp.com/validated-patterns/vault/vault-kubernetes-auth Deploy the application using the specified YAML file and stream its logs to verify the secret mounting process. ```bash $ kubectl apply -f csi-deploy.yaml -n app-1 $ kubectl logs deploy/my-app -n app-1 --follow ``` -------------------------------- ### Pull Request Description Example Source: https://developer.hashicorp.com/validated-patterns/terraform/upgrade-terraform-provider This example pull request description details the refactoring of Terraform code to address deprecated attributes and provider versioning issues. It highlights the addition of new resources, the use of import blocks, and a lifecycle rule to mitigate a known bug. ```markdown This PR refactors the Terraform code to replace the deprecated `versioning` block in the `aws_s3_bucket` resource with the recommended `aws_s3_bucket_versioning` resource. While testing, I have found an additional deprecation warning that was unexpected. To address these warnings, two resources were added to align with the latest AWS provider recommendations. These resources will be imported during the next `terraform apply` to ensure there are no unexpected errors. I've also found a [bug](https://github.com/hashicorp/terraform-provider-aws/issues/28701) in this provider version so a `lifecycle` block has been added to ignore misleading changes in the plan output. Please review for accuracy and adherence to best practices. ``` -------------------------------- ### Run Vault Benchmark Source: https://developer.hashicorp.com/validated-patterns/vault/vault-migrate-storage-raft Execute the vault-benchmark tool using the specified configuration file. This command initiates the performance testing against Vault cluster A. ```bash $ vault-benchmark run -config=config_cluster_a.hcl ``` ```bash $ vault-benchmark run -config=config_cluster_a.hcl ``` -------------------------------- ### Prepare Kubernetes Service Account Source: https://developer.hashicorp.com/validated-patterns/vault/vault-pki-with-vso-and-openshift Create a new OpenShift project and define a Kubernetes Service Account for VSO to use for authentication. ```bash oc new-project app-1 \ --display-name="Application 1" \ --description="My example application" cat < ``` ```bash Warning: Deprecated modules found, consider installing an updating version. The following are affected: Version X.X.X of ``` -------------------------------- ### HCP Vault JWT Authentication Request with Correlation ID Source: https://developer.hashicorp.com/validated-patterns/vault/ai-agent-identity-with-hashicorp-vault Example request payload for HCP Vault JWT authentication, demonstrating the inclusion of the X-Correlation-ID header. ```json "request": { "headers": { "x-correlation-id": [ "45df07b9-bd1d-43b6-897a-3e743c9565fb-Bob" ] }, "mount_point": "admin/auth/jwt/", "path": "auth/jwt/login", "remote_address": "42.61.153.210", ... ``` -------------------------------- ### Verify Served Certificate via OpenSSL Source: https://developer.hashicorp.com/validated-patterns/vault/vault-pki-with-vso-and-openshift Use `openssl s_client` to connect to the application's route and `openssl x509` to display the served certificate's subject and issuer, confirming it matches the expected values. ```bash openssl s_client -connect foo.tenant-1.example.com:443 -servername foo.tenant-1.example.com /dev/null | openssl x509 -noout -subject -issuer ``` -------------------------------- ### MCP Server Log Entry with Correlation ID Source: https://developer.hashicorp.com/validated-patterns/vault/ai-agent-identity-with-hashicorp-vault Example log entry from the MCP server showing a product fetching operation, including the correlation ID. ```log 2025-09-02 08:22:01,390 - INFO - product_service: 45df07b9-bd1d-43b6-897a-3e743c9565fb-Bob - Fetching products ```