### Install the YAML package Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/gopkg.in/yaml.v2/README.md Use the go get command to download the package into your Go environment. ```bash go get gopkg.in/yaml.v2 ``` -------------------------------- ### Install Doublestar Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Install the doublestar package using the go get command. ```bash go get github.com/bmatcuk/doublestar ``` -------------------------------- ### Create Root Logger Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/go-logr/logr/README.md Initialize the root logger early in an application's lifecycle. This example uses a hypothetical 'logimpl' implementation. ```go func main() { // ... other setup code ... // Create the "root" logger. We have chosen the "logimpl" implementation, // which takes some initial parameters and returns a logr.Logger. logger := logimpl.New(param1, param2) // ... other setup code ... } ``` -------------------------------- ### Install Bundler and Build OS Image Inside Container Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Install Bundler and then build the OS image within the Docker container. This command uses a rake task and requires specifying the OS, codename, and output path for the image. ```bash gem install bundler bundle install # Build OS image bundle exec rake stemcell:build_os_image[ubuntu,jammy,${PWD}/tmp/ubuntu_base_image.tgz] ``` -------------------------------- ### Convert klog format strings to structured logr Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/go-logr/logr/README.md Examples showing the transformation of klog.Infof calls into structured logger methods. ```go klog.V(4).Infof("Client is returning errors: code %v, error %v", responseCode, err) ``` ```go logger.Error(err, "client returned an error", "code", responseCode) ``` ```go klog.V(4).Infof("Got a Retry-After %ds response for attempt %d to %v", seconds, retries, url) ``` ```go logger.V(4).Info("got a retry-after response when requesting url", "attempt", retries, "after seconds", seconds, "url", url) ``` -------------------------------- ### Ginkgo Test Suite for Library Operations Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/onsi/ginkgo/v2/README.md This example demonstrates a comprehensive test suite for a library system using Ginkgo and Gomega. It covers scenarios like checking out books, handling unavailable books, and managing holds. Ensure Ginkgo and Gomega are imported. ```go import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" ... ) var _ = Describe("Checking books out of the library", Label("library"), func() { var library *libraries.Library var book *books.Book var valjean *users.User BeforeEach(func() { library = libraries.NewClient() book = &books.Book{ Title: "Les Miserables", Author: "Victor Hugo", } valjean = users.NewUser("Jean Valjean") }) When("the library has the book in question", func() { BeforeEach(func(ctx SpecContext) { Expect(library.Store(ctx, book)).To(Succeed()) }) Context("and the book is available", func() { It("lends it to the reader", func(ctx SpecContext) { Expect(valjean.Checkout(ctx, library, "Les Miserables")).To(Succeed()) Expect(valjean.Books()).To(ContainElement(book)) Expect(library.UserWithBook(ctx, book)).To(Equal(valjean)) }, SpecTimeout(time.Second * 5)) }) Context("but the book has already been checked out", func() { var javert *users.User BeforeEach(func(ctx SpecContext) { javert = users.NewUser("Javert") Expect(javert.Checkout(ctx, library, "Les Miserables")).To(Succeed()) }) It("tells the user", func(ctx SpecContext) { err := valjean.Checkout(ctx, library, "Les Miserables") Expect(err).To(MatchError("Les Miserables is currently checked out")) }, SpecTimeout(time.Second * 5)) It("lets the user place a hold and get notified later", func(ctx SpecContext) { Expect(valjean.Hold(ctx, library, "Les Miserables")).To(Succeed()) Expect(valjean.Holds(ctx)).To(ContainElement(book)) By("when Javert returns the book") Expect(javert.Return(ctx, library, book)).To(Succeed()) By("it eventually informs Valjean") notification := "Les Miserables is ready for pick up" Eventually(ctx, valjean.Notifications).Should(ContainElement(notification)) Expect(valjean.Checkout(ctx, library, "Les Miserables")).To(Succeed()) Expect(valjean.Books(ctx)).To(ContainElement(book)) Expect(valjean.Holds(ctx)).To(BeEmpty()) }, SpecTimeout(time.Second * 10)) }) }) When("the library does not have the book in question", func() { It("tells the reader the book is unavailable", func(ctx SpecContext) { err := valjean.Checkout(ctx, library, "Les Miserables") Expect(err).To(MatchError("Les Miserables is not in the library catalog")) }, SpecTimeout(time.Second * 5)) }) }) ``` -------------------------------- ### Build Stemcell with StemcellBuilder Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Orchestrates the complete stemcell build process, including OS extraction, kernel installation, agent setup, and image creation. Requires definition, environment, runner, and stage collection. ```ruby require 'bosh/stemcell/stemcell_builder' require 'bosh/stemcell/build_environment' require 'bosh/stemcell/stage_collection' require 'bosh/stemcell/stage_runner' require 'bosh/stemcell/definition' definition = Bosh::Stemcell::Definition.for('google', 'kvm', 'ubuntu', 'jammy') environment = Bosh::Stemcell::BuildEnvironment.new( ENV.to_hash, definition, '1.0.0', '/tmp/ubuntu_base_image.tgz' ) collection = Bosh::Stemcell::StageCollection.new(definition) runner = Bosh::Stemcell::StageRunner.new( build_path: environment.build_path, command_env: environment.command_env, settings_file: environment.settings_path, work_path: environment.work_path ) builder = Bosh::Stemcell::StemcellBuilder.new( environment: environment, runner: runner, collection: collection ) # Build the stemcell (runs all stages) builder.build ``` -------------------------------- ### Log Message from Application Object Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/go-logr/logr/README.md An example of how an application object uses its logr.Logger to emit log messages. The logger is typically received during initialization. ```go type appObject struct { // ... other fields ... logger logr.Logger // ... other fields ... } func (app *appObject) Run() { app.logger.Info("starting up", "timestamp", time.Now()) // ... app code ... } ``` -------------------------------- ### Identify Missing OVF Tool Error Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Example of an error message encountered when the required ovftool installer is missing during the Docker build process. ```shell ADD failed: failed to compute cache key: "/VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle": not found ``` -------------------------------- ### Rebuild the Docker Image Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Builds the stemcell builder container using specific CLI tool versions and base images. Ensure the ovftool installer is present in the specified directory before running. ```shell export short_name="jammy" docker build \ --platform linux/amd64 \ --build-arg BASE_IMAGE="ubuntu:${short_name}" \ --build-arg META4_CLI_URL="https://github.com/dpb587/metalink/releases/download/v0.5.0/meta4-0.5.0-linux-amd64" \ --build-arg SYFT_CLI_URL="https://github.com/anchore/syft/releases/download/v1.42.3/syft_1.42.3_linux_amd64.tar.gz" \ --build-arg YQ_CLI_URL="https://github.com/mikefarah/yq/releases/download/v4.52.5/yq_linux_amd64" \ --build-arg RUBY_INSTALL_URL="https://github.com/postmodern/ruby-install/releases/download/v0.10.2/ruby-install-0.10.2.tar.gz" \ --build-arg RUBY_VERSION="$(cat .ruby-version)" \ --build-arg GEM_HOME="/usr/local/bundle" \ --build-arg OVF_TOOL_INSTALLER="VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle" \ --build-arg OVF_TOOL_INSTALLER_SHA1="6c24e473be49c961cfc3bb16774b52b48e822991" \ -t bosh/os-image-stemcell-builder:${short_name} \ ci/docker/os-image-stemcell-builder/ ``` -------------------------------- ### Build BOSH Linux Stemcell Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/docs/develop.md Run the stemcell builder container and execute the rake task to build the OS image. Ensure you have the necessary dependencies like bucc and virtualbox installed. ```bash cd ~/workspace/bosh-linux-stemcell-builder/ci/docker ./run os-image-stemcell-builder-impish bundle exec rake stemcell:build_os_image[ubuntu,impish,$PWD/tmp/ubuntu_base_image.tgz] ``` -------------------------------- ### Get Infrastructure Configuration Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt The `Bosh::Stemcell::Infrastructure` module provides infrastructure-specific settings. You can retrieve configurations for supported IaaS providers and access their properties like hypervisor, disk formats, and default disk size. ```ruby require 'bosh/stemcell/infrastructure' # Get infrastructure configuration aws = Bosh::Stemcell::Infrastructure.for('aws') aws.name # => 'aws' aws.hypervisor # => 'xen' aws.default_disk_size # => 5120 (MB) aws.disk_formats # => ['raw'] aws.stemcell_formats # => ['aws-raw'] aws.additional_cloud_properties # => {'root_device_name' => '/dev/sda1'} # vSphere configuration vsphere = Bosh::Stemcell::Infrastructure.for('vsphere') vsphere.disk_formats # => ['ovf'] vsphere.stemcell_formats # => ['vsphere-ova', 'vsphere-ovf'] ``` -------------------------------- ### Test Ubuntu OS Image with RSpec Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Execute RSpec tests against a built Ubuntu OS image to verify its configuration and installed packages. Ensure the OS_IMAGE environment variable is set to the path of the OS image tarball. ```bash cd /opt/bosh/bosh-stemcell OS_IMAGE=/opt/bosh/tmp/ubuntu_base_image.tgz \ bundle exec rspec -fd spec/os_image/ubuntu_spec.rb ``` -------------------------------- ### Run mkall.sh to Generate Files (Old Build System) Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/golang.org/x/sys/unix/README.md Use this command to generate Go files for your current OS and architecture with the old build system. Ensure GOOS and GOARCH are set correctly. ```bash mkall.sh ``` -------------------------------- ### RSpec Package Installation Test Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Tests for the installation status of packages. This snippet checks if the 'auditd' package is installed and if the 'sendmail' package is not installed. ```ruby describe package('auditd') do it { should be_installed } end describe package('sendmail') do it { should_not be_installed } end ``` -------------------------------- ### Show mkall.sh Commands (Old Build System) Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/golang.org/x/sys/unix/README.md View the commands that will be executed by mkall.sh without actually running them. Useful for understanding the build process. ```bash mkall.sh -n ``` -------------------------------- ### Build a Stemcell from Local OS Image Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Adds the BOSH agent and virtualization tools to a previously built OS image. Rebuild this when testing BOSH-specific changes. ```bash export short_name="jammy" export build_number="0.0.8" bundle exec rake stemcell:build_with_local_os_image[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz,${build_number}] ``` -------------------------------- ### Upload VMware ovftool to GCP Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Uploads the required VMware ovftool installer to the designated GCP bucket for LTS stemcell creation. ```shell gsutil cp MY_OVFTOOL_FILE gs://bosh-vmware-ovftool/MY_OS/ ``` ```shell export short_name="jammy" gsutil cp VMware-ovftool-4.4.3-18663434-lin.x86_64.bundle gs://bosh-vmware-ovftool/${short_name}/ ``` -------------------------------- ### Run Gomega linter Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/onsi/gomega/CONTRIBUTING.md Verify code quality by running the Go linter locally. ```bash go vet ./... ``` -------------------------------- ### Build Ubuntu Base OS Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Use this Rake task to create a base OS image tarball. It requires the OS name, version, and an output path for the tarball. ```bash # Build an Ubuntu Jammy base OS image export short_name="jammy" bundle exec rake stemcell:build_os_image[ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] # Arguments: # 1. operating_system_name: Type of OS (currently only 'ubuntu' supported) # 2. operating_system_version: Ubuntu codename (e.g., 'jammy', 'focal') # 3. os_image_path: Output path for the tarball # Example output: tmp/ubuntu_base_image.tgz (~500MB compressed) ``` -------------------------------- ### Run Stemcell Build Stages Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Initialize `StageRunner` with build paths and settings, then configure and apply stages sequentially or resume from a failed stage. ```ruby require 'bosh/stemcell/stage_runner' runner = Bosh::Stemcell::StageRunner.new( build_path: '/mnt/stemcells/vsphere/esxi/ubuntu/build/build', command_env: 'env HTTP_PROXY=... HTTPS_PROXY=...', settings_file: '/mnt/.../build/etc/settings.bash', work_path: '/mnt/stemcells/vsphere/esxi/ubuntu/work/work' ) # Configure and apply stages stages = [:base_debootstrap, :base_apt, :base_ssh] runner.configure_and_apply(stages) # Resume from a failed stage runner.configure_and_apply(stages, 'base_apt') # Each stage runs: # 1. config.sh (if exists) - configures stage settings # 2. apply.sh - applies changes to the chroot/image ``` -------------------------------- ### Initialize Build Environment Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Create a `BuildEnvironment` instance with stemcell definition, version, and OS image path. Accesses build paths and generates RSpec commands. ```ruby require 'bosh/stemcell/build_environment' require 'bosh/stemcell/definition' definition = Bosh::Stemcell::Definition.for('vsphere', 'esxi', 'ubuntu', 'jammy') environment = Bosh::Stemcell::BuildEnvironment.new( ENV.to_hash, # Environment variables definition, # Stemcell definition '1.0.0', # Version number '/path/to/os_image.tgz' # OS image tarball path ) # Access build paths environment.build_path # => '/mnt/stemcells/vsphere/esxi/ubuntu/build/build' environment.work_path # => '/mnt/stemcells/vsphere/esxi/ubuntu/work/work' environment.chroot_dir # => '/mnt/stemcells/vsphere/esxi/ubuntu/work/work/chroot' environment.settings_path # => '/mnt/.../build/etc/settings.bash' environment.stemcell_tarball_path # => work_path # Get RSpec test commands environment.os_image_rspec_command # => "cd /opt/bosh/bosh-stemcell; OS_IMAGE=/path/to/os_image.tgz bundle exec rspec -fd spec/os_image/ubuntu_spec.rb" environment.stemcell_rspec_command # => "cd /opt/bosh/bosh-stemcell; STEMCELL_IMAGE=... bundle exec rspec -fd spec/os_image/ubuntu_spec.rb spec/stemcells/..." # Prepare build environment environment.prepare_build # Creates directories, copies builder scripts, persists settings ``` -------------------------------- ### Run Gomega tests Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/onsi/gomega/CONTRIBUTING.md Execute all unit tests locally to ensure project stability before submission. ```bash ginkgo -r -p ``` -------------------------------- ### Configure Azure Stemcell Options Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Instantiate Azure infrastructure configuration to determine the hypervisor and available disk formats. ```ruby azure = Bosh::Stemcell::Infrastructure.for('azure') azure.hypervisor # => 'hyperv' azure.disk_formats # => ['vhd'] ``` -------------------------------- ### Building OS Images Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt This Rake task creates a base OS image tarball. It supports Ubuntu and allows specifying the OS version and output path. ```APIDOC ## Build OS Image ### Description Creates a base OS image tarball containing the operating system filesystem without infrastructure-specific components. ### Method `bundle exec rake` ### Endpoint `stemcell:build_os_image[operating_system_name,operating_system_version,os_image_path]` ### Parameters #### Path Parameters - **operating_system_name** (string) - Required - Type of OS (currently only 'ubuntu' supported). - **operating_system_version** (string) - Required - Ubuntu codename (e.g., 'jammy', 'focal'). - **os_image_path** (string) - Required - Output path for the tarball. ### Request Example ```bash export short_name="jammy" bundle exec rake stemcell:build_os_image[ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] ``` ### Response #### Success Response (200) - **os_image_path** (string) - Path to the created OS image tarball (e.g., `tmp/ubuntu_base_image.tgz`). ``` -------------------------------- ### Syscall Entry Points in Assembly Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/golang.org/x/sys/unix/README.md Defines the three primary entry points for system call dispatch in the hand-written assembly file. These are used for standard and low-level system calls. ```go func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) ``` ```go func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr) ``` ```go func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) ``` -------------------------------- ### Glob Filesystem Paths Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Use Glob to find all files and directories matching a pattern. It can be relative or absolute and is a replacement for filepath.Glob. ```go func Glob(pattern string) ([]string, error) ``` -------------------------------- ### PathMatch for System Separators Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Use PathMatch to match patterns using the system's path separator. It's a replacement for filepath.Match. ```go func PathMatch(pattern, name string) (bool, error) ``` -------------------------------- ### Build an OS Image Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Generates a base OS image tarball. This should be rebuilt when modifying OS packages or their configurations. ```bash export short_name="jammy" bundle exec rake stemcell:build_os_image[ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] ``` -------------------------------- ### Configure Google Cloud Stemcell Options Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Instantiate Google Cloud infrastructure configuration to specify the hypervisor and disk formats. ```ruby google = Bosh::Stemcell::Infrastructure.for('google') google.hypervisor # => 'kvm' google.disk_formats # => ['rawdisk'] ``` -------------------------------- ### Build a Stemcell Locally Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Initializes the build environment using Docker and executes the Rake tasks to build the OS image and the final stemcell. ```bash export short_name="jammy" git clone git@github.com:cloudfoundry/bosh-linux-stemcell-builder.git cd bosh-linux-stemcell-builder git checkout ubuntu-${short_name} mkdir -p tmp docker build \ --platform linux/amd64 \ --build-arg SYFT_VERSION=v1.42.3 \ -t bosh/os-image-stemcell-builder:${short_name} \ ci/docker/os-image-stemcell-builder/ docker run \ --platform linux/amd64 \ --privileged \ -v "$(pwd):/opt/bosh" \ --workdir /opt/bosh \ --user=1000:1000 \ -it \ bosh/os-image-stemcell-builder:${short_name} # You're now in the Docker container gem install bundler bundle install # build OS image bundle exec rake stemcell:build_os_image[ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] # build vSphere stemcell bundle exec rake stemcell:build_with_local_os_image[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] ``` -------------------------------- ### Create Integration Subnets Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Creates a new subnet for BOSH integration tests. ```shell # branch: ubuntu-${short_name} gcloud compute networks subnets create --network default --range 10.100.0.0/24 bosh-integration-0 ``` -------------------------------- ### Custom OS Interface for Filesystem Operations Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Implement the OS interface to abstract filesystem operations, useful for testing or custom environments. StandardOS uses the standard library's os package. ```go type OS interface { Lstat(name string) (os.FileInfo, error) Open(name string) (*os.File, error) PathSeparator() rune Stat(name string) (os.FileInfo, error) } ``` -------------------------------- ### Match Path Pattern Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Use Match to check if a given name matches a pattern. It splits patterns by '/' and is a replacement for path.Match. ```go func Match(pattern, name string) (bool, error) ``` -------------------------------- ### Create GCP Storage Buckets Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Initializes GCS buckets for stemcell artifacts and terraform state in the europe-west4 region. ```shell gsutil mb -l europe-west4 gs://bosh-aws-light-stemcells gsutil mb -l europe-west4 gs://bosh-aws-light-stemcells-candidate gsutil mb -l europe-west4 gs://bosh-gce-light-stemcell-ci-terraform-state gsutil mb -l europe-west4 gs://bosh-gce-light-stemcells gsutil mb -l europe-west4 gs://bosh-gce-light-stemcells-candidate gsutil mb -l europe-west4 gs://bosh-gce-raw-stemcells-new gsutil mb -l europe-west4 gs://bosh-core-stemcells gsutil mb -l europe-west4 gs://bosh-core-stemcells-candidate gsutil mb -l europe-west4 gs://bosh-os-images gsutil mb -l europe-west4 gs://bosh-stemcell-triggers ``` -------------------------------- ### Enable Bucket Versioning Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Enables versioning for the stemcell triggers bucket. ```shell gsutil versioning set on gs://bosh-stemcell-triggers ``` -------------------------------- ### Infrastructure Configuration API Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt The `Bosh::Stemcell::Infrastructure` module provides infrastructure-specific settings for each supported IaaS, allowing access to details like hypervisor, disk formats, and default disk size. ```APIDOC ## Infrastructure Configuration API ### Description Provides infrastructure-specific settings for each supported IaaS. Allows retrieval of configuration details for various cloud providers. ### Module `Bosh::Stemcell::Infrastructure` ### Methods #### `for(infrastructure_name)` Gets the infrastructure configuration object for the given infrastructure name. - **infrastructure_name** (string) - Required - The name of the infrastructure (e.g., 'aws', 'vsphere'). ### Properties - **`infrastructure.name`** (string) - The name of the infrastructure. - **`infrastructure.hypervisor`** (string) - The default hypervisor for the infrastructure. - **`infrastructure.default_disk_size`** (integer) - The default disk size in MB. - **`infrastructure.disk_formats`** (array) - Supported disk formats for the infrastructure. - **`infrastructure.stemcell_formats`** (array) - Supported stemcell formats for the infrastructure. - **`infrastructure.additional_cloud_properties`** (object) - Additional cloud properties required for the infrastructure. ### Request Example ```ruby require 'bosh/stemcell/infrastructure' # Get AWS configuration aws = Bosh::Stemcell::Infrastructure.for('aws') puts aws.name # => 'aws' puts aws.hypervisor # => 'xen' puts aws.default_disk_size # => 5120 (MB) puts aws.disk_formats # => ['raw'] puts aws.stemcell_formats # => ['aws-raw'] puts aws.additional_cloud_properties # => {'root_device_name' => '/dev/sda1'} # Get vSphere configuration vsphere = Bosh::Stemcell::Infrastructure.for('vsphere') puts vsphere.disk_formats # => ['ovf'] puts vsphere.stemcell_formats # => ['vsphere-ova', 'vsphere-ovf'] ``` ### Response Example ```json { "name": "aws", "hypervisor": "xen", "default_disk_size": 5120, "disk_formats": ["raw"], "stemcell_formats": ["aws-raw"], "additional_cloud_properties": {"root_device_name": "/dev/sda1"} } ``` ``` -------------------------------- ### Load Slim-Sprig FuncMap Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/go-task/slim-sprig/v3/README.md The FuncMap must be registered with the template object before parsing any template files. ```go import ( "html/template" "github.com/go-task/slim-sprig" ) // This example illustrates that the FuncMap *must* be set before the // templates themselves are loaded. tpl := template.Must( template.New("base").Funcs(sprig.FuncMap()).ParseGlob("*.html") ) ``` -------------------------------- ### Mount and Unmount Disk Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Handles mounting and unmounting raw disk images for inspection. Uses `losetup` and `kpartx` to map partitions and can mount EFI partitions. Ensures proper unmounting and cleanup of loop devices. ```ruby require 'bosh/stemcell/disk_image' disk_image = Bosh::Stemcell::DiskImage.new( image_file_path: '/mnt/stemcells/vsphere/esxi/ubuntu/work/work/vsphere-esxi-ubuntu.raw', image_mount_point: '/mnt/stemcell_mount', verbose: true ) # Mount the disk image disk_image.mount # Uses losetup and kpartx to map partitions # Mounts EFI partition if present # Access mount point disk_image.image_mount_point # => '/mnt/stemcell_mount' # Unmount when done disk_image.unmount # Unmounts filesystems and removes loop device mappings ``` -------------------------------- ### Build Stemcell with Remote OS Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt This Rake task automates stemcell creation by downloading a base OS image from a metalink. It then invokes the local build task. ```bash # Build an AWS stemcell (downloads OS image automatically) bundle exec rake stemcell:build[aws,xen-hvm,ubuntu,jammy,1.0.0] # This task: # 1. Downloads the OS image via metalink from bosh-stemcell/image-metalinks/ # 2. Stores it in tmp/base_os_image.tgz # 3. Invokes build_with_local_os_image with the downloaded image ``` -------------------------------- ### Build Stemcell with Local OS Image Inside Container Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Build a stemcell using a pre-built local OS image. This command uses a rake task and requires specifying the CPI, hypervisor, OS, codename, OS image path, and stemcell version. ```bash bundle exec rake stemcell:build_with_local_os_image[vsphere,esxi,ubuntu,jammy,${PWD}/tmp/ubuntu_base_image.tgz,1.0.0] ``` -------------------------------- ### Run Docker Build Container Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Execute the Docker container for building stemcells. This command mounts the current directory into the container and sets the working directory and user. It requires the Docker image to be built first. ```bash docker run \ --platform linux/amd64 \ --privileged \ -v "$(pwd):/opt/bosh" \ --workdir /opt/bosh \ --user=1000:1000 \ -it \ bosh/os-image-stemcell-builder:${short_name} ``` -------------------------------- ### Create Stemcell Definition Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Use the `Bosh::Stemcell::Definition` class to define stemcell configurations. It allows accessing properties like infrastructure, hypervisor, disk formats, and generating stemcell names. ```ruby require 'bosh/stemcell/definition' # Create a stemcell definition definition = Bosh::Stemcell::Definition.for( 'vsphere', # infrastructure_name 'esxi', # hypervisor_name 'ubuntu', # operating_system_name 'jammy' # operating_system_version ) # Access definition properties definition.infrastructure.name # => 'vsphere' definition.infrastructure.hypervisor # => 'esxi' definition.infrastructure.disk_formats # => ['ovf'] definition.infrastructure.default_disk_size # => 5120 definition.operating_system.name # => 'ubuntu' definition.operating_system.version # => 'jammy' # Generate stemcell name definition.stemcell_name('ovf') # => 'vsphere-esxi-ubuntu-jammy-go_agent' # With FIPS variant fips_definition = Bosh::Stemcell::Definition.for('aws', 'xen-hvm', 'ubuntu', 'jammy-fips') fips_definition.operating_system.variant # => 'fips' ``` -------------------------------- ### Unmarshal and Marshal YAML Data in Go Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/gopkg.in/yaml.v3/README.md Demonstrates unmarshalling YAML data into a struct and a map, and then marshalling them back to YAML. Ensure struct fields are public for correct unmarshalling. ```Go package main import ( "fmt" "log" "gopkg.in/yaml.v3" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` // Note: struct fields must be public in order for unmarshal to // correctly populate the data. type T struct { A string B struct { RenamedC int `yaml:"c"` D []int `yaml:",flow"` } } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` -------------------------------- ### Run ShelloutTypes Unit Tests Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Executes unit tests for the ShelloutTypes library. Requires root privileges and an Ubuntu chroot environment. ```shell bundle install --local cd /opt/bosh/bosh-stemcell OS_IMAGE=/opt/bosh/tmp/ubuntu_base_image.tgz bundle exec rspec spec/ --tag shellout_types ``` ```shell OSX=true OS_IMAGE=/opt/bosh/tmp/ubuntu_base_image.tgz bundle exec rspec spec/ --tag shellout_types ``` -------------------------------- ### Import Doublestar Package Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/bmatcuk/doublestar/README.md Import the doublestar package into your Go code to use its functionalities. ```go import "github.com/bmatcuk/doublestar" ``` -------------------------------- ### Configure OpenStack Stemcell Options Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Instantiate OpenStack infrastructure configuration to access disk formats and additional cloud properties. ```ruby openstack = Bosh::Stemcell::Infrastructure.for('openstack') openstack.disk_formats # => ['qcow2', 'raw'] openstack.additional_cloud_properties # => {'auto_disk_config' => true} ``` -------------------------------- ### Configure the aggregated pipeline Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/docs/new_stemcell_line.md Execute the configuration script to update the pipeline with the new branch details. ```bash ./ci/{os_name}-{os_version}/configure-aggregated-pipeline.sh ``` -------------------------------- ### Clone Bucc and Prepare Operators Directory Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/docs/develop.md Clone the bucc repository and create an operators directory to store deployment configuration files. This is a prerequisite for deploying a BOSH director with custom stemcells. ```bash mkdir -p ~/workspace git clone https://github.com/starkandwayne/bucc cd bucc mkdir operators ``` -------------------------------- ### Switch to master branch Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/docs/new_stemcell_line.md Return to the master branch after pushing the new feature or release branch. ```bash git checkout master ``` -------------------------------- ### Build Stemcell from Local OS Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt This Rake task constructs a stemcell for a specific infrastructure using a pre-built OS image. Provide the infrastructure, hypervisor, OS details, OS image path, and build number. ```bash # Build a vSphere stemcell from local OS image export short_name="jammy" export build_number="0.0.8" bundle exec rake stemcell:build_with_local_os_image[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz,${build_number}] # Arguments: # 1. infrastructure_name: Target IaaS (aws, azure, google, vsphere, openstack, etc.) # 2. hypervisor_name: Hypervisor type (xen-hvm, hyperv, kvm, esxi, boshlite) # 3. operating_system_name: OS type ('ubuntu') # 4. operating_system_version: OS version with optional variant ('jammy' or 'jammy-fips') # 5. os_image_path: Path to base OS image tarball # 6. build_number: Stemcell version number (defaults to '0000') # Output: tmp/bosh-stemcell-0.0.8-vsphere-esxi-ubuntu-jammy-go_agent.tgz ``` -------------------------------- ### Building Stemcells with Remote OS Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt This Rake task automatically downloads a pre-built OS image from a metalink and then builds the stemcell. It simplifies the process by handling OS image retrieval. ```APIDOC ## Build Stemcell with Remote OS Image ### Description Automatically downloads a pre-built OS image from a metalink and builds the stemcell. This task invokes `build_with_local_os_image` internally after downloading the OS image. ### Method `bundle exec rake` ### Endpoint `stemcell:build[infrastructure_name,hypervisor_name,operating_system_name,operating_system_version,build_number]` ### Parameters #### Path Parameters - **infrastructure_name** (string) - Required - Target IaaS (aws, azure, google, vsphere, openstack, etc.). - **hypervisor_name** (string) - Required - Hypervisor type (xen-hvm, hyperv, kvm, esxi, boshlite). - **operating_system_name** (string) - Required - OS type ('ubuntu'). - **operating_system_version** (string) - Required - Ubuntu codename (e.g., 'jammy', 'focal'). - **build_number** (string) - Required - Stemcell version number. ### Request Example ```bash # Build an AWS stemcell (downloads OS image automatically) bundle exec rake stemcell:build[aws,xen-hvm,ubuntu,jammy,1.0.0] ``` ### Response #### Success Response (200) - **stemcell_path** (string) - Path to the created stemcell tarball. ``` -------------------------------- ### Handle format strings in structured logs Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/go-logr/logr/README.md Use fmt.Sprintf when a format string is strictly necessary within a key-value pair. ```go log.Printf("unable to reflect over type %T") ``` ```go logger.Info("unable to reflect over type", "type", fmt.Sprintf("%T")) ``` -------------------------------- ### Create Git Commit and Release Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/onsi/gomega/RELEASING.md After updating the version and changelog, commit the changes, push to the remote repository, and create a new GitHub release using the GitHub CLI. ```bash git commit -m "vM.m.p" git push gh release create "vM.m.p" git fetch --tags origin master ``` -------------------------------- ### Building Stemcells with Local OS Image Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt This Rake task builds a complete stemcell from a pre-built OS image for a specific infrastructure. It requires infrastructure, hypervisor, OS details, OS image path, and build number. ```APIDOC ## Build Stemcell with Local OS Image ### Description Creates a complete stemcell from a pre-built OS image for a specific infrastructure. ### Method `bundle exec rake` ### Endpoint `stemcell:build_with_local_os_image[infrastructure_name,hypervisor_name,operating_system_name,operating_system_version,os_image_path,build_number]` ### Parameters #### Path Parameters - **infrastructure_name** (string) - Required - Target IaaS (aws, azure, google, vsphere, openstack, etc.). - **hypervisor_name** (string) - Required - Hypervisor type (xen-hvm, hyperv, kvm, esxi, boshlite). - **operating_system_name** (string) - Required - OS type ('ubuntu'). - **operating_system_version** (string) - Required - OS version with optional variant ('jammy' or 'jammy-fips'). - **os_image_path** (string) - Required - Path to base OS image tarball. - **build_number** (string) - Required - Stemcell version number (defaults to '0000'). ### Request Example ```bash export short_name="jammy" export build_number="0.0.8" bundle exec rake stemcell:build_with_local_os_image[vsphere,esxi,ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz,${build_number}] ``` ### Response #### Success Response (200) - **stemcell_path** (string) - Path to the created stemcell tarball (e.g., `tmp/bosh-stemcell-0.0.8-vsphere-esxi-ubuntu-jammy-go_agent.tgz`). ``` -------------------------------- ### Collect Stemcell Build Stages Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Instantiate `StageCollection` with a stemcell definition to retrieve lists of OS image, agent, and infrastructure-specific stages. ```ruby require 'bosh/stemcell/stage_collection' require 'bosh/stemcell/definition' definition = Bosh::Stemcell::Definition.for('aws', 'xen-hvm', 'ubuntu', 'jammy') collection = Bosh::Stemcell::StageCollection.new(definition) # OS image stages (common to all infrastructures) collection.operating_system_stages # => [:base_debootstrap, :base_ubuntu_firstboot, :base_apt, :base_ubuntu_build_essential, # :base_ubuntu_packages, :base_file_permission, :base_ssh, :bosh_sysstat, # :bosh_environment, :bosh_sysctl, :bosh_limits, :bosh_users, :bosh_monit, # :bosh_ntp, :bosh_sudoers, :bosh_systemd, :password_policies, :restrict_su_command, # :tty_config, :rsyslog_config, :delay_monit_start, :system_grub, :vim_tiny, # :cron_config, :escape_ctrl_alt_del, :bosh_audit_ubuntu, :bosh_log_audit_start, # :clean_machine_id] # Agent stages (BOSH agent and tools) collection.agent_stages # => [:bosh_go_agent, :blobstore_clis, :logrotate_config, :dev_tools_config, :static_libraries_config] # Infrastructure-specific stemcell image stages collection.build_stemcell_image_stages # For AWS # => [:system_network, :system_aws_modules, :system_parameters, :bosh_clean, # :bosh_harden, :bosh_aws_agent_settings, :bosh_clean_ssh, :udev_aws_rules, # :restore_apt_sources, :image_create, :image_install_grub, :sbom_create, # :bosh_package_list] # Package stages by disk format collection.package_stemcell_stages('raw') # => [:prepare_raw_image_stemcell] collection.package_stemcell_stages('qcow2') # => [:prepare_qcow2_image_stemcell] collection.package_stemcell_stages('ovf') # => [:image_ovf_vmx, :image_ovf_generate, :prepare_ovf_image_stemcell] ``` -------------------------------- ### Unmarshal and Marshal YAML data Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/gopkg.in/yaml.v2/README.md Demonstrates unmarshalling YAML into a struct or map, and marshalling them back into YAML format. Struct fields must be exported for the unmarshaller to populate them. ```Go package main import ( "fmt" "log" "gopkg.in/yaml.v2" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` // Note: struct fields must be public in order for unmarshal to // correctly populate the data. type T struct { A string B struct { RenamedC int `yaml:"c"` D []int `yaml:",flow"` } } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` -------------------------------- ### Run Ginkgo Specs in Parallel Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/acceptance-tests/vendor/github.com/onsi/ginkgo/v2/README.md Execute Ginkgo tests in parallel using the '-p' flag. This is useful for speeding up test execution in larger suites. ```bash ginkgo -p ``` -------------------------------- ### Build Docker Image for Stemcell Builder Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Build the Docker image used for creating stemcells. This command requires setting the short_name build argument for the OS distribution codename. ```bash export short_name="jammy" docker build \ --platform linux/amd64 \ --build-arg SYFT_VERSION=v1.42.3 \ -t bosh/os-image-stemcell-builder:${short_name} \ ci/docker/os-image-stemcell-builder/ ``` -------------------------------- ### Run BOSH Linux Stemcell Builder Tests Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Executes the primary test suite for the BOSH Linux Stemcell Builder codebase. ```shell bundle install --local cd /opt/bosh/bosh-stemcell bundle exec rspec spec/ ``` -------------------------------- ### Resume Failed OS Image Build Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Resume a failed stemcell build process from a specific stage. This is useful for continuing a build after fixing an issue without restarting from the beginning. Use with caution as stages are not guaranteed to be idempotent. ```bash # If a build fails at the 'rsyslog_config' stage, resume from there: bundle exec rake stemcell:build_os_image[ubuntu,jammy,${PWD}/tmp/ubuntu_base_image.tgz] \ resume_from=rsyslog_config ``` -------------------------------- ### Push the new branch Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/docs/new_stemcell_line.md Push the newly created branch to the remote repository. ```bash git push origin <> ``` -------------------------------- ### List Uploaded Stemcells in BOSH Director Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt List all stemcells that have been uploaded to the BOSH Director. This command provides an overview of available stemcells and their versions. ```bash bosh stemcells ``` -------------------------------- ### Upload Local Stemcell to BOSH Director Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Upload a locally built stemcell to the BOSH Director. This command requires the path to the stemcell tarball. ```bash bosh upload-stemcell tmp/bosh-stemcell-1.0.0-vsphere-esxi-ubuntu-jammy-go_agent.tgz ``` -------------------------------- ### Package Stemcell with StemcellPackager Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Packages a built stemcell image into a distributable tarball. Requires stemcell definition, version, paths, disk size, runner, and stage collection. Outputs a tarball with manifest, image, and package lists. ```ruby require 'bosh/stemcell/stemcell_packager' packager = Bosh::Stemcell::StemcellPackager.new( definition: definition, version: '1.0.0', work_path: '/mnt/stemcells/google/kvm/ubuntu/work/work', tarball_path: '/mnt/stemcells/google/kvm/ubuntu/work/work', disk_size: 5120, runner: runner, collection: collection ) # Package stemcell for specific disk format tarball_path = packager.package('rawdisk') ``` -------------------------------- ### Resume Build from Specific Stage Source: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/README.md Resumes the stemcell build process from a specific stage after a failure. Use with caution as stages may not be idempotent. ```shell export short_name="jammy" bundle exec rake stemcell:build_os_image[ubuntu,${short_name},${PWD}/tmp/ubuntu_base_image.tgz] resume_from=rsyslog_config ``` -------------------------------- ### Test Complete Stemcell with RSpec Source: https://context7.com/cloudfoundry/bosh-linux-stemcell-builder/llms.txt Run comprehensive RSpec tests on a complete stemcell, including OS image and stemcell-specific tests. This command requires setting STEMCELL_IMAGE, STEMCELL_WORKDIR, and OS_NAME environment variables. ```bash STEMCELL_IMAGE=/mnt/stemcells/vsphere/esxi/ubuntu/work/work/vsphere-esxi-ubuntu.raw \ STEMCELL_WORKDIR=/mnt/stemcells/vsphere/esxi/ubuntu/work/work/chroot \ OS_NAME=ubuntu \ bundle exec rspec -fd \ --tag ~exclude_on_vsphere \ spec/os_image/ubuntu_spec.rb \ spec/stemcells/ubuntu_spec.rb \ spec/stemcells/go_agent_spec.rb \ spec/stemcells/vsphere_spec.rb \ spec/stemcells/stig_spec.rb \ spec/stemcells/cis_spec.rb ```