### Install difflib Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/aryann/difflib/README.md Use the go get command to download and install the library into your workspace. ```bash go get github.com/aryann/difflib ``` -------------------------------- ### Create and Run a SOCKS5 Server Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/cloudfoundry/go-socks5/README.md This example demonstrates how to create a new SOCKS5 server with default configuration and start listening for incoming TCP connections on a specified address and port. Ensure the 'socks5' package is imported. ```go // Create a SOCKS5 server conf := &socks5.Config{} server, err := socks5.New(conf) if err != nil { panic(err) } // Create SOCKS5 proxy on localhost port 8000 if err := server.ListenAndServe("tcp", "127.0.0.1:8000"); err != nil { panic(err) } ``` -------------------------------- ### Clone and Run Go Sigar Examples Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/cloudfoundry/gosigar/README.md Clone the Go Sigar repository and navigate to the examples directory to run provided examples like uptime, df, free, and ps. ```bash git clone https://github.com/cloudfoundry/gosigar.git cd gosigar/examples go run uptime.go go run df.go go run free.go go run ps.go ``` -------------------------------- ### Install go-isatty Package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mattn/go-isatty/README.md Install the go-isatty package using the go get command. This command fetches and installs the package, making it available for use in your Go projects. ```bash go get github.com/mattn/go-isatty ``` -------------------------------- ### Install the YAML package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v1/README.md Use the go get command to download the package into your workspace. ```bash go get gopkg.in/yaml.v1 ``` -------------------------------- ### Install doublestar Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Install the doublestar package using 'go get'. This command fetches and installs the package and its dependencies. ```bash go get github.com/bmatcuk/doublestar ``` -------------------------------- ### Install and Import kubernetes-sigs/yaml Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/sigs.k8s.io/yaml/README.md Install the package using go get and import it in your Go program. ```bash $ go get sigs.k8s.io/yaml ``` ```go import "sigs.k8s.io/yaml" ``` -------------------------------- ### Install Backoff library Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jpillora/backoff/README.md Command to install the backoff package via go get. ```bash $ go get -v github.com/jpillora/backoff ``` -------------------------------- ### Installation and Import Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Instructions on how to install the doublestar package and import it into your Go project. ```APIDOC ## Installation **doublestar** can be installed via `go get`: ```bash go get github.com/bmatcuk/doublestar ``` To use it in your code, you must import it: ```go import "github.com/bmatcuk/doublestar" ``` ``` -------------------------------- ### Install godbus/dbus Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/godbus/dbus/v5/README.md Use the go get command to install the library. Requires Go 1.12 or later. ```bash go get github.com/godbus/dbus/v5 ``` -------------------------------- ### Install go-colorable Package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mattn/go-colorable/README.md Install the go-colorable package using the go get command. This is a prerequisite for using the package in your Go projects. ```bash $ go get github.com/mattn/go-colorable ``` -------------------------------- ### Install the YAML package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v2/README.md Use the go get command to fetch the package dependency. ```bash go get gopkg.in/yaml.v2 ``` -------------------------------- ### Install ansi Go Package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mgutz/ansi/README.md Use 'go get' to install the ansi package. This command fetches and installs the package and its dependencies. ```sh go get -u github.com/mgutz/ansi ``` -------------------------------- ### Create a web server with pat Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmizerany/pat/README.md Example of setting up a basic HTTP server with a named URL parameter using pat. ```go package main import ( "io" "net/http" "github.com/bmizerany/pat" "log" ) // hello world, the web server func HelloServer(w http.ResponseWriter, req *http.Request) { io.WriteString(w, "hello, "+req.URL.Query().Get(":name")+"!\n") } func main() { m := pat.New() m.Get("/hello/:name", http.HandlerFunc(HelloServer)) // Register this pat with the default serve mux so that other packages // may also be exported. (i.e. /debug/pprof/*) http.Handle("/", m) err := http.ListenAndServe(":12345", nil) if err != nil { log.Fatal("ListenAndServe: ", err) } } ``` -------------------------------- ### Install pat library Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmizerany/pat/README.md Command to fetch the pat package using the Go toolchain. ```bash $ go get github.com/bmizerany/pat ``` -------------------------------- ### Install gouuid package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/nu7hatch/gouuid/README.md Use the go tool to fetch the package from the repository. ```bash $ go get github.com/nu7hatch/gouuid ``` -------------------------------- ### Example Commit Message Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/godbus/dbus/v5/CONTRIBUTING.md A concrete example of a commit message following the project's convention. ```text scripts: add the test-cluster command this uses tmux to setup a test cluster that you can easily kill and start for debugging. Fixes #38 ``` -------------------------------- ### Use backoff with net package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jpillora/backoff/README.md Example of using backoff to manage reconnection attempts for a network connection. ```go b := &backoff.Backoff{ Max: 5 * time.Minute, } for { conn, err := net.Dial("tcp", "example.com:5309") if err != nil { d := b.Duration() fmt.Printf("%s, reconnecting in %s", err, d) time.Sleep(d) continue } //connected b.Reset() conn.Write([]byte("hello world!")) // ... Read ... Write ... etc conn.Close() //disconnected } ``` -------------------------------- ### Install Go TOML Package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/BurntSushi/toml/README.md Add the TOML package to your go.mod file. Requires Go 1.18 or newer. ```bash % go get github.com/BurntSushi/toml@latest ``` -------------------------------- ### ANSI Color and Style Combinations Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mgutz/ansi/README.md Examples of using the Color function with various foreground colors, attributes, and background colors. ```go Color(s, "red") // red Color(s, "red+d") // red dim Color(s, "red+b") // red bold Color(s, "red+B") // red blinking Color(s, "red+u") // red underline Color(s, "red+bh") // red bold bright Color(s, "red:white") // red on white Color(s, "red+b:white+h") // red bold on white bright Color(s, "red+B:white+h") // red blink on white bright Color(s, "off") // turn off ansi codes ``` -------------------------------- ### Identify Go Version Errors Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/hashicorp/go-multierror/README.md Example of compile errors encountered when using an unsupported Go version. ```text /go/src/github.com/hashicorp/go-multierror/multierror.go:112:9: undefined: errors.As /go/src/github.com/hashicorp/go-multierror/multierror.go:117:9: undefined: errors.Is ``` -------------------------------- ### YAML processing output Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v3/README.md The expected console output after running the provided Go example. ```text --- t: {Easy! {2 [3 4]}} --- t dump: a: Easy! b: c: 2 d: [3, 4] --- m: map[a:Easy! b:map[c:2 d:[3 4]]] --- m dump: a: Easy! b: c: 2 d: - 3 - 4 ``` -------------------------------- ### Install TOML Validator CLI Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/BurntSushi/toml/README.md Install the TOML validator command-line tool. This tool can be used to validate TOML files. ```bash % go install github.com/BurntSushi/toml/cmd/tomlv@latest % tomlv some-toml-file.toml ``` -------------------------------- ### Update Cgroup Resources Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Updates the resources applied to an existing cgroup. This example updates CPU shares. ```go shares = uint64(200) if err := control.Update(&specs.LinuxResources{ CPU: &specs.LinuxCPU{ Shares: &shares, }, }); err != nil { } ``` -------------------------------- ### YAML processing output Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v2/README.md The expected console output after running the YAML unmarshal and marshal example. ```text --- t: {Easy! {2 [3 4]}} --- t dump: a: Easy! b: c: 2 d: [3, 4] --- m: map[a:Easy! b:map[c:2 d:[3 4]]] --- m dump: a: Easy! b: c: 2 d: - 3 - 4 ``` -------------------------------- ### Get Cgroup Stats Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Retrieves statistics for a cgroup. Can optionally ignore non-existent files, such as swap memory stats when swap is not enabled. ```go stats, err := control.Stat() ``` ```go stats, err := control.Stat(cgroups.IgnoreNotExist) ``` -------------------------------- ### Custom Unmarshaling with encoding.TextUnmarshaler Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/BurntSushi/toml/README.md Implement the encoding.TextUnmarshaler interface to parse values into custom types, such as mail.Address. This example decodes a list of contact strings into mail.Address objects. ```toml contacts = [ "Donald Duck ", "Scrooge McDuck ", ] ``` ```go // Create address type which satisfies the encoding.TextUnmarshaler interface. type address struct { *mail.Address } func (a *address) UnmarshalText(text []byte) error { var err error a.Address, err = mail.ParseAddress(string(text)) return err } // Decode it. func decode() { blob := ` contacts = [ "Donald Duck ", "Scrooge McDuck ", ] ` var contacts struct { Contacts []address } _, err := toml.Decode(blob, &contacts) if err != nil { log.Fatal(err) } for _, c := range contacts.Contacts { fmt.Printf("%#v\n", c.Address) } // Output: // &mail.Address{Name:"Donald Duck", Address:"donald@duckburg.com"} // &mail.Address{Name:"Scrooge McDuck", Address:"scrooge@duckburg.com"} } ``` -------------------------------- ### Run the Demo Application Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/aryann/difflib/README.md Execute the demo server to visualize diff results between two files in a web browser. ```bash go run src/github.com/aryann/difflib/difflib_server/difflib_demo.go ``` -------------------------------- ### Build All Files (Old System) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/golang.org/x/sys/unix/README.md Run this script to generate Go files for your current OS and architecture using the old build system. Ensure GOOS and GOARCH are set correctly. Requires bash and go. ```bash mkall.sh ``` -------------------------------- ### Implement simple backoff Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jpillora/backoff/README.md Demonstrates basic usage of the Backoff struct with default settings. ```go b := &backoff.Backoff{ //These are the defaults Min: 100 * time.Millisecond, Max: 10 * time.Second, Factor: 2, Jitter: false, } fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) fmt.Printf("Reset!\n") b.Reset() fmt.Printf("%s\n", b.Duration()) ``` -------------------------------- ### Show Build Commands (Old System) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/golang.org/x/sys/unix/README.md Use this flag with mkall.sh to see the commands that would be run without actually executing them. Useful for debugging or understanding the build process. ```bash mkall.sh -n ``` -------------------------------- ### Colorize String with ANSI Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mgutz/ansi/README.md Demonstrates basic string coloring using ansi.Color. For performance, consider using ansi.ColorFunc to create reusable color functions. ```go import "github.com/mgutz/ansi" // colorize a string, SLOW msg := ansi.Color("foo", "red+b:white") // create a FAST closure function to avoid computation of ANSI code phosphorize := ansi.ColorFunc("green+h:black") msg = phosphorize("Bring back the 80s!") msg2 := phospohorize("Look, I\'m a CRT!") // cache escape codes and build strings manually lime := ansi.ColorCode("green+h:black") reset := ansi.ColorCode("reset") fmt.Println(lime, "Bring back the 80s!", reset) ``` -------------------------------- ### Create New Cgroup (v1 Static Path) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Creates a new cgroup using a static path for all subsystems under /test. Specifies CPU shares as a resource constraint and uses the v1 implementation. ```go shares := uint64(100) control, err := cgroups.New(cgroups.V1, cgroups.StaticPath("/test"), &specs.LinuxResources{ CPU: &specs.LinuxCPU{ Shares: &shares, }, }) defer control.Delete() ``` -------------------------------- ### Load and Query BOSH Release Index in Go Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Demonstrates loading a BOSH release index from a YAML file and querying it for GitHub organizations and specific releases. Ensure the index file path is correct. ```go package main import ( "fmt" "log" "github.com/bosh-io/worker/src/worker/releases" ) func main() { // Load the releases index from a YAML file index, err := releases.LoadFile("../bosh-io-releases/index.yml") if err != nil { log.Fatalf("failed to load index: %v", err) } // Get all unique GitHub organizations from the index orgs := index.GitHubOrgs() fmt.Printf("Found %d organizations: %v\n", len(orgs), orgs) // Output: Found 5 organizations: [cloudfoundry cloudfoundry-community pivotal-cf ...] // Filter releases by a specific organization cloudfoundryReleases := index.FilterByOrg("cloudfoundry") fmt.Printf("CloudFoundry has %d releases\n", len(cloudfoundryReleases)) // Access release details for _, release := range cloudfoundryReleases { fmt.Printf("Release: %s/%s (min_version: %s)\n", release.GitHubOrg(), release.GitHubRepo(), release.MinVersion) } } ``` -------------------------------- ### Create New Cgroup (v1 Systemd Slice) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Creates a new cgroup with systemd slice support. Requires the 'specs' package for resource definitions. ```go control, err := cgroups.New(cgroups.Systemd, cgroups.Slice("system.slice", "runc-test"), &specs.LinuxResources{ CPU: &specs.CPU{ Shares: &shares, }, }) ``` -------------------------------- ### Import difflib Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/aryann/difflib/README.md Include the library in your Go source files to access diffing functions. ```go import ( ... "fmt" "github.com/aryann/difflib" ... ) ``` -------------------------------- ### Abstracting the os package Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Information on using `GlobOS` and `PathMatchOS` for custom filesystem implementations. ```APIDOC ## Abstracting the `os` package **doublestar** by default uses the `Open`, `Stat`, and `Lstat`, functions and `PathSeparator` value from the standard library's `os` package. To abstract this, for example to be able to perform tests of Windows paths on Linux, or to interoperate with your own filesystem code, it includes the functions `GlobOS` and `PathMatchOS` which are identical to `Glob` and `PathMatch` except that they operate on an `OS` interface: ```go type OS interface { Lstat(name string) (os.FileInfo, error) Open(name string) (*os.File, error) PathSeparator() rune Stat(name string) (os.FileInfo, error) } ``` `StandardOS` is a value that implements this interface by calling functions in the standard library's `os` package. ``` -------------------------------- ### Import Release Notes from PostgreSQL Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md This command imports release notes from a PostgreSQL database into a specified index file. It queries the 'release_notes' table and pipes the output to a Go script for processing. ```bash $ psql db-name -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from release_notes;"| go run import-release-notes.go ~/workspace/bosh-io/releases-index/ ``` -------------------------------- ### Unmarshal and Marshal YAML in Go Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v1/README.md Demonstrates how to unmarshal YAML data into a struct or a map, and how to marshal those objects back into YAML format. ```go package main import ( "fmt" "log" "gopkg.in/yaml.v1" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` type T struct { A string B struct{C int; D []int ",flow"} } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` -------------------------------- ### Define Options Struct Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jessevdk/go-flags/README.md Use struct tags to define command line options with short and long names. ```go type Options struct { Verbose []bool `short:"v" long:"verbose" description:"Show verbose debug information"` } ``` -------------------------------- ### Login to Concourse and Sync Pipelines Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md Log in to the Concourse CI instance and then sync the pipelines for bosh.io releases. Ensure both 'bosh-io-worker' and 'bosh-io-releases' repositories are cloned in the same parent directory. ```bash $ fly -t bosh-io login -c https://bosh.ci.cloudfoundry.org// -n x-bosh-io $ ./bin/sync-pipelines ``` -------------------------------- ### Parse Complex Command Line Arguments Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jessevdk/go-flags/README.md Demonstrates advanced usage including callbacks, required flags, choice restrictions, and environment variable integration. ```go var opts struct { // Slice of bool will append 'true' each time the option // is encountered (can be set multiple times, like -vvv) Verbose []bool `short:"v" long:"verbose" description:"Show verbose debug information"` // Example of automatic marshalling to desired type (uint) Offset uint `long:"offset" description:"Offset"` // Example of a callback, called each time the option is found. Call func(string) `short:"c" description:"Call phone number"` // Example of a required flag Name string `short:"n" long:"name" description:"A name" required:"true"` // Example of a flag restricted to a pre-defined set of strings Animal string `long:"animal" choice:"cat" choice:"dog"` // Example of a value name File string `short:"f" long:"file" description:"A file" value-name:"FILE"` // Example of a pointer Ptr *int `short:"p" description:"A pointer to an integer"` // Example of a slice of strings StringSlice []string `short:"s" description:"A slice of strings"` // Example of a slice of pointers PtrSlice []*string `long:"ptrslice" description:"A slice of pointers to string"` // Example of a map IntMap map[string]int `long:"intmap" description:"A map from string to int"` // Example of env variable Thresholds []int `long:"thresholds" default:"1" default:"2" env:"THRESHOLD_VALUES" env-delim:","` } // Callback which will invoke callto: to call a number. // Note that this works just on OS X (and probably only with // Skype) but it shows the idea. opts.Call = func(num string) { cmd := exec.Command("open", "callto:"+num) cmd.Start() cmd.Process.Release() } // Make some fake arguments to parse. args := []string{ "-vv", "--offset=5", "-n", "Me", "--animal", "dog", // anything other than "cat" or "dog" will raise an error "-p", "3", "-s", "hello", "-s", "world", "--ptrslice", "hello", "--ptrslice", "world", "--intmap", "a:1", "--intmap", "b:5", "arg1", "arg2", "arg3", } // Parse flags from `args'. Note that here we use flags.ParseArgs for // the sake of making a working example. Normally, you would simply use // flags.Parse(&opts) which uses os.Args args, err := flags.ParseArgs(&opts, args) if err != nil { panic(err) } fmt.Printf("Verbosity: %v\n", opts.Verbose) fmt.Printf("Offset: %d\n", opts.Offset) fmt.Printf("Name: %s\n", opts.Name) fmt.Printf("Animal: %s\n", opts.Animal) fmt.Printf("Ptr: %d\n", *opts.Ptr) fmt.Printf("StringSlice: %v\n", opts.StringSlice) fmt.Printf("PtrSlice: [%v %v]\n", *opts.PtrSlice[0], *opts.PtrSlice[1]) fmt.Printf("IntMap: [a:%v b:%v]\n", opts.IntMap["a"], opts.IntMap["b"]) fmt.Printf("Remaining args: %s\n", strings.Join(args, " ")) ``` -------------------------------- ### System Call Dispatch Entry Points Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/golang.org/x/sys/unix/README.md These are the entry points in the hand-written assembly file for system call dispatch. Syscall and Syscall6 handle standard calls, while RawSyscall is for low-level use without scheduler notification. ```go func Syscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) ``` ```go func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2, err uintptr) ``` ```go func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2, err uintptr) ``` -------------------------------- ### Load Existing Cgroup (v1) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Loads an existing cgroup using its v1 static path. ```go control, err = cgroups.Load(cgroups.V1, cgroups.StaticPath("/test")) ``` -------------------------------- ### Process BOSH Releases with Version Filtering Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Use this tool to build release tarballs, create metalinks, and commit release metadata to an index repository. It filters releases based on a minimum version requirement. ```go // Usage: go run create-releases.go // Example: go run create-releases.go ./bosh ./releases-index/github.com/cloudfoundry/bosh "270.0" "s3://s3-external-1.amazonaws.com/boshio-release-tarballs" package main import ( "fmt" "os" "path/filepath" semver "github.com/cppforlife/go-semi-semantic/version" . "github.com/bosh-io/worker/src/worker/misc" ) func main() { // Initialize release index with directory and minimum version index := &ReleaseIndex{ DirPath: os.Args[2], // e.g., "releases-index/github.com/cloudfoundry/bosh" MinVersion: semver.MustNewVersionFromString(os.Args[3]), // e.g., "270.0" } // Configure metalink destination for S3 uploads meta4 := Meta4{Dst: os.Args[4]} // e.g., "s3://bucket/path" // Process all releases in the release directory err := process(os.Args[1], index, meta4) if err != nil { panic(fmt.Sprintf("Failed: %s", err)) } } // Expected output during processing: // [bosh-280.0.1] skipping (already indexed) // [bosh-280.1.0] importing // [bosh-280.1.0] processed tarball // [bosh-280.1.0] created metalink // [bosh-280.1.0] imported // Done ``` -------------------------------- ### Implementing the Wrapper Interface Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/hashicorp/errwrap/README.md Shows how to implement the Wrapper interface on a custom error type to enable compatibility with errwrap functions. ```go type AppError { Code ErrorCode Err error } func (e *AppError) WrappedErrors() []error { return []error{e.Err} } ``` -------------------------------- ### Import Release Versions from PostgreSQL Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md This command imports release versions from a PostgreSQL database into a specified index file. It queries the 'release_versions' table and pipes the output to a Go script for processing. ```bash $ psql db-name -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from release_versions;"| go run import-release-versions.go ~/workspace/bosh-io/releases-index/ ``` -------------------------------- ### Load Existing Cgroup (v2 Systemd) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Loads an existing systemd cgroup slice using the v2 unified hierarchy. ```go m, err := cgroupsv2.LoadSystemd("/", "my-cgroup-abc.slice") if err != nil { return err } ``` -------------------------------- ### Create a Request Generator Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/tedsuo/rata/README.md Initialize a RequestGenerator on the client side using the server's base URL and the defined routes. This helps in creating valid HTTP requests that conform to the defined routes. ```go requestGenerator := rata.NewRequestGenerator(server.URL, petRoutes) ``` -------------------------------- ### Commit, Push, and Create GitHub Release Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/onsi/gomega/RELEASING.md Standard Git commands to commit changes, push to the remote repository, and create a new tagged release on GitHub. Ensure the version number (vM.m.p) is correctly formatted. ```bash git commit -m "vM.m.p" git push gh release create "vM.m.p" git fetch --tags origin master ``` -------------------------------- ### Basic Error Wrapping and Checking Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/hashicorp/errwrap/README.md Demonstrates wrapping an error using Wrapf and checking for its presence or type using Contains and ContainsType. ```go // A function that always returns an error, but wraps it, like a real // function might. func tryOpen() error { _, err := os.Open("/i/dont/exist") if err != nil { return errwrap.Wrapf("Doesn't exist: {{err}}", err) } return nil } func main() { err := tryOpen() // We can use the Contains helpers to check if an error contains // another error. It is safe to do this with a nil error, or with // an error that doesn't even use the errwrap package. if errwrap.Contains(err, "does not exist") { // Do something } if errwrap.ContainsType(err, new(os.PathError)) { // Do something } // Or we can use the associated `Get` functions to just extract // a specific error. This would return nil if that specific error doesn't // exist. perr := errwrap.GetType(err, new(os.PathError)) } ``` -------------------------------- ### List Processes in Cgroup Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Lists all processes within a cgroup, with an option for recursive listing. ```go processes, err := control.Processes(cgroups.Devices, recursive) ``` -------------------------------- ### OS Interface for Filesystem Operations Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md The OS interface abstracts standard 'os' package functions like Open, Stat, Lstat, and PathSeparator. This allows for custom filesystem implementations, useful for testing or specific environments. ```go type OS interface { Lstat(name string) (os.FileInfo, error) Open(name string) (*os.File, error) PathSeparator() rune Stat(name string) (os.FileInfo, error) } ``` -------------------------------- ### Create New Cgroup (v2 Systemd Slice) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Creates a new systemd v2 cgroup slice. Note that systemd slices have specific naming conventions, and '-' is a special character. ```go import ( cgroupsv2 "github.com/containerd/cgroups/v2" specs "github.com/opencontainers/runtime-spec/specs-go" ) res := cgroupsv2.Resources{} // dummy PID of -1 is used for creating a "general slice" to be used as a parent cgroup. // see https://github.com/containerd/cgroups/blob/1df78138f1e1e6ee593db155c6b369466f577651/v2/manager.go#L732-L735 m, err := cgroupsv2.NewSystemd("/", "my-cgroup-abc.slice", -1, &res) if err != nil { return err } ``` -------------------------------- ### Import Stemcell Notes from PostgreSQL Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md This command imports stemcell notes from a PostgreSQL database into a specified index file. It queries the 'stemcell_notes' table and pipes the output to a Go script for processing. ```bash $ psql db-name -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from stemcell_notes;"|go run import-stemcell-notes.go ~/workspace/bosh-io/stemcells-legacy-index/ ``` -------------------------------- ### Import Release Job Metadata from PostgreSQL Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt A Go utility to import job metadata from a PostgreSQL database export into the releases index. It reads pipe-delimited data from stdin and writes job specifications to the appropriate index directories. Requires the path to the releases index directory. ```go // Usage: psql db -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from jobs;" | go run import-release-jobs.go package main import ( "bufio" "encoding/json" "fmt" "io" "os" "path/filepath" "strings" ) type ReleaseJobs struct { ReleasesIndexDir string } type key struct { Source string // e.g., "github.com/cloudfoundry/bosh" VersionRaw string // e.g., "280.1.0" } // Input format (pipe-delimited JSON): // {"Source":"github.com/cloudfoundry/bosh","VersionRaw":"280.1.0"} | [{"Name":"director",...}] func (r ReleaseJobs) Import(data io.Reader) error { rd := bufio.NewReader(data) for { line, err := rd.ReadString('\n') if err == io.EOF { break } pieces := strings.SplitN(line, "|", 2) var k key json.Unmarshal([]byte(pieces[0]), &k) var js []interface{} json.Unmarshal([]byte(pieces[1]), &js) // Find matching release directory paths, _ := filepath.Glob(filepath.Join( r.ReleasesIndexDir, k.Source, "*- "+k.VersionRaw, )) // Write jobs metadata jobsPath := filepath.Join(paths[0], "jobs.v1.yml") jsBytes, _ := json.MarshalIndent(js, "", " ") os.WriteFile(jobsPath, jsBytes, 0644) fmt.Printf("[%s@%s] processed\n", k.Source, k.VersionRaw) } return nil } ``` -------------------------------- ### Define Pet Resource Routes Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/tedsuo/rata/README.md Define a set of routes for a 'pet' resource using rata.Routes. Each route specifies a name, HTTP method, and a path pattern with parameters. ```go petRoutes := rata.Routes{ {Name: "get_pet", Method: rata.GET, Path: "/people/:owner_id/pets/:pet_id"}, {Name: "create_pet", Method: rata.POST, Path: "/people/:owner_id/pets"}, {Name: "update_pet", Method: rata.PUT, Path: "/people/:owner_id/pets/:pet_id"}, {Name: "delete_pet", Method: rata.DELETE, Path: "/people/:owner_id/pets/:pet_id"}, } ``` -------------------------------- ### Create a New Rata Router Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/tedsuo/rata/README.md Instantiate a new Rata router by combining the defined routes and handlers. The router implements http.Handler and can be used with standard Go http servers. ```go router, err := rata.NewRouter(petRoutes, petHandlers) if err != nil { panic(err) } // The router is just an http.Handler, so it can be used to create a server in the usual fashion: server := httptest.NewServer(router) ``` -------------------------------- ### Vendor Go Modules for Dependency Resolution Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md If encountering 'cannot find module providing package github.com/pkg/errors: import lookup disabled by -mod=vendor' errors, navigate to the 'worker/src' directory and run 'go mod vendor'. This command downloads and Vends all dependencies. ```bash cd src/worker go mod vendor cd ../.. ``` -------------------------------- ### Define HTTP Handlers for Pet Routes Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/tedsuo/rata/README.md Create a map of handler functions corresponding to the defined route names. Each handler function should implement the logic for its specific route. ```go petHandlers := rata.Handlers{ "get_pet": newGetPetHandler(), "create_pet": newCreatePetHandler(), "update_pet": newUpdatePetHandler(), "delete_pet": newDeletePetHandler() } ``` -------------------------------- ### Create Metalink File for Release Distribution Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Handles the creation and upload of metalink files, which describe release tarballs with checksums and download URLs for S3 distribution. Requires a destination S3 bucket URL and file details. ```go package misc // Meta4 handles metalink file operations for release distribution type Meta4 struct { Dst string // Destination S3 bucket URL } // File represents a release tarball type File struct { Path string // Local filesystem path Name string // Filename (e.g., "bosh-280.1.0.tgz") Version string // Release version } // Usage example: func createMetalink() { meta4 := Meta4{ Dst: "s3://s3-external-1.amazonaws.com/boshio-release-tarballs", } file := File{ Path: "/tmp/release-uuid-12345", Name: "bosh-280.1.0.tgz", Version: "280.1.0", } // Creates metalink, imports file with checksums, uploads to S3 meta4Path, err := meta4.Create(file) // Returns path to generated metalink file: /tmp/metalink-uuid-67890 // Generated metalink contains: // - File checksums (SHA-256, SHA-512) // - S3 download URL // - Version metadata } ``` -------------------------------- ### PathMatch: System-aware pattern matching Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md PathMatch checks if a name matches a pattern, automatically using the system's path separator. It serves as a replacement for filepath.Match. ```go func PathMatch(pattern, name string) (bool, error) ``` -------------------------------- ### Create and Process BOSH Release Objects Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Utilize the ReleaseFactory to create Release objects for processing BOSH release manifests. This includes building tarballs and extracting metadata for indexing. ```go package misc // ReleaseFactory creates Release instances with proper dependencies type ReleaseFactory struct{} // Release represents a BOSH release to be processed type Release struct { DirPath string // Path to release repository MFPath string // Path to release manifest YAML } // Usage example: func processRelease() { factory := ReleaseFactory{} // Create a new release from a manifest path release := factory.New( "/workspace/bosh-release", "/workspace/bosh-release/releases/bosh/bosh-280.1.0.yml", ) // Get release version version, err := release.Version() // Returns: semver.Version{280, 1, 0} // Get release name-version string nameVersion := release.NameVersion() // Returns: "bosh-280.1.0" // Process release: builds tarball, extracts metadata relMeta, jobsMeta, file, err := release.Process() // relMeta contains release manifest data // jobsMeta contains job specifications // file contains tarball path and name } ``` -------------------------------- ### Glob: Find files matching a pattern Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Glob searches the filesystem for files and directories that match the provided pattern. The pattern can be relative or absolute. This function is a replacement for filepath.Glob. ```go func Glob(pattern string) ([]string, error) ``` -------------------------------- ### Marshal and Unmarshal Go Structs to/from YAML Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/sigs.k8s.io/yaml/README.md Demonstrates marshaling a Go struct to YAML and unmarshaling YAML back into a struct. JSON struct tags are used to influence YAML field names. ```go package main import ( "fmt" "sigs.k8s.io/yaml" ) type Person struct { Name string `json:"name"` // Affects YAML field names too. Age int `json:"age"` } func main() { // Marshal a Person struct to YAML. p := Person{"John", 30} y, err := yaml.Marshal(p) if err != nil { fmt.Printf("err: %v\n", err) return } fmt.Println(string(y)) /* Output: age: 30 name: John */ // Unmarshal the YAML back into a Person struct. var p2 Person err = yaml.Unmarshal(y, &p2) if err != nil { fmt.Printf("err: %v\n", err) return } fmt.Println(p2) /* Output: {John 30} */ } ``` -------------------------------- ### Import Release Jobs from PostgreSQL Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md This command imports release jobs from a PostgreSQL database into a specified index file. It queries the 'jobs' table and pipes the output to a Go script for processing. ```bash $ psql db-name -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from jobs;"| go run import-release-jobs.go ~/workspace/bosh-io/releases-index/ ``` -------------------------------- ### Unmarshal and Marshal YAML data Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/gopkg.in/yaml.v2/README.md Demonstrates unmarshalling YAML into a struct or a map, and marshalling data back into YAML format. Struct fields must be exported to be populated during unmarshalling. ```go package main import ( "fmt" "log" "gopkg.in/yaml.v2" ) var data = ` a: Easy! b: c: 2 d: [3, 4] ` // Note: struct fields must be public in order for unmarshal to // correctly populate the data. type T struct { A string B struct { RenamedC int `yaml:"c"` D []int `yaml:",flow"` } } func main() { t := T{} err := yaml.Unmarshal([]byte(data), &t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t:\n%v\n\n", t) d, err := yaml.Marshal(&t) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- t dump:\n%s\n\n", string(d)) m := make(map[interface{}]interface{}) err = yaml.Unmarshal([]byte(data), &m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m:\n%v\n\n", m) d, err = yaml.Marshal(&m) if err != nil { log.Fatalf("error: %v", err) } fmt.Printf("--- m dump:\n%s\n\n", string(d)) } ``` -------------------------------- ### Run ANSI Tests Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/mgutz/ansi/README.md Execute the package tests to verify ANSI code functionality. This is typically done from the project directory. ```sh go test ``` -------------------------------- ### Create Subcgroup Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Creates a new subcgroup under the current cgroup, applying specified resources. ```go subCgroup, err := control.New("child", resources) ``` -------------------------------- ### Generate Concourse Pipeline for Organization in Go Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Generates a Concourse pipeline configuration in JSON format for a given GitHub organization, including resources, jobs, and tasks for building and uploading BOSH releases. Requires the release index file path as a command-line argument. ```go package main import ( "fmt" "log" "os" "github.com/bosh-io/worker/src/worker/pipelines" "github.com/bosh-io/worker/src/worker/releases" ) func main() { // Load the releases index index, err := releases.LoadFile(os.Args[1]) if err != nil { log.Fatalf("loading index: %v", err) } // Create a new pipeline for an organization org := pipelines.NewOrgPipeline("cloudfoundry") // Add all releases belonging to this organization for _, release := range index.FilterByOrg(org.Name()) { org.AddRelease(release) } // Output the pipeline as JSON (for use with fly set-pipeline) pipelineJSON := org.PipelineBytes() fmt.Println(string(pipelineJSON)) // Output: {"groups":[{"name":"all","jobs":["bosh","cf-deployment",...]}], // "resources":[...], "jobs":[...]} } ``` -------------------------------- ### Match a name against a pattern Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Use Match to determine if a given name conforms to a specified pattern. This function is a direct replacement for path.Match and always uses '/' as the path separator. ```go func Match(pattern, name string) (bool, error) ``` -------------------------------- ### Automate Concourse Pipeline Deployment Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt A bash script to automate Concourse pipeline deployment. It iterates over organizations in the releases index and deploys corresponding Concourse pipelines using the `fly` CLI. Ensure `fly` CLI is authenticated before running. ```bash #!/bin/bash # bin/sync-pipelines - Deploy Concourse pipelines for all organizations set -eu worker_dir="$( cd "$( dirname "$0" )" && cd .. && pwd )" releases_index="${worker_dir}/../bosh-io-releases/index.yml" # Ensure fly CLI is authenticated echo "If this fails run: fly -t bosh-io login -c https://bosh.ci.cloudfoundry.org/ -n x-bosh-io" fly -t bosh-io status # Pull latest releases index echo "Pulling latest releases/index.yml..." pushd "$( dirname "${releases_index}" )" >/dev/null git pull --ff-only popd >/dev/null # Generate and deploy pipeline for each organization echo "Synchronizing pipelines..." pushd ./src/worker >/dev/null while read org; do lowerOrg=$(echo "${org}" | tr '[:upper:]' '[:lower:]') echo "$org..." # Generate pipeline JSON and deploy via fly fly -t bosh-io set-pipeline -n \ -p "releases-${lowerOrg}" \ -c <( go run -mod=vendor main/generatepipeline/main.go "${releases_index}" "$org" ) done < <( set -eu go run -mod=vendor main/indexorgs/main.go "${releases_index}" ) popd >/dev/null echo "Done" # Example output: # Pulling latest releases/index.yml... # Synchronizing pipelines... # cloudfoundry... # pipeline created! # cloudfoundry-community... # pipeline created! # Done ``` -------------------------------- ### Import doublestar in Go code Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/bmatcuk/doublestar/README.md Import the doublestar package into your Go source files to use its functionalities for path matching and globbing. ```go import "github.com/bmatcuk/doublestar" ``` -------------------------------- ### Generate HTML Diff Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/aryann/difflib/README.md Compare two string slices and output the result as an HTML-formatted diff. ```go fmt.Println(difflib.HTMLDiff([]string{"one", "two", "three"}, []string{"two", "four", "three"})) ``` -------------------------------- ### Import Release Tarballs from PostgreSQL Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/README.md This command imports release tarballs from a PostgreSQL database into a specified index file. It queries the 'release_tarballs' table and pipes the output to a Go script for processing. ```bash $ psql db-name -t -c "select convert_from(key, 'utf-8'), convert_from(value, 'utf-8') from release_tarballs;"| go run import-release-tarballs.go ~/workspace/bosh-io/releases-index/ ``` -------------------------------- ### BOSH Release Index YAML Format Source: https://context7.com/cloudfoundry/bosh-io-worker/llms.txt Defines the structure for a BOSH release index YAML file, specifying GitHub repository URLs, categories, minimum version, and homepage visibility for each release. ```yaml # index.yml - Release definitions for bosh.io - url: https://github.com/cloudfoundry/bosh categories: - bosh min_version: "270.0" homepage: true - url: https://github.com/cloudfoundry/cf-deployment categories: - cf - deployment min_version: "1.0" homepage: true - url: https://github.com/cloudfoundry-community/postgres-boshrelease categories: - databases min_version: "0" homepage: false ``` -------------------------------- ### Enable jitter for randomized backoff Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/jpillora/backoff/README.md Configures the backoff instance to use jitter for randomized duration intervals. ```go import "math/rand" b := &backoff.Backoff{ Jitter: true, } rand.Seed(42) fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) fmt.Printf("Reset!\n") b.Reset() fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) fmt.Printf("%s\n", b.Duration()) ``` -------------------------------- ### Move Process to Another Cgroup Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Moves a process from its current cgroup to a specified destination cgroup. ```go err := control.MoveTo(destination) ``` -------------------------------- ### Register for Memory Pressure Event (v1) Source: https://github.com/cloudfoundry/bosh-io-worker/blob/main/src/worker/vendor/github.com/containerd/cgroups/README.md Registers for memory pressure events for v1 cgroups. Notifies via an eventfd when memory pressure reaches a specified level. ```go event := cgroups.MemoryPressureEvent(cgroups.MediumPressure, cgroups.DefaultMode) efd, err := control.RegisterMemoryEvent(event) ```