### Install Chaos Monkey Binary Source: https://github.com/netflix/chaosmonkey/blob/master/README.md Use 'go get' to install the Chaos Monkey binary locally. Ensure you have Go installed and configured. ```bash go get github.com/netflix/chaosmonkey/cmd/chaosmonkey ``` -------------------------------- ### Setup MySQL Database Source: https://context7.com/netflix/chaosmonkey/llms.txt Initialize the MySQL backend required for storing schedules and termination logs. ```bash # Start MySQL locally using Docker (for development) docker run -e MYSQL_ROOT_PASSWORD=password -p3306:3306 mysql:8.0 ``` ```sql -- Connect to MySQL and create the chaosmonkey database mysql> CREATE DATABASE chaosmonkey; ``` ```bash # Apply database migrations to create required tables chaosmonkey migrate ``` -------------------------------- ### Install Chaos Monkey Binary Source: https://context7.com/netflix/chaosmonkey/llms.txt Use the Go toolchain to install the binary and verify the installation version. ```bash # Install the chaosmonkey binary using go get go get github.com/netflix/chaosmonkey/cmd/chaosmonkey # Verify the installation chaosmonkey --version # Output: 2.0.2 ``` -------------------------------- ### Start Local MySQL Container Source: https://github.com/netflix/chaosmonkey/blob/master/docs/Running-locally.md Use this command to start a MySQL container locally. The root password will be set to 'password'. Ensure Docker is installed and running. ```bash docker run -e MYSQL_ROOT_PASSWORD=password -p3306:3306 mysql:8.0 ``` -------------------------------- ### Run All Unit Tests Source: https://github.com/netflix/chaosmonkey/blob/master/docs/dev/Running-tests.md Execute all unit tests in the project. Ensure you have Go installed and configured. ```bash go test ./... ``` -------------------------------- ### Example Chaos Monkey App Configuration Output Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md This is an example of the output when successfully fetching Chaos Monkey's configuration for an application. It details enabled status, kill intervals, and grouping. ```go (*chaosmonkey.AppConfig)(0xc4202ec0c0)({ Enabled: (bool) true, RegionsAreIndependent: (bool) true, MeanTimeBetweenKillsInWorkDays: (int) 2, MinTimeBetweenKillsInWorkDays: (int) 1, Grouping: (chaosmonkey.Group) cluster, Exceptions: ([]chaosmonkey.Exception) { } }) ``` -------------------------------- ### Cron Job Setup Source: https://context7.com/netflix/chaosmonkey/llms.txt Instructions for setting up cron jobs to automate Chaos Monkey scheduling and termination scripts. ```APIDOC ## Cron Job Setup ### Scheduling Script Create `/apps/chaosmonkey/chaosmonkey-schedule.sh`: ```bash #!/bin/bash # chaosmonkey-schedule.sh - Daily scheduler script /apps/chaosmonkey/chaosmonkey schedule >> /var/log/chaosmonkey-schedule.log 2>&1 ``` ### Termination Script Create `/apps/chaosmonkey/chaosmonkey-terminate.sh`: ```bash #!/bin/bash # chaosmonkey-terminate.sh - Termination wrapper script /apps/chaosmonkey/chaosmonkey terminate "$@" >> /var/log/chaosmonkey-terminate.log 2>&1 ``` ### Cron Configuration Create `/etc/cron.d/chaosmonkey-schedule`: ```bash # Run the Chaos Monkey scheduler at 5AM PDT (12:00 UTC) every weekday # Must run as root to write to /etc/cron.d/chaosmonkey-daily-terminations # min hour dom month day user command 0 12 * * 1-5 root /apps/chaosmonkey/chaosmonkey-schedule.sh ``` ``` -------------------------------- ### Example Chaos Monkey TOML Configuration Source: https://github.com/netflix/chaosmonkey/blob/master/docs/Configuration-file-format.md This is an example of a Chaos Monkey configuration file in TOML format. It includes settings for the chaos monkey itself, database connection, and Spinnaker integration. Note that 'encrypted_password' fields should contain the unencrypted password. ```toml [chaosmonkey] enabled = true schedule_enabled = true leashed = false accounts = ["production", "test"] [database] host = "dbhost.example.com" name = "chaosmonkey" user = "chaosmonkey" encrypted_password = "securepasswordgoeshere" [spinnaker] endpoint = "http://spinnaker.example.com:8084" ``` -------------------------------- ### Configure Chaos Monkey via YAML Source: https://context7.com/netflix/chaosmonkey/llms.txt Example configuration for Chaos Monkey settings, including kill frequency, grouping, and exception rules. ```yaml chaosMonkey: enabled: true meanTimeBetweenKillsInWorkDays: 2 # Average days between kills minTimeBetweenKillsInWorkDays: 1 # Minimum days between kills grouping: cluster # Options: app, stack, cluster regionsAreIndependent: true # Treat regions separately exceptions: - account: prod region: us-west-2 stack: staging detail: "" # Blank detail field - account: test region: "*" # Wildcard matches all stack: "*" detail: "*" ``` -------------------------------- ### Chaos Monkey Default Configuration Values Source: https://github.com/netflix/chaosmonkey/blob/master/docs/Configuration-file-format.md This example displays all default values for Chaos Monkey configuration parameters. It covers settings for instance termination, scheduling, time zones, paths, and integration with database and Spinnaker. Many parameters like 'decryptor' and 'trackers' have no-op implementations. ```toml [chaosmonkey] enabled = false # if false, won't terminate instances when invoked leashed = true # if true, terminations are only simulated (logged only) schedule_enabled = false # if true, will generate schedule of terminations each weekday accounts = [] # list of Spinnaker accounts with chaos monkey enabled, e.g.: ["prod", "test"] start_hour = 9 # time during day when starts terminating end_hour = 15 # time during day when stops terminating # tzdata format, see TZ column in https://en.wikipedia.org/wiki/List_of_tz_database_time_zones # Other allowed values: "UTC", "Local" time_zone = "America/Los_Angeles" # time zone used by start.hour and end.hour term_account = "root" # account used to run the term_path command max_apps = 2147483647 # max number of apps Chaos Monkey will schedule terminations for # location of command Chaos Monkey uses for doing terminations term_path = "/apps/chaosmonkey/chaosmonkey-terminate.sh" # cron file that Chaos Monkey writes to each day for scheduling kills cron_path = "/etc/cron.d/chaosmonkey-daily-terminations" # decryption system for encrypted_password fields for spinnaker and database decryptor = "" # event tracking systems that records chaos monkey terminations trackers = [] # metric collection systems that track errors for monitoring/alerting error_counter = "" # outage checking system that tells chaos monkey if there is an ongoing outage outage_checker = "" [database] host = "" # database host port = 3306 # tcp port that the database is listening on user = "" # database user encrypted_password = "" # password for database auth, encrypted by decryptor name = "" # name of database that contains chaos monkey data [spinnaker] endpoint = "" # spinnaker api url certificate = "" # path to p12 file when using client-side tls certs encrypted_password = "" # password used for p12 certificate, encrypted by decryptor user = "" # user associated with terminations, sent in API call to terminate # For dynamic configuration options, see viper docs [dynamic] provider = "" # options: "etcd", "consul" endpoint = "" # url for dynamic provider path = "" # path for dynamic provider ``` -------------------------------- ### Run MySQL Integration Tests with Docker Source: https://github.com/netflix/chaosmonkey/blob/master/docs/dev/Running-tests.md Execute unit tests that interact with MySQL using Docker. The tests will automatically manage the MySQL container lifecycle. Requires Docker to be installed and running. ```bash go test -tags docker ./... ``` -------------------------------- ### Run MySQL Integration Tests without Docker Container Management Source: https://github.com/netflix/chaosmonkey/blob/master/docs/dev/Running-tests.md Execute MySQL integration tests while keeping a Docker container running or using a native MySQL instance. This is useful for faster test runs or when testing with a pre-existing MySQL setup. Requires the 'docker' and 'dockerup' tags. ```bash go test -tags "docker dockerup" ./... ``` -------------------------------- ### Implement Custom Chaos Monkey Constrainer in Go Source: https://github.com/netflix/chaosmonkey/blob/master/docs/plugins/Constrainer.md Define a custom constrainer to filter out specific termination candidates. This example prevents terminations for apps with 'foo' in their name. Ensure the constrainer is registered via deps.GetConstrainer. ```go package constrainer import ( "github.com/Netflix/chaosmonkey/deps" "github.com/Netflix/chaosmonkey/config" "github.com/Netflix/chaosmonkey/schedule" "strings" ) func init() { deps.GetConstrainer = getConstrainer() } type noFoo struct {} func getConstrainer(cfg *config.Monkey) (schedule.Constrainer, error) { return noFoo{}, nil } func (n noFoo) Filter(s schedule.Schedule) schedule.Schedule { result := schedule.New() for _, entry := range s.Entries() { if !strings.Contains(entry.Group.App(), "foo") { result.Add(entry.Time, entry.Group) } } return result } ``` -------------------------------- ### Initialize AppConfig in Go Source: https://context7.com/netflix/chaosmonkey/llms.txt Create a new application configuration instance with custom exception rules. ```go package main import ( "fmt" "github.com/Netflix/chaosmonkey/v2" ) func main() { // Create a new AppConfig with default settings exceptions := []chaosmonkey.Exception{ {Account: "test", Stack: "*", Detail: "*", Region: "*"}, } config := chaosmonkey.NewAppConfig(exceptions) // Default values: // - Enabled: true // - RegionsAreIndependent: true // - MeanTimeBetweenKillsInWorkDays: 5 // - Grouping: Cluster fmt.Printf("Enabled: %v\n", config.Enabled) fmt.Printf("Mean time between kills: %d days\n", config.MeanTimeBetweenKillsInWorkDays) fmt.Printf("Grouping: %s\n", config.Grouping) // Output: cluster } ``` -------------------------------- ### Load Custom Plugins in Main Source: https://github.com/netflix/chaosmonkey/blob/master/docs/plugins/index.md Register custom plugins by importing them with an underscore in the main package to trigger their init functions. ```go package main import ( "github.com/Netflix/chaosmonkey/command" _ "example.com/chaosmonkey/constrainer" ) func main() { command.Execute() } ``` -------------------------------- ### Build a Custom Chaos Monkey Binary Source: https://context7.com/netflix/chaosmonkey/llms.txt Create a main entry point that imports built-in and custom plugins for side-effect registration. ```go package main import ( "github.com/Netflix/chaosmonkey/v2/command" // Import built-in plugins _ "github.com/Netflix/chaosmonkey/v2/constrainer" _ "github.com/Netflix/chaosmonkey/v2/decryptor" _ "github.com/Netflix/chaosmonkey/v2/env" _ "github.com/Netflix/chaosmonkey/v2/errorcounter" _ "github.com/Netflix/chaosmonkey/v2/outage" _ "github.com/Netflix/chaosmonkey/v2/tracker" // Import your custom plugins (anonymous import for side effects) _ "example.com/chaosmonkey/myconstrainer" _ "example.com/chaosmonkey/mytracker" ) func main() { command.Execute() } ``` ```bash # Build custom chaosmonkey binary go build -o chaosmonkey ./cmd/chaosmonkey # Or install to $GOBIN go install ./cmd/chaosmonkey ``` -------------------------------- ### Migrate Database Schema for Chaos Monkey Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Run this command to create the necessary tables in the 'chaosmonkey' database. Ensure your MySQL credentials are correctly configured in chaosmonkey.toml before running this. ```bash chaosmonkey migrate ``` -------------------------------- ### Implement Custom Tracker in Go Source: https://context7.com/netflix/chaosmonkey/llms.txt Create a custom tracker by implementing the Tracker interface to record termination events. ```go package tracker import ( "log" "github.com/Netflix/chaosmonkey/v2" ) // SyslogTracker records terminations to syslog type SyslogTracker struct{} // Track implements the chaosmonkey.Tracker interface func (t SyslogTracker) Track(trm chaosmonkey.Termination) error { log.Printf("CHAOS_MONKEY_TERMINATION: app=%s account=%s region=%s instance=%s leashed=%v", trm.Instance.AppName(), trm.Instance.AccountName(), trm.Instance.RegionName(), trm.Instance.ID(), trm.Leashed) return nil } // Register the tracker with the dependency system func init() { // Registration code to make this tracker available // when "syslog" is specified in config } ``` -------------------------------- ### Build Custom Chaos Monkey Binary Source: https://github.com/netflix/chaosmonkey/blob/master/docs/plugins/index.md Compile the custom binary using the go build command pointing to your local main package. ```bash go build example.com/chaosmonkey/cmd/chaosmonkey ``` -------------------------------- ### Fetch Chaos Monkey App Configuration Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Verify Chaos Monkey's Spinnaker integration by fetching the configuration for a specific application. Successful execution shows the application's Chaos Monkey settings. ```bash chaosmonkey config ``` -------------------------------- ### Verify Database Connectivity Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Use the fetch-schedule command to confirm Chaos Monkey can communicate with the database. ```bash chaosmonkey fetch-schedule ``` ```text [69400] 2016/09/30 23:41:03 chaosmonkey fetch-schedule starting [69400] 2016/09/30 23:41:03 Writing /etc/cron.d/chaosmonkey-daily-terminations [69400] 2016/09/30 23:41:03 chaosmonkey fetch-schedule done ``` ```text [69668] 2016/09/30 23:43:50 chaosmonkey fetch-schedule starting [69668] 2016/09/30 23:43:50 FATAL: could not fetch schedule: failed to retrieve schedule for 2016-09-30 23:43:50.953795019 -0700 PDT: dial tcp 127.0.0.1:3306: getsockopt: connection refused ``` -------------------------------- ### Create Chaos Monkey MySQL Database Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Execute this SQL command to create the 'chaosmonkey' database. Chaos Monkey requires this database for storing termination schedules and enforcing kill intervals. ```sql mysql> CREATE DATABASE chaosmonkey; ``` -------------------------------- ### Create Termination Script Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Create a wrapper script for instance termination to be called by cron jobs. ```bash #!/bin/bash /apps/chaosmonkey/chaosmonkey terminate "$@" >> /var/log/chaosmonkey-terminate.log 2>&1 ``` -------------------------------- ### Create Daily Schedule Script Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Create a shell script to invoke the scheduler and log output. ```bash #!/bin/bash /apps/chaosmonkey/chaosmonkey schedule >> /var/log/chaosmonkey-schedule.log 2>&1 ``` -------------------------------- ### List Chaos Monkey Clusters Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists clusters for a given app and account. ```bash chaosmonkey clusters myapp production ``` -------------------------------- ### Define Grouping Types in Go Source: https://context7.com/netflix/chaosmonkey/llms.txt Demonstrates the available grouping options for determining the scope of instance termination. ```go package main import ( "fmt" "github.com/Netflix/chaosmonkey/v2" ) func main() { // Available grouping options groups := []chaosmonkey.Group{ chaosmonkey.App, // Kill one instance per app per day chaosmonkey.Stack, // Kill one instance per stack per day chaosmonkey.Cluster, // Kill one instance per cluster per day } for _, g := range groups { fmt.Printf("Grouping: %s\n", g.String()) } // Output: // Grouping: app // Grouping: stack // Grouping: cluster } ``` -------------------------------- ### AppConfig Configuration Source: https://context7.com/netflix/chaosmonkey/llms.txt Defines the core configuration structure for per-app Chaos Monkey settings. ```APIDOC ## AppConfig Structure ### Description The AppConfig structure manages the operational settings for Chaos Monkey, including enabled status, timing intervals, and grouping strategies. ### Fields - **Enabled** (bool) - Whether Chaos Monkey is active for the app. - **MeanTimeBetweenKillsInWorkDays** (int) - Average days between termination events. - **Grouping** (string) - The scope of termination (app, stack, or cluster). - **Exceptions** ([]Exception) - List of exclusion rules for specific environments. ``` -------------------------------- ### chaosmonkey fetch-schedule Source: https://context7.com/netflix/chaosmonkey/llms.txt Retrieves the existing termination schedule from the database and configures the system's cron jobs accordingly. ```APIDOC ## chaosmonkey fetch-schedule ### Description Fetches the termination schedule from the database and sets up the necessary cron jobs to execute these terminations automatically. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters None ### Request Example ```bash chaosmonkey fetch-schedule ``` ### Response #### Success Response (Output) - **Log messages** - Indicates the start and completion of the fetch-schedule process, including confirmation of cron job file creation. #### Response Example ``` [12345] 2024/01/15 09:00:00 chaosmonkey fetch-schedule starting [12345] 2024/01/15 09:00:00 Writing /etc/cron.d/chaosmonkey-daily-terminations [12345] 2024/01/15 09:00:00 chaosmonkey fetch-schedule done ``` ``` -------------------------------- ### Query Chaos Monkey Configuration Source: https://context7.com/netflix/chaosmonkey/llms.txt Displays Chaos Monkey configuration. Use 'chaosmonkey config' to dump monkey-level configuration or 'chaosmonkey config ' to query Spinnaker for app-specific settings. ```bash # Dump monkey-level configuration chaosmonkey config ``` ```bash # Query Spinnaker for app-specific config chaosmonkey config myapp ``` -------------------------------- ### Spinnaker Configuration Source: https://context7.com/netflix/chaosmonkey/llms.txt Instructions for enabling and configuring Chaos Monkey within the Spinnaker UI. ```APIDOC ## Spinnaker Configuration ### Enabling Chaos Monkey in Spinnaker Deck Edit `/var/www/settings.js` in Spinnaker Deck: ```javascript // settings.js - Enable Chaos Monkey feature flag window.spinnakerSettings = { feature: { pipelines: true, notifications: false, fastProperty: true, chaosMonkey: true, // Add this line to enable Chaos Monkey UI // ... other features }, // ... other settings }; ``` ### App Configuration via Spinnaker UI Configure termination frequency and exceptions through Spinnaker's Config tab: ```yaml ``` -------------------------------- ### Define Termination Structure in Go Source: https://context7.com/netflix/chaosmonkey/llms.txt Create a termination object to track instance shutdown events, including leashed mode for simulation. ```go package main import ( "fmt" "time" "github.com/Netflix/chaosmonkey/v2" ) // MockInstance implements chaosmonkey.Instance interface type MockInstance struct{} func (m MockInstance) AppName() string { return "myapp" } func (m MockInstance) AccountName() string { return "production" } func (m MockInstance) RegionName() string { return "us-east-1" } func (m MockInstance) StackName() string { return "prod" } func (m MockInstance) ClusterName() string { return "myapp-prod" } func (m MockInstance) ASGName() string { return "myapp-prod-v001" } func (m MockInstance) ID() string { return "i-abc123def456" } func (m MockInstance) CloudProvider() string { return "aws" } func main() { instance := MockInstance{} termination := chaosmonkey.Termination{ Instance: instance, Time: time.Now(), Leashed: false, // Set to true to simulate without killing } fmt.Printf("Terminating instance %s at %v (leashed: %v)\n", termination.Instance.ID(), termination.Time, termination.Leashed) } ``` -------------------------------- ### Match Exceptions in Go Source: https://context7.com/netflix/chaosmonkey/llms.txt Use the Exception type to define and verify exclusion rules for specific infrastructure targets. ```go package main import ( "fmt" "github.com/Netflix/chaosmonkey/v2" ) func main() { // Define an exception that opts out prod/us-west-2/staging clusters ex := chaosmonkey.Exception{ Account: "prod", Stack: "staging", Detail: "*", // Wildcard matches any detail Region: "us-west-2", } // Check if an ASG matches this exception matches := ex.Matches("prod", "staging", "web", "us-west-2") fmt.Printf("Matches exception: %v\n", matches) // Output: true // Different region won't match matches = ex.Matches("prod", "staging", "web", "us-east-1") fmt.Printf("Matches exception: %v\n", matches) // Output: false // Wildcard exception for entire test account testEx := chaosmonkey.Exception{ Account: "test", Stack: "*", Detail: "*", Region: "*", } matches = testEx.Matches("test", "anything", "whatever", "any-region") fmt.Printf("Test account exception matches: %v\n", matches) // Output: true } ``` -------------------------------- ### Implement a Custom Decryptor Source: https://context7.com/netflix/chaosmonkey/llms.txt Create a struct that satisfies the chaosmonkey.Decryptor interface to handle custom secret decryption logic. ```go package decryptor import ( "encoding/base64" "github.com/Netflix/chaosmonkey/v2" ) // Base64Decryptor implements simple base64 decoding type Base64Decryptor struct{} // Decrypt implements the chaosmonkey.Decryptor interface func (d Base64Decryptor) Decrypt(ciphertext string) (string, error) { decoded, err := base64.StdEncoding.DecodeString(ciphertext) if err != nil { return "", err } return string(decoded), nil } // GetDecryptor returns the decryptor based on config func GetDecryptor(name string) (chaosmonkey.Decryptor, error) { switch name { case "base64": return Base64Decryptor{}, nil default: // Return no-op decryptor that passes through plaintext return NoOpDecryptor{}, nil } } ``` -------------------------------- ### List Chaos Monkey Regions Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists regions for a given cluster and account. ```bash chaosmonkey regions myapp-prod production ``` -------------------------------- ### Configure Chaos Monkey Source: https://context7.com/netflix/chaosmonkey/llms.txt Define the application behavior, database connection, and Spinnaker integration in the TOML configuration file. ```toml # chaosmonkey.toml - Basic configuration example [chaosmonkey] enabled = true # Enable instance terminations schedule_enabled = true # Generate termination schedules each weekday leashed = false # Set to true to simulate terminations without actually killing accounts = ["production", "test"] # Spinnaker accounts with chaos monkey enabled start_hour = 9 # Start terminating at 9 AM end_hour = 15 # Stop terminating at 3 PM time_zone = "America/Los_Angeles" # Time zone for start/end hours term_account = "root" # Account used to run termination commands term_path = "/apps/chaosmonkey/chaosmonkey-terminate.sh" cron_path = "/etc/cron.d/chaosmonkey-daily-terminations" max_apps = 2147483647 # Max number of apps to schedule terminations for # Optional plugins decryptor = "" trackers = [] error_counter = "" outage_checker = "" [database] host = "dbhost.example.com" port = 3306 name = "chaosmonkey" user = "chaosmonkey" encrypted_password = "yourpassword" # Plain text despite the name (no-op decryptor) [spinnaker] endpoint = "http://spinnaker.example.com:8084" certificate = "" # Path to p12 file for TLS client certs encrypted_password = "" # Password for p12 certificate user = "chaosmonkey@example.com" # User associated with terminations # Optional: Dynamic configuration via etcd or Consul [dynamic] provider = "" # Options: "etcd", "consul" endpoint = "" path = "" ``` -------------------------------- ### List Chaos Monkey Eligible Instances Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists instances eligible for termination for a given app and account. Filters can be applied by --region, --stack, or --cluster. ```bash # List all eligible instances for an app and account chaosmonkey eligible myapp production ``` ```bash # Filter by region chaosmonkey eligible myapp production --region=us-east-1 ``` ```bash # Filter by stack chaosmonkey eligible myapp production --stack=prod ``` ```bash # Filter by cluster chaosmonkey eligible myapp production --cluster=myapp-prod ``` -------------------------------- ### Generate Chaos Monkey Schedule Source: https://context7.com/netflix/chaosmonkey/llms.txt Generates a termination schedule. Use --apps to specify applications, --max-apps to limit the number of apps, and --no-record-schedule for a dry run without database recording. ```bash # Generate schedule for specific apps only (useful for debugging) chaosmonkey schedule --apps=myapp,otherapp ``` ```bash # Limit to first N apps (useful for testing) chaosmonkey schedule --max-apps=10 ``` ```bash # Generate schedule without recording to database (dry run) chaosmonkey schedule --no-record-schedule ``` -------------------------------- ### Exception Matching Source: https://context7.com/netflix/chaosmonkey/llms.txt Defines how to create and evaluate exception rules to prevent Chaos Monkey from terminating specific instances. ```APIDOC ## Exception Matching ### Description Exceptions allow fine-grained control over which instances are excluded from termination based on account, stack, detail, and region. ### Methods - **Matches(account, stack, detail, region)** (bool) - Evaluates if the provided instance attributes match the exception rule, supporting wildcards ('*'). ``` -------------------------------- ### Fetch Chaos Monkey Schedule Source: https://context7.com/netflix/chaosmonkey/llms.txt Retrieves the existing termination schedule from the database and configures cron jobs for daily terminations. ```bash # Fetch today's schedule from database and configure cron chaosmonkey fetch-schedule ``` -------------------------------- ### Termination Tracking Source: https://context7.com/netflix/chaosmonkey/llms.txt Defines the structure for recording instance terminations and implementing custom tracking logic. ```APIDOC ## Termination and Tracker ### Description Represents a termination event and provides an interface for custom tracking implementations. ### Termination Fields - **Instance** (Instance) - The target instance being terminated. - **Time** (time.Time) - Timestamp of the event. - **Leashed** (bool) - If true, simulates the termination without actually killing the instance. ### Tracker Interface - **Track(Termination)** (error) - Interface method to implement custom logging or reporting for termination events. ``` -------------------------------- ### Look Up Cloud Provider Source: https://context7.com/netflix/chaosmonkey/llms.txt Looks up the cloud provider associated with an account name. ```bash chaosmonkey provider production ``` -------------------------------- ### Generate Termination Schedule Source: https://context7.com/netflix/chaosmonkey/llms.txt Execute the schedule command to generate daily termination plans and update local cron jobs. ```bash # Generate a full termination schedule for all apps chaosmonkey schedule ``` -------------------------------- ### Configure Cron Job for Scheduling Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Define a cron entry to run the scheduler script on weekdays. ```cron # Run the Chaos Monkey scheduler at 5AM PDT (4AM PST) every weekday # This corresponds to: 12:00 UTC # Because system clock runs UTC, time change affects when job runs # The scheduler must run as root because it needs root permissions to write # to the file /etc/cron.d/chaosmonkey-daily-terminations # min hour dom month day user command 0 12 * * 1-5 root /apps/chaosmonkey/chaosmonkey-schedule.sh ``` -------------------------------- ### Terminate Instance with Chaos Monkey Source: https://context7.com/netflix/chaosmonkey/llms.txt Terminates a randomly selected instance from a given app and account. Filters can be applied using --region, --cluster, and --stack for precise targeting. Use --leashed to simulate termination without actual killing. ```bash # Basic termination - randomly selects instance from app in account chaosmonkey terminate myapp production ``` ```bash # Terminate instance from specific region chaosmonkey terminate myapp production --region=us-east-1 ``` ```bash # Terminate instance from specific cluster chaosmonkey terminate myapp production --cluster=myapp-prod ``` ```bash # Terminate from specific stack chaosmonkey terminate myapp production --stack=prod ``` ```bash # Combine filters for precise targeting chaosmonkey terminate myapp production --region=us-east-1 --stack=prod --cluster=myapp-prod ``` ```bash # Run in leashed mode (simulate without actual termination) chaosmonkey terminate myapp production --leashed ``` -------------------------------- ### chaosmonkey clusters Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists all clusters associated with a given application and account. ```APIDOC ## chaosmonkey clusters ### Description Lists the clusters available for a specified application and account. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **app** (string) - Required - The name of the application. - **account** (string) - Required - The name of the cloud account. ### Request Example ```bash chaosmonkey clusters myapp production ``` ### Response #### Success Response (Output) - **Cluster names** - A list of cluster names belonging to the application and account. #### Response Example ``` myapp-prod myapp-staging myapp-canary ``` ``` -------------------------------- ### chaosmonkey config Source: https://context7.com/netflix/chaosmonkey/llms.txt Queries and displays Chaos Monkey configuration, either globally or for a specific application. ```APIDOC ## chaosmonkey config ### Description Retrieves and displays the current configuration of Chaos Monkey. This can be the overall monkey-level configuration or specific configuration for an application. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **app** (string) - Optional - The name of the application to query configuration for. If omitted, displays global configuration. ### Request Example ```bash chaosmonkey config chaosmonkey config myapp ``` ### Response #### Success Response (Output) - **Configuration details** - Displays the configuration parameters, which may include enabled status, kill settings, grouping, and exceptions. #### Response Example ``` (*chaosmonkey.AppConfig)(0xc4202ec0c0)({ Enabled: (bool) true, RegionsAreIndependent: (bool) true, MeanTimeBetweenKillsInWorkDays: (int) 2, MinTimeBetweenKillsInWorkDays: (int) 1, Grouping: (chaosmonkey.Group) cluster, Exceptions: ([]chaosmonkey.Exception) { } }) ``` ``` -------------------------------- ### Terminate Instance Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Manually invoke the termination command for a specific cluster and region. ```bash chaosmonkey terminate chaosguineapig test --cluster=chaosguineapig --region=us-east-1 ``` -------------------------------- ### chaosmonkey provider Source: https://context7.com/netflix/chaosmonkey/llms.txt Looks up the cloud provider (e.g., AWS) by providing the account name. ```APIDOC ## chaosmonkey provider ### Description Retrieves the cloud provider associated with a given account name. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **account_name** (string) - Required - The name of the cloud account. ### Request Example ```bash chaosmonkey provider production ``` ### Response #### Success Response (Output) - **Provider name** - The name of the cloud provider (e.g., 'aws'). #### Response Example ``` aws ``` ``` -------------------------------- ### chaosmonkey eligible Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists all instances that are eligible for termination based on the specified application, account, and optional filters. ```APIDOC ## chaosmonkey eligible ### Description Lists instances that are eligible for termination. This command can filter eligible instances by region, stack, and cluster. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **app** (string) - Required - The name of the application. - **account** (string) - Required - The name of the cloud account. #### Query Parameters - **--region** (string) - Optional - Filters eligible instances by the specified region. - **--stack** (string) - Optional - Filters eligible instances by the specified stack. - **--cluster** (string) - Optional - Filters eligible instances by the specified cluster. ### Request Example ```bash chaosmonkey eligible myapp production chaosmonkey eligible myapp production --region=us-east-1 chaosmonkey eligible myapp production --stack=prod chaosmonkey eligible myapp production --cluster=myapp-prod ``` ### Response #### Success Response (Output) - **Instance IDs** - A list of instance IDs that are eligible for termination. #### Response Example ``` i-abc123def456 i-789ghi012jkl i-345mno678pqr ``` ``` -------------------------------- ### Look Up Cloud Account ID Source: https://context7.com/netflix/chaosmonkey/llms.txt Looks up a cloud account ID by its name. ```bash chaosmonkey account production ``` -------------------------------- ### Generate Termination Schedule Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Manually trigger schedule generation, optionally limiting the number of applications processed. ```bash chaosmonkey schedule --no-record-schedule --max-apps=10 ``` -------------------------------- ### chaosmonkey schedule Source: https://context7.com/netflix/chaosmonkey/llms.txt Generates and optionally records a schedule for terminating instances. Useful for debugging and testing termination strategies. ```APIDOC ## chaosmonkey schedule ### Description Generates a schedule for terminating instances. Can be used to target specific applications, limit the number of instances, or perform a dry run without recording. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Query Parameters - **--apps** (string) - Optional - Comma-separated list of application names to include in the schedule. - **--max-apps** (integer) - Optional - Limits the schedule generation to the first N apps. - **--no-record-schedule** (boolean) - Optional - If set, the schedule will not be recorded to the database (dry run). ### Request Example ```bash chaosmonkey schedule --apps=myapp,otherapp chaosmonkey schedule --max-apps=10 chaosmonkey schedule --no-record-schedule ``` ### Response #### Success Response (Output) - **Log messages** - Indicates the start and end of the schedule generation process, along with details of scheduled terminations. #### Response Example ``` [12345] 2024/01/15 09:00:00 chaosmonkey schedule starting myapp-prod mtbk=5 kill=true myapp-test mtbk=5 kill=false [12345] 2024/01/15 09:00:05 chaosmonkey schedule done ``` ``` -------------------------------- ### chaosmonkey terminate Source: https://context7.com/netflix/chaosmonkey/llms.txt Terminates a specific instance from an application and account. Supports filtering by region, cluster, and stack, and can run in a simulated 'leashed' mode. ```APIDOC ## chaosmonkey terminate ### Description Terminates an instance from a given app and account. Allows for precise targeting using various filters and can simulate termination without actual impact. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **app** (string) - Required - The name of the application. - **account** (string) - Required - The name of the cloud account. #### Query Parameters - **--region** (string) - Optional - The specific region to terminate an instance from. - **--cluster** (string) - Optional - The specific cluster to terminate an instance from. - **--stack** (string) - Optional - The specific stack to terminate an instance from. - **--leashed** (boolean) - Optional - If set, simulates termination without actually killing the instance. ### Request Example ```bash chaosmonkey terminate myapp production chaosmonkey terminate myapp production --region=us-east-1 chaosmonkey terminate myapp production --cluster=myapp-prod chaosmonkey terminate myapp production --stack=prod chaosmonkey terminate myapp production --region=us-east-1 --stack=prod --cluster=myapp-prod chaosmonkey terminate myapp production --leashed ``` ### Response #### Success Response (Output) - **Log messages** - Indicates the termination process, including instance details or simulation information if `--leashed` is used. #### Response Example (Leashed) ``` [12345] 2024/01/15 14:30:00 leashed=true, not killing instance i-abc123def ``` ``` -------------------------------- ### Cron Configuration for Chaos Monkey Scheduler Source: https://context7.com/netflix/chaosmonkey/llms.txt Cron job configuration to run the Chaos Monkey scheduler daily at 5 AM PDT (12:00 UTC) on weekdays. Requires root privileges to write to /etc/cron.d. ```cron # min hour dom month day user command 0 12 * * 1-5 root /apps/chaosmonkey/chaosmonkey-schedule.sh ``` -------------------------------- ### Pull MySQL Docker Image Source: https://github.com/netflix/chaosmonkey/blob/master/docs/dev/Running-tests.md Download the MySQL 8.0 Docker image, which is used for integration tests. This version ensures compatibility with Amazon Aurora. ```bash docker pull mysql:8.0 ``` -------------------------------- ### chaosmonkey regions Source: https://context7.com/netflix/chaosmonkey/llms.txt Lists all regions associated with a given cluster and account. ```APIDOC ## chaosmonkey regions ### Description Lists the regions available for a specified cluster and account. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **cluster** (string) - Required - The name of the cluster. - **account** (string) - Required - The name of the cloud account. ### Request Example ```bash chaosmonkey regions myapp-prod production ``` ### Response #### Success Response (Output) - **Region names** - A list of region names where the cluster is deployed. #### Response Example ``` us-east-1 us-west-2 eu-west-1 ``` ``` -------------------------------- ### Define Database Schema Source: https://context7.com/netflix/chaosmonkey/llms.txt The database schema includes tables for tracking daily schedules and historical termination events. ```sql -- schedules table: Stores daily termination schedules CREATE TABLE IF NOT EXISTS schedules ( id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, date DATE NOT NULL, -- date of termination schedule, in local time zone time DATETIME NOT NULL, -- time in UTC app VARCHAR(512) NOT NULL, account VARCHAR(100) NOT NULL, region VARCHAR(50) NOT NULL, stack VARCHAR(255) NOT NULL, cluster VARCHAR(768) NOT NULL, INDEX date_index (date) ) ENGINE=InnoDB; -- terminations table: Records actual terminations CREATE TABLE IF NOT EXISTS terminations ( id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, app VARCHAR(512) NOT NULL, account VARCHAR(100) NOT NULL, stack VARCHAR(255) NOT NULL, cluster VARCHAR(768) NOT NULL, region VARCHAR(50) NOT NULL, asg VARCHAR(1000) NOT NULL, instance_id VARCHAR(48) NOT NULL, killed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, leashed BOOLEAN NOT NULL DEFAULT FALSE, INDEX app_killed_at_index (app,killed_at) ) ENGINE=InnoDB; ``` -------------------------------- ### chaosmonkey account Source: https://context7.com/netflix/chaosmonkey/llms.txt Looks up the cloud account ID by providing the account name. ```APIDOC ## chaosmonkey account ### Description Retrieves the cloud account ID associated with a given account name. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters #### Path Parameters - **account_name** (string) - Required - The name of the cloud account. ### Request Example ```bash chaosmonkey account production ``` ### Response #### Success Response (Output) - **Account ID** - The unique identifier for the cloud account. #### Response Example ``` 123456789012 ``` ``` -------------------------------- ### Implement a Custom Constrainer Source: https://context7.com/netflix/chaosmonkey/llms.txt Define a struct implementing the schedule.Constrainer interface to filter out specific termination times. ```go package constrainer import ( "log" "github.com/Netflix/chaosmonkey/v2/schedule" ) // TimeOfDayConstrainer prevents terminations during specific hours type TimeOfDayConstrainer struct { BlockedHours []int // Hours (0-23) when terminations are blocked } // Filter implements the schedule.Constrainer interface func (c TimeOfDayConstrainer) Filter(s schedule.Schedule) schedule.Schedule { result := schedule.New() for _, entry := range s.Entries() { hour := entry.Time.Hour() blocked := false for _, blockedHour := range c.BlockedHours { if hour == blockedHour { blocked = true break } } if blocked { log.Printf("Filtered out termination at %v (blocked hour)", entry.Time) continue } result.Add(entry.Time, entry.Group) } return *result } ``` -------------------------------- ### Chaos Monkey Scheduling Script Source: https://context7.com/netflix/chaosmonkey/llms.txt A bash script to run the Chaos Monkey scheduler and log its output. This script is intended to be called by cron. ```bash #!/bin/bash # chaosmonkey-schedule.sh - Daily scheduler script /apps/chaosmonkey/chaosmonkey schedule >> /var/log/chaosmonkey-schedule.log 2>&1 ``` -------------------------------- ### Enable Chaos Monkey in Spinnaker Settings Source: https://context7.com/netflix/chaosmonkey/llms.txt JavaScript code snippet to enable the Chaos Monkey feature flag within Spinnaker's settings.js file, allowing access to Chaos Monkey features in the Spinnaker UI. ```javascript // settings.js - Enable Chaos Monkey feature flag window.spinnakerSettings = { feature: { pipelines: true, notifications: false, fastProperty: true, chaosMonkey: true, // Add this line to enable Chaos Monkey UI // ... other features }, // ... other settings }; ``` -------------------------------- ### Enable Chaos Monkey Feature in Spinnaker Deck Source: https://github.com/netflix/chaosmonkey/blob/master/docs/How-to-deploy.md Add the 'chaosMonkey: true' flag to the 'feature' object in Spinnaker's Deck configuration file (/var/www/settings.js). This enables the Chaos Monkey checkbox in the New Application modal. ```javascript feature: { pipelines: true, notifications: false, fastProperty: true, ... chaosMonkey: true ``` -------------------------------- ### Check Chaos Monkey Outage Status Source: https://context7.com/netflix/chaosmonkey/llms.txt Checks if there is an ongoing outage. Chaos Monkey will not terminate instances during detected outages. ```bash chaosmonkey outage ``` -------------------------------- ### Chaos Monkey Termination Wrapper Script Source: https://context7.com/netflix/chaosmonkey/llms.txt A bash script that wraps the Chaos Monkey terminate command, passing all arguments and logging output. This script is intended for use with cron. ```bash #!/bin/bash # chaosmonkey-terminate.sh - Termination wrapper script /apps/chaosmonkey/chaosmonkey terminate "$@" >> /var/log/chaosmonkey-terminate.log 2>&1 ``` -------------------------------- ### chaosmonkey outage Source: https://context7.com/netflix/chaosmonkey/llms.txt Checks if there is an ongoing outage, during which Chaos Monkey will not perform terminations. ```APIDOC ## chaosmonkey outage ### Description Checks for ongoing outages. Chaos Monkey will refrain from terminating instances if an outage is detected. ### Method CLI Command ### Endpoint N/A (CLI command) ### Parameters None ### Request Example ```bash chaosmonkey outage ``` ### Response #### Success Response (Output) - **Boolean** - Returns `true` if an outage is detected, `false` otherwise. #### Response Example ``` false ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.