### Infrastructure Setup Failure Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of a failure during K0s/infrastructure installation, indicating the 'Point of No Return'. Requires a full reset and reinstallation.

```bash
✔  Initialization complete
✔  Host preflights passed
◓  Installing node

Error: Node installation failed: K0s failed to start: context deadline exceeded

To collect diagnostic information, run: embedded-cluster support-bundle
To retry installation, run: embedded-cluster reset and wait for server reboot
```

--------------------------------

### Headless Installation Command

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example command to perform a headless installation with specified configuration and credentials.

```bash
$ embedded-cluster install --headless --target linux --license license.yaml --config-values config.yaml --admin-console-password password --yes
```

--------------------------------

### Installation Settings Validation Failure Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of installation settings validation failures reported by the API before installation begins. Correct network settings and retry.

```bash
◓  Initializing

Error: installation settings validation failed:
  - Pod CIDR 10.96.0.0/12 overlaps with service CIDR 10.96.0.0/16
  - Network interface 'eth1' not found on host

For configuration options, run: embedded-cluster install --help
Please correct the above issues and retry
```

--------------------------------

### Go: Helm Install Implementation Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/helm_binary_migration.md

This Go code snippet demonstrates the implementation of the `Install` function using the Helm binary executor. It handles chart source resolution and constructs arguments for the `helm install` command.

```go
// Example: Install implementation
func (c *HelmClient) Install(ctx context.Context, opts InstallOptions) (*release.Release, error) {
    args := []string{"install", opts.ReleaseName}

    // Handle chart source
    if c.airgapPath != "" {
        // Use chart from airgap path
    } else if !strings.HasPrefix(opts.ChartPath, "/") {
        // Pull chart with retries (includes oci:// prefix)
    } else {
        // Use local chart path
    }
    
    // Add all helm install flags: --namespace, --create-namespace, --wait, etc.
    // Add values file if provided
    // Add labels if provided
    
    // Execute helm command
    stdout, stderr, err := c.executor.ExecuteCommand(ctx, nil, c.helmPath, args...)
    
    // Parse release from JSON output
    return &release, nil
}
```

--------------------------------

### Go Example: Setting Up Infrastructure with Polling

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

This example demonstrates triggering infrastructure setup and then using the `pollUntilComplete` function to monitor its progress until completion or failure. It includes spinner feedback for the user.

```go
func (o *orchestrator) setupInfrastructure(ctx context.Context, ignoreHostPreflights bool) error {
    o.logger.Debug("Starting infrastructure setup")

    loading := spinner.Start(spinner.WithWriter(o.progressWriter))
    loading.Infof("Setting up infrastructure...")

    // Trigger setup (non-blocking)
    infra, err := o.apiClient.SetupLinuxInfra(ignoreHostPreflights)
    if err != nil {
        loading.ErrorClosef("Infrastructure setup failed: %v", err)
        return fmt.Errorf("setup linux infra: %w", err)
    }

    getStatus := func() (types.State, string, error) {
        infra, err = o.apiClient.GetLinuxInfraStatus()
        if err != nil {
            return types.State{}, "", err
        }
        return infra.Status.State, infra.Status.Description, nil
    }

    // Poll for completion using api/client.Client
    err = pollUntilComplete(ctx, getStatus)
    if err != nil {
        loading.ErrorClosef("Infrastructure setup failed: %v", err)
        return fmt.Errorf("poll until complete: %w", err)
    }

    loading.Closef("Infrastructure setup complete")
    o.logger.Debug("Infrastructure setup complete")
    return nil
}
```

--------------------------------

### Full Embedded Cluster Recovery Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Demonstrates a complete recovery workflow starting from a failed headless installation due to bad configuration, followed by a reset and a successful re-installation.

```bash
# Initial install fails due to bad config
$ embedded-cluster install --headless --target linux --config-values config.yaml --license license.yaml --admin-console-password pass123
✔  Initialization complete
✔  Host preflights passed
◓  Installing node

Error: Node installation failed: K0s failed to start: context deadline exceeded

# Reset the system
$ embedded-cluster reset

✔  Reset complete

# Fix config.yaml with correct values
$ vim config.yaml

# Retry installation
$ embedded-cluster install --headless --target linux --config-values config.yaml --license license.yaml --admin-console-password pass123

✔  Initialization complete
✔  Host preflights passed
✔  Node is ready
✔  Storage is ready
✔  Runtime Operator is ready
✔  Disaster Recovery is ready
✔  Admin Console is ready
✔  Additional components are ready
✔  Application preflights passed
✔  Application is ready
```

--------------------------------

### Example Headless Install Log Output

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Shows typical log entries during a headless installation, including progress, debug information, and errors.

```log
2025-10-21T10:30:15Z INFO  Headless install started target=linux
2025-10-21T10:30:15Z DEBUG Config values loaded from ./config.yaml
2025-10-21T10:30:16Z INFO  Authentication successful
2025-10-21T10:30:17Z DEBUG Patching config values via API endpoint=/api/v3/linux/install/app/config-values
2025-10-21T10:30:18Z INFO  Config validation passed
2025-10-21T10:30:20Z ERROR Infrastructure setup failed error="K0s failed to start" exit_code=3
```

--------------------------------

### Successful Headless Installation Output

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example output demonstrating a successful headless installation, showing all stages completing with checkmarks.

```bash
$ ./my-app install --target linux --license license.yaml --config-values config.yaml --admin-console-password password --yes --headless

✔  Initialization complete
✔  Host preflights passed
✔  Node is ready
✔  Storage is ready
✔  Runtime Operator is ready
✔  Disaster Recovery is ready
✔  Admin Console is ready
✔  Installing additional components (2/2)
✔  Application is ready

Installation completed successfully
```

--------------------------------

### Example JSON Error Response: Infrastructure Setup Failure

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Presents a JSON error response for a failure during infrastructure setup, such as K0s failing to start.

```JSON
{
  "statusCode": 500,
  "message": "infrastructure setup failed",
  "errors": [
    {
      "message": "K0s failed to start: context deadline exceeded"
    }
  ]
}
```

--------------------------------

### RunHeadlessInstall Method Implementation

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Implements the headless installation workflow, executing steps like configuration, preflights, infrastructure setup, and application installation. Returns true if a reset is needed after failure.

```go
func (o *orchestrator) RunHeadlessInstall(ctx context.Context, opts HeadlessInstallOptions) (bool, error) {
    // Configure application with config values
    if err := o.configureApplication(ctx, opts); err != nil {
        return false, err // Can retry without reset
    }

    // Configure installation
    if err := o.configureInstallation(ctx, opts); err != nil {
        return false, err // Can retry without reset
    }

    // Run host preflights (allow bypass when --ignore-host-preflights flag is set)
    if err := o.runHostPreflights(ctx, opts.IgnoreHostPreflights); err != nil {
        return false, err // Can retry without reset
    }

    // Setup infrastructure (POINT OF NO RETURN)
    // After this point, any failure requires running 'embedded-cluster reset'
    if err := o.setupInfrastructure(ctx, opts.IgnoreHostPreflights); err != nil {
        return true, err // Reset required
    }

    // Process airgap if needed
    if opts.AirgapBundle != "" {
        if err := o.processAirgap(ctx); err != nil {
            return true, err // Reset required
        }
    }

    // Run app preflights (allow bypass when --ignore-app-preflights flag is set)
    if err := o.runAppPreflights(ctx, opts.IgnoreAppPreflights); err != nil {
        return true, err // Reset required
    }

    // Install application
    if err := o.installApp(ctx, opts.IgnoreAppPreflights); err != nil {
        return true, err // Reset required
    }

    return false, nil // Success
}
```

--------------------------------

### Example ConfigValues File for Headless Install

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

An example of a ConfigValues file used to provide custom settings for a headless installation, specifying values for hostname and password.

```yaml
# Example ConfigValues file for headless install (provided by user)
apiVersion: kots.io/v1beta1
kind: ConfigValues
metadata:
  name: my-app-config
spec:
  values:
    hostname:
      value: "postgres.example.com"
    pw:
      value: "secretpassword"

```

--------------------------------

### Headless Installation Output with Bypassed Preflight Failures

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example output for a headless installation where preflight checks failed but were bypassed, showing warnings and continued installation.

```bash
$ ./my-app install --target linux --license license.yaml --config-values config.yaml --admin-console-password password --yes --headless --ignore-host-preflights

✔  Initialization complete
✗  Host preflights completed with failures

⚠ Warning: Host preflight checks completed with failures

  [ERROR] Insufficient disk space: 10GB available, 50GB required
  [WARN] CPU count below recommended: 2 cores, 4 recommended

Installation will continue, but the system may not meet requirements (failures bypassed with flag).

✔  Node is ready
✔  Storage is ready
✔  Runtime Operator is ready
✔  Disaster Recovery is ready
✔  Admin Console is ready
✔  Installing additional components (2/2)
✔  Application is ready

Installation completed successfully
```

--------------------------------

### GetInstallationConfig Method

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/kurl-migration-api-foundation.md

Retrieves and merges installation configuration by calling manager methods to get kURL and EC defaults, then merging them.

```go
// GetInstallationConfig retrieves and merges installation configuration
func (mc *MigrationController) GetInstallationConfig(ctx context.Context) (*types.LinuxInstallationConfigResponse, error) {
    // Call manager.GetKurlConfig() to extract kURL config with non-overlapping CIDRs
    // Call manager.GetECDefaults() to get EC defaults
    // Merge configs (kURL > defaults)
    // Return response with values/defaults/resolved
}
```

--------------------------------

### Setup Infrastructure API

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Sets up the necessary infrastructure for the installation. This is an asynchronous operation that requires polling for status.

```APIDOC
## POST /api/linux/install/infra/setup

### Description
Sets up the necessary infrastructure for the installation. This is an asynchronous operation that requires polling for status.

### Method
POST

### Endpoint
/api/linux/install/infra/setup

### Parameters
#### Request Body
- **ignoreHostPreflights** (bool) - Required - Whether to ignore host preflight failures during infrastructure setup.

### Response
#### Success Response (200)
- **Infra** - Information about the set up infrastructure.

### Response Example
(Infra structure not detailed in source)
```

```APIDOC
## GET /api/linux/install/infra/status

### Description
Retrieves the status of the infrastructure setup. This endpoint is polled to track the progress of asynchronous operations.

### Method
GET

### Endpoint
/api/linux/install/infra/status

### Response
#### Success Response (200)
- **Infra** - The current status of the infrastructure setup.

### Response Example
(Infra structure not detailed in source)
```

--------------------------------

### Install LXD

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/e2e/README.md

Install LXD using snap and initialize it automatically. This is a prerequisite for integration tests.

```bash
$ snap install lxd
$ lxd init --auto
```

--------------------------------

### Setup Infrastructure and Monitor State

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Initiates infrastructure setup and polls for completion using the API client. Includes error handling for setup and polling phases.

```Go
// Another example showing state monitoring
func (o *orchestrator) setupInfrastructure(ctx context.Context, ignoreHostPreflights bool) error {
    o.logger.Debug("Starting infrastructure setup")

    loading := spinner.Start(spinner.WithWriter(o.progressWriter))
    loading.Infof("Setting up infrastructure...")

    // Initiate infra setup using api/client.Client
    infra, err := o.apiClient.SetupLinuxInfra(ignoreHostPreflights)
    if err != nil {
        loading.ErrorClosef("Infrastructure setup failed: %v", err)
        return fmt.Errorf("setup linux infra: %w", err)
    }

    getStatus := func() (types.State, string, error) {
        infra, err = o.apiClient.GetLinuxInfraStatus()
        if err != nil {
            return types.State{}, "", err
        }
        return infra.Status.State, infra.Status.Description, nil
    }

    // Poll for completion using api/client.Client
    err = pollUntilComplete(ctx, getStatus)
    if err != nil {
        loading.ErrorClosef("Infrastructure setup failed: %v", err)
        return fmt.Errorf("poll until complete: %w", err)
    }

    loading.Closef("Infrastructure setup complete")
    o.logger.Debug("Infrastructure setup complete")
    return nil
}
```

--------------------------------

### Run Headless Installation (V3)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Executes the headless installation process, including signal handling, orchestrator building, and error reporting.

```go
func runV3InstallHeadless(
    ctx context.Context,
    cancel context.CancelFunc,
    flags installFlags,
    installCfg *installConfig,
    apiOpts apiOptions,
) error {
    // Setup signal handler
    signalHandler(ctx, cancel, func(ctx context.Context, sig os.Signal) {
        apiOpts.MetricsReporter.ReportSignalAborted(ctx, sig)
    })

    // Build orchestrator
    orchestrator, err := buildOrchestrator(installCfg, apiOpts)
    if err != nil {
        return fmt.Errorf("failed to build orchestrator: %w", err)
    }

    // Build install options
    opts := buildHeadlessInstallOptions(flags, apiOpts)

    resetNeeded, err := orchestrator.RunHeadlessInstall(ctx, opts)
    if err != nil {
        if errors.Is(err, terminal.InterruptErr) {
            apiOpts.MetricsReporter.ReportSignalAborted(ctx, syscall.SIGINT)
        } else {
            apiOpts.MetricsReporter.ReportInstallationFailed(ctx, err)
        }

        // Print error and recovery instructions
        logrus.Errorf("\nError: %v\n", err)

        if resetNeeded {
            logrus.Info("To collect diagnostic information, run: embedded-cluster support-bundle")
            logrus.Info("To retry installation, run: embedded-cluster reset and wait for server reboot")
        } else {
            logrus.Info("Please correct the above issues and retry")
        }

        return NewErrorNothingElseToAdd(err)
    }

    // Display success message
    logrus.Info("\nInstallation completed successfully")

    apiOpts.MetricsReporter.ReportInstallationSucceeded(ctx)
    return nil
}
```

--------------------------------

### Application Installation Failures

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example output when the final application installation phase fails, typically due to timeouts. Recovery requires running 'embedded-cluster reset'.

```bash
✔  Initialization complete
✔  Host preflights passed
✔  Node is ready
✔  Storage is ready
✔  Runtime Operator is ready
✔  Disaster Recovery is ready
✔  Admin Console is ready
✔  Additional components are ready
◓  Installing application

Error: Application installation failed: timeout waiting for pods to become ready

To collect diagnostic information, run: embedded-cluster support-bundle
To retry installation, run: embedded-cluster reset and wait for server reboot
```

--------------------------------

### Build Headless Install Options

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Constructs HeadlessInstallOptions for the installation process using provided flags and API options.

```go
// buildHeadlessInstallOptions (Hop) creates HeadlessInstallOptions from CLI inputs.
func buildHeadlessInstallOptions(
    flags installFlags,
    apiOpts apiOptions,
) install.HeadlessInstallOptions {
    // Build Linux installation config from flags
    linuxInstallationConfig := apitypes.LinuxInstallationConfig{
        AdminConsolePort:        flags.adminConsolePort,
        DataDirectory:           flags.dataDir,
        LocalArtifactMirrorPort: flags.localArtifactMirrorPort,
        HTTPProxy:               flags.httpProxy,
        HTTPSProxy:              flags.httpsProxy,
        NoProxy:                 flags.noProxy,
        NetworkInterface:        flags.networkInterface,
        PodCIDR:                 flags.podCIDR,
        ServiceCIDR:             flags.serviceCIDR,
        GlobalCIDR:              flags.globalCIDR,
    }

    return install.HeadlessInstallOptions{
        ConfigValues:            apiOpts.ConfigValues,
        LinuxInstallationConfig: linuxInstallationConfig,
        IgnoreHostPreflights:    flags.ignoreHostPreflights,
        IgnoreAppPreflights:     flags.ignoreAppPreflights,
        AirgapBundle:            flags.airgapBundle,
    }
}
```

--------------------------------

### Host Preflight Failure Example (Not Bypassed)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of host preflight checks failing before infrastructure setup. Correct host issues or re-run with --ignore-host-preflights.

```bash
✔  Initialization complete
✗  Host preflights completed with failures

⚠ Warning: Host preflight checks completed with failures

  - [ERROR] Insufficient disk space: 10GB available, 50GB required
  - [WARN] CPU count below recommended: 2 cores, 4 recommended

Please correct the above issues and retry, or run with --ignore-host-preflights to bypass (not recommended)
```

--------------------------------

### Run Manager Experience Install

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Determines whether to run the headless installation flow based on flags.

```go
func runManagerExperienceInstall(...) (finalErr error) {
    // ... existing setup code ...

    if flags.headless {
        return runV3InstallHeadless(ctx, cancel, flags, installCfg, apiOpts)
    }

    // ... existing UI flow ...
}
```

--------------------------------

### Install Dagger CLI

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/dagger/README.md

Installs the Dagger CLI using Homebrew. Ensure Dagger is installed and configured before proceeding with development.

```bash
brew install dagger/tap/dagger
```

--------------------------------

### Headless Install Options Struct

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Defines the configuration options for a headless installation, including config values, installation settings, and preflight check bypass flags.

```go
package install

import (
	"github.com/replicatedhq/kots/pkg/api/apitypes"
)

// HeadlessInstallOptions contains the configuration options for a headless installation
type HeadlessInstallOptions struct {
	// ConfigValues are the application config values to use for installation
	ConfigValues apitypes.AppConfigValues

	// LinuxInstallationConfig contains the installation settings for the Linux target
	LinuxInstallationConfig apitypes.LinuxInstallationConfig

	// IgnoreHostPreflights indicates whether to bypass host preflight check failures
	IgnoreHostPreflights bool

	// IgnoreAppPreflights indicates whether to bypass app preflight check failures
	IgnoreAppPreflights bool

	// AirgapBundle is the path to the airgap bundle file (empty string for online installs)
	AirgapBundle string
}
```

--------------------------------

### Headless Install Dry-Run Test Pattern

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

This Go code snippet illustrates the general pattern for dry-run integration tests in the headless installer. It shows how to configure the installer with various flags and verifies that these configurations propagate correctly to different components.

```go
func TestV3HeadlessInstall_<FlagName>(t *testing.T) {
    // GIVEN: Headless install with specific flag configuration
    dr := dryrunInstall(t, client,
        "--headless",
        "--target", "linux",
        "--config-values", configValuesFile,
        "--admin-console-password", "password",
        "--<flag>", "<value>",
        "--yes",
    )

    // THEN: Verify flag propagates to all relevant components
    // - K0s configuration
    // - Addon Helm values
    // - Host preflights
    // - Environment variables
    // - Commands
}
```

--------------------------------

### Run Online Installation Tests

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Executes both browser-based and headless installation tests for the 'online' scenario. It first runs the browser test and then the headless test, returning an error if either fails. This function is used to ensure both installation methods work correctly in an online environment.

```go
func (m *TestModule) RunOnlineTests(ctx context.Context) error {
    // Run browser-based test
    browserResults, err := m.RunBrowserBasedTest(ctx, "online")
    if err != nil {
        return err
    }

    // Run headless test
    headlessResults, err := m.RunHeadlessTest(ctx, "online")
    if err != nil {
        return err
    }

    return m.reportResults(ctx, browserResults, headlessResults)
}
```

--------------------------------

### Install App API

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Installs the application. This is an asynchronous operation that requires polling for status.

```APIDOC
## POST /api/linux/install/app/install

### Description
Installs the application. This is an asynchronous operation that requires polling for status.

### Method
POST

### Endpoint
/api/linux/install/app/install

### Response
#### Success Response (200)
- **AppInstall** - Information about the application installation.

### Response Example
(AppInstall structure not detailed in source)
```

```APIDOC
## GET /api/linux/install/app/status

### Description
Retrieves the status of the application installation. This endpoint is polled to track the progress of asynchronous operations.

### Method
GET

### Endpoint
/api/linux/install/app/status

### Response
#### Success Response (200)
- **AppInstall** - The current status of the application installation.

### Response Example
(AppInstall structure not detailed in source)
```

--------------------------------

### Log Installation Blocked by Strict App Preflights

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/strict_preflight_blocking.md

Logs a warning when an installation is blocked due to strict app preflight failures. This is useful for monitoring and debugging installation issues.

```go
logger.Warn("Installation blocked due to strict app preflight failures",
    "failedChecks", getFailedStrictChecks(output),
    "installID", installID)
```

--------------------------------

### Initial Release Creation (Online)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/README.md

Create the initial release for an online installation. This command is part of the V2 installation process.

```bash
make initial-release
```

--------------------------------

### HTTP Handler for GetInstallationConfig

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/kurl-migration-api-foundation.md

Implements the HTTP handler for retrieving the kURL installation configuration. It calls the controller to get the configuration and uses utility functions for JSON responses and error handling.

```go
type Handler struct {
    logger              *logrus.Logger
    migrationController Controller
    migrationStore      Store
}

func NewHandler(controller Controller, store Store, logger *logrus.Logger) *Handler

// GetInstallationConfig returns kURL config merged with EC defaults (values/defaults/resolved)
// @Router /api/kurl-migration/config [get]
func (h *Handler) GetInstallationConfig(w http.ResponseWriter, r *http.Request) {
    // Call controller.GetInstallationConfig(r.Context())
    // Use utils.JSON() to return LinuxInstallationConfigResponse with 200
    // Use utils.JSONError() to handle errors (controller returns typed errors)
```

--------------------------------

### Node Creation (Online)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/README.md

Create node0 for an online installation. This command is part of the V2 installation process.

```bash
make create-node0
```

--------------------------------

### Run Headless (CLI) Installation Test

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Executes a headless installation test via the command-line interface without Playwright. It provisions a CMX VM, runs the installation using CLI commands, and validates the outcome. This function is ideal for testing automated or non-interactive installation flows.

```go
func (m *TestModule) RunHeadlessTest(
    ctx context.Context,
    // Installation scenario to test. Valid values: "online", "airgap"
    scenario string,
) (*TestResults, error) {
    // Get secrets from 1Password
    secrets := m.secrets.GetBatch(ctx, []string{
        "replicated_api_token",
        "cmx_api_token",
    })

    // Provision CMX VM for headless installation test
    vm := m.provisionCMXVM(ctx, CMXConfig{
        OS:       "ubuntu-22.04",
        Memory:   "8GB",
        CPUs:     4,
        Secrets:  secrets,
        Scenario: scenario, // "online", "airgap"
    })
    defer vm.Cleanup()

    // Run headless installation via CLI
    // Note: No Playwright for headless tests
    result := m.installHeadless(ctx, vm, HeadlessConfig{
        Scenario:   scenario,
        AppVersion: m.appVersion,
        License:    m.getLicenseForScenario(scenario),
        LicenseID:  m.getLicenseIDForScenario(scenario),
    })

    // Validate installation succeeded using Kube client
    // Note: Both browser-based and headless tests use Kube client for validation
    // This performs comprehensive validation (see Installation Validation section)
    validationResults := m.validate(ctx, scenario, vm, result)

    return &TestResults{
        Scenario: scenario,
        Mode:     "headless",
        Success:  validationResults.Success,
        Details:  validationResults,
    }, nil
}
```

--------------------------------

### Helm Install Operation

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/helm_binary_migration_research.md

Used for installing addons and applications. Returns complete release metadata.

```go
hcli.Install(ctx, helm.InstallOptions{...})
```

--------------------------------

### Configure Installation API

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Configures the installation settings for a Linux environment. This is an asynchronous operation that requires polling for status.

```APIDOC
## POST /api/linux/install/installation/configure

### Description
Configures the installation settings for a Linux environment. This is an asynchronous operation that requires polling for status.

### Method
POST

### Endpoint
/api/linux/install/installation/configure

### Parameters
#### Request Body
- **config** (LinuxInstallationConfig) - Required - The installation configuration details.

### Response
#### Success Response (200)
- **Status** - The status of the configuration operation.

### Response Example
(Status response structure not detailed in source)
```

```APIDOC
## GET /api/linux/install/installation/status

### Description
Retrieves the current status of the Linux installation configuration. This endpoint is polled to track the progress of asynchronous operations.

### Method
GET

### Endpoint
/api/linux/install/installation/status

### Response
#### Success Response (200)
- **Status** - The current status of the installation.

### Response Example
(Status response structure not detailed in source)
```

--------------------------------

### Install Release (Online)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/README.md

Install the release using the embedded-cluster binary for an online deployment. Ensure the CUSTOMER_LICENSE_FILE environment variable is set.

```bash
output/bin/embedded-cluster install --license "$CUSTOMER_LICENSE_FILE"
```

--------------------------------

### Run Browser-Based Installation Test

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Executes a browser-based installation test using Playwright. It provisions a CMX VM, runs the installation via the browser, and validates the outcome using the Kube client. This function is suitable for testing the interactive installation flow.

```go
func (m *TestModule) RunBrowserBasedTest(
    ctx context.Context,
    // Installation scenario to test. Valid values: "online", "airgap"
    scenario string,
) (*TestResults, error) {
    // Get secrets from 1Password
    secrets := m.secrets.GetBatch(ctx, []string{
        "replicated_api_token",
        "cmx_api_token",
    })

    // Provision CMX VM for browser based installation test
    vm := m.provisionCMXVM(ctx, CMXConfig{
        OS:       "ubuntu-22.04",
        Memory:   "8GB",
        CPUs:     4,
        Secrets:  secrets,
        Scenario: scenario, // "online", "airgap"
    })
    defer vm.Cleanup()

    // Run browser based Playwright installation
    result := m.installBrowserBased(ctx, vm, PlaywrightConfig{
        Scenario:   scenario,
        AppVersion: m.appVersion,
        License:    m.getLicenseForScenario(scenario),
        LicenseID:  m.getLicenseIDForScenario(scenario),
    })

    // Validate installation succeeded using Kube client
    // Note: Both browser-based and headless tests use Kube client for validation
    // This performs comprehensive validation (see Installation Validation section)
    validationResults := m.validate(ctx, scenario, vm, result)

    return &TestResults{
        Scenario: scenario,
        Mode:     "browser-based",
        Success:  validationResults.Success,
        Details:  validationResults,
    }, nil
}
```

--------------------------------

### Initial Release Creation (Airgap)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/README.md

Create the initial release for an airgap installation, including uploading binaries. This command is part of the V2 installation process.

```bash
make initial-release UPLOAD_BINARIES=1
```

--------------------------------

### Test Helm Client Install Operation

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/helm_binary_migration.md

Unit test for the HelmClient Install method using a mock binary executor. Verifies that the correct command and arguments are passed for installing a Helm release.

```go
import (
    "context"
    "testing"
    "time"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/mock"
    "github.com/stretchr/testify/require"
)

// MockBinaryExecutor is a mock type for BinaryExecutor
type MockBinaryExecutor struct {
    mock.Mock
}

// ExecuteCommand is a mock implementation of BinaryExecutor.ExecuteCommand
func (_m *MockBinaryExecutor) ExecuteCommand(ctx context.Context, env []string, binaryPath string, args ...string) (string, string, error) {
    ret := _m.Called(ctx, env, binaryPath, args)
    return ret.Get(0).(string), ret.Get(1).(string), ret.Error(2)
}

// HelmClient represents the Helm client
type HelmClient struct {
    helmPath string
    executor BinaryExecutor
}

// InstallOptions holds options for Helm installation
type InstallOptions struct {
    ReleaseName string
    ChartPath   string
    Namespace   string
    Timeout     time.Duration
}

// Release represents a Helm release
type Release struct {
    Name string
}

// Using mockery-generated mock
func TestHelmClient_Install(t *testing.T) {
    mockExec := new(MockBinaryExecutor)
    client := &HelmClient{
        helmPath: "/usr/local/bin/helm",
        executor: mockExec,
    }
    
    testReleaseJSON := `{"name": "myrelease"}`

    mockExec.On("ExecuteCommand", 
        mock.Anything,  // context
        mock.Anything,  // env
        "/usr/local/bin/helm",
        "install", "myrelease", "/path/to/chart",
        "--namespace", "default",
        "--create-namespace",
        "--wait",
        "--wait-for-jobs",
        "--timeout", "5m0s",
        "--replace",
        "--output", "json",
    ).Return(testReleaseJSON, "", nil)
    
    release, err := client.Install(context.Background(), InstallOptions{
        ReleaseName: "myrelease",
        ChartPath:   "/path/to/chart",
        Namespace:   "default",
        Timeout:     5 * time.Minute,
    })
    
    require.NoError(t, err)
    assert.Equal(t, "myrelease", release.Name)
    mockExec.AssertExpectations(t)
}
```

--------------------------------

### Build Orchestrator

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Creates an orchestrator instance from installation configuration and API options. Supports only the Linux install target.

```go
// buildOrchestrator (Hop) creates an orchestrator from CLI inputs.
func buildOrchestrator(
    installCfg *installConfig,
    apiOpts apiOptions,
) (install.Orchestrator, error) {
    // Construct API URL from manager port
    apiURL := fmt.Sprintf("https://localhost:%d", installCfg.managerPort)

    // We do not yet support the "kubernetes" target
    if target != apitypes.InstallTargetLinux {
        return nil, fmt.Errorf("%s target not supported", target)
    }

    // Create HTTP client with InsecureSkipVerify for localhost
    // Since the API server is in-process and on localhost only, certificate
    // validation is not critical for this use case
    httpClient := &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            Proxy: nil,  // No proxy for localhost
            TLSClientConfig: &tls.Config{
                InsecureSkipVerify: true,  // Acceptable for localhost in-process API
            },
        },
    }

    // Create API client
    apiClient := client.New(
        apiURL,  // e.g., "https://localhost:30000"
        client.WithHTTPClient(httpClient),
    )

    // Create orchestrator
    orchestrator, err := install.NewOrchestrator(
        apiClient,
        apiOpts.Password,
        apiOpts.InstallTarget,
    )
    if err != nil {
        return nil, fmt.Errorf("failed to create orchestrator: %w", err)
    }

    return orchestrator, nil
}
```

--------------------------------

### Run Airgap Installation Tests

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Executes both browser-based and headless installation tests for the 'airgap' scenario. Similar to `RunOnlineTests`, it runs both test types sequentially and reports any errors. This function is crucial for verifying installation in environments without direct internet access.

```go
func (m *TestModule) RunAirgapTests(ctx context.Context) error {
    // Run browser-based test
    browserResults, err := m.RunBrowserBasedTest(ctx, "airgap")
    if err != nil {
        return err
    }

    // Run headless test
    headlessResults, err := m.RunHeadlessTest(ctx, "airgap")
    if err != nil {
        return err
    }

    return m.reportResults(ctx, browserResults, headlessResults)
}
```

--------------------------------

### Test Headless Install with Custom TLS Certificate

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Verifies that custom TLS certificates and hostnames are correctly configured for the admin console and KOTS during headless installation.

```go
func TestV3HeadlessInstall_CustomTLSCertificate(t *testing.T) {
    // GIVEN: Headless install with custom TLS configuration
    //        --tls-cert /path/to/cert.pem
    //        --tls-key /path/to/key.pem
    //        --hostname myapp.example.com
    // WHEN: Installation is run via dryrunInstall
    // THEN: TLS configuration propagates to:
    //   - Admin Console addon creates TLS secret with provided cert/key
    //   - KOTS admin console uses provided certificate
    //   - Ingress resources reference correct secret
    //   - Certificate is stored in appropriate namespace
    //   - Installation URL uses specified hostname
}
```

--------------------------------

### Config Value Parse Failure Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of a YAML parsing error during CLI startup before the API starts. Fix YAML syntax and re-run the install command.

```bash
Error: failed to load config values from ./config.yaml: yaml: line 5: mapping values are not allowed in this context
```

--------------------------------

### Go Example: Running Host Preflights with Progress

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Illustrates how to use the spinner package to display progress messages while running host preflight checks.

```Go
// Example usage in orchestrator methods:
func (o *orchestrator) runHostPreflights(ctx context.Context, ignoreFailures bool) error {
    o.logger.Debug("Starting host preflights")

    loading := spinner.Start(spinner.WithWriter(o.progressWriter))
    loading.Infof("Running host preflights...")

    // Trigger preflights
    resp, err := o.apiClient.RunLinuxInstallHostPreflights(ignoreFailures)
    if err != nil {
        loading.ErrorClosef("Host preflights failed: %v", err)
        return fmt.Errorf("run linux install host preflights: %w", err)
    }

    getStatus := func() (types.State, string, error) {
        resp, err = o.apiClient.GetLinuxInstallHostPreflightsStatus()
        if err != nil {
            return types.State{}, "", err
        }
        return resp.Status.State, resp.Status.Description, nil
    }

```

--------------------------------

### Perform Browser-Based Installation

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Installs embedded-cluster using a browser-based method via Playwright. This involves downloading the release, starting the installation in headless mode, and then using Playwright to interact with the UI for final configuration.

```go
// installBrowserBased performs a browser-based installation using Playwright
func (m *TestModule) installBrowserBased(ctx context.Context, vm *CMXInstance, config PlaywrightConfig) (*InstallResult, error) {
    // Download and prepare embedded-cluster release
    if err := m.downloadAndPrepareRelease(ctx, vm, config.AppVersion, config.LicenseID); err != nil {
        return nil, err
    }

    // Start embedded-cluster install (headless mode to get to UI)
    // This starts the installation and makes the UI available on port 30000
    installCmd := []string{
        "/usr/local/bin/embedded-cluster-smoke-test-staging-app", "install",
        "--target", "linux",
        "--headless",
        "--yes",
    }

    if config.Scenario == "airgap" {
        installCmd = append(installCmd, "--airgap-bundle", "/tmp/airgap-bundle.tar.gz")
    }

    if _, _, err := vm.RunCommandAsync(ctx, installCmd); err != nil {
        return nil, err
    }

    // Wait for UI to be available
    if err := vm.WaitForPort(ctx, 30000, 2*time.Minute); err != nil {
        return nil, fmt.Errorf("UI did not become available: %w", err)
    }

    // Run Playwright tests to complete installation via UI
    // This includes: setting admin password, uploading license, deploying app
    playwrightResult, err := m.runPlaywrightInstallation(ctx, vm, config)
    if err != nil {
        return nil, fmt.Errorf("playwright installation failed: %w", err)
    }

    return playwrightResult, nil
}
```

--------------------------------

### Host Preflight Failure Example (Bypassed)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of host preflight checks failing but being bypassed with the --ignore-host-preflights flag. Installation continues with warnings.

```bash
✔  Initialization complete
✗  Host preflights completed with failures

⚠ Warning: Host preflight checks completed with failures

  [ERROR] Insufficient disk space: 10GB available, 50GB required
  [WARN] CPU count below recommended: 2 cores, 4 recommended

Installation will continue, but the system may not meet requirements.

✔  Node is ready
...
```

--------------------------------

### StartMigration Method Logic

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/kurl-migration-api-foundation.md

Initializes and starts the migration process, including validation of transfer mode and configuration merging.

```go
// StartMigration initializes and starts the migration process
func (mc *MigrationController) StartMigration(ctx context.Context, transferMode string, config *types.LinuxInstallationConfig) (string, error) {
    // Check if migration already exists (return types.NewConflictError(ErrMigrationAlreadyStarted))
    // Default transferMode to "copy" if empty
    // Validate transfer mode using manager.ValidateTransferMode() (return types.NewBadRequestError(err) if invalid)
    // Get base config (kURL + defaults) using mc.GetInstallationConfig()
    // Merge with user config (user > kURL > defaults) using manager.MergeConfigs()
    // Validate final config using installationManager.ValidateConfig() (return types.NewBadRequestError(err) if invalid)
    // Generate migration ID (uuid)
    // Initialize migration in store
    // Launch background goroutine mc.runMigration()
    // Return migration ID immediately
}
```

--------------------------------

### Config Value Validation Failure Example

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example of config values validation failures reported by the API before installation begins. Correct the specified fields and retry.

```bash
◓  Initializing

Error: config values validation failed:
  - Field 'database_host': required field missing
  - Field 'replica_count': value "10" exceeds maximum allowed value 5
  - Field 'enable_ssl': validation rule failed: SSL requires cert_path to be set

Please correct the above issues and retry
```

--------------------------------

### List Available Distributions

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/README.md

Run this command to see a list of all available operating system distributions for creating nodes.

```bash
make list-distros
```

--------------------------------

### Helm Install Command Options

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/helm_binary_migration.md

This shows the available options for the `helm install` command when migrating to binary execution. Ensure all necessary flags are included for proper chart deployment.

```bash
helm install [NAME] [CHART] \
  --namespace <namespace> \
  --create-namespace \
  --wait \
  --wait-for-jobs \
  --timeout <duration> \
  --values <file> \
  --set key=value \
  --atomic=false \
  --replace \
  --output json
```

--------------------------------

### App Preflight Failures (Bypassed)

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Example output when application preflight checks fail but are bypassed using the '--ignore-app-preflights' flag. Installation continues with a warning.

```bash
✔  Initialization complete
✔  Host preflights passed
✔  Node is ready
✔  Storage is ready
✔  Runtime Operator is ready
✔  Disaster Recovery is ready
✔  Admin Console is ready
✔  Additional components are ready
✗  App preflights completed with failures

⚠ Warning: Application preflight checks completed with failures

  [ERROR] Cannot connect to required database host postgres.example.com:5432
  [WARN] PVC storage class 'fast' not available

Installation will continue, but the application may not function correctly (failures bypassed with flag).

✔  Application is ready
```

--------------------------------

### Register Migration Routes

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/kurl-migration-api-foundation.md

Registers the migration-related API routes under the /api/kurl-migration/ path. It includes routes for getting configuration, starting a migration, and polling migration status, all protected by authentication middleware.

```go
// Register migration routes under /api/kurl-migration/ with auth middleware
// GET  /config - Get installation configuration
// POST /start  - Start migration with transfer mode and optional config
// GET  /status - Poll migration status
```

--------------------------------

### Test Migration API Endpoints in Dryrun

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/kurl-migration-api-foundation.md

Extends an existing upgrade test to cover the new migration API endpoints. It simulates starting a migration via POST and checking its status via GET.

```go
func TestUpgradeKURLMigration(t *testing.T) {
    // Existing setup: ENABLE_V3=1, mock kURL kubeconfig, dryrun.KubeUtils
    // Existing setup: Create kurl-config ConfigMap in kube-system namespace
    // Existing test: Verify CLI upgrade detection and messaging

    // NEW: Test Migration API endpoints
    t.Run("migration API skeleton", func(t *testing.T) {
        // Start the API server with migration mode
        // POST /api/kurl-migration/start with transferMode="copy"
        // Verify response: migrationID returned with 200

        // GET /api/kurl-migration/status
        // Verify response: state=Failed, error contains "migration phase execution not yet implemented"
        // This validates ErrMigrationPhaseNotImplemented is properly returned
    })
}
```

--------------------------------

### Develop with Dagger

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/dagger/README.md

Runs the `dagger develop` command to initialize the Dagger SDK, ensuring it is installed, configured, and its files are regenerated.

```bash
dagger develop
```

--------------------------------

### Update install controller to block installation on strict failures

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/strict_preflight_blocking.md

Ensures the install controller prevents installation when strict preflight failures are detected, regardless of the 'ignoreAppPreflights' flag. This enforces the strict blocking mechanism.

```go
// In install controller logic:
if resp.HasStrictAppPreflightFailures {
	// Block installation
}
```

--------------------------------

### Headless Installation Function

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/v3_e2e_tests.md

Installs the embedded cluster using a headless (CLI/API) approach. It handles downloading the release, preparing configuration files, and running the installation command. Use this for automated or script-based installations.

```go
func (m *TestModule) installHeadless(ctx context.Context, vm *CMXInstance, config HeadlessConfig) (*InstallResult, error) {
    // Download and prepare embedded-cluster release
    if err := m.downloadAndPrepareRelease(ctx, vm, config.AppVersion, config.LicenseID); err != nil {
        return nil, err
    }

    // Create headless config file if provided
    if config.ConfigFile != "" {
        if err := vm.UploadFile(ctx, config.ConfigFile, "/tmp/install-config.yaml"); err != nil {
            return nil, err
        }
    }

    // Build install command
    installCmd := []string{
        "/usr/local/bin/embedded-cluster-smoke-test-staging-app", "install",
        "--license", config.License,
        "--target", "linux",
        "--headless",
        "--yes",
    }

    if config.Scenario == "airgap" {
        installCmd = append(installCmd, "--airgap-bundle", "/tmp/airgap-bundle.tar.gz")
    }

    if config.ConfigFile != "" {
        installCmd = append(installCmd, "--config", "/tmp/install-config.yaml")
    }

    // Run installation command
    stdout, stderr, err := vm.RunCommand(ctx, installCmd, RunOptions{
        Timeout: 30 * time.Minute,
    })
    if err != nil {
        return nil, fmt.Errorf("installation failed: %w\nstdout: %s\nstderr: %s", err, stdout, stderr)
    }

    return &InstallResult{
        Success:         true,
        KubeconfigPath:  "/var/lib/embedded-cluster-smoke-test-staging-app/k0s/pki/admin.conf",
        InstallationLog: stdout,
    }, nil
}
```

--------------------------------

### Existing API Client Interface Methods

Source: https://github.com/replicatedhq/embedded-cluster/blob/main/proposals/headless_install.md

Defines the existing Go interface methods for interacting with the installation API. Includes authentication, installation, app configuration, infrastructure, and preflight checks.

```go
type Client interface {
    // Authentication
    Authenticate(password string) error

    // Installation configuration
    GetLinuxInstallationConfig() (types.LinuxInstallationConfigResponse, error)
    ConfigureLinuxInstallation(config types.LinuxInstallationConfig) (types.Status, error)
    GetLinuxInstallationStatus() (types.Status, error)

    // App configuration
    GetLinuxInstallAppConfigValues() (types.AppConfigValues, error)
    PatchLinuxInstallAppConfigValues(types.AppConfigValues) (types.AppConfigValues, error)
    TemplateLinuxInstallAppConfig(values types.AppConfigValues) (types.AppConfig, error)

    // Infrastructure
    SetupLinuxInfra(ignoreHostPreflights bool) (types.Infra, error)
    GetLinuxInfraStatus() (types.Infra, error)

    // App preflights
    RunLinuxInstallAppPreflights() (types.InstallAppPreflightsStatusResponse, error)
    GetLinuxInstallAppPreflightsStatus() (types.InstallAppPreflightsStatusResponse, error)

    // App installation
    InstallLinuxApp() (types.AppInstall, error)
    GetLinuxAppInstallStatus() (types.AppInstall, error)
}
```