### Install First Release Source: https://github.com/werf/nelm/blob/main/README.md Deploys the Helm chart as a new release to the Kubernetes cluster. This command installs the 'cert-manager' chart with specified configurations. ```bash nelm release install -n myproject -r myproject ``` -------------------------------- ### Nelm Release Planning Command Source: https://github.com/werf/nelm/blob/main/README.md The `nelm release plan install` command provides a detailed explanation of upcoming cluster changes for a release. It shows accurate diffs between current and target resource versions using dry-run Server-Side Apply. ```APIDOC Command: nelm release plan install Purpose: Explains exactly what will happen in a cluster on the next release. Features: - Shows 100% accurate diffs between current and to-be resource versions. - Utilizes robust dry-run Server-Side Apply instead of client-side trickery. ``` -------------------------------- ### Define Chart Dependencies (Chart.yaml) Source: https://github.com/werf/nelm/blob/main/README.md Specifies the chart's API version, name, version, and external dependencies. This example includes cert-manager from a specific repository. ```yaml apiVersion: v2 name: mychart version: 1.0.0 dependencies: - name: cert-manager version: 1.13.3 repository: https://charts.jetstack.io ``` -------------------------------- ### Configure Chart Values (values.yaml) Source: https://github.com/werf/nelm/blob/main/README.md Defines default configuration values for the chart. This example customizes cert-manager settings like CRD installation and startup API check. ```yaml cert-manager: installCRDs: true startupapicheck: enabled: false ``` -------------------------------- ### Nelm CRD Management Improvements Source: https://github.com/werf/nelm/blob/main/README.md Nelm enhances Custom Resource Definition (CRD) management by ensuring CRDs are deployed not only on the initial install but also on upgrades. Additionally, CRDs can now be updated, not just created. ```APIDOC Feature: Improved CRD Management CRD Deployment: - CRDs from the `crds/` directory are deployed on the very first release install. - CRDs are also deployed on release upgrades. CRD Operations: - CRDs can be created. - CRDs can be updated. ``` -------------------------------- ### Helm vs Nelm Command Equivalents Source: https://github.com/werf/nelm/blob/main/README.md Illustrates common Helm CLI commands and their equivalent Nelm CLI commands for migration and usage comparison. ```bash | Helm command | Nelm command equivalent | |--------------------------------------------------------|-------------------------------------------------------------| | `helm upgrade --install --atomic --wait -n ns release ./chart` | `nelm release install --auto-rollback -n ns -r release ./chart` | | `helm uninstall -n ns release` | `nelm release uninstall -n ns -r release` | | `helm template ./chart` | `nelm chart render ./chart` | | `helm dependency build` | `nelm chart dependency download` | ``` -------------------------------- ### Download Chart Dependencies Source: https://github.com/werf/nelm/blob/main/README.md Fetches and downloads the dependencies specified in Chart.yaml, creating a Chart.lock file to ensure reproducible builds. ```bash nelm chart dependency download ``` -------------------------------- ### Create Helm Chart Directory Source: https://github.com/werf/nelm/blob/main/README.md Initializes a new directory for a Helm chart. This is the first step in creating a new chart structure. ```bash mkdir mychart cd mychart ``` -------------------------------- ### Nelm CLI Command Overview Source: https://github.com/werf/nelm/blob/main/README.md A comprehensive list of Nelm CLI commands categorized by their functionality, including release management, chart operations, secret handling, dependency management, and repository interactions. ```APIDOC Release Commands: release install Deploy a chart to Kubernetes. release rollback Rollback to a previously deployed release. release plan install Plan a release install to Kubernetes. release uninstall Uninstall a Helm Release from Kubernetes. release list List all releases in a namespace. release history Show release history. release get Get information about a deployed release. Chart Commands: chart lint Lint a chart. chart render Render a chart. chart download Download a chart from a repository. chart upload Upload a chart to a repository. chart pack Pack a chart into an archive to distribute via a repository. Secret Commands: chart secret key create Create a new chart secret key. chart secret key rotate Reencrypt secret files with a new secret key. chart secret values-file edit Interactively edit encrypted values file. chart secret values-file encrypt Encrypt values file and print result to stdout. chart secret values-file decrypt Decrypt values file and print result to stdout. chart secret file edit Interactively edit encrypted file. chart secret file encrypt Encrypt file and print result to stdout. chart secret file decrypt Decrypt file and print result to stdout. Dependency Commands: chart dependency download Download chart dependencies from Chart.lock. chart dependency update Update Chart.lock and download chart dependencies. Repo Commands: repo add Set up a new chart repository. repo remove Remove a chart repository. repo update Update info about available charts for all chart repositories. repo login Log in to an OCI registry with charts. repo logout Log out from an OCI registry with charts. Other Commands: completion bash Generate the autocompletion script for bash completion fish Generate the autocompletion script for fish completion powershell Generate the autocompletion script for powershell completion zsh Generate the autocompletion script for zsh version Show version. ``` -------------------------------- ### Kubernetes Annotations for Deployment Control Source: https://github.com/werf/nelm/blob/main/README.md This section details various werf-specific annotations used to control the deployment process, including resource ordering, dependency management, and handling of sensitive data. ```APIDOC Annotation `werf.io/weight`: Format: `` Default: `0` Example: `werf.io/weight: "10"`, `werf.io/weight: "-10"` Description: Controls deployment order. Resources with the same weight are grouped and deployed in parallel. Groups are deployed sequentially from low to high weight. Has higher priority than `helm.sh/hook-weight`, but lower than `werf.io/deploy-dependency-`. Annotation `werf.io/deploy-dependency-`: Format: `state=ready|present[,name=][,namespace=][,kind=][,group=][,version=]` Example: `werf.io/deploy-dependency-db: state=ready,kind=StatefulSet,name=postgres`, `werf.io/deploy-dependency-app: state=present,kind=Deployment,group=apps,version=v1,name=app,namespace=app` Description: Ensures a resource deploys only after its dependencies are satisfied (either `present` or `ready`). Offers more power than hooks and `werf.io/weight`. Can only point to resources within the same release. Has higher priority than `werf.io/weight` and `helm.sh/hook-weight`. Annotation `.external-dependency.werf.io/resource`: Format: `[..]/` Example: `secret.external-dependency.werf.io/resource: secret/config`, `someapp.external-dependency.werf.io/resource: deployments.v1.apps/app` Description: Manages deployment based on external dependencies (resources outside the current release). Waits for the specified resource to be `present` and `ready`. Annotation `.external-dependency.werf.io/name`: Format: `` Example: `someapp.external-dependency.werf.io/name: someapp-production` Description: Sets the namespace for the external dependency defined by `.external-dependency.werf.io/resource`. The `` must match across both annotations. Defaults to the release namespace if not specified. Annotation `werf.io/sensitive`: Format: `true|false` Default: `false` (but `true` for `v1/Secret`) Example: `werf.io/sensitive: "true" Description: DEPRECATED. Use `werf.io/sensitive-paths` instead. Prevents diffs for the resource. Behavior can be altered by the `NELM_FEAT_FIELD_SENSITIVE` feature gate. Annotation `werf.io/sensitive-paths`: Format: `JSONPath,JSONPath,...` Example: `werf.io/sensitive-paths: "$.spec.template.spec.containers[*].env[*].value,$.data.*" Description: Prevents diffs for specific resource fields matching the provided JSONPath expressions. Overrides `werf.io/sensitive`. Annotation `werf.io/track-termination-mode`: Format: `WaitUntilResourceReady|NonBlocking` Default: `WaitUntilResourceReady` Example: `werf.io/track-termination-mode: NonBlocking` Description: Configures when resource readiness tracking stops. `WaitUntilResourceReady` waits for the resource to be `ready`. `NonBlocking` does not wait. Annotation `werf.io/fail-mode`: Format: `FailWholeDeployProcessImmediately|IgnoreAndContinueDeployProcess` Default: `FailWholeDeployProcessImmediately` Example: `werf.io/fail-mode: IgnoreAndContinueDeployProcess` Description: Determines behavior when resource tracking errors exceed `werf.io/failures-allowed-per-replica`. `FailWholeDeployProcessImmediately` fails the release. `IgnoreAndContinueDeployProcess` continues the deployment. ``` -------------------------------- ### Plan Release Update Source: https://github.com/werf/nelm/blob/main/README.md Prepares for a release update by specifying changes, such as increasing the replica count for a deployment. This command shows what will change without applying it. ```bash nelm release plan install -n myproject -r myproject --set cert-manager.replicaCount=2 ``` -------------------------------- ### Nelm Resource State Tracking Features Source: https://github.com/werf/nelm/blob/main/README.md Nelm provides robust resource tracking capabilities, including reliable detection of resource states (readiness, presence, absence, failures), heuristic readiness detection for Custom Resources, and automatic tracking of dependent resources like Pods. ```APIDOC Feature: Resource State Tracking Capabilities: - Reliable detection of resource readiness, presence, absence, or failures. - Heuristically determined readiness of Custom Resources by analyzing status fields (works for ~50% of CRs, no false positives). - Automatic tracking of dependent resources (e.g., Pods of Deployments). User Feedback: - Table with tracked resources current information (statuses, errors) printed periodically during deployment. Configuration: - Tracking can be configured per resource using annotations. ``` -------------------------------- ### Nelm Resource Ordering Annotations Source: https://github.com/werf/nelm/blob/main/README.md Nelm utilizes specific annotations to control the order and dependencies of resource deployments within a release. These annotations allow for fine-grained control over deployment sequences, including parallel execution and external dependencies. ```APIDOC Annotation: werf.io/weight Description: Similar to helm.sh/hook-weight, but works for non-hook resources. Resources with the same weight are deployed in parallel. Annotation: werf.io/deploy-dependency- Description: Makes Nelm wait for the readiness or presence of another resource in a release before deploying the annotated resource. This is a powerful mechanism for resource ordering. Annotation: .external-dependency.werf.io/resource Description: Allows Nelm to wait for the readiness of non-release resources, such as those created by third-party operators. ``` -------------------------------- ### Deployment Logs and Status Source: https://github.com/werf/nelm/blob/main/README.md Shows the output from a deployment, including logs from pods and the progress status of Kubernetes resources. This helps in monitoring the deployment's health. ```smalltalk Starting release "myproject" (namespace: "myproject") ┌ Logs for Pod/myproject-cert-manager-webhook-76c89cc4c7-d8xn4, container/cert-manager-webhook │ W0324 14:49:12.719893 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. │ I0324 14:49:12.743617 1 webhook.go:128] "cert-manager/webhook: using dynamic certificate generating using CA stored in Secret resource" secret_namespace="myproject" secret_name="myproject-cert-manager-webhook-ca" │ I0324 14:49:12.743756 1 server.go:147] "cert-manager: listening for insecure healthz connections" address=":6080" │ I0324 14:49:12.744309 1 server.go:213] "cert-manager: listening for secure connections" address=":10250" │ I0324 14:49:13.747685 1 dynamic_source.go:255] "cert-manager: Updated cert-manager webhook TLS certificate" DNSNames=["myproject-cert-manager-webhook","myproject-cert-manager-webhook.myproject","myproject-cert-manager-webhook.myproject.svc"] └ Logs for Pod/myproject-cert-manager-webhook-76c89cc4c7-d8xn4, container/cert-manager-webhook ┌ Progress status │ RESOURCE (→READY) STATE INFO │ Deployment/myproject-cert-manager-webhook WAITING Ready:0/1 │ • Pod/myproject-cert-manager-webhook-76c89cc4c7-d8xn4 UNKNOWN Status:Running │ ClusterRole/myproject-cert-manager-cainjector READY │ ClusterRole/myproject-cert-manager-cluster-view READY │ Role/myproject-cert-manager:leaderelection READY Namespace:kube-system │ Role/myproject-cert-manager-webhook:dynamic-serving READY │ RoleBinding/myproject-cert-manager-cainjector:leaderelection WAITING Namespace:kube-system │ RoleBinding/myproject-cert-manager:leaderelection WAITING Namespace:kube-system │ RoleBinding/myproject-cert-manager-webhook:dynamic-serving WAITING │ Service/myproject-cert-manager READY │ Service/myproject-cert-manager-webhook READY │ ServiceAccount/myproject-cert-manager READY │ ServiceAccount/myproject-cert-manager-cainjector READY │ ServiceAccount/myproject-cert-manager-webhook READY │ ValidatingWebhookConfiguration/myproject-cert-manager-webhook READY ... (truncated) └ Progress status ┌ Completed operations │ Create resource: ClusterRole/myproject-cert-manager-cainjector │ Create resource: ClusterRole/myproject-cert-manager-cluster-view │ Create resource: ClusterRole/myproject-cert-manager-controller-approve:cert-manager-io │ Create resource: ClusterRole/myproject-cert-manager-controller-certificates │ Create resource: ClusterRole/myproject-cert-manager-controller-certificatesigningrequests │ Create resource: ClusterRole/myproject-cert-manager-controller-challenges ... (truncated) └ Completed operations Succeeded release "myproject" (namespace: "myproject") ``` -------------------------------- ### Nelm Deployment Output Control Annotations Source: https://github.com/werf/nelm/blob/main/README.md Annotations that control how Nelm handles sensitive information, logs, events, and errors during deployment. These settings allow for fine-grained management of deployment visibility and behavior. ```APIDOC Annotation `werf.io/sensitive`: - Description: Marks a resource or its fields as sensitive, preventing their values from being displayed in logs or output. - Usage: Protects sensitive data like passwords or API keys. Annotation `werf.io/sensitive-paths`: - Description: Specifies exact paths within a resource's spec or status to be treated as sensitive. - Usage: Allows selective masking of sensitive fields. Annotation `werf.io/track-termination-mode`: - Description: Configures how Nelm tracks the termination of a resource, influencing deployment status updates. - Usage: Can be set to modes like 'wait' or 'skip' for specific termination behaviors. Annotation `werf.io/fail-mode`: - Description: Defines the behavior when a resource fails to deploy or update. - Usage: Options might include 'fail' (stop deployment), 'ignore' (continue), or 'warn'. Annotation `werf.io/failures-allowed-per-replica`: - Description: Specifies the number of replica failures allowed before considering the resource deployment as failed. - Usage: Useful for stateful applications or deployments with rolling updates. Annotation `werf.io/no-activity-timeout`: - Description: Sets a timeout for resources that show no activity (e.g., no new events or status changes). - Usage: Helps detect stalled deployments. Annotation `werf.io/log-regex`: - Description: A regular expression used to filter or highlight specific log messages from pods. - Usage: Can be used to extract relevant information or suppress noise. Annotation `werf.io/log-regex-for-`: - Description: Similar to `werf.io/log-regex`, but specifically targets logs from a named container within a pod. - Usage: For multi-container pods, allows targeted log analysis. Annotation `werf.io/skip-logs`: - Description: Prevents Nelm from collecting and displaying logs for the annotated resource. - Usage: Useful for resources that generate excessive or irrelevant logs. Annotation `werf.io/skip-logs-for-containers`: - Description: Skips log collection for specific containers within a pod. - Usage: Fine-grained control over log output per container. Annotation `werf.io/show-logs-only-for-containers`: - Description: Configures Nelm to only show logs for specified containers, hiding logs from others. - Usage: Focuses log output on critical components. Annotation `werf.io/show-service-messages`: - Description: Controls whether Nelm displays internal service messages or status updates during deployment. - Usage: Can be toggled to provide more or less verbose output. ``` -------------------------------- ### Deploy Updated Release Source: https://github.com/werf/nelm/blob/main/README.md Applies the planned changes to the Kubernetes cluster, deploying the updated release. This command executes the modifications reviewed in the planning stage. ```bash nelm release install -n myproject -r myproject --set cert-manager.replicaCount=2 ``` -------------------------------- ### Release Update Plan Status Source: https://github.com/werf/nelm/blob/main/README.md Displays the planned changes for a release update, highlighting differences in resource specifications. This allows for review before deployment. ```smalltalk Planning release install "myproject" (namespace: "myproject") ┌ Update Deployment/myproject-cert-manager │ namespace: myproject │ spec: │ progressDeadlineSeconds: 600 │ - replicas: 1 │ + replicas: 2 │ revisionHistoryLimit: 10 │ selector: │ matchLabels: └ Update Deployment/myproject-cert-manager Planned changes summary for release "myproject" (namespace: "myproject"): - update: 1 resource(s) ``` -------------------------------- ### Nelm Resource Ordering Annotations Source: https://github.com/werf/nelm/blob/main/README.md Annotations used to define advanced resource ordering and dependencies within Kubernetes deployments managed by Nelm. These annotations control the sequence and relationships between resources during the deployment process. ```APIDOC Annotation `werf.io/weight`: - Description: Assigns a weight to a resource, influencing its deployment order relative to other resources. - Usage: Can be used to prioritize or de-prioritize specific resources. Annotation `werf.io/deploy-dependency-`: - Description: Defines an explicit dependency of the annotated resource on another resource identified by ``. - Usage: Ensures that the resource with `` is successfully deployed or ready before the annotated resource proceeds. Annotation `.external-dependency.werf.io/resource`: - Description: Specifies an external resource that the current resource depends on, identified by its name and namespace. - Usage: Useful for managing dependencies on resources managed outside of the current Helm chart. Annotation `.external-dependency.werf.io/name`: - Description: Provides the name of the external resource for dependency tracking. - Usage: Works in conjunction with `.external-dependency.werf.io/resource` to define external dependencies. ``` -------------------------------- ### Nelm Deployment Strategy: Server-Side Apply Source: https://github.com/werf/nelm/blob/main/README.md Nelm replaces Helm's traditional 3-Way Merge with Kubernetes Server-Side Apply (SSA). SSA is a server-side mechanism that resolves issues related to client-side patch assumptions, ensuring more reliable resource updates. ```APIDOC Feature: Server-Side Apply (SSA) Replaces: Helm 3-Way Merge (3WM) Description: SSA is a client-side mechanism to make a patch for updating a resource in a cluster. Its issues stem from the fact that it has to assume that all previous release manifests were successfully applied to the cluster, which is not always the case. SSA makes patches server-side by Kubernetes instead of client-side by Helm, solving issues of 3WM. Benefits: Solves issues of 3WM, widely adopted by other deployment tools like Flux. Adoption: Nelm adopted SSA from the beginning due to a rewrite of the deployment subsystem. ``` -------------------------------- ### werf/nelm Resource Tracking Annotations Source: https://github.com/werf/nelm/blob/main/README.md Kubernetes annotations for controlling resource tracking behavior in werf/nelm. These annotations allow fine-tuning of error tolerance, timeouts, log filtering, and log display for deployed resources. ```APIDOC Annotation: werf.io/failures-allowed-per-replica Format: Default: `1` Example: `werf.io/failures-allowed-per-replica: "0"` Description: Set the number of allowed errors during resource tracking. When exceeded, act according to `werf.io/fail-mode`. Annotation: werf.io/no-activity-timeout Format: (reference: https://pkg.go.dev/time#ParseDuration) Default: `4m` Example: `werf.io/no-activity-timeout: 8m30s` Description: Take it as a resource tracking error if no new events or resource updates are received during resource tracking for the specified time. Annotation: werf.io/log-regex Format: (reference: https://github.com/google/re2/wiki/Syntax) Example: `werf.io/log-regex: ".*ERR|err|WARN|warn.*"` Description: Only show log lines that match the specified regex. Annotation: werf.io/log-regex-for- Format: (reference: https://github.com/google/re2/wiki/Syntax) Example: `werf.io/log-regex-for-backend: ".*ERR|err|WARN|warn.*"` Description: For the specified container, only show log lines that match the specified regex. Annotation: werf.io/skip-logs Format: `true|false` Default: `false` Example: `werf.io/skip-logs: "true"` Description: Don't print container logs during resource tracking. Annotation: werf.io/skip-logs-for-containers Format: `[,...]` Example: `werf.io/skip-logs-for-containers: "backend,frontend"` Description: Don't print logs for specified containers during resource tracking. Annotation: werf.io/show-logs-only-for-containers Format: `[,...]` Example: `werf.io/show-logs-only-for-containers: "backend,frontend"` Description: Print logs only for specified containers during resource tracking. Annotation: werf.io/show-service-messages Format: `true|false` Default: `false` Example: `werf.io/show-service-messages: "true"` Description: Show resource events during resource tracking. ``` -------------------------------- ### Nelm Log and Event Printing During Deploy Source: https://github.com/werf/nelm/blob/main/README.md During deployment, Nelm automatically finds Pods of deployed release resources and periodically prints their container logs. Resource events can also be printed using a specific annotation. ```APIDOC Feature: Log and Event Printing During Deployment: - Nelm finds Pods of deployed release resources. - Periodically prints their container logs. Event Printing: - Annotation: `werf.io/show-service-messages: "true"` enables printing resource events. Configuration: - Log/event printing can be tuned with annotations. ``` -------------------------------- ### nelm: Create Secret Key Source: https://github.com/werf/nelm/blob/main/README.md Generates a secret key required for encrypting and decrypting sensitive values and files used with nelm. This key should be exported as an environment variable. ```bash export NELM_SECRET_KEY="$(nelm chart secret key create)" ``` -------------------------------- ### Nelm Feature Gates (Environment Variables) Source: https://github.com/werf/nelm/blob/main/README.md Environment variables that control the enablement of experimental or specific features within Nelm. These allow users to opt-in to new functionalities or modify existing behavior. ```APIDOC Environment Variable `NELM_FEAT_PREVIEW_V2`: - Description: Enables preview features related to version 2 of some functionality. - Usage: Set to `1` or `true` to enable. Environment Variable `NELM_FEAT_REMOTE_CHARTS`: - Description: Enables support for fetching and deploying Helm charts directly from remote repositories. - Usage: Set to `1` or `true` to enable. Environment Variable `NELM_FEAT_NATIVE_RELEASE_LIST`: - Description: Enables a native implementation for listing Helm releases, potentially offering better performance or compatibility. - Usage: Set to `1` or `true` to enable. Environment Variable `NELM_FEAT_NATIVE_RELEASE_UNINSTALL`: - Description: Enables a native implementation for uninstalling Helm releases. - Usage: Set to `1` or `true` to enable. Environment Variable `NELM_FEAT_PERIODIC_STACK_TRACES`: - Description: Configures Nelm to periodically capture and report stack traces for debugging long-running operations. - Usage: Set to `1` or `true` to enable. Environment Variable `NELM_FEAT_FIELD_SENSITIVE`: - Description: Enables enhanced sensitivity detection for fields within Kubernetes resources, potentially flagging more fields as sensitive by default. - Usage: Set to `1` or `true` to enable. ``` -------------------------------- ### werf/nelm Template Functions Source: https://github.com/werf/nelm/blob/main/README.md Helper functions for werf/nelm templating, used for accessing secrets, debugging template rendering, and including other templates. ```go Function: werf_secret_file Format: werf_secret_file "" Example: config: {{ werf_secret_file "config.yaml" | nindent 4 }} Description: Read the specified secret file from the `secret/` directory of the Helm chart. Function: dump_debug Format: dump_debug "" Example: {{ dump_debug $ }} Description: If the log level is `debug`, then pretty-dumps the passed value to the logs. Handles any kind of complex types, including .Values, or event root context. Never prints to the templating output. Function: printf_debug Format: printf_debug "" Example: {{ printf_debug "myval: %s" .Values.myval }} Description: If the log level is `debug`, then prints the result to the logs. Never prints to the templating output. Function: include_debug Format: include_debug "