### Local Development Setup with Minikube

Source: https://docs.kubernetes-tenants.org/installation

Scripts for setting up and deploying the Tenant Operator locally using Minikube. The `setup-minikube.sh` script automatically installs cert-manager, and `deploy-to-minikube.sh` builds and deploys the operator.

```bash
# Quick setup (cert-manager included)
./scripts/setup-minikube.sh      # Create cluster with cert-manager
./scripts/deploy-to-minikube.sh  # Build and deploy operator
```

--------------------------------

### Setup Minikube Cluster for Tenant Operator

Source: https://docs.kubernetes-tenants.org/quickstart

This script creates a Minikube cluster with specified resources, installs cert-manager, sets up CRDs, and creates necessary namespaces. It ensures the environment is ready for Tenant Operator deployment.

```bash
cd /path/to/tenant-operator

# Run setup script
./scripts/setup-minikube.sh
```

--------------------------------

### Install Tenant Operator from Source

Source: https://docs.kubernetes-tenants.org/installation

Installs the Tenant Operator by building and deploying from its source code. This involves cloning the repository, installing Custom Resource Definitions (CRDs), and then deploying the operator. cert-manager must be installed before deploying the operator.

```bash
# Clone repository
git clone https://github.com/kubernetes-tenants/tenant-operator.git
cd tenant-operator

# Install CRDs
make install

# Install cert-manager first if not already installed
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Deploy operator
make deploy IMG=ghcr.io/kubernetes-tenants/tenant-operator:latest
```

--------------------------------

### Variable Collection Example

Source: https://docs.kubernetes-tenants.org/templates

Shows an example of variables collected from a database row.

```text
uid = "acme-corp"
hostOrUrl = "https://acme.example.com"
activate = true
planId = "enterprise"
```

--------------------------------

### Install Go Dependencies

Source: https://docs.kubernetes-tenants.org/development

Download and install Go module dependencies required for the Tenant Operator project. Must be run from the project root.

```bash
go mod download
```

--------------------------------

### Install cert-manager for Webhook TLS

Source: https://docs.kubernetes-tenants.org/installation

Guides users on installing cert-manager, a prerequisite for automatically managing webhook TLS certificates. This includes applying the cert-manager manifest, waiting for its deployment, and restarting the operator.

```bash
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Wait for cert-manager to be ready
kubectl wait --for=condition=Available --timeout=300s -n cert-manager deployment/cert-manager

# Restart operator to pick up certificates
kubectl rollout restart -n tenant-operator-system deployment/tenant-operator-controller-manager
```

--------------------------------

### Setup Minikube with Tenant Operator

Source: https://docs.kubernetes-tenants.org/quickstart

Configures Minikube with specified CPU and memory resources before setting up the Tenant Operator. It uses environment variables for customization.

```bash
MINIKUBE_PROFILE=my-cluster \
MINIKUBE_CPUS=8 \
MINIKUBE_MEMORY=16384 \
./scripts/setup-minikube.sh
```

--------------------------------

### Install Tenant Operator with Helm

Source: https://docs.kubernetes-tenants.org/installation

Installs the Tenant Operator using Helm. This method involves adding the Helm repository, updating it, and then performing the installation with specified namespace configurations. cert-manager must be installed prior to this.

```bash
# Step 1: Install cert-manager (REQUIRED)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Step 2: Wait for cert-manager to be ready
kubectl wait --for=condition=Available --timeout=300s -n cert-manager \
  deployment/cert-manager \
  deployment/cert-manager-webhook \
  deployment/cert-manager-cainjector

# Step 3: Add Helm repository
helm repo add tenant-operator https://kubernetes-tenants.github.io/tenant-operator
helm repo update

# Step 4: Install Tenant Operator
helm install tenant-operator tenant-operator/tenant-operator \
  --namespace tenant-operator-system \
  --create-namespace
```

--------------------------------

### Basic Template Syntax Examples

Source: https://docs.kubernetes-tenants.org/templates

Demonstrates basic variable substitution, function usage, conditional logic, and default value assignment within the template syntax. These examples show how to construct dynamic values for Kubernetes resource names.

```yaml
# Basic variable substitution
nameTemplate: "{{ .uid }}-app"

# With function
nameTemplate: "{{ .uid | trunc63 }}"

# Conditional
nameTemplate: "{{ if .region }}{{ .region }}-{{ end }}{{ .uid }}"

# Default value
nameTemplate: "{{ .uid }}-{{ .planId | default \"basic\" }}"
```

--------------------------------

### Install Tenant Operator with Kustomize

Source: https://docs.kubernetes-tenants.org/installation

Installs the Tenant Operator using Kustomize. This method requires cert-manager to be installed beforehand for managing webhook TLS certificates. The command applies the Kustomize configuration from the operator's repository.

```bash
# Step 1: Install cert-manager (if not already installed)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Step 2: Wait for cert-manager to be ready
kubectl wait --for=condition=Available --timeout=300s -n cert-manager deployment/cert-manager
kubectl wait --for=condition=Available --timeout=300s -n cert-manager deployment/cert-manager-webhook

# Step 3: Install Tenant Operator
# cert-manager will automatically issue and manage webhook TLS certificates
kubectl apply -k https://github.com/kubernetes-tenants/tenant-operator/config/default
```

--------------------------------

### Auto-Processing Example

Source: https://docs.kubernetes-tenants.org/templates

Illustrates automatic processing of variables.

```text
.host = "acme.example.com"  # extracted from .hostOrUrl
```

--------------------------------

### Install cert-manager using kubectl

Source: https://docs.kubernetes-tenants.org/installation

Installs the cert-manager component, which is required for all Tenant Operator installations to manage webhook TLS certificates. This command applies the cert-manager manifest from a specific release URL.

```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml
```

--------------------------------

### Create Example Files in Bash

Source: https://docs.kubernetes-tenants.org/contributing-datasource

Bash commands to create example directories and files for a custom datasource in the Kubernetes Tenants project.

```bash
# Create example directory
mkdir -p config/samples/yourdatasource

# Add example files
touch config/samples/yourdatasource/registry.yaml
touch config/samples/yourdatasource/secret.yaml
touch config/samples/yourdatasource/template.yaml
```

--------------------------------

### Clone Tenant Operator Repository

Source: https://docs.kubernetes-tenants.org/development

Clone the Tenant Operator Git repository and navigate to the project directory. Requires Git to be installed.

```bash
git clone https://github.com/kubernetes-tenants/tenant-operator.git
cd tenant-operator
```

--------------------------------

### Test and Build Project with Bash

Source: https://docs.kubernetes-tenants.org/development

Runs make commands for testing, linting, and building the project. Requires make and project dependencies. Inputs: None, outputs test results and build artifacts. Limitations: Assumes all dependencies are installed.

```bash
make test
make lint
make build
```

--------------------------------

### List Script Locations and Descriptions

Source: https://docs.kubernetes-tenants.org/quickstart

Displays the directory structure of the scripts used in the project, along with a brief description of each script's purpose.

```bash
scripts/
├── setup-minikube.sh          # Step 1: Cluster setup
├── deploy-to-minikube.sh      # Step 2: Operator deployment
├── deploy-mysql.sh            # Step 3: MySQL database
├── deploy-tenantregistry.sh   # Step 4: TenantRegistry CR
├── deploy-tenanttemplate.sh   # Step 5: TenantTemplate CR
└── cleanup-minikube.sh        # Cleanup
```

--------------------------------

### Troubleshoot Operator Startup Issues

Source: https://docs.kubernetes-tenants.org/quickstart

Diagnostic commands to check the status of the Tenant Operator and its dependencies (cert-manager). Useful when the operator fails to start properly.

```bash
# Check operator pod
kubectl get pods -n tenant-operator-system

# Check logs
kubectl logs -n tenant-operator-system -l control-plane=controller-manager

# Check cert-manager
kubectl get pods -n cert-manager
kubectl get certificate -n tenant-operator-system
```

--------------------------------

### Deploy MySQL Database for Tenants

Source: https://docs.kubernetes-tenants.org/quickstart

Sets up a MySQL database instance within a specified namespace, allowing for the configuration of the root password.

```bash
MYSQL_NAMESPACE=my-test-ns \
MYSQL_ROOT_PASSWORD=mypassword \
./scripts/deploy-mysql.sh
```

--------------------------------

### Updating Resources: Delete/Recreate vs. In-place Update (Bash)

Source: https://docs.kubernetes-tenants.org/policies

Compares two methods for updating Kubernetes resources managed by TenantRegistry/TenantTemplate. The 'DON'T' example shows the cascading deletion issue with delete/recreate, while the 'DO' example demonstrates the preferred in-place update using `kubectl apply`.

```bash
# ❌ DON'T: Delete and recreate (causes cascade deletion)
kubectl delete tenantregistry my-registry
kubectl apply -f updated-registry.yaml

# ✅ DO: Update in place
kubectl apply -f updated-registry.yaml

```

--------------------------------

### Troubleshoot Tenant Creation Issues

Source: https://docs.kubernetes-tenants.org/quickstart

Commands to inspect the status of the TenantRegistry and TenantTemplate resources. Useful when tenants are not being created as expected.

```bash
# Check registry status
kubectl get tenantregistry test-registry -o yaml

# Check template status
kubectl get tenanttemplate test-template -o yaml
```

--------------------------------

### Resource Creation Process

Source: https://docs.kubernetes-tenants.org/templates

Describes the server-side apply process.

```text
Rendered resource is applied to Kubernetes using Server-Side Apply.
```

--------------------------------

### Test Against Local Kind Cluster

Source: https://docs.kubernetes-tenants.org/development

Create a local Kubernetes cluster using Kind and test the operator against it. Requires Kind to be installed.

```bash
# Create kind cluster
kind create cluster --name tenant-operator-dev

# Install CRDs
make install

# Run operator
make run
```

--------------------------------

### Install CRDs and Run Controller Locally

Source: https://docs.kubernetes-tenants.org/development

Install Custom Resource Definitions (CRDs) and run the controller locally using kubectl configuration. Webhooks are not available when running locally.

```bash
# Install CRDs
make install

# Run controller locally (uses ~/.kube/config)
make run

# Run with debug logging
LOG_LEVEL=debug make run
```

--------------------------------

### Check Tenant Operator Metrics

Source: https://docs.kubernetes-tenants.org/quickstart

Forwards the metrics endpoint of the Tenant Operator and retrieves tenant-related metrics. Requires curl to be installed locally for viewing the metrics.

```bash
# Port-forward metrics endpoint
kubectl port-forward -n tenant-operator-system deployment/tenant-operator-controller-manager 8080:8080

# View metrics
curl http://localhost:8080/metrics | grep tenant
```

--------------------------------

### Execute Development Workflow with Bash

Source: https://docs.kubernetes-tenants.org/development

Provides a full shell script workflow for developing and integrating a new datasource. Requires make, vim, kubectl, and project setup. Inputs: None, outputs deployed resources. Limitations: Assumes local Kubernetes cluster and samples directory.

```bash
# 1. Create adapter file
touch internal/datasource/postgres.go

# 2. Implement interface
# (Copy mysql.go as template)

# 3. Register in factory
vim internal/datasource/interface.go

# 4. Add API types
vim api/v1/tenantregistry_types.go

# 5. Generate manifests
make manifests

# 6. Write tests
touch internal/datasource/postgres_test.go

# 7. Test
make test

# 8. Lint
make lint

# 9. Build
make build

# 10. Test locally
make install
make run
kubectl apply -f config/samples/postgres/
```

--------------------------------

### Example Tenant event messages (Bash)

Source: https://docs.kubernetes-tenants.org/monitoring

Sample output of typical Tenant events emitted by the operator, illustrating success, conflict, and deletion scenarios. These messages can be captured via the event‑viewing commands above for troubleshooting and audit purposes.

```bash
# Success
TemplateAppliedComplete: Applied 10 resources (10 ready, 0 failed, 2 changed)

# ConflictConflict: Resource conflict detected for default/acme-app (Kind: Deployment Policy: Stuck).\nAnother controller or user may be managing this resource.

# Deletion
TenantDeleting: Deleting Tenant 'acme-prod-template' (template: prod-template, uid: acme) -\nno longer in active dataset. This could be due to: row deletion, activate=false, or template change.
```

--------------------------------

### Sprig String Functions

Source: https://docs.kubernetes-tenants.org/templates

Provides examples of common Sprig string manipulation functions, including converting to uppercase/lowercase, trimming whitespace, replacing substrings, and quoting strings.

```yaml
# Uppercase/lowercase
nameTemplate: "{{ .uid | upper }}"
nameTemplate: "{{ .uid | lower }}"

# Trim whitespace
value: "{{ .name | trim }}"

# Replace
value: "{{ .uid | replace \".\" \"-\" }}"

# Quote
value: "{{ .name | quote }}"
```

--------------------------------

### Policy Combination for Init Job

Source: https://docs.kubernetes-tenants.org/policies

Combines Once creation, Delete deletion, Force conflict, and replace patch policies for jobs to handle one-time initialization exclusively by the operator. Ensures cleanup post-deletion and exact spec matching. Suitable for setup tasks; limitations include potential overwrites.

```yaml
jobs:
  - id: init
    creationPolicy: Once        # Run only once
    deletionPolicy: Delete      # Clean up after tenant deletion
    conflictPolicy: Force       # Re-create if needed
    patchStrategy: replace      # Exact job spec
    nameTemplate: "{{ .uid }}-init"
    spec:
      # ... job spec
```

--------------------------------

### Pattern: ConfigMap Before Deployment (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

Illustrates the pattern of creating a ConfigMap before a deployment that relies on its configuration. This is crucial for ensuring that applications start with the correct settings loaded from the ConfigMap.

```yaml
configMaps:
  - id: app-config
    nameTemplate: "{{ .uid }}-config"

deployments:
  - id: app
    dependIds: ["app-config"]  # Wait for configmap
```

--------------------------------

### Run Linting and Formatting

Source: https://docs.kubernetes-tenants.org/development

Perform code quality checks and formatting using golangci-lint and Go tools. Can automatically fix some linting issues.

```bash
# Run linter
make lint

# Auto-fix issues
golangci-lint run --fix

# Format code
go fmt ./...

# Or use goimports
goimports -w .
```

--------------------------------

### Verify Tenant Operator Deployment

Source: https://docs.kubernetes-tenants.org/installation

Commands to verify that the Tenant Operator is running correctly within the Kubernetes cluster. It checks the status of the operator's deployment and allows viewing its logs in real-time.

```bash
# Check operator deployment
kubectl get deployment -n tenant-operator-system tenant-operator-controller-manager

# Check operator logs
kubectl logs -n tenant-operator-system deployment/tenant-operator-controller-manager -f
```

--------------------------------

### Verify Tenant Operator Deployment

Source: https://docs.kubernetes-tenants.org/quickstart

Commands to check the status of the Tenant Operator pod and view its logs for debugging and verification purposes.

```bash
# Check operator pod
kubectl get pods -n tenant-operator-system

# View operator logs
kubectl logs -n tenant-operator-system -l control-plane=controller-manager -f
```

--------------------------------

### Template Evaluation Process

Source: https://docs.kubernetes-tenants.org/templates

Describes the template evaluation process.

```text
For each resource in the template:
  * Render `nameTemplate` → resource name
  * Render `labelsTemplate` → labels
  * Render `annotationsTemplate` → annotations
  * Render `spec` → recursively render all string values in the resource
```

--------------------------------

### Deploy Application with Kubernetes Policies (YAML)

Source: https://docs.kubernetes-tenants.org/policies

Example of deploying an application using Kubernetes tenant policies. It specifies creation, deletion, conflict, and patch strategies for the deployment. Policies ensure the deployment is kept updated, cleaned up on deletion, and handles conflicts safely.

```yaml
deployments:
  - id: app
    creationPolicy: WhenNeeded  # Keep updated
    deletionPolicy: Delete      # Clean up on deletion
    conflictPolicy: Stuck       # Safe default
    patchStrategy: apply        # Kubernetes best practice
    nameTemplate: "{{ .uid }}-app"
    spec:
      # ... deployment spec
```

--------------------------------

### Example Registry Configuration in YAML

Source: https://docs.kubernetes-tenants.org/contributing-datasource

Example YAML configuration for a TenantRegistry with a custom datasource, including host, port, credentials, and value mappings.

```yaml
apiVersion: operator.kubernetes-tenants.org/v1
kind: TenantRegistry
metadata:
  name: yourdatasource-registry
  namespace: default
spec:
  source:
    type: yourdatasource
    yourdatasource:
      host: your-host.example.com
      port: 5432
      username: tenant_reader
      passwordRef:
        name: yourdatasource-credentials
        key: password
      database: tenants_db
      table: tenants
    syncInterval: 1m
  valueMappings:
    uid: id
    hostOrUrl: url
    activate: active
```

--------------------------------

### Rollback Operator Deployment

Source: https://docs.kubernetes-tenants.org/installation

Explains how to roll back the operator deployment to a previous version using `kubectl rollout undo`. It also includes a command to check the status of the rollout process.

```bash
# Rollback to previous version
kubectl rollout undo -n tenant-operator-system \
  deployment/tenant-operator-controller-manager

# Check rollout status
kubectl rollout status -n tenant-operator-system \
  deployment/tenant-operator-controller-manager
```

--------------------------------

### Upgrade Operator Deployment Image

Source: https://docs.kubernetes-tenants.org/installation

Details how to update the operator's deployment to a new image version. This can be achieved using `kubectl set image` or the `make deploy` command with a specified image tag.

```bash
# Update operator deployment
kubectl set image -n tenant-operator-system \
  deployment/tenant-operator-controller-manager \
  manager=ghcr.io/kubernetes-tenants/tenant-operator:v1.1.0

# Or use make
make deploy IMG=ghcr.io/kubernetes-tenants/tenant-operator:v1.1.0
```

--------------------------------

### Check Operator Namespace and RBAC

Source: https://docs.kubernetes-tenants.org/installation

Verifies the operator's presence within its designated namespace and inspects the associated ClusterRoles and ClusterRoleBindings. This helps confirm the operator has the necessary permissions to function correctly.

```bash
# Check operator namespace
kubectl get all -n tenant-operator-system

# View RBAC
kubectl get clusterrole | grep tenant-operator
kubectl get clusterrolebinding | grep tenant-operator
```

--------------------------------

### Generate Kubernetes Code with Bash

Source: https://docs.kubernetes-tenants.org/development

Runs make commands to generate manifests and code for the Kubernetes project. Requires make build system and proper project setup. Inputs: None, outputs generated files in the project. Limitations: Assumes Kubebuilder or similar tools are configured.

```bash
make generate
make manifests
```

--------------------------------

### Kustomization for Webhook Patching

Source: https://docs.kubernetes-tenants.org/installation

Demonstrates how to enable webhook patches in the Kustomize configuration for the Kubernetes Tenants Operator. These patches are essential for managing webhook TLS certificates and CA injection, often handled by cert-manager.

```yaml
# config/default/kustomization.yaml
# Webhook patches are enabled by default
patches:
- path: manager_webhook_patch.yaml
- path: webhookcainjection_patch.yaml
```

--------------------------------

### Minimal Dependencies Best Practice (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

Advises specifying only the essential dependencies for a resource to avoid unnecessary delays. This example contrasts a good practice (listing only the `secret`) with a bad practice (listing an `unrelated-resource` unnecessarily).

```yaml
# Good:
- id: app
  dependIds: ["secret"]  # Only essential dependency

# Bad:
- id: app
  dependIds: ["secret", "unrelated-resource"]  # Unnecessary wait
```

--------------------------------

### PromQL for Kubernetes Tenant Observability (PromQL)

Source: https://docs.kubernetes-tenants.org/policies

PromQL queries for monitoring Kubernetes tenant operations and policies. These examples show how to count apply attempts by policy and track failed reconciliations, aiding in performance analysis and error detection.

```promql
# Count apply attempts by policy
apply_attempts_total{kind="Deployment",result="success",conflict_policy="Stuck"}

# Failed reconciliations
tenant_reconcile_duration_seconds{result="error"}
```

--------------------------------

### Update Go Dependencies and Check Vulnerabilities

Source: https://docs.kubernetes-tenants.org/security

Bash commands for managing Go project dependencies. It includes updating all dependencies (`go get -u ./...`), tidying the module (`go mod tidy`), and checking for vulnerabilities using `nancy sleuth`.

```bash
# Update Go dependencies
go get -u ./...
go mod tidy

# Check for vulnerabilities
go list -json -m all | nancy sleuth
```

--------------------------------

### Build Tenant Operator Binary

Source: https://docs.kubernetes-tenants.org/development

Compile the Tenant Operator binary for the current platform. The binary is output to bin/manager and can be executed directly.

```bash
# Build for current platform
make build

# Binary output: bin/manager
./bin/manager --help
```

--------------------------------

### Run Unit Tests

Source: https://docs.kubernetes-tenants.org/development

Execute unit tests for the Tenant Operator with optional coverage reporting. Uses the standard Go testing framework.

```bash
# Run all unit tests
make test

# Run with coverage
make test-coverage

# View coverage report
go tool cover -html=cover.out
```

--------------------------------

### Simple Tenant Table Creation and Registry Configuration

Source: https://docs.kubernetes-tenants.org/datasource

This snippet creates a basic MySQL table for storing tenant configurations including ID, URL, activation status, plan, and region, with sample data insertion. It pairs with a YAML registry configuration to map fields for Kubernetes tenant integration. Dependencies include MySQL; inputs are tenant details; outputs are a populated table and mapped values for truthy activation ('1' or '0'). Limitations: Assumes standard MySQL setup without advanced indexing.

```sql
CREATE TABLE tenant_configs (
    tenant_id VARCHAR(255) PRIMARY KEY,
    tenant_url VARCHAR(500) NOT NULL,
    is_active TINYINT(1) DEFAULT 0,
    subscription_plan VARCHAR(50),
    deployment_region VARCHAR(50),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Sample data
INSERT INTO tenant_configs (tenant_id, tenant_url, is_active, subscription_plan, deployment_region) VALUES
    ('acme-corp', 'https://acme.example.com', 1, 'enterprise', 'us-east-1'),
    ('beta-inc', 'https://beta.example.com', 1, 'startup', 'eu-west-1'),
    ('gamma-llc', 'https://gamma.example.com', 0, 'trial', 'ap-south-1');  -- Not active
```

```yaml
valueMappings:
  uid: tenant_id
  hostOrUrl: tenant_url
  activate: is_active
extraValueMappings:
  planId: subscription_plan
  region: deployment_region
```

--------------------------------

### Deploy Tenant Operator to Minikube

Source: https://docs.kubernetes-tenants.org/quickstart

Deploys the Tenant Operator to a Minikube cluster. It allows specifying the image tag for the operator and the Minikube profile.

```bash
IMG=tenant-operator:my-tag \
MINIKUBE_PROFILE=my-cluster \
./scripts/deploy-to-minikube.sh
```

--------------------------------

### Example structured log entry (JSON)

Source: https://docs.kubernetes-tenants.org/monitoring

Shows a typical JSON‑formatted log line emitted by the controller manager. It includes timestamp, log level, message, tenant identifier, and resource counters. Such logs can be parsed by log aggregation tools for monitoring and alerting.

```json
{
  "level": "info",
  "ts": "2025-01-15T10:30:00.000Z",
  "msg": "Reconciliation completed",
  "tenant": "acme-prod-template",
  "ready": 10,
  "failed": 0,
  "changed": 2
}
```

--------------------------------

### Kubernetes Resource Name Template Example

Source: https://docs.kubernetes-tenants.org/glossary

Examples of nameTemplate syntax for generating Kubernetes resource names. It utilizes template variables and the `trunc63` function to ensure names adhere to Kubernetes naming constraints (lowercase, alphanumeric, '-', max 63 characters).

```yaml
nameTemplate: "{{ .uid }}-app"
nameTemplate: "{{ .uid | trunc63 }}"
```

--------------------------------

### Deploy Tenant Operator to Minikube

Source: https://docs.kubernetes-tenants.org/quickstart

This script builds the Tenant Operator Docker image, loads it into Minikube's internal registry, and deploys the operator to the tenant-operator-system namespace. It waits for the operator to be ready before completion.

```bash
./scripts/deploy-to-minikube.sh
```

--------------------------------

### Example Secret Configuration in YAML

Source: https://docs.kubernetes-tenants.org/contributing-datasource

Example YAML configuration for a Kubernetes Secret containing datasource credentials.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: yourdatasource-credentials
  namespace: default
type: Opaque
stringData:
  password: "your-password-here"
```

--------------------------------

### Verify CRDs Installation

Source: https://docs.kubernetes-tenants.org/installation

Checks if the necessary Custom Resource Definitions (CRDs) for the Kubernetes Tenants Operator are installed in the cluster. This is a crucial step before proceeding with other operations to ensure the operator's custom resources are recognized.

```bash
# Verify CRDs are installed
kubectl get crd | grep operator.kubernetes-tenants.org
```

--------------------------------

### Watch for Rendering Errors (bash)

Source: https://docs.kubernetes-tenants.org/templates

Shows commands to check operator logs and tenant events

```bash
# Check operator logs
kubectl logs -n tenant-operator-system deployment/tenant-operator-controller-manager -f | grep "render"

# Check Tenant events
kubectl describe tenant <tenant-name>
```

--------------------------------

### Deploy to Local Cluster with Webhooks

Source: https://docs.kubernetes-tenants.org/development

Deploy the operator to a local cluster with cert-manager for complete webhook functionality. Includes validation and defaulting.

```bash
# See Local Development with Minikube guide
./scripts/deploy-to-minikube.sh  # Includes cert-manager and webhooks
```

--------------------------------

### Run Integration Tests

Source: https://docs.kubernetes-tenants.org/development

Execute integration tests that require a running Kubernetes cluster. These tests create and mutate cluster resources.

```bash
# Run integration tests (requires cluster)
make test-integration
```

--------------------------------

### Verify external-dns Installation (Bash)

Source: https://docs.kubernetes-tenants.org/integration-external-dns

These bash commands are used to verify the installation of the external-dns component in the Kubernetes cluster. They check for the running external-dns pod in the 'kube-system' namespace and allow inspection of its logs.

```bash
# Check ExternalDNS pod
kubectl get pods -n kube-system -l app=external-dns

# Check logs
kubectl logs -n kube-system -l app=external-dns
```

--------------------------------

### SQL for referencing sensitive data

Source: https://docs.kubernetes-tenants.org/security

Illustrates the recommended SQL approach for handling sensitive data by storing references instead of the actual sensitive values.

```sql
-- Good
api_key_ref = "secret-acme-api-key"

-- Bad
api_key = "sk-abc123..."
```

--------------------------------

### Upgrade CRDs using Make or Kubectl

Source: https://docs.kubernetes-tenants.org/installation

Provides commands to upgrade the Custom Resource Definitions (CRDs) for the Kubernetes Tenants Operator. This operation is safe and preserves existing data. It can be performed using `make install` or `kubectl apply -f`.

```bash
# Upgrade CRDs (safe, preserves existing data)
make install

# Or with kubectl
kubectl apply -f config/crd/bases/
```

--------------------------------

### View Reference Implementation with Bash

Source: https://docs.kubernetes-tenants.org/development

Uses cat to display the MySQL adapter code as a reference for new implementations. Requires Bash and file access. Inputs: None, outputs file contents. Limitations: File must exist in the path.

```bash
# View the implementation
cat internal/datasource/mysql.go

# Key sections:
# - NewMySQLAdapter(): Connection setup
# - QueryTenants(): Query + mapping + filtering
# - Close(): Resource cleanup
# - Helper functions: joinColumns(), isActive()
```

--------------------------------

### Kubernetes SaaS Deployment and Verification Commands

Source: https://docs.kubernetes-tenants.org/integration-external-dns

This bash script outlines the steps to deploy the SaaS platform using kubectl, monitor tenant creation, verify DNS records, and test the application's HTTPS endpoint.

```bash
# Apply template
kubectl apply -f saas-platform-template.yaml

# Wait for tenants to be created
kubectl get tenant -n default -w

# Check DNS records (example with Route53)
aws route53 list-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --query "ResourceRecordSets[?contains(Name, 'tenant')]"

# Test DNS resolution
nslookup tenant-alpha.saas.example.com
nslookup tenant-beta.saas.example.com

# Test HTTPS
curl https://tenant-alpha.saas.example.com
curl https://tenant-beta.saas.example.com
```

--------------------------------

### Verify New Tenant Creation in Kubernetes

Source: https://docs.kubernetes-tenants.org/quickstart

Checks that the new tenant has been created as a Tenant CR and that associated resources (Deployment, Service) have been provisioned. Should be run after the sync interval (default 30 seconds).

```bash
# New Tenant CR appears
kubectl get tenant delta-co-test-template

# New resources created
kubectl get deployments,services delta-co-app delta-co-app
```

--------------------------------

### Quoting String Values in YAML

Source: https://docs.kubernetes-tenants.org/templates

Explains why quoting string values in YAML is crucial to prevent parsing errors. Shows both good and bad examples.

```yaml
value: "{{ .uid }}"
```

```yaml
value: {{ .uid }}
```

--------------------------------

### Implement Datasource Adapter in Go

Source: https://docs.kubernetes-tenants.org/development

Creates an adapter implementing the Datasource interface for querying tenants. Requires Go and database connection. Inputs: Query configuration, outputs list of tenant rows. Limitations: Must handle connection and mapping correctly.

```go
// internal/datasource/your_adapter.go
package datasource

type YourAdapter struct {
    conn *YourConnection
}

// QueryTenants retrieves tenant data
func (a *YourAdapter) QueryTenants(ctx context.Context, config QueryConfig) ([]TenantRow, error) {
    // 1. Build query using config.Table, config.ValueMappings, config.ExtraMappings
    // 2. Execute query
    // 3. Map results to []TenantRow
    // 4. Filter active tenants
    return tenants, nil
}

// Close cleans up resources
func (a *YourAdapter) Close() error {
    return a.conn.Close()
}
```

--------------------------------

### Conditional Resources in YAML

Source: https://docs.kubernetes-tenants.org/templates

Shows how to conditionally create Kubernetes resources based on the plan ID. This example creates a Redis deployment only for premium or enterprise plans.

```yaml
{{- if or (eq .planId \"premium\") (eq .planId \"enterprise\") }}
deployments:
  - id: redis
    nameTemplate: "{{ .uid }}-redis"
    spec:
      apiVersion: apps/v1
      kind: Deployment
      spec:
        replicas: 1
        template:
          spec:
            containers:
            - name: redis
              image: redis:7-alpine
{{- end }}
```

--------------------------------

### Full Cleanup Including Minikube Cluster

Source: https://docs.kubernetes-tenants.org/quickstart

Runs a script to perform complete cleanup of the environment, including the Minikube cluster. The script interactively prompts for various cleanup options.

```bash
./scripts/cleanup-minikube.sh
```

--------------------------------

### Reference Secrets in Templates

Source: https://docs.kubernetes-tenants.org/security

Example of referencing a Kubernetes Secret within a template, specifically for environment variables in a Pod. It uses `valueFrom.secretKeyRef` to fetch the secret's key.

```yaml
env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: "{{ .uid }}-secrets"
      key: api-key
```

--------------------------------

### Kubernetes Secret for Database Credentials

Source: https://docs.kubernetes-tenants.org/security

Example of creating a Kubernetes Secret to store sensitive database credentials. This secret is of type Opaque and contains a stringData field for the password.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql-credentials
  namespace: default
type: Opaque
stringData:
  password: your-secure-password
```

--------------------------------

### Pattern: Secrets Before Apps (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

A common pattern in Kubernetes deployments where a secret is created before the application deployment that depends on it. This ensures sensitive information is available when the application starts.

```yaml
secrets:
  - id: api-secret
    nameTemplate: "{{ .uid }}-secret"
    # No dependencies

deployments:
  - id: app
    dependIds: ["api-secret"]  # Wait for secret
```

--------------------------------

### Kubernetes Role for Operator Permissions

Source: https://docs.kubernetes-tenants.org/security

Example of a Kubernetes Role definition that grants specific permissions to the Tenant Operator within a particular namespace. It includes rules for managing Deployments and other resources.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role  # Not ClusterRole
metadata:
  name: tenant-operator-role
  namespace: production  # Specific namespace
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["*"]
```

--------------------------------

### Create Diagnostic Bundle (Bash)

Source: https://docs.kubernetes-tenants.org/runbooks/tenant-not-ready

Initialize a directory to collect diagnostic information for tenant issues. This is the first step in comprehensive diagnostics when troubleshooting.

```bash
# Create diagnostic bundle
mkdir -p tenant-diagnostics
```

--------------------------------

### Define Resource Requests (YAML)

Source: https://docs.kubernetes-tenants.org/configuration

Shows how to define resource requests and limits for the controller manager container. It ensures the controller can start on smaller nodes and accommodates bursts of resource usage.

```yaml
resources:\n  limits:\n    cpu: 1000m\n    memory: 1Gi\n  requests:\n    cpu: 200m\n    memory: 256Mi
```

--------------------------------

### Combine Readiness and Dependencies (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

Explains how to use both `dependIds` for creation order and `waitForReady: true` to ensure a resource is not only created but also in a ready state before dependent resources are processed. This is useful for databases or other critical services.

```yaml
deployments:
  - id: db
    waitForReady: true
    timeoutSeconds: 300

deployments:
  - id: app
    dependIds: ["db"]  # Wait for db to exist AND be ready
    waitForReady: true
```

--------------------------------

### TenantTemplate Resource with Retain DeletionPolicy (YAML)

Source: https://docs.kubernetes-tenants.org/policies

Example YAML configuration for a TenantTemplate resource. It demonstrates how to set `deletionPolicy: Retain` for deployments, services, and persistentVolumeClaims to ensure these resources are not automatically deleted when the TenantTemplate or TenantRegistry is removed.

```yaml
apiVersion: operator.kubernetes-tenants.org/v1
kind: TenantTemplate
metadata:
  name: my-template
spec:
  registryId: my-registry

  # Set Retain for ALL resources
  deployments:
    - id: app
      deletionPolicy: Retain  # ✅ Keeps deployment
      nameTemplate: "{{ .uid }}-app"
      spec:
        # ... deployment spec

  services:
    - id: svc
      deletionPolicy: Retain  # ✅ Keeps service
      nameTemplate: "{{ .uid }}-svc"
      spec:
        # ... service spec

  persistentVolumeClaims:
    - id: data
      deletionPolicy: Retain  # ✅ Keeps PVC and data
      nameTemplate: "{{ .uid }}-data"
      spec:
        # ... PVC spec

```

--------------------------------

### Uninstall Operator and CRDs

Source: https://docs.kubernetes-tenants.org/installation

Provides commands for completely removing the Kubernetes Tenants Operator and its associated CRDs from the cluster. This includes deleting the deployment and CRDs, with a warning about data loss when deleting CRDs.

```bash
# Delete operator deployment
kubectl delete -k config/default

# Or with make
make undeploy

# Delete CRDs (WARNING: This deletes all Tenant data!)
make uninstall

# Or with kubectl
kubectl delete crd tenantregistries.operator.kubernetes-tenants.org
kubectl delete crd tenanttemplates.operator.kubernetes-tenants.org
kubectl delete crd tenants.operator.kubernetes-tenants.org
```

--------------------------------

### Identify Not-Ready Resources

Source: https://docs.kubernetes-tenants.org/runbooks/tenant-not-ready

Commands to list and inspect resources owned by a tenant that might be causing readiness issues. Replace <tenant-name> and <namespace> with actual values.

```bash
# List all resources owned by tenant
kubectl get all -n <namespace> -l kubernetes-tenants.org/tenant=<tenant-name>

# Check pod status specifically
kubectl get pods -n <namespace> -l kubernetes-tenants.org/tenant=<tenant-name>

# Check other resources
kubectl get deployments,statefulsets,jobs,services,ingresses -n <namespace> \
  -l kubernetes-tenants.org/tenant=<tenant-name>
```

--------------------------------

### Pattern: PVC Before StatefulSet (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

Demonstrates the common practice of creating a PersistentVolumeClaim (PVC) before a StatefulSet that will use it for persistent storage. This guarantees that storage is provisioned and available when the stateful application starts.

```yaml
persistentVolumeClaims:
  - id: data-pvc
    # No dependencies

statefulSets:
  - id: stateful-app
    dependIds: ["data-pvc"]  # Wait for PVC
```

--------------------------------

### Build and Push Container Image

Source: https://docs.kubernetes-tenants.org/development

Build a container image for the Tenant Operator and optionally push it to a registry. Supports multi-platform builds using Docker buildx.

```bash
# Build image
make docker-build IMG=myregistry/tenant-operator:dev

# Push image
make docker-push IMG=myregistry/tenant-operator:dev

# Build multi-platform
docker buildx build --platform linux/amd64,linux/arm64 \
  -t myregistry/tenant-operator:dev \
  --push .
```

--------------------------------

### Check Tenant Operator Metrics Service and ServiceMonitor (Bash)

Source: https://docs.kubernetes-tenants.org/monitoring

These commands check if the metrics service for Tenant Operator is exposed and if a ServiceMonitor resource is deployed. The ServiceMonitor check requires the prometheus-operator to be installed.

```bash
# Check if metrics port is exposed
kubectl get svc -n tenant-operator-system tenant-operator-controller-manager-metrics-service

# Check if ServiceMonitor is deployed (requires prometheus-operator)
kubectl get servicemonitor -n tenant-operator-system

```

--------------------------------

### Creating New Datasource Adapter Files in Bash

Source: https://docs.kubernetes-tenants.org/contributing-datasource

Demonstrates the bash commands to create new file stubs for different datasource adapters within the `internal/datasource/` directory. This is the initial step for implementing support for new data sources like PostgreSQL, MongoDB, or REST APIs.

```bash
# Example: PostgreSQL
touch internal/datasource/postgres.go

# Example: MongoDB
touch internal/datasource/mongodb.go

# Example: REST API
touch internal/datasource/rest.go
```

--------------------------------

### Adjust Readiness Probe Settings (Bash)

Source: https://docs.kubernetes-tenants.org/runbooks/tenant-not-ready

Edit tenant templates to adjust readiness probe settings like initial delay, period, and failure thresholds. This is useful for slow-starting applications.

```bash
kubectl edit tenanttemplate <template-name> -n <namespace>
# Increase initialDelaySeconds, periodSeconds, or failureThreshold
```

--------------------------------

### Deploy TenantTemplate Custom Resource

Source: https://docs.kubernetes-tenants.org/quickstart

Deploys a TenantTemplate custom resource, used for defining tenant configurations. It requires specifying the template name and the associated registry name.

```bash
TEMPLATE_NAME=my-template \
REGISTRY_NAME=my-registry \
./scripts/deploy-tenanttemplate.sh
```

--------------------------------

### Register Controller with Manager in Go

Source: https://docs.kubernetes-tenants.org/development

Registers the custom reconciler with the Kubernetes manager. Requires controller-runtime setup. Inputs: Manager instance, outputs potential error. Limitations: Manager must be initialized beforehand.

```go
// cmd/main.go
if err = (&controller.MyResourceReconciler{
    Client: mgr.GetClient(),
    Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
    // Handle error
}
```

--------------------------------

### Edit Tenant Template

Source: https://docs.kubernetes-tenants.org/quickstart

Opens the tenant template for editing. Changes to the template will automatically apply to all existing tenants. Can be used to add new resources like Ingress or ConfigMap.

```bash
kubectl edit tenanttemplate test-template
```

--------------------------------

### Operator Concurrency Settings

Source: https://docs.kubernetes-tenants.org/installation

Configures the concurrency for various reconciliation processes within the Kubernetes Tenants Operator. This includes settings for tenant, template, and registry reconciliations, as well as enabling leader election for high availability.

```yaml
spec:
  template:
    spec:
      containers:
      - name: manager
        args:
        - --tenant-concurrency=10        # Concurrent Tenant reconciliations (default: 10)
        - --template-concurrency=5       # Concurrent Template reconciliations (default: 5)
        - --registry-concurrency=3       # Concurrent Registry syncs (default: 3)
        - --leader-elect                 # Enable leader election
```

--------------------------------

### Deploy TenantRegistry Custom Resource

Source: https://docs.kubernetes-tenants.org/quickstart

Deploys a TenantRegistry custom resource, which is essential for managing tenant information. It allows configuration of the registry name, MySQL namespace, and sync interval.

```bash
REGISTRY_NAME=my-registry \
MYSQL_NAMESPACE=my-test-ns \
SYNC_INTERVAL=1m \
./scripts/deploy-tenantregistry.sh
```

--------------------------------

### Resource Limits Configuration

Source: https://docs.kubernetes-tenants.org/installation

Defines resource limits and requests for the Kubernetes Tenants Operator's manager container. These settings can be adjusted based on cluster size and the expected load from managing tenants, templates, and registries.

```yaml
# config/manager/manager.yaml
resources:
  limits:
    cpu: 500m      # Increase for large clusters
    memory: 512Mi  # Increase for many tenants
  requests:
    cpu: 100m
    memory: 128Mi
```

--------------------------------

### Detect Circular Dependencies (YAML)

Source: https://docs.kubernetes-tenants.org/dependencies

Demonstrates how Tenant Operator rejects circular dependencies using a DAG. A circular dependency will block reconciliation and result in a `DependencyError`. This example shows a simple two-resource cycle.

```yaml
# ❌ This will fail
- id: a
  dependIds: ["b"]
- id: b
  dependIds: ["a"]
```

--------------------------------

### Get Tenant Details using Kubectl

Source: https://docs.kubernetes-tenants.org/runbooks/tenant-not-ready

Retrieves detailed configuration and state of a specific Kubernetes tenant in YAML format and a human-readable description. This is useful for understanding the tenant's current setup and identifying potential misconfigurations.

```bash
kubectl get tenant <tenant-name> -n <namespace> -o yaml > tenant-diagnostics/tenant.yamlkubectl describe tenant <tenant-name> -n <namespace> > tenant-diagnostics/tenant-describe.txt
```

--------------------------------

### Sprig Math Functions

Source: https://docs.kubernetes-tenants.org/templates

Shows how to perform basic arithmetic operations (addition, multiplication) and find minimum or maximum values using Sprig's math functions.

```yaml
# Arithmetic
value: "{{ add .basePort 1000 }}"
value: "{{ mul .cpuLimit 2 }}"

# Min/max
value: "{{ max .minReplicas 3 }}"
```

--------------------------------

### Create and Register Go Datasource Adapter

Source: https://docs.kubernetes-tenants.org/datasource

This Go code demonstrates how to create a custom datasource adapter for the Kubernetes Tenant Operator. It includes defining the adapter structure, implementing connection and query functions, and registering the adapter in the factory. This requires a Go environment and knowledge of the Tenant Operator's interface.

```go
package datasource

type YourAdapter struct {
    conn *YourConnection
}

func NewYourAdapter(config Config) (*YourAdapter, error) {
    // Connect to your datasource
    return &YourAdapter{conn: conn}, nil
}

func (a *YourAdapter) QueryTenants(ctx context.Context, config QueryConfig) ([]TenantRow, error) {
    // Query and return tenant data
    return tenants, nil
}

func (a *YourAdapter) Close() error {
    // Cleanup
    return a.conn.Close()
}

// Register in factory (internal/datasource/interface.go)
case SourceTypeYours:
    return NewYourAdapter(config)
```

--------------------------------

### Troubleshoot Resources Not Appearing in Kubernetes

Source: https://docs.kubernetes-tenants.org/quickstart

Provides commands to check the status of Tenant Custom Resources (CR), related events, and operator logs to diagnose why resources might not be appearing.

```bash
# Check Tenant CR status
kubectl get tenant <tenant-name> -o yaml

# Check events
kubectl get events --sort-by='.lastTimestamp' | grep <tenant-name>

# Check operator logs
kubectl logs -n tenant-operator-system -l control-plane=controller-manager | grep <tenant-name>
```

--------------------------------

### Configure Sync Intervals in YAML

Source: https://docs.kubernetes-tenants.org/datasource

Shows example YAML configurations for setting the 'syncInterval' property, which controls the frequency of tenant synchronization. It provides recommendations for high, medium, and low frequencies based on development, production, and large-scale deployment scenarios.

```yaml
# High-frequency (more API calls, faster sync)
syncInterval: 30s   # For development/testing

# Medium-frequency (balanced)
syncInterval: 1m    # Recommended for production

# Low-frequency (fewer API calls, slower sync)
syncInterval: 5m    # For large deployments (1000+ tenants)
```

--------------------------------

### Manually Test Readiness Probe (Bash)

Source: https://docs.kubernetes-tenants.org/runbooks/tenant-not-ready

Execute a command within a pod's container to manually test the readiness probe endpoint. Replace the example endpoint with the actual one.

```bash
kubectl exec <pod-name> -n <namespace> -- wget -qO- http://localhost:8080/health
# Or whatever the readiness probe endpoint is
```