### Setup and Run Toxiproxy Chaos Scenario

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/chaos-tester/README.md

This sequence sets up the E2E environment, starts toxiproxy, runs the chaos utility to break connections, starts the Argo CD agent, and then runs E2E tests until failure. Ensure the toxiproxy connection code is uncommented in main.go.

```bash
# Initial setup
make setup-e2e

# Run the following in different windows/tabs, for convenience...

# Window A: start toxiproxy
podman run --rm --net=host -it ghcr.io/shopify/toxiproxy
# You should see 'Starting Toxiproxy' message

# Window B: start chaos utility, which will break connection between principal/agent via toxiproxy
cd hack/chaos-tester
# In 'hack/chaos-tester/main.go', ensure the toxiproxy connection code is uncommented.
# - Note the other ACTION-REQUIRED configurables within the code.
go run .
# You should see 'Waiting for toxiproxy server'/'Creating proxy'


# Window C: start agent itself via goreman
ARGOCD_AGENT_REMOTE_PORT=8475 make start-e2e
# Verify you see messages from agent/managed indicating they are occasionally losing connection to principal
# - For example, "Auth failure: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:8475: connect: connection refused\" (retrying in 1s)"

# Window D:
# This will run the E2E tests over and over until they fail. 
hack/chaos-tester/until-fail.sh make test-e2e

# For any tests that fail, you can further investigate them by running them over and over until fail.
# e.g.
"hack/chaos-tester/until-fail.sh" go test -count=1 -v -v -timeout 60m -run  TestSyncTestSuite/Test_SyncManaged github.com/argoproj-labs/argocd-agent/test/e2e
```

--------------------------------

### Two-Region HA Setup - Region A (Primary)

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/ha.md

Example configuration for Region A, setting it as the preferred primary principal in an HA setup. It enables HA, sets the preferred role, specifies the peer address in Region B, and allows replication clients from Region B.

```bash
argocd-agent principal \
  --ha-enabled \
  --ha-preferred-role=primary \
  --ha-peer-address=principal.region-b.internal:8443 \
  --ha-allowed-replication-clients=region-b
```

--------------------------------

### Two-Region HA Setup - Region B (Replica)

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/ha.md

Example configuration for Region B, setting it as the preferred replica principal in an HA setup. It enables HA, sets the preferred role, specifies the peer address in Region A, and allows replication clients from Region A.

```bash
argocd-agent principal \
  --ha-enabled \
  --ha-preferred-role=replica \
  --ha-peer-address=principal.region-a.internal:8443 \
  --ha-allowed-replication-clients=region-a
```

--------------------------------

### Setup Reverse Tunnel Environment

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/reverse-tunnel/README.md

Execute this command to set up the development environment, including installing Rathole, updating cluster secrets, and starting the local rathole container.

```shell
# Setup environment as usual
make setup-e2e

# Installs Rathole Deployment/Services on vcluster, update 'managed-agent' cluster Secret to point to Rathole Deployment, and start local rathole container
./hack/dev-env/reverse-tunnel/setup.sh

```

--------------------------------

### Setup and Run Process Restart Chaos Scenario

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/chaos-tester/README.md

This sequence sets up the E2E environment, runs the chaos utility to restart processes via goreman, starts the Argo CD agent, and then runs E2E tests until failure. Ensure the process restart code is uncommented in main.go. Note that some E2E tests that also restart processes are incompatible with this scenario.

```bash
# Initial setup
make setup-e2e

# Run the following in different windows/tabs, for convenience...

# Window A: start chaos utility, which will start/stop processes via goreman
cd hack/chaos-tester
# In 'hack/chaos-tester/main.go', ensure the process restart code is uncommented.
# - Note the other ACTION-REQUIRED configurables within the code.
go run .

# Window B: start agent itself via goreman
make start-e2e
# Verify that you see principal/agent processes occasionally being terminated via chaos-tester utility
# For example: 
# 02:07:45        principal | signal: interrupt
# 02:07:45        principal | Terminating principal
# (...) 
# 02:07:53        principal | Starting principal on port 5000

# Window C:
# This will run the E2E tests over and over until they fail. 
hack/chaos-tester/until-fail.sh make test-e2e


# For any tests that fail, you can further investigate them by running them over and over until fail.
# e.g.
 "hack/chaos-tester/until-fail.sh" go test -count=1 -v -v -timeout 60m -run  TestSyncTestSuite/Test_SyncManaged github.com/argoproj-labs/argocd-agent/test/e2e
```

--------------------------------

### Start E2E Test Principal and Agents

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/test/e2e/README.md

Execute this command from the repository root to start the principal and agents for the end-to-end tests.

```shell
make start-e2e
```

--------------------------------

### Install Istio and Set Environment Variables

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

Download the Istio distribution and set the ISTIO_DIR environment variable to point to the Istio installation directory. This also adds Istio's bin directory to your PATH for easy access to istioctl.

```bash
curl -L https://istio.io/downloadIstio | sh -
export ISTIO_DIR=$PWD/$(ls -d istio-* | head -1)
export PATH="$PATH:$ISTIO_DIR/bin"
```

--------------------------------

### Install Istio on Workload Cluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

Installs Istio on the workload cluster using the 'demo' profile. It first creates the 'istio-system' namespace and configures a shared CA certificate secret before running the istioctl install command.

```bash
kubectl create namespace istio-system --context kind-$AGENT_CLUSTER_NAME
```

```bash
kubectl create secret generic cacerts -n istio-system --context kind-$AGENT_CLUSTER_NAME \
  --from-file=ca-cert.pem=/tmp/istio-certs/cluster2/ca-cert.pem \
  --from-file=ca-key.pem=/tmp/istio-certs/cluster2/ca-key.pem \
  --from-file=root-cert.pem=/tmp/istio-certs/cluster2/root-cert.pem \
  --from-file=cert-chain.pem=/tmp/istio-certs/cluster2/cert-chain.pem
```

```bash
istioctl install --context kind-$AGENT_CLUSTER_NAME --set profile=demo -y
```

--------------------------------

### Install Argo CD with ApplicationSet for Workload Cluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Installs Argo CD along with the ApplicationSet controller onto the workload cluster using server-side apply. This is an alternative to the standard install and is required for Autonomous mode when using ApplicationSets.

```bash
kubectl apply -n $NAMESPACE_NAME --server-side 
  -k "https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/agent-autonomous-appset?ref=$RELEASE_BRANCH" 
  --context kind-$AGENT_CLUSTER_NAME
```

--------------------------------

### Development Setup: Flexible Agent Namespace Configuration

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

Enable flexible agent namespace management for development environments. This configuration allows any namespace starting with 'agent-*' and enables auto-creation with specific labels.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-agent-params
  namespace: argocd
data:
  principal.allowed-namespaces: "agent-*"
  principal.namespace-create.enable: "true"
  principal.namespace-create.pattern: "^agent-.*$"
  principal.namespace-create.labels: "environment=development,managed-by=argocd-agent"
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/secret-to-kubeconf/README.md

Install the necessary Python packages for the script to function. This can be done using a requirements file or by installing packages individually.

```bash
pip install -r requirements.txt
```

```bash
pip install kubernetes PyYAML
```

--------------------------------

### Helm Install with Custom values.yaml File

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/install/helm-repo/docs/install-agent.md

Install the agent using a custom YAML file to specify multiple configuration overrides.

```bash
helm install argocd-agent ghcr.io/argoproj-labs/argocd-agent/argocd-agent-agent-helm --version 0.1.0 \
  -f my-custom-values.yaml
```

--------------------------------

### Example Output with Full-Detail Logging

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/observability.md

Demonstrates the log output when full-detail logging is enabled with the 'actions' category, including the 'detail' field with resource information.

```text
level=info msg="Created application default/my-app" action=create detail="{\"metadata\":{\"name\":\"my-app\",...}}" log_category=actions name=my-app namespace=default resource_type=application
```

--------------------------------

### Start MicroK8s Cluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

Starts the MicroK8s Kubernetes cluster. This is a prerequisite for setting up the development environment.

```shell
sudo microk8s start
```

--------------------------------

### Setup E2E Test Environment

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/test/e2e/README.md

Execute this command from the repository root to set up the test environment on your cluster. Ensure vcluster and LoadBalancer functionality are available.

```shell
make setup-e2e
```

--------------------------------

### Install Agent using Helm Chart

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/adding-agents.md

Install the agent using the Helm chart, specifying the agent server address, port, and mode via `--set` flags. Ensure you are in the correct directory or provide the chart path.

```bash
helm install argocd-agent-agent ./charts/agent \
  --namespace argocd \
  --kube-context <workload-cluster-context> \
  --set agent.server.address=<principal-address> \
  --set agent.server.port=<principal-port> \
  --set agent.mode=<autonomous|managed>
```

--------------------------------

### Example Application Naming Convention (Managed Mode)

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/applications.md

Employ consistent naming conventions for your applications within each namespace for managed agents.

```yaml
   metadata:
     name: frontend-prod
     namespace: production-east
```

--------------------------------

### Start E2E Tests

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/reverse-tunnel/README.md

Run this command after setting up the reverse tunnel to initiate the end-to-end tests.

```shell
# Start E2E tests
make start-e2e

```

--------------------------------

### Install Argo CD App in Managed Vcluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

Use these commands to switch to the control plane context and apply the managed guestbook application manifest. This installs the application within the managed vcluster.

```shell
kubectl config use-context vcluster-control-plane
kubectl apply -f hack/dev-env/apps/managed-guestbook.yaml
```

--------------------------------

### Coordinator Configuration Example

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/proposals/ha-failover.md

Illustrates how to configure the principal to use an external coordinator for HA. This example shows settings for AWS Route53 ARC, but other types like Consul or Kubernetes Lease are supported.

```yaml
principal:
  ha:
    enabled: true
    mode: coordinator        # "manual" (default) or "coordinator"
    coordinator:
      type: aws-arc          # or: consul, k8s-lease
      pollInterval: 10s
      aws:
        routingControlArn: arn:aws:route53-recovery-control::123456789012:...
```

--------------------------------

### Install Istio with Shared CA on Control Plane Cluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

Installs Istio on the control plane cluster after creating the 'istio-system' namespace and a Kubernetes secret containing the shared CA certificates. Uses the 'demo' profile for installation.

```bash
kubectl create namespace istio-system --context kind-$PRINCIPAL_CLUSTER_NAME

kubectl create secret generic cacerts -n istio-system --context kind-$PRINCIPAL_CLUSTER_NAME \
  --from-file=ca-cert.pem=/tmp/istio-certs/cluster1/ca-cert.pem \
  --from-file=ca-key.pem=/tmp/istio-certs/cluster1/ca-key.pem \
  --from-file=root-cert.pem=/tmp/istio-certs/cluster1/root-cert.pem \
  --from-file=cert-chain.pem=/tmp/istio-certs/cluster1/cert-chain.pem

istioctl install --context kind-$PRINCIPAL_CLUSTER_NAME --set profile=demo -y
```

--------------------------------

### Install Argo CD App in Autonomous Vcluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

Use these commands to switch to the autonomous agent context and apply the autonomous guestbook application manifest. This installs the application within the autonomous vcluster.

```shell
kubectl config use-context vcluster-agent-autonomous
kubectl apply -f ./hack/dev-env/apps/autonomous-guestbook.yaml
```

--------------------------------

### Install Argo CD for Autonomous Agents

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Installs Argo CD components on the workload cluster for autonomous agent mode. Replace '<release-branch>' with the desired release branch.

```bash
# For autonomous agents (alternative)
# kubectl apply -n argocd \
#   --server-side \
#   -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/agent-autonomous?ref=<release-branch>' \
#   --context <workload-cluster-context>
```

--------------------------------

### Create Virtual Clusters and Install Argo CD

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

Run this script to create three vclusters and install an opinionated Argo CD into each. Ensure vcluster is in your PATH and your kubeconfig is set to a cluster admin context. Requires vcluster version 0.20-beta4 or newer.

```shell
chmod +x ./hack/dev-env/setup-vcluster-env.sh
./hack/dev-env/setup-vcluster-env.sh create
```

--------------------------------

### Install Argo CD for Control Plane

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Install the Principal-specific Argo CD configuration using server-side apply to avoid annotation size limits. This includes essential components like argocd-server and argocd-repo-server, but excludes argocd-application-controller.

```bash
kubectl apply -n $NAMESPACE_NAME --server-side \
  -k "https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/principal?ref=$RELEASE_BRANCH" \
  --context kind-$PRINCIPAL_CLUSTER_NAME
```

--------------------------------

### Start Jaeger with Docker

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/operations/tracing.md

Launches a Jaeger all-in-one instance using Docker for local development and testing of OpenTelemetry traces.

```bash
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest
```

--------------------------------

### Example Output Without Full-Detail Logging

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/observability.md

Shows the log output when full-detail logging is disabled, omitting the 'detail' field that contains verbose resource payloads.

```text
level=info msg="Created application default/my-app" action=create log_category=actions name=my-app namespace=default resource_type=application
```

--------------------------------

### Configure Allowed Namespaces (Wildcard Pattern)

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

Use this command to allow the principal to operate in any namespace that starts with 'agent-'.

```bash
argocd-agent principal --allowed-namespaces='agent-*'
```

--------------------------------

### Install Customized Argo CD

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Installs a customized Argo CD instance excluding the application controller, suitable for control plane clusters. Server-side apply is required due to manifest size. Replace `<release-branch>` with the desired release tag.

```bash
# Apply the principal-specific Argo CD configuration
kubectl apply -n argocd \
  --server-side  \
  -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/principal?ref=<release-branch>' \
  --context <control-plane-context>
```

--------------------------------

### AppProject Configuration for Destination-Based Mapping

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/concepts/agent-mapping.md

Example AppProject manifest on the agent, configured to allow applications from any namespace, which is required for destination-based mapping.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  sourceNamespaces:
    - <namespaces>  # Required for destination-based mapping
  # ... other configuration
```

--------------------------------

### Principal ConfigMap Configuration

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/index.md

Example of a ConfigMap for configuring the principal component, including listen host/port, log level, namespace, and TLS settings.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-agent-params
  namespace: argocd
data:
  principal.listen.host: "0.0.0.0"
  principal.listen.port: "8443"
  principal.log.level: "info"
  principal.namespace: "argocd"
  principal.metrics.port: "8000"
  principal.healthz.port: "8003"
  principal.tls.secret-name: "argocd-agent-principal-tls"
  principal.tls.server.root-ca-secret-name: "argocd-agent-ca"
  principal.tls.client-cert.require: "true"
  principal.auth: "mtls:CN=([^,]+)"
```

--------------------------------

### Minimal Principal Configuration

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/index.md

Provides the essential ConfigMap entries for a minimal principal setup, focusing on namespace, authentication, and client certificate requirements.

```yaml
# ConfigMap
principal.namespace: "argocd"
principal.auth: "mtls:CN=([^,]+)"
principal.tls.client-cert.require: "true"
```

--------------------------------

### Call BaseSuite's SetupSuite Method

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/test/e2e/README.md

If defining a custom SetupSuite method, ensure you call the BaseSuite's method first to leverage its automatic setup.

```go
func (suite *MyTestSuite) SetupSuite() {
	suite.BaseSuite.SetupSuite()
    ...
}
```

--------------------------------

### Example Namespace Organization for Managed Agents

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/applications.md

Use clear, descriptive namespace names that match your agent names for better organization in managed mode.

```plaintext
   production-east
   production-west
   staging-cluster
   development-cluster
```

--------------------------------

### List Argo CD Clusters

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/migration.md

Use this command to list all clusters currently configured in your Argo CD installation. This is a key step in assessing your current setup.

```bash
argocd cluster list
```

--------------------------------

### Generate Agent Credentials

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

Run this script to generate necessary credentials for the agents before starting any agent or principal components. Ensure the script has execute permissions.

```shell
chmod +x ./hack/dev-env/gen-creds.sh
./hack/dev-env/gen-creds.sh
```

--------------------------------

### Initialize PKI and Issue Certificates with argocd-agentctl

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/adding-agents.md

Use `argocd-agentctl` to initialize the Certificate Authority, issue principal server and resource proxy certificates, and create a JWT signing key. Ensure you replace placeholder values with your actual configuration details.

```bash
argocd-agentctl pki init \
  --principal-context <control-plane-context> \
  --principal-namespace argocd

argocd-agentctl pki issue principal \
  --principal-context <control-plane-context> \
  --ip <principal-ip-addresses> \
  --dns <principal-dns-names> \
  --upsert

argocd-agentctl pki issue resource-proxy \
  --principal-context <control-plane-context> \
  --ip <resource-proxy-ip-addresses> \
  --dns <resource-proxy-dns-names> \
  --upsert

argocd-agentctl jwt create-key \
  --principal-context <control-plane-context> \
  --principal-namespace argocd \
  --upsert
```

--------------------------------

### Import Logging Packages

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/contributing/logging.md

Import the necessary logging packages at the beginning of your Go file.

```go
import (
    "github.com/argoproj-labs/argocd-agent/internal/logging"
    "github.com/argoproj-labs/argocd-agent/internal/logging/logfields"
)
```

--------------------------------

### Create Virtual Clusters with Specific Argo CD Version

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/hack/dev-env/README.md

To install a specific Argo CD version, set the ARGOCD_VERSION environment variable before running the setup script. This can be 'stable', 'latest', or a specific version like 'v3.2.2'.

```shell
ARGOCD_VERSION=latest ./hack/dev-env/setup-vcluster-env.sh create
```

--------------------------------

### HA Configuration Flags for Primary Region

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/proposals/ha-failover.md

Example configuration flags for setting up the preferred primary region in a high availability setup. Includes enabling HA, setting preferred role, and specifying the peer address and allowed clients.

```text
--ha-enabled
--ha-preferred-role=primary
--ha-peer-address=principal.region-b.internal:8443

# Peer identity allowlist (replication uses the server's --auth method)
--ha-allowed-replication-clients=region-b
```

--------------------------------

### Configure Agent Namespaces with Wildcards

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

Use wildcards in ConfigMaps to manage agent namespaces. This example sets up both Argo CD and the principal to recognize namespaces starting with 'agent-*' and enables automatic creation of new agent namespaces matching a specific pattern.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  application.namespaces: "agent-*"

---
# argocd-agent-params - Principal
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-agent-params
  namespace: argocd
data:
  principal.allowed-namespaces: "agent-*"
  principal.namespace-create.enable: "true"
  principal.namespace-create.pattern: "^agent-[a-z0-9-]+$"
  principal.namespace-create.labels: "managed-by=argocd-agent"
```

--------------------------------

### Initialize PKI

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Initializes the Public Key Infrastructure for the Argo CD Agent principal. Ensure you replace `<control-plane-context>` and `<principal-namespace>` with your specific values.

```bash
# Initialize the PKI
argocd-agentctl pki init \
  --principal-context <control-plane-context> \
  --principal-namespace argocd
```

--------------------------------

### JSON Log Example

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/observability.md

An example of a log message formatted in JSON, suitable for structured logging and aggregation.

```json
{"level":"info","ts":"2024-01-15T10:30:45.123Z","msg":"Agent connected","agent":"production-cluster"}
```

--------------------------------

### Correct Logging Approach with Context

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/contributing/logging.md

Use specialized loggers and include relevant fields like method, duration, and status for detailed debugging.

```go
logger := logging.GetDefaultLogger().ApplicationLogger("AppProcessor", appName)
logger.WithFields(logrus.Fields{
    logfields.Method:    "ProcessApp",
    logfields.Duration:  duration,
    logfields.Status:    "completed",
}).Info("Application processing completed")
```

--------------------------------

### Text Log Example

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/observability.md

An example of a log message formatted in human-readable text, useful for development and manual inspection.

```text
INFO[0001] Agent connected                               agent=production-cluster
```

--------------------------------

### Troubleshoot Agent Creation: Inspect PKI

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/adding-agents.md

Verify that the Public Key Infrastructure (PKI) is properly initialized by inspecting it with `argocd-agentctl pki inspect`.

```bash
argocd-agentctl pki inspect
```

--------------------------------

### Create Test Application on Principal

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Create a sample application named 'test-app' within the 'my-first-agent' namespace on the principal. This application will be used to test synchronization to the agent.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: my-first-agent
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://<principal-external-ip:port>?agentName=my-first-agent # For example, https://172.18.255.200:443?agentName=my-first-agent
    namespace: guestbook
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
```

--------------------------------

### Initialize PKI for Argo CD Agent Principal

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/openshift/index.md

Use this command to initialize the Public Key Infrastructure (PKI) for the Argo CD-agent principal in non-production scenarios. Be aware that this method stores sensitive PKI components unprotected.

```bash
argocd-agentctl --principal-context <control plane context> pki init
```

--------------------------------

### Verify Agent Certificate Installation

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Verifies the installation of agent client certificates by checking the secrets `argocd-agent-client-tls` and `argocd-agent-ca` in the specified namespace and context.

```bash
kubectl get secret argocd-agent-client-tls -n $NAMESPACE_NAME --context kind-$AGENT_CLUSTER_NAME

# Expected result
# NAME                      TYPE                DATA   AGE
# argocd-agent-client-tls   kubernetes.io/tls   2      7s

kubectl get secret argocd-agent-ca -n $NAMESPACE_NAME --context kind-$AGENT_CLUSTER_NAME

# Expected result
# NAME              TYPE     DATA   AGE
# argocd-agent-ca   Opaque   1      15s
```

--------------------------------

### Istio Principal Deployment Example

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/service-mesh.md

A complete Kubernetes Deployment manifest for the Argo CD Agent principal when integrated with Istio. This includes setting the image, arguments for principal configuration, and enabling Istio sidecar injection.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-agent-principal
  namespace: argocd
spec:
  template:
    metadata:
      labels:
        app: argocd-agent-principal
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: principal
        image: ghcr.io/argoproj-labs/argocd-agent:latest
        args:
        - principal
        - --listen-host=127.0.0.1
        - --insecure-plaintext=true
        - --auth=header:x-forwarded-client-cert:^.*URI=spiffe://[^/]+/ns/[^/]+/sa/([^,;]+)
        ports:
        - containerPort: 8443
```

--------------------------------

### Verify Certificate Installation with kubectl

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Verify that the agent client certificate and CA certificate secrets have been successfully installed in the agent's namespace.

```bash
# Verify the client certificate secret exists
kubectl get secret argocd-agent-client-tls -n argocd --context <workload-cluster-context>

# Verify the CA certificate secret exists
kubectl get secret argocd-agent-ca -n argocd --context <workload-cluster-context>
```

--------------------------------

### Enable Full-Detail Logging on Agent via Command Line

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/observability.md

Enable detailed logging for all categories on the agent component using the command-line flag `--full-detail=all`.

```bash
argocd-agent agent --full-detail=all
```

--------------------------------

### Helm Install with --set for Individual Values

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/install/helm-repo/docs/install-agent.md

Override specific configuration values during Helm installation using the --set flag.

```bash
helm install argocd-agent ghcr.io/argoproj-labs/argocd-agent/argocd-agent-agent-helm --version 0.1.0 \
  --set logLevel="debug" \
  --set agentMode="managed" \
  --set server="https://my-argocd-server.com"
```

--------------------------------

### Install ArgoCD Agent Helm Chart

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/install/helm-repo/docs/install-agent.md

Basic command to install the argocd-agent-agent-helm chart from GHCR. Ensure Helm CLI and Kubernetes cluster are set up.

```bash
helm install argocd-agent ghcr.io/argoproj-labs/argocd-agent/argocd-agent-agent-helm --version 0.1.0
```

--------------------------------

### Configure Default Logger

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/internal/logging/README.md

Configure the default logger at application startup with specified log level and format. Uses stdout by default.

```go
import "github.com/argoproj-labs/argocd-agent/internal/logging"

// Configure logging at application startup
err := logging.GetDefaultLogger().SetupLogging(
    logging.LogLevelInfo,
    logging.LogFormatJSON,
    nil, // Use default output (stdout)
)
if err != nil {
    panic(err)
}
```

--------------------------------

### Example Project Management for Autonomous Agents

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/applications.md

Be mindful of project names in autonomous agents, as they may be prefixed on the principal. For example, 'microservices' might become 'agent-name-microservices'.

```yaml
   spec:
     project: microservices  # Becomes "agent-name-microservices" on principal
```

--------------------------------

### Install Argo CD for Managed Agents

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Installs Argo CD components on the workload cluster for managed agent mode. Replace '<release-branch>' with the desired release branch.

```bash
# For managed agents
kubectl apply -n argocd \
  --server-side \
  -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/agent-managed?ref=<release-branch>' \
  --context <workload-cluster-context>
```

--------------------------------

### List Argo CD Applications

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/hybrid-architecture.md

Use the `argocd app list` command to verify that traditional applications on the hub cluster remain unaffected and are still in a Synced/Healthy state after the hybrid setup.

```bash
# Traditional apps should still be Synced/Healthy
argocd app list

```

--------------------------------

### Application-Specific Resource Permissions Examples

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/live-resources.md

Examples of additional RBAC permissions required for applications that deploy custom resources or use specific APIs, such as cert-manager, Istio, or monitoring stacks.

```yaml
# Example: For applications using cert-manager
- apiGroups: ["cert-manager.io"]
  resources: ["certificates", "issuers", "clusterissuers"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# Example: For applications using Istio
- apiGroups: ["networking.istio.io"]
  resources: ["virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# Example: For applications using monitoring stack
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors", "prometheusrules"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
```

--------------------------------

### Multi-Tenant Setup: Pattern-Based Namespace Restrictions

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

Configure multi-tenant environments to allow team-specific agent namespaces. This setup uses patterns to restrict namespace creation to a defined convention, such as 'team-a-agent-*'.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-agent-params
  namespace: argocd
data:
  # Allow team-specific patterns
  principal.allowed-namespaces: "team-a-*,team-b-*"
  principal.namespace-create.enable: "true"
  # Only allow namespaces matching team naming convention
  principal.namespace-create.pattern: "^team-[a-z]+-agent-[a-z0-9-]+$"
```

--------------------------------

### Verify Application Reference to AppProjects

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/migration.md

Use Argo CD CLI commands to list applications and get application details, ensuring they can correctly reference the newly configured AppProjects.

```bash
# Verify Applications can find their AppProjects
argocd app list --context <control-plane-context>
argocd app get <app-name> --context <control-plane-context>
```

--------------------------------

### Create and Deploy Test Application

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

This snippet retrieves the principal's external IP and NodePort, then applies a sample ArgoCD Application manifest targeting the agent. Ensure the agent's destination server is correctly configured.

```bash
PRINCIPAL_EXTERNAL_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' argocd-hub-control-plane)
PRINCIPAL_NODE_PORT=$(kubectl get svc argocd-agent-principal -n $NAMESPACE_NAME --context kind-$PRINCIPAL_CLUSTER_NAME -o jsonpath='{.spec.ports[0].nodePort}')
cat <<EOF | kubectl apply -f - --context kind-$PRINCIPAL_CLUSTER_NAME
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: $AGENT_APP_NAME
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://$PRINCIPAL_EXTERNAL_IP:$PRINCIPAL_NODE_PORT?agentName=$AGENT_APP_NAME
    namespace: guestbook
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
EOF
```

--------------------------------

### Production Setup: Explicit Allowed Agent Namespaces

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

For production, use an explicit list of allowed agent namespaces in the principal's configuration. Auto-creation is typically disabled by default in this restrictive setup.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-agent-params
  namespace: argocd
data:
  principal.allowed-namespaces: "agent-prod-us,agent-prod-eu,agent-prod-asia"
  # Auto-creation disabled (default)
```

--------------------------------

### Enable Apps in Any Namespace

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Configures Argo CD to allow applications in any namespace and restarts the server deployment to apply the changes. Replace `<control-plane-context>` with your cluster context.

```bash
kubectl patch configmap argocd-cmd-params-cm -n argocd --context <control-plane-context> \
  --patch '{"data":{"application.namespaces":"*"}}'

# Restart the server to apply changes
kubectl rollout restart deployment argocd-server -n argocd --context <control-plane-context>
```

--------------------------------

### Create Test Application for Synchronization

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Defines and applies a sample Argo CD Application manifest to the principal. This application will be managed and synchronized by the agent.

```bash
# Check NodePort
PRINCIPAL_EXTERNAL_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' argocd-hub-control-plane)
echo "<principal-external-ip>: $PRINCIPAL_EXTERNAL_IP"

PRINCIPAL_NODE_PORT=$(kubectl get svc argocd-agent-principal -n $NAMESPACE_NAME --context kind-$PRINCIPAL_CLUSTER_NAME -o jsonpath='{.spec.ports[0].nodePort}'
)
echo "<principal-node-port>: $PRINCIPAL_NODE_PORT"

cat <<EOF | kubectl apply -f - --context kind-$PRINCIPAL_CLUSTER_NAME
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: $AGENT_APP_NAME
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
    path: guestbook
  destination:
    server: "https://$PRINCIPAL_EXTERNAL_IP:$PRINCIPAL_NODE_PORT?agentName=$AGENT_APP_NAME"
    namespace: guestbook
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
EOF
```

--------------------------------

### Generate Shared Root CA and Cluster Certificates

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

Generates a shared root CA and cluster-specific CA certificates required for Istio installations across multiple clusters. This must be performed before installing Istio.

```bash
mkdir -p /tmp/istio-certs && cd /tmp/istio-certs
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk root-ca
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f $ISTIO_DIR/tools/certs/Makefile.selfsigned.mk cluster2-cacerts
```

--------------------------------

### Verify Principal Installation

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Checks the status and logs of the Argo CD Agent principal deployment to ensure it has been installed and is running correctly. This includes verifying pod status and examining logs for expected output.

```bash
kubectl get pods -n $NAMESPACE_NAME --context kind-$PRINCIPAL_CLUSTER_NAME | grep principal

kubectl logs -n $NAMESPACE_NAME deployment/argocd-agent-principal --context kind-$PRINCIPAL_CLUSTER_NAME

# Expected logs:
# argocd-agent-principal-785cd96ddc-sm44r            1/1     Running   0          7s
# {"level":"info","msg":"Setting loglevel to info","time":"2025-09-13T07:25:38Z"}
# time="2025-09-13T07:25:38Z" level=info msg="Loading gRPC TLS certificate from secret argocd/argocd-agent-principal-tls" - Loading TLS certificate for gRPC communication from Kubernetes Secret
# ...
# time="2025-09-13T07:25:38Z" level=info msg="This server will require TLS client certs as part of authentication" module=server - TLS client certificates required for client authentication
# ...
# time="2025-09-13T07:25:38Z" level=info msg="Starting argocd-agent (server) v0.0.1-alpha (ns=$NAMESPACE_NAME, allowed_namespaces=[])" module=server - Starting argocd-agent server (version, namespace, allowed namespaces list)
# ...
# time="2025-09-13T07:25:38Z" level=info msg="Now listening on [::]:8443" module=server
# time="2025-09-13T07:25:38Z" level=info msg="Application informer synced and ready" module=server
# time="2025-09-13T07:25:38Z" level=info msg="AppProject informer synced and ready" module=server
# time="2025-09-13T07:25:38Z" level=info msg="Repository informer synced and ready" module=server
```

--------------------------------

### Set Preferred Role for Principal

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/ha.md

Configure the preferred role for a principal on startup using `--ha-preferred-role`. Use `primary` to start in ACTIVE or `replica` to start in SYNCING. This can also be set via `ARGOCD_PRINCIPAL_HA_PREFERRED_ROLE`.

```bash
argocd-agent principal \
  --ha-preferred-role=primary
```

```bash
argocd-agent principal \
  --ha-preferred-role=replica
```

--------------------------------

### Apply Argo CD Agent Installation Manifests

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/openshift/index.md

Apply the Kubernetes manifests for the Argo CD Agent. Change `<release-branch>` to the desired release version. This command installs the agent resources into the specified workload namespace.

```bash
oc apply -n $(workload-namespace) -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/agent?ref=<release-branch>'
```

```bash
oc apply -n argocd -k 'https://github.com/argoproj-labs/argocd-agent/install/kubernetes/agent?ref=<release-branch>'
```

--------------------------------

### Bulk Agent Creation Script

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/adding-agents.md

Use this bash script to automate the creation of multiple agents, including their configurations and necessary Kubernetes resources.

```bash
#!/bin/bash
AGENTS=("prod-east" "prod-west" "staging" "dev")

for agent in "${AGENTS[@]}"; do
  echo "Creating agent: $agent"
  
  # Create agent configuration
  argocd-agentctl agent create "$agent" \
    --resource-proxy-server <resource-proxy-service-name>:9090

  # Issue client certificate
  argocd-agentctl pki issue agent "$agent" \
    --agent-context "cluster-$agent" \
    --agent-namespace argocd \
    --upsert
  
  # Create namespace for managed agents
  kubectl create namespace "$agent" --context control-plane || true
  
  echo "Agent $agent created successfully"
done
```

--------------------------------

### Install Argo CD Agent with Server-Side Apply

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/istio/index.md

Installs the Argo CD agent components on the workload cluster using server-side apply to avoid annotation size limits. This command deploys the application controller, repo server, and redis.

```bash
kubectl apply -n $NAMESPACE_NAME --server-side \
  -k "https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/agent-$AGENT_MODE?ref=$RELEASE_BRANCH" \
  --context kind-$AGENT_CLUSTER_NAME
```

--------------------------------

### Create Agent Configuration

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/index.md

Use this command to create a new agent configuration on the principal. Ensure you replace placeholders with your actual control plane context and IP addresses.

```bash
# Create agent configuration on the principal
argocd-agentctl agent create my-first-agent \
  --principal-context <control-plane-context> \
  --principal-namespace argocd \
  --resource-proxy-server <principal-external-ip>:9090 \
  --resource-proxy-username my-first-agent \
  --resource-proxy-password "$(openssl rand -base64 32)"
```

--------------------------------

### Install Argo CD for Workload Cluster

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/getting-started/kubernetes/kind/index.md

Installs Argo CD components, including the application controller, repo server, and redis, onto the workload cluster using server-side apply. Server-side apply is required due to the size of the Argo CD manifest.

```bash
kubectl apply -n $NAMESPACE_NAME --server-side 
  -k "https://github.com/argoproj-labs/argocd-agent/install/kubernetes/argo-cd/agent-$AGENT_MODE?ref=$RELEASE_BRANCH" 
  --context kind-$AGENT_CLUSTER_NAME
```

--------------------------------

### Check AppProject Configuration

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/repository.md

Verify the configuration of a specific AppProject using kubectl. Ensure it exists and has correct patterns for agent synchronization.

```bash
# Check specific AppProject configuration
kubectl get appproject my-project -n argocd -o yaml
```

--------------------------------

### Configure Allowed Namespaces (Multiple Patterns)

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/configuration/namespaces.md

Use this command to specify multiple wildcard patterns for allowed namespaces.

```bash
argocd-agent principal --allowed-namespaces='agent-*,cluster-*'
```

--------------------------------

### Successful Agent Connection Log Messages

Source: https://github.com/argoproj-labs/argocd-agent/blob/main/docs/user-guide/adding-agents.md

Example log messages indicating a successful agent connection to the principal.

```text
INFO[0001] Starting argocd-agent (agent) v0.1.0 (ns=argocd, mode=autonomous, auth=userpass)
INFO[0002] Authentication successful
INFO[0003] Connected to argocd-agent-principal v0.1.0
```