### Install EgressGateway

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Aliyun.en.md

Install the EgressGateway chart using Helm.

```shell
helm install egress --wait --debug egressgateway/egressgateway
```

--------------------------------

### Install ipset on Linux

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

Install the ipset utility on CentOS, RHEL, Rocky Linux, or AlmaLinux.

```bash
yum install ipset
```

--------------------------------

### Install ipset tool

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

If the 'ipset' tool is not installed, use the appropriate package manager command for your distribution. This tool is needed to check ipset rules on the host.

```bash
# Ubuntu/Debian
apt-get install ipset
```

--------------------------------

### Start test server

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Aliyun.en.md

Run a nettools server to verify egress IP functionality.

```shell
docker run -d --net=host ghcr.io/spidernet-io/egressgateway-nettools:latest /usr/bin/nettools-server -protocol web -webPort 8080
```

--------------------------------

### Install EgressGateway via Helm

Source: https://github.com/spidernet-io/egressgateway/blob/main/charts/README.md

Commands to add the EgressGateway Helm repository and install the chart into the kube-system namespace.

```shell
helm repo add egressgateway https://spidernet-io.github.io/egressgateway/
helm install egressgateway egressgateway/egressgateway --namespace kube-system
```

--------------------------------

### Install EgressGateway via Helm

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Commands to add the repository and install EgressGateway with basic or high-availability configurations.

```bash
# Add the EgressGateway Helm repository
helm repo add egressgateway https://spidernet-io.github.io/egressgateway/
helm repo update

# Install EgressGateway with basic configuration
helm install egressgateway egressgateway/egressgateway \
  -n kube-system \
  --set feature.tunnelIpv4Subnet="192.200.0.1/16" \
  --wait --debug

# Install with IPv6 support and high availability
helm install egressgateway egressgateway/egressgateway \
  -n kube-system \
  --set feature.enableIPv4=true \
  --set feature.enableIPv6=true \
  --set feature.tunnelIpv4Subnet="192.200.0.1/16" \
  --set feature.tunnelIpv6Subnet="fd11::/112" \
  --set controller.replicas=2 \
  --wait --debug

# Verify installation
kubectl get pod -n kube-system | grep egressgateway
# Output:
# egressgateway-agent-29lt5                  1/1     Running   0          9h
# egressgateway-agent-94n8k                  1/1     Running   0          9h
# egressgateway-controller-5754f6658-7pn4z   1/1     Running   0          9h
```

--------------------------------

### Install EgressGateway Helm Chart

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.en.md

Install the EgressGateway Helm chart, enabling IPv4, disabling IPv6, and optionally specifying an extra cluster CIDR for EgressPolicy behavior.

```shell
helm install egress --wait \
  --debug egressgateway/egressgateway \
  --set feature.enableIPv4=true \
  --set feature.enableIPv6=false \
  --set feature.clusterCIDR.extraCidr[0]=172.16.0.0/16
```

--------------------------------

### Setup E2E Cluster

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Develop.en.md

Initializes the end-to-end testing cluster. For developers in China, it supports using a China image registry and an HTTP proxy for chart pulls.

```bash
# setup the cluster
make e2e_init
```

```bash
# for china developer, use china image registry, use HTTP_PROXY to pull chart 
make e2e_init -e E2E_CHINA_IMAGE_REGISTRY=true -e HTTP_PROXY=http://10.0.0.1:7890
```

--------------------------------

### Install EgressGateway Helm Chart

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.zh.md

Install the EgressGateway Helm chart with specific configurations for IPv4 enablement, IPv6 disablement, and setting an extra cluster CIDR. The `feature.clusterCIDR.extraCidr` parameter influences EgressPolicy behavior for traffic not matching `spec.destSubnet`.

```shell
helm install egress --wait \
 --debug egressgateway/egressgateway \
 --set feature.enableIPv4=true \
 --set feature.enableIPv6=false \
 --set feature.clusterCIDR.extraCidr[0]=172.16.0.0/16
```

--------------------------------

### Enable Prometheus Metrics

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Helm installation command to enable Prometheus metrics for both the controller and agent components.

```bash
helm install egressgateway egressgateway/egressgateway \
  -n kube-system \
  --set feature.tunnelIpv4Subnet="192.200.0.1/16" \
  --set agent.prometheus.enabled=true \
  --set agent.prometheus.port=5811 \
  --set agent.prometheus.serviceMonitor.install=true \
  --set controller.prometheus.enabled=true \
  --set controller.prometheus.port=5821 \
  --set controller.prometheus.serviceMonitor.install=true
```

--------------------------------

### Install EgressGateway using Helm

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Install.en.md

Installs EgressGateway on a Kubernetes cluster using Helm. Key parameters include `feature.tunnelIpv4Subnet`, `feature.tunnelDetectMethod`, `feature.enableIPv6`, `feature.tunnelIpv6Subnet`, `controller.replicas`, and `feature.enableGatewayReplyRoute`. Ensure tunnel subnets do not conflict with existing cluster addresses.

```shell
helm install egressgateway egressgateway/egressgateway \
	  -n kube-system \
		--set feature.tunnelIpv4Subnet="192.200.0.1/16" \
		--wait --debug
```

--------------------------------

### Example iptables PREROUTING rules

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

This shows the expected output for iptables 'PREROUTING' rules when EgressGateway is configured correctly, indicating that EgressGateway rules are present and traffic is being marked.

```bash
$ iptables -t mangle -nvL PREROUTING

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target                      prot opt in     out     source        destination
    6    6K EGRESSGATEWAY-MARK-REQUEST  0    --  *      *       0.0.0.0/0     0.0.0.0/0     /* egw:Lh98b3mb9WlZrgw7 */ /* Checking for EgressPolicy matched traffic */
    0     0 ACCEPT                      0    --  *      *       0.0.0.0/0     0.0.0.0/0     /* egw:4vaggeYl6c-Gn0Yv */ /* EgressGateway traffic accept datapath rule */ mark match 0x26000000/0xff000000
```

--------------------------------

### Configure CNI Plugins

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Specific configuration patches and installation flags required for Calico, Spiderpool, and Cilium CNI integration.

```bash
kubectl patch felixconfigurations default --type='merge' \
  -p '{"spec":{"chainInsertMode":"Append"}}'
```

```bash
kubectl get FelixConfiguration default -o yaml
```

```bash
kubectl patch spidercoordinators default --type='merge' \
  -p '{"spec": {"hijackCIDR": ["1.1.1.1/32", "2.2.2.2/32"]}}'
```

```bash
cilium install --wait \
  --set enable-ipv4=true \
  --set ipv4NativeRoutingCIDR=$IPV4_POD_CIDR \
  --set autoDirectNodeRoutes=true \
  --set routingMode="native" \
  --set bpf.masquerade=false
```

--------------------------------

### Example EgressPolicy YAML

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Uninstall.en.md

A sample EgressPolicy resource definition.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  name: ns-policy
  namespace: default
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: mock-app
  egressGatewayName: egressgateway
status:
  eip:
    ipv4: 10.6.1.55
    ipv6: fd00::55
  node: workstation2
```

--------------------------------

### Query installed EgressGateway

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Uninstall.en.md

List all Helm releases related to EgressGateway.

```shell
helm ls -A | grep -i egress
```

--------------------------------

### Install Cilium with Native Routing

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Install.en.md

Installs Cilium in `Native` routing mode, which is required for EgressGateway compatibility. Ensure `enable-ipv4`, `ipv4NativeRoutingCIDR`, `autoDirectNodeRoutes`, `routingMode`, and `bpf.masquerade` are set appropriately for your environment.

```shell
cilium install --wait --set enable-ipv4=true \
    --set ipv4NativeRoutingCIDR=$IPV4_POD_CIDR \
    --set autoDirectNodeRoutes=true \
    --set routingMode="native" \
    --set bpf.masquerade=false
```

--------------------------------

### Define an EgressPolicy resource

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/reference/EgressPolicy.en.md

Example configuration for an EgressPolicy, specifying the gateway, egress IP settings, target Pods, and destination subnets.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  namespace: "default"
  name: "policy-test"
spec:
  egressGatewayName: "eg1" 
  egressIP:
    ipv4: ""                            
    ipv6: ""
    useNodeIP: false       
  appliedTo:                
    podSelector:
      matchLabels:
        app: "shopping"
    podSubnet:              
    - "172.29.16.0/24"
    - 'fd00:1/126'
  destSubnet:
    - "10.6.1.92/32"
    - "fd00::92/128"
  priority: 100
```

--------------------------------

### Get Specific Egress Policy Details

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

Fetch the detailed YAML configuration for a specific Egress Policy. Review 'egressGatewayName', 'appliedTo.podSelector', 'destSubnet', and 'egressIP' for correctness.

```bash
$ kubectl get egresspolicy test -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  generation: 1
  name: test
  namespace: default
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: visitor
  destSubnet:
  - 172.16.25.183/32
  egressGatewayName: default
  egressIP:
    allocatorPolicy: default
    useNodeIP: false
status:
  eip:
    ipv4: 172.16.25.189
  node: node2
```

--------------------------------

### Define an EgressClusterPolicy resource

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/reference/EgressClusterPolicy.en.md

Example YAML configuration for an EgressClusterPolicy, including pod and namespace selectors and destination subnet definitions.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterPolicy
metadata:
  name: "policy-test"
spec:
  priority: 100
  egressGatewayName: "eg1"
  egressIP:
    ipv4: ""
    ipv6: ""
    useNodeIP: false
  appliedTo:
    podSelector:
      matchLabels:
        app: "shopping"
    podSubnet:
      - "172.29.16.0/24"
      - 'fd00:1/126'
    namespaceSelector:   # (1)
      matchLabels:
        app: "shopping"
  destSubnet:
    - "10.6.1.92/32"
    - "fd00::92/128"
```

--------------------------------

### Define an EgressGateway resource

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/reference/EgressGateway.en.md

Example YAML configuration for an EgressGateway resource, including IP pool definitions and node selection criteria.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: "eg1"
spec:
  ippools:                     
    ipv4:                       
      - "10.6.1.55"
      - "10.6.1.60-10.6.1.65"
      - "10.6.1.70/28"
    ipv6:                      
      - ""
    ipv4DefaultEIP: ""
    ipv6DefaultEIP: ""
  nodeSelector:
    selector:
      matchLabels:
        egress: "true"
    policy: "doing"
status:
  nodeList:
    - name: "node1"
      status: "Ready"
      epis:
        - ipv4: "10.6.1.55"
          ipv6: "fd00::55"
          policies:
            - name: "app"
              namespace: "default"
```

--------------------------------

### Create and Configure Dummy NIC for Egress Gateway

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/03-egress-ip/README.en.md

Creates a dummy network interface named 'egress.eip' and brings it up. This is a prerequisite for setting up EIPs on the node.

```shell
ip link add egress.eip type dummy
ip link set egress.eip up
```

--------------------------------

### Configure IP Sets for Policy Matching

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/03-egress-ip/README.en.md

Creates and populates ipsets for source and destination IP matching.

```shell
IPSET_RULE_DEST_NAME=egress-dest-uuid
ipset x $IPSET_RULE_DEST_NAME
ipset create $IPSET_RULE_DEST_NAME hash:net
ipset add $IPSET_RULE_DEST_NAME 10.6.105.150/32

IPSET_RULE_SRC_NAME=egress-src-uuid
ipset x $IPSET_RULE_SRC_NAME
ipset create $IPSET_RULE_SRC_NAME hash:net
ipset add $IPSET_RULE_SRC_NAME 172.29.234.173/32
```

--------------------------------

### Deploy Test Application

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Install.en.md

Creates a deployment named visitor using the nginx image.

```shell
kubectl create deployment visitor --image nginx
```

--------------------------------

### Configure Policy-Matched IP Sets

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/concepts/Datapath.en.md

Creates ipsets for tracking source and destination IPs matching the egress policy.

```shell
IPSET_RULE_DEST_NAME=egress-dest-uuid

ipset x $IPSET_RULE_DEST_NAME

ipset create $IPSET_RULE_DEST_NAME hash:net

ipset add $IPSET_RULE_DEST_NAME 10.6.105.150/32


IPSET_RULE_SRC_NAME=egress-src-uuid

ipset x $IPSET_RULE_SRC_NAME

ipset create $IPSET_RULE_SRC_NAME hash:net

ipset add $IPSET_RULE_SRC_NAME 172.29.234.173/32
```

--------------------------------

### Configure IPSet for Policy

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/01-egress-gateway/EgressGateway.md

Commands to create and populate an IPSet for destination policy matching.

```bash
sudo ipset create egress-dst-policy-name
sudo ipset add egress-dest-policy-name 172.16.1.1/32
```

--------------------------------

### Clone Repository

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Contribute.en.md

Clone your forked repository to your local machine to begin development.

```shell
git clone https://github.com/your-username/your-project-name.git
```

--------------------------------

### Configure Network Interface Offloading

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Troubleshooting.en.md

Commands to enable or verify network interface offloading settings to improve VXLAN throughput.

```bash
ethtool --offload host-interface-name rx on tx on
```

```bash
ethtool -k egress.vxlan
```

--------------------------------

### EgressEndpointSlice Resource Definition

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Example of an EgressEndpointSlice resource showing the mapping of Pods to nodes.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressEndpointSlice
metadata:
  name: visitor-policy-kvlp6
  namespace: default
endpoints:
  - ipv4:
      - "172.40.14.195"
    node: worker-1
    ns: default
    pod: visitor-6764bb48cc-29vq9
  - ipv4:
      - "172.40.14.196"
    node: worker-2
    ns: default
    pod: visitor-6764bb48cc-x8k2m
```

--------------------------------

### Basic Helm Upgrade Command Format

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Upgrade.en.md

This is the general syntax for performing a Helm upgrade. Replace [RELEASE] with the application name, [CHART] with the chart reference, and [flags] with any necessary options.

```shell
helm upgrade [RELEASE] [CHART] [flags]
```

--------------------------------

### Controller Implementation - Initialization

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/02-egress-node/EgressTunnel.en.md

Details the initialization process of the EgressTunnel controller.

```APIDOC
## Controller Initialization

### Description
Handles the initial setup and creation of EgressTunnel resources.

1. Get IPv4, IPv6 and corresponding CIDR from CM.
2. Check if the node has a corresponding EgressTunnel. If not, create one with status set to "pending".
3. If a tunnel IP already exists, bind the IP to the node after checking its legality. If illegal, set the status to "Pending".
```

--------------------------------

### Define EgressGateway CRD

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Example configuration for an EgressGateway resource defining IP pools and node selection criteria.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: default
spec:
  # Set this gateway as the cluster default (optional)
  clusterDefault: true
  ippools:
    # IPv4 egress IP pool - supports single IPs, ranges, and CIDR notation
    ipv4:
      - "10.6.1.55"
      - "10.6.1.60-10.6.1.65"
      - "10.6.1.70/28"
    # IPv6 egress IP pool
    ipv6:
      - "fd00::55"
      - "fd00::56"
    # Default IPv4 EIP used when EgressPolicy doesn't specify one
    ipv4DefaultEIP: "10.6.1.55"
    # Default IPv6 EIP
    ipv6DefaultEIP: "fd00::55"
  nodeSelector:
    selector:
      matchLabels:
        egress: "true"
# Status is populated by the controller
status:
  nodeList:
    - name: node1
      status: Ready
      eips:
        - ipv4: "10.6.1.55"
          ipv6: "fd00::55"
          policies:
            - name: policy1
              namespace: default
    - name: node2
      status: Ready
  ipUsage:
    ipv4Total: 20
    ipv4Free: 18
    ipv6Total: 2
    ipv6Free: 1
```

--------------------------------

### Check Pod Status

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.en.md

Verifies that all pods, including the newly created 'mock-app', are in the 'Running' state and displays their assigned IP and node.

```shell
~ kubectl get pods -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP               NODE                             NOMINATED NODE   READINESS GATES
egressgateway-agent-zw426                   1/1     Running   0          15m   172.16.103.117   ip-172-16-103-117.ec2.internal   <none>           <none>
egressgateway-agent-zw728                   1/1     Running   0          15m   172.16.61.234    ip-172-16-61-234.ec2.internal    <none>           <none>
egressgateway-controller-6cc84c6985-9gbgd   1/1     Running   0          15m   172.16.51.178    ip-172-16-61-234.ec2.internal    <none>           <none>
mock-app                                    1/1     Running   0          12m   172.16.51.74     ip-172-16-61-234.ec2.internal    <none>           <none>
```

--------------------------------

### Verify Egress Source ipset Rules

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

List and inspect the egress-src ipset rules to verify if the Pod's IP address is included.

```bash
$ ipset list | grep egress-src-
Name: egress-src-v4-738bb014438bdbfe7

$ ipset list egress-src-v4-738bb014438bdbfe7
Name: egress-src-v4-738bb014438bdbfe7
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0x498b6df2
Size in memory: 504
References: 1
Number of entries: 1
Members:
10.200.0.22
```

--------------------------------

### Add Helm Repository

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/release.md

Command to add the Helm repository for retrieving packaged charts.

```bash
helm repo add $REPO_NAME https://kdoctor-io.github.io/$REPO_NAME
```

--------------------------------

### Define EgressClusterInfo CRD

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/reference/EgressClusterInfo.en.md

Example configuration for the EgressClusterInfo resource, demonstrating auto-detection settings for cluster IP, node IP, and pod CIDR.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterInfo
metadata:
  name: default  # (1)
spec:
  autoDetect:
    clusterIP: true # (2)
    nodeIP: true # (3)
    podCidrMode: auto # (4)
  extraCidr: # (5)
  - 10.10.10.1
status:
  clusterIP: # (6)
    ipv4:
    - 172.41.0.0/16
    ipv6:
    - fd41::/108
  extraCidr: # (7)
  - 10.10.10.1
  nodeIP: # (8)
    egressgateway-control-plane:
      ipv4:
      - 172.18.0.3
      ipv6:
      - fc00:f853:ccd:e793::3
    egressgateway-worker:
      ipv4:
      - 172.18.0.2
      ipv6:
      - fc00:f853:ccd:e793::2
    egressgateway-worker2:
      ipv4:
      - 172.18.0.4
      ipv6:
      - fc00:f853:ccd:e793::4
  podCIDR: # (9)
    default-ipv4-ippool:
      ipv4:
      - 172.40.0.0/16
    default-ipv6-ippool:
      ipv6:
      - fd40::/48
    test-ippool:
      ipv4:
      - 177.70.0.0/16
  podCidrMode: calico # (10)
```

--------------------------------

### Verify EgressGateway Pods

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Install.en.md

Checks the status of EgressGateway pods in the `kube-system` namespace. This command helps confirm that the EgressGateway agents and controller are running correctly after installation.

```shell
$ kubectl get pod -n kube-system | grep egressgateway
egressgateway-agent-29lt5                  1/1     Running   0          9h
egressgateway-agent-94n8k                  1/1     Running   0          9h
egressgateway-agent-klkhf                  1/1     Running   0          9h
egressgateway-controller-5754f6658-7pn4z   1/1     Running   0          9h
```

--------------------------------

### Create and Add to ipset for Destination IPs

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/concepts/Datapath.en.md

Creates an ipset for destination IPs and adds a specific network to it. Ensure the ipset name is unique and managed.

```shell
IPSET_RULE_DEST_NAME=egress-dest-uuid

ipset x $IPSET_RULE_DEST_NAME

ipset create $IPSET_RULE_DEST_NAME hash:net

ipset add $IPSET_RULE_DEST_NAME 10.6.105.150/32
```

--------------------------------

### EgressPolicy CRD Example

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/05-egress-gateway-policy-cluster/README.en.md

Defines an EgressPolicy with specific appliedTo and destSubnet configurations. Use this for namespace-level egress policies. The `priority` field allows for policy prioritization.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  namespace: "default"
  name: "policy-test"
spec:
  priority: 100             # 1
  egressGatewayName: "eg1"
  egressIP:
    ipv4: ""
    ipv6: ""
    useNodeIP: false
  appliedTo:
    podSelector:
      matchLabels:
        app: "shopping"
    podSubnet:
    - "172.29.16.0/24"
    - 'fd00:1/126'
  destSubnet:
    - "10.6.1.92/32"
    - "fd00::92/128"
```

--------------------------------

### Create Test Pod for Egress Testing

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.zh.md

Create a simple Nginx pod named `mock-app` and schedule it to run on a specific node (`ip-172-16-61-234.ec2.internal`). This pod will be used to test EgressGateway policies.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: mock-app
  labels:
    app: mock-app
spec:
  nodeName: ip-172-16-61-234.ec2.internal
  containers:
  - name: nginx
    image: nginx
```

--------------------------------

### EgressClusterPolicy CRD Example

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/05-egress-gateway-policy-cluster/README.en.md

Defines an EgressClusterPolicy, enabling cluster-level egress policies. It includes `namespaceSelector` for applying policies to specific namespaces across the cluster and a `priority` field for conflict resolution.

```yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterPolicy
metadata:
  namespace: "default"
  name: "policy-test"
spec:
  priority: 100             # 1
  egressGatewayName: "eg1"
  egressIP:
    ipv4: ""
    ipv6: ""
    useNodeIP: false
  appliedTo:
    podSelector:
      matchLabels:
        app: "shopping"
    podSubnet:
    - "172.29.16.0/24"
    - 'fd00:1/126'
    namespaceSelector:      # 1
      matchLabels:
        app: "shopping"
  destSubnet:
    - "10.6.1.92/32"
    - "fd00::92/128"
```

--------------------------------

### Build Local CI Image

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Develop.en.md

Builds the local CI image for the project. This is typically the first step in the development workflow.

```bash
make build_local_image
```

--------------------------------

### Configure KUBECONFIG and View Nodes

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Develop.en.md

Sets the KUBECONFIG environment variable to point to the Egress Gateway's test configuration and then lists the nodes in the cluster.

```bash
export KUBECONFIG=$(pwd)/test/runtime/kubeconfig_egressgateway.config
kubectl get node
```

--------------------------------

### Verify EgressPolicy Using Default Gateway

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/ClusterDefaultEgressGateway.en.md

After creating an EgressPolicy without specifying an EgressGateway, use `kubectl get` to confirm that the `egressGatewayName` field has been automatically populated with the cluster's default gateway.

```shell
$ kubectl get egresspolicies mock-app -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  creationTimestamp: "2023-08-09T11:54:34Z"
  generation: 1
  name: mock-app
  namespace: default
  resourceVersion: "6233341"
  uid: 5692c5e6-a72b-41bd-a611-1106abd41bc2
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: mock-app
  destSubnet:
  - 10.6.1.92/32
  - fd00::92/128
  - 172.30.40.0/21
  egressGatewayName: default
```

--------------------------------

### List cluster nodes

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Aliyun.en.md

Verify the nodes available in the current cluster.

```shell
$ kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   66m   v1.30.0
node2   Ready    <none>          66m   v1.30.0
```

--------------------------------

### Add and Update EgressGateway Helm Repository

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.zh.md

Add the EgressGateway Helm repository and update your local Helm chart repository to fetch the latest EgressGateway charts. This is the first step in installing EgressGateway using Helm.

```shell
helm repo add egressgateway https://spidernet-io.github.io/egressgateway/
helm repo update
```

--------------------------------

### Manage EgressGateway via CLI

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Commands to create an EgressGateway resource, label nodes, and check status.

```bash
# Create an EgressGateway
kubectl apply -f - <<EOF
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: default
spec:
  ippools:
    ipv4:
      - "172.22.0.100-172.22.0.110"
  nodeSelector:
    selector:
      matchLabels:
        egressgateway: "true"
EOF

# Label nodes to be egress gateways
kubectl label node worker-1 egressgateway="true"
kubectl label node worker-2 egressgateway="true"

# Check EgressGateway status
kubectl get egressgateways
# NAME      ipv4DefaultEIP   ipv6DefaultEIP   clusterDefault   ipv4Total   ipv4Free
# default   172.22.0.110                      false            11          10

kubectl get egw default -o yaml
```

--------------------------------

### Perform EgressGateway Configuration Upgrade

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Upgrade.en.md

This command upgrades the EgressGateway configuration, for example, setting the agent log level to debug. It uses `--set` for specific value overrides and `--reuse-values` to maintain existing configurations.

```shell
helm upgrade \
  egress \
  egressgateway/egressgateway \
  --set agent.debug.logLevel=debug \
  --reuse-values
```

--------------------------------

### Create New Branch

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Contribute.en.md

Create a new branch for each feature or bugfix to keep changes organized. Use a descriptive name.

```shell
git checkout -b your-new-branch-name
```

--------------------------------

### Create and Add to ipset for Source IPs

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/concepts/Datapath.en.md

Creates an ipset for source IPs and adds a specific network to it. This is used in conjunction with destination ipsets for policy matching.

```shell
IPSET_RULE_SRC_NAME=egress-src-uuid

ipset x $IPSET_RULE_SRC_NAME

ipset create $IPSET_RULE_SRC_NAME hash:net

ipset add $IPSET_RULE_SRC_NAME 172.29.234.173/32
```

--------------------------------

### List Kubernetes Nodes

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/AwsWithCilium.en.md

Lists all nodes in the cluster with detailed information. Useful for identifying available nodes for pod placement.

```shell
~ kubectl get nodes -A -owide
NAME                             STATUS   ROLES    AGE   VERSION               INTERNAL-IP      EXTERNAL-IP                         
ip-172-16-103-117.ec2.internal   Ready    <none>   25m   v1.30.0-eks-036c24b   172.16.103.117   34.239.162.85  
ip-172-16-61-234.ec2.internal    Ready    <none>   25m   v1.30.0-eks-036c24b   172.16.61.234    54.147.15.230
ip-172-16-62-200.ec2.internal    Ready    <none>   25m   v1.30.0-eks-036c24b   172.16.62.200    54.147.16.130  
```

--------------------------------

### Initialize Traffic Tagging Chain

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/03-egress-ip/README.en.md

Creates the mangle table chain for marking egress requests.

```shell
iptables -t mangle -N EGRESSGATEWAY-MARK-REQUEST
iptables -t mangle -I PREROUTING 1 -j EGRESSGATEWAY-MARK-REQUEST -m comment --comment "egress gateway: mark egress packet"
```

--------------------------------

### Verify EgressPolicy Configuration

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/NamespaceDefaultEgressGateway.en.md

Inspect the EgressPolicy to confirm the default gateway has been automatically applied.

```shell
$ kubectl get egresspolicies mock-app -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  creationTimestamp: "2023-08-09T10:54:34Z"
  generation: 1
  name: mock-app
  namespace: default
  resourceVersion: "6233341"
  uid: 5692c5e6-a71b-41bd-a611-1106abd41ba3
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: mock-app
  destSubnet:
  - 10.6.1.92/32
  - fd00::92/128
  - 172.30.40.0/21
  egressGatewayName: egressgateway
```

--------------------------------

### Egress Gateway Project Structure

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Develop.en.md

Visual representation of the Go package structure for the Egress Gateway project. It outlines the organization of directories and sub-packages.

```tree
.
├── api
├── charts
├── cmd
├── docs
├── images
├── output
├── pkg
│   ├── agent
│   ├── coalescing
│   ├── config
│   ├── constant
│   ├── controller
│   ├── egressgateway
│   ├── ethtool
│   ├── ipset
│   ├── iptables
│   ├── k8s
│   ├── layer2
│   ├── lock
│   ├── logger
│   ├── markallocator
│   ├── profiling
│   ├── schema
│   ├── types
│   └── utils
├── test
│   ├── doc
│   ├── e2e
│   ├── kindconfig
│   ├── scripts
│   └── yaml
├── tools
│   ├── golang
│   ├── images
│   └── scripts
└── vendor
```

--------------------------------

### Enable Host Network Card Offload

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Troubleshooting.zh.md

To improve VXLAN performance, ensure that offload settings are enabled on the host's network interface. This command enables both receive and transmit offloading.

```bash
ethtool --offload host-interface-name rx on tx on
```

--------------------------------

### Create EgressGateway Instance

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Install.en.md

Defines a group of nodes as an egress gateway. Ensure the ippools CIDR matches the gateway node's egress interface subnet.

```shell
cat <<EOF | kubectl apply -f -
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: default
spec:
  ippools:
    ipv4:
    - "172.22.0.100-172.22.0.110"
  nodeSelector:
    selector:
      matchLabels:
        egressgateway: "true"
EOF
```

--------------------------------

### Agent Configuration Parameters

Source: https://github.com/spidernet-io/egressgateway/blob/main/charts/README.md

Configuration options for the egress gateway agent.

```APIDOC
## Agent Configuration

### Description
This section details the configuration parameters for the egress gateway agent.

### Parameters
#### Configuration Fields
- **agent.prometheus.grafanaDashboard.annotations** (object) - The additional annotations of egressgateway agent grafanaDashboard.
- **agent.prometheus.grafanaDashboard.labels** (object) - The additional label of egressgateway agent grafanaDashboard.
- **agent.debug.logLevel** (string) - The log level of egress gateway agent [`debug`, `info`, `warn`, `error`, `fatal`, `panic`]. Default: `info`.
- **agent.debug.logEncoder** (string) - Set the type of log encoder (`json`, `console`). Default: `json`.
- **agent.debug.logWithCaller** (boolean) - Enable or disable logging with caller information (`true`/`false`). Default: `true`.
- **agent.debug.logUseDevMode** (boolean) - Enable or disable development mode for logging (`true`/`false`). Default: `true`.
- **agent.debug.gopsPort** (integer) - The port used by gops tool for process monitoring and performance tuning. Default: `5812`.
- **agent.debug.pyroscopeServerAddr** (string) - The address of the Pyroscope server. Default: `""`.
```

--------------------------------

### Configure Routing Rules

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/01-egress-gateway/EgressGateway.md

Commands to manage routing tables and multipath policies for egress traffic.

```bash
ip rule add fwmark 0x11000000 table 100
ip route f table 100
ip route add default via 20.0.0.85 dev egress-vxlan-v4 onlink table 100
```

```bash
sysctl -w net.ipv4.fib_multipath_hash_policy=1
```

```bash
ip rule add fwmark 0x11000000 table 100
ip route f table 100
ip route add table 100 default \
  nexthop via 20.0.0.85 dev egress-vxlan onlink \
  nexthop via 20.0.0.90 dev egress-vxlan onlink
```

--------------------------------

### Go Runtime Metrics

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/reference/metrics.en.md

Metrics related to the Go runtime environment, including garbage collection and memory statistics.

```APIDOC
## Go Runtime Metrics

### Description
Metrics related to the Go runtime environment, including garbage collection and memory statistics.

### Metrics
- **`go_memstats_next_gc_bytes`** (gauge) - Number of heap bytes when next garbage collection will take place
- **`go_memstats_other_sys_bytes`** (gauge) - Number of bytes used for other system allocations
- **`go_memstats_stack_inuse_bytes`** (gauge) - Number of bytes in use by the stack allocator
- **`go_memstats_stack_sys_bytes`** (gauge) - Number of bytes obtained from system for stack allocator
- **`go_memstats_sys_bytes`** (gauge) - Number of bytes obtained from system
- **`go_threads`** (gauge) - Number of OS threads created
```

--------------------------------

### Verify Egress Destination ipset Rules

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

List and inspect the egress-dst ipset rules to verify if the destination subnet's IP address is included.

```shell
$ ipset list | grep egress-dst-
Name: egress-dst-v4-738bb014438bdbfe7

$ ipset list egress-dst-v4-738bb014438bdbfe7
Name: egress-dst-v4-738bb014438bdbfe7
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 65536 bucketsize 12 initval 0x14fe6b9e
Size in memory: 504
References: 1
Number of entries: 1
Members:
172.16.25.183
```

--------------------------------

### Initialize Egress Gateway Reset Mark Chain

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/concepts/Datapath.en.md

Creates the mangle table chain for resetting marks on egress packets.

```shell
iptables -t mangle -N EGRESSGATEWAY-RESET-MARK
iptables -t mangle -I FORWARD 1 -j EGRESSGATEWAY-RESET-MARK -m comment --comment "egress gateway: mark egress packet"

iptables -t mangle -A EGRESSGATEWAY-RESET-MARK \
    -m mark --mark $NODE_MARK/0x26000000 \
    -j MARK --set-mark 0x12000000 \
    -m comment --comment "egress gateway: change mark"
\ -m -comment "egress gateway: change mark
```

--------------------------------

### Configure IPTables for Policy Marking

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/01-egress-gateway/EgressGateway.md

Commands to set up IPTables rules for marking packets based on source and destination sets.

```bash
iptables -t mangle -F EGRESSGATEWAY-MARK-REQUEST-POLICY-NAME
iptables -t mangle -X EGRESSGATEWAY-MARK-REQUEST-POLICY-NAME
iptables -t mangle -N EGRESSGATEWAY-MARK-REQUEST-POLICY-NAME

iptables -A EGRESSGATEWAY-MARK-REQUEST-POLICY-NAME \
  -t mangle \
  -m conntrack --ctdir ORIGINAL \
  -m set --match-set egress-dst-policy-name dst \
  -m set --match-set egress-src-policy-name src \
  -j MARK --set-mark 0x11000000 \
  -m comment --comment "rule uuid: mark request packet"
```

--------------------------------

### Efficiently Update ipset with New IP Range

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/03-egress-ip/README.en.md

Creates a temporary ipset, adds a new IP range to it, and then swaps it with the active ipset. This method simplifies ipset operations and improves efficiency for updating destination IP lists.

```shell
ipset create egress-dst-v4-xxx-tmp 
ipset add egress-dst-v4-xxx-tmp $NEW_IP_RANGE
ipset swap egress-dst-v4-xxx egress-dst-v4-xxx-tmp
```

--------------------------------

### Initialize SNAT EIP Chain

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/03-egress-ip/README.en.md

Creates the nat table chain for SNAT EIP operations.

```shell
iptables -t nat -N EGRESSGATEWAY-SNAT-EIP
# Need to insert after rules that don't require SNAT to keep the chain at the top
iptables -t nat -I POSTROUTING 1 -j EGRESSGATEWAY-SNAT-EIP -m comment --comment "egress gateway: snat EIP"
```

--------------------------------

### Run E2E Tests

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Develop.en.md

Executes the end-to-end tests for the Egress Gateway project.

```bash
make e2e_run
```

--------------------------------

### List All Egress Policies

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

Retrieve a list of all Egress Policies across all namespaces. This helps in identifying the policy that might be causing issues.

```bash
$ kubectl get egresspolicy -A
NAMESPACE   NAME   GATEWAY   IPV4            IPV6   EGRESSNODE
default     test   default   172.16.25.189          node2
```

--------------------------------

### Check Cluster Policies

Source: https://context7.com/spidernet-io/egressgateway/llms.txt

Lists all cluster-wide egress policies and displays their assigned gateway, IPv4, and egress tunnel information.

```bash
# Check cluster policies
kubectl get egressclusterpolicies
# NAME                GATEWAY   IPV4           IPV6   EGRESSTUNNEL
# production-egress   default   172.22.0.100          worker-2
```

```bash
kubectl get egcp production-egress -o yaml
```

--------------------------------

### Verify Routing Table and Gateway Status

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/troubleshooting/Troubleshooting.en.md

Check IP rules, routing tables, and EgressGateway resources to confirm traffic forwarding paths.

```bash
$ ip rule
0:	from all lookup local
99:	from all fwmark 0x26b7b4e4 lookup 649573604
32766:	from all lookup main
32767:	from all lookup default

$ ip r show table 649573604
default via 192.200.201.40 dev egress.vxlan

$ ip r show table 649573604
default via 192.200.201.40 dev egress.vxlan

$ kubectl get egt
NAME    TUNNELMAC           TUNNELIPV4       TUNNELIPV6   MARK         PHASE
node1   66:c5:d6:cc:29:54   192.200.142.89                0x2698ee37   Ready
node2   66:66:41:3e:23:6b   192.200.201.40                0x26b7b4e4   Ready

$ kubectl get egp
NAME   GATEWAY   IPV4            IPV6   EGRESSNODE
test   default   172.16.25.189          node2
```

--------------------------------

### Define test Pod

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Aliyun.en.md

YAML configuration for a test Nginx Pod deployed on node1.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
    - image: nginx
      imagePullPolicy: IfNotPresent
      name: nginx
      resources: {}
  nodeName: node1
```

--------------------------------

### Stage and Commit Changes

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/develop/Contribute.en.md

Stage all modified files and commit them with a clear and concise message indicating the purpose of the changes.

```shell
git add .
git commit -m "Your commit message"
```

--------------------------------

### Update Local Helm Repository

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/usage/Upgrade.en.md

Run this command before upgrading to ensure your local Helm chart repository is up-to-date with the latest available versions.

```shell
helm repo update
```

--------------------------------

### Configure Egress Gateway Routing Rules

Source: https://github.com/spidernet-io/egressgateway/blob/main/docs/proposal/06-underlay-cni-supports/README.en.md

Sets up iptables marking and routing rules on the egress gateway node to ensure return traffic passes through the host network stack.

```shell
On the egress gateway node, new connections coming through the tunnel are marked as EgressGateway hits.
iptables -t mangle -A PREROUTING -i egress.vxlan -m conntrack --ctstate NEW -j MARK --set-mark 0x27

Add mark to the connection tracking table for recovery on packet return
iptables -t mangle -A PREROUTING -m mark --mark 0x27 -j CONNMARK --save-mark


ESTABLISHED connections, packets need to be recovered based on the contents of the connection tracking table Mark
iptables -t mangle -A PREROUTING -m conntrack --ctstate ESTABLISHED -j CONNMARK --restore-mark


Add routes to tunnel the packets back, one for each of them
ip rule add from all fwmark 0x27 lookup 600
ip r add <Pod IP> via <Pod Node> dev egress.vxlan t 600

Clear the mark of the inner packets to avoid interfering with the outer packets
iptables -t mangle -A POSTROUTING -m mark --mark 0x27 -j MARK --set-mark 0x00
```