Kube-router

https://www.kube-router.io/docs
Admin
Kube-router is a lean and fast Kubernetes network solution that provides BGP routing, IPVS/load...
Tokens:22,557
Snippets:176
Trust Score:8.1
Update:11 months ago
Show doc for...
Tokens:
Raw
### Run Kube-router using Containerd (ctr)

Source: https://www.kube-router.io/docs/user-guide

This section shows how to deploy Kube-router using the `ctr` command with Containerd. It involves pulling the image, then running the container with similar privileged and network configurations as the Docker example, including specific mount options for read-only and read-write access.

```shell
$ ctr image pull docker.io/cloudnativelabs/kube-router:latest
$ ctr run --privileged -t --net-host \
--mount type=bind,src=/lib/modules,dst=/lib/modules,options=rbind:ro \
--mount type=bind,src=/run/xtables.lock,dst=/run/xtables.lock,options=rbind:rw \
docker.io/cloudnativelabs/kube-router:latest kube-router-cleanup /usr/local/bin/kube-router --cleanup-config

```

--------------------------------

### Example Kube-router Startup Configuration

Source: https://www.kube-router.io/docs/health

This example shows a typical startup configuration for kube-router, enabling various controllers. The health of these controllers is monitored, and failures in their main loops will be reflected in the overall health status.

```text
--run-router=true
--run-firewall=true
--run-service-proxy=true
--run-loadbalancer=true
```

--------------------------------

### Run Kube-router using Docker

Source: https://www.kube-router.io/docs/user-guide

This snippet demonstrates how to run Kube-router as a Docker container. It utilizes privileged mode, host networking, and bind mounts for necessary kernel modules and locks. The `--cleanup-config` flag is used for initial setup or cleanup.

```docker
docker run --privileged --net=host \
--mount type=bind,source=/lib/modules,target=/lib/modules,readonly \
--mount type=bind,source=/run/xtables.lock,target=/run/xtables.lock,bind-propagation=rshared \
cloudnativelabs/kube-router /usr/local/bin/kube-router --cleanup-config

```

--------------------------------

### Build kube-router Binary

Source: https://www.kube-router.io/docs/developing

Builds the kube-router binary using Go modules. Requires Go version 1.19 or above. Dependencies are automatically fetched. This command compiles the main executable.

```bash
make kube-router
go build pkg/cmd/kube-router.go
```

--------------------------------

### Deploy Kube-router as a DaemonSet

Source: https://www.kube-router.io/docs/user-guide

Applies a Kubernetes DaemonSet YAML file to automatically deploy Kube-router pods on each node in the cluster. This is the recommended method for quick deployment.

```Kubernetes
kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kube-router-all-service-daemonset.yaml
```

--------------------------------

### Set Up Local Kube-router Repository (Bash)

Source: https://www.kube-router.io/docs/CONTRIBUTING

Commands to clone the Kube-router repository, set up an upstream remote, and rebase the local master branch with the upstream master. This is essential for starting local development.

```bash
$ git clone https://github.com/YOUR_ACCOUNT/kube-router.git
$ cd kube-router
$ git remote add upstream https://github.com/cloudnativelabs/kube-router
$ git checkout master
$ git fetch upstream
$ git rebase upstream/master
```

--------------------------------

### Run Kube-router as an Agent for Firewall Service

Source: https://www.kube-router.io/docs/user-guide

Starts Kube-router in agent mode, specifically configured to provide ingress firewall services for pods while disabling other features like service proxy and routing.

```Shell
kube-router --master=http://192.168.1.99:8080/ --run-firewall=true --run-service-proxy=false --run-router=false
```

--------------------------------

### Install kube-router for Pod Networking and Network Policy with Kubeadm

Source: https://www.kube-router.io/docs/kubeadm

Applies the kube-router daemonset configuration for pod networking and network policy when using kubeadm for Kubernetes cluster setup. This command ensures basic pod communication and network segmentation.

```bash
KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml
```

--------------------------------

### Build Docker Image with Custom Options

Source: https://www.kube-router.io/docs/developing

Builds a Docker image for kube-router with customizable image registry, namespace, and tag. This allows users to specify where the image should be built and how it should be named, useful for testing or deploying to non-default repositories.

```bash
make container IMG_FQDN=quay.io IMG_NAMESPACE=bzub IMG_TAG=custom
```

--------------------------------

### Download CNI Configuration for Kube-router

Source: https://www.kube-router.io/docs/user-guide

Downloads the necessary CNI configuration file for Kube-router to manage pod-to-pod network connectivity. This file is typically placed in /etc/cni/net.d/.

```Shell
wget -O /etc/cni/net.d/10-kuberouter.conf https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/cni/10-kuberouter.conf
```

--------------------------------

### Enable Hairpin Mode for a Service using Kube-router

Source: https://www.kube-router.io/docs/user-guide

This example demonstrates how to enable hairpin mode for a specific Kubernetes service (`my-service`) using `kubectl annotate`. It shows annotations for enabling general hairpin traffic and specifically for external IPs associated with the service.

```shell
kubectl annotate service my-service "kube-router.io/service.hairpin="

kubectl annotate service my-service "kube-router.io/service.hairpin.externalips="

```

--------------------------------

### Kube-router API Server Connection Options

Source: https://www.kube-router.io/docs/user-guide

Specifies how Kube-router connects to the Kubernetes API server. It can use a direct master URL or a kubeconfig file.

```Shell
kube-router --master=http://192.168.1.99:8080/
kube-router --kubeconfig=<path to kubeconfig file>
```

--------------------------------

### Install kube-router for Full Features (Service Proxy, Firewall, Pod Networking) with Kubeadm

Source: https://www.kube-router.io/docs/kubeadm

Applies the comprehensive kube-router daemonset configuration, enabling pod networking, network policy, and its advanced service proxy features. This is used as an alternative to kube-proxy.

```bash
KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter-all-features.yaml
```

--------------------------------

### Annotate Service for IP Advertising with Kube-router

Source: https://www.kube-router.io/docs/user-guide

This example shows how to use `kubectl annotate` to selectively enable or disable the advertising of Cluster, External, and LoadBalancer IPs for specific services using Kube-router. This complements global configuration flags.

```shell
$ kubectl annotate service my-advertised-service "kube-router.io/service.advertise.clusterip=true" "kube-router.io/service.advertise.externalip=true" "kube-router.io/service.advertise.loadbalancerip=true"
$ kubectl annotate service my-non-advertised-service "kube-router.io/service.advertise.clusterip=false" "kube-router.io/service.advertise.externalip=false" "kube-router.io/service.advertise.loadbalancerip=false"

```

--------------------------------

### Switch between Kube-proxy and Kube-router for Service Proxy

Source: https://www.kube-router.io/docs/user-guide

This sequence illustrates how to transition between Kube-proxy and Kube-router for service proxy functionality. It involves cleaning up existing iptables rules for Kube-proxy, then launching Kube-router with specific flags to handle service proxying, firewall, and routing. Reverting involves cleaning Kube-router's configuration.

```shell
kube-proxy --cleanup-iptables

```

```shell
kube-router --master=http://192.168.1.99:8080/ --run-service-proxy=true --run-firewall=false --run-router=false

```

```shell
kube-router --cleanup-config

```

--------------------------------

### Push Docker Image for kube-router

Source: https://www.kube-router.io/docs/developing

Pushes the built container image to a Docker registry. By default, it uses the cloudnativelabs/kube-router repository on Docker Hub. The target repository can be customized using image options.

```bash
make push
```

--------------------------------

### Configure Kube-router Service Schedulers with Annotations

Source: https://www.kube-router.io/docs/user-guide

This snippet demonstrates how to configure different load balancing scheduling algorithms for Kubernetes services using kubectl annotations. These annotations are applied to a service to specify the desired scheduling method, such as least connection, round-robin, or various hashing methods.

```bash
#For least connection scheduling use:
$ kubectl annotate service my-service "kube-router.io/service.scheduler=lc"

#For round-robin scheduling use:
$ kubectl annotate service my-service "kube-router.io/service.scheduler=rr"

#For source hashing scheduling use:
$ kubectl annotate service my-service "kube-router.io/service.scheduler=sh"

#For destination hashing scheduling use:
$ kubectl annotate service my-service "kube-router.io/service.scheduler=dh"

#For maglev scheduling use:
$ kubectl annotate service my-service "kube-router.io/service.scheduler=mh"

# The maglev scheduler can be further tuned with additional options.
#To use the maglev scheduler's fallback option use:
$ kubectl annotate service my-service "kube-router.io/service.schedflags=flag-1"
#To use the maglev scheduler's port option use:
$ kubectl annotate service my-service "kube-router.io/service.schedflags=flag-2"
#To use the maglev scheduler's port and fallback option use:
$ kubectl annotate service my-service "kube-router.io/service.schedflags=flag-1,flag-2"
```

--------------------------------

### Build Docker Image for kube-router

Source: https://www.kube-router.io/docs/developing

Compiles kube-router and builds a Docker image. The image is tagged with the last release version and current commit ID by default. This process involves multiple steps defined in the Dockerfile, including setting build arguments for base images and copying the compiled binary.

```bash
make container
```

--------------------------------

### Configure Kube-router CNI for Hairpin Mode

Source: https://www.kube-router.io/docs/user-guide

This JSON configuration snippet is for a Kube-router CNI setup, specifically enabling `hairpinMode`. This setting is crucial for allowing pods to communicate with services on their own ClusterIP, ensuring proper traffic flow and bypassing potential network restrictions.

```json
{
    "name":"mynet",
    "type":"bridge",
    "bridge":"kube-bridge",
    "isDefaultGateway":true,
    "hairpinMode":true,
    "ipam": {
        "type":"host-local"
     }
}

```

--------------------------------

### Kube-router DaemonSet Deployment (Kubernetes)

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-hostport

This DaemonSet ensures that the kube-router agent runs on each node in the cluster. It includes specifications for the container image, arguments for enabling router, firewall, and service proxy functionalities, environment variables, probes, resource requests, security context, and volume mounts for CNI configuration and other necessary files. It also defines init containers for CNI setup and host-related configurations.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-router
      tier: node
  template:
    metadata:
      labels:
        k8s-app: kube-router
        tier: node
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: kube-router
      serviceAccount: kube-router
      containers:
      - name: kube-router
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        args:
        - --run-router=true
        - --run-firewall=true
        - --run-service-proxy=true
        - --bgp-graceful-restart=true
        - --kubeconfig=/var/lib/kube-router/kubeconfig
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBE_ROUTER_CNI_CONF_FILE
          value: /etc/cni/net.d/10-kuberouter.conflist
        livenessProbe:
          httpGet:
            path: /healthz
            port: 20244
          initialDelaySeconds: 10
          periodSeconds: 3
        resources:
          requests:
            cpu: 250m
            memory: 250Mi
        securityContext:
          privileged: true
        volumeMounts:
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kubeconfig
          mountPath: /var/lib/kube-router
          readOnly: true
        - name: xtables-lock
          mountPath: /run/xtables.lock
          readOnly: false
      initContainers:
      - name: install-cni
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        command:
        - /bin/sh
        - -c
        - set -e -x;
          if [ ! -f /etc/cni/net.d/10-kuberouter.conflist ]; then
            if [ -f /etc/cni/net.d/*.conf ]; then
              rm -f /etc/cni/net.d/*.conf;
            fi;
            TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
            cp /etc/kube-router/cni-conf.json ${TMP};
            mv ${TMP} /etc/cni/net.d/10-kuberouter.conflist;
          fi;
          if [ -x /usr/local/bin/cni-install ]; then
            /usr/local/bin/cni-install;
          fi;
        volumeMounts:
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kube-router-cfg
          mountPath: /etc/kube-router
        - name: host-opt
          mountPath: /opt
      hostNetwork: true
      hostPID: true
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: cni-conf-dir
        hostPath:
          path: /etc/cni/net.d
      - name: kube-router-cfg
        configMap:
          name: kube-router-cfg
      - name: kubeconfig
        configMap:
          name: kube-proxy
          items:
          - key: kubeconfig.conf
            path: kubeconfig
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: host-opt
        hostPath:
          path: /opt
```

--------------------------------

### Kubernetes DaemonSet Configuration with POD_NAME Environment Variable

Source: https://www.kube-router.io/docs/load-balancer-allocator

Example snippet from a Kubernetes DaemonSet definition for kube-router, illustrating how the POD_NAME environment variable is set using the downward API to identify the lease for leader election.

```yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router
  namespace: kube-system
spec:
  ...
  template:
    metadata:
      ....
    spec:
        ...
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
...

```

--------------------------------

### Configuring Kubernetes Controller Manager for Dual-Stack

Source: https://www.kube-router.io/docs/ipv6

This example shows how to configure the Kubernetes controller manager to support both IPv4 and IPv6 CIDRs for cluster networking, which is a prerequisite for dual-stack functionality in kube-router.

```bash
--cluster-cidr=10.242.0.0/16,2001:db8:42:1000::/56
```

--------------------------------

### Configure HTTP Proxy for Kube-router Deployment (YAML)

Source: https://www.kube-router.io/docs/user-guide

This snippet shows how to inject an HTTP_PROXY environment variable into a Kube-router deployment configuration. This is typically used in environments like AWS that require an explicit proxy server.

```yaml
env:
- name: HTTP_PROXY
  value: "http://proxy.example.com:80"
```

--------------------------------

### Watch kube-router Pods for Changes (kubectl)

Source: https://www.kube-router.io/docs/upgrading

This command continuously watches for changes in kube-router pods across all nodes. It's used to monitor the rollout and ensure new pods start and remain running.

```bash
kubectl -n kube-system get pods -o wide -w
```

--------------------------------

### Get Default Containerd CRI Configuration

Source: https://www.kube-router.io/docs/generic

Retrieves the default configuration for the Containerd CRI plugin, which specifies paths for CNI binary and configuration directories. This is essential for ensuring Containerd correctly loads CNI configurations.

```bash
containerd config default

```

--------------------------------

### Annotate Node for Kube-Router Route Reflector Client

Source: https://www.kube-router.io/docs/bgp

This command annotates a Kubernetes node to configure it as a Route Reflector client. Clients will peer with Route Reflector servers in Kube-Router's setup. It requires kubectl.

```bash
kubectl annotate node <kube-node> "kube-router.io/rr.client=42"
```

--------------------------------

### Kube-router CNI Configuration for HostPort Support

Source: https://www.kube-router.io/docs/user-guide

This JSON configuration snippet defines the CNI (Container Network Interface) settings required for Kube-router to support HostPort functionality. It includes the CNI version, network name, and plugins like 'kubernetes' and 'portmap' with capabilities for SNAT and port mappings.

```json
{
   "cniVersion":"0.3.0",
   "name":"mynet",
   "plugins":[
      {
         "name":"kubernetes",
         "type":"bridge",
         "bridge":"kube-bridge",
         "isDefaultGateway":true,
         "ipam":{
            "type":"host-local"
         }
      },
      {
         "type":"portmap",
         "capabilities":{
            "snat":true,
            "portMappings":true
         }
      }
   ]
}
```

--------------------------------

### Annotate Node for Kube-Router Route Reflector Server

Source: https://www.kube-router.io/docs/bgp

This command annotates a Kubernetes node to configure it as a Route Reflector server. This is used in conjunction with Kube-Router's route reflector setup to centralize peering. It requires kubectl.

```bash
kubectl annotate node <kube-node> "kube-router.io/rr.server=42"
```

--------------------------------

### Kube-Router ClusterRole Definition (YAML)

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-dsr

Defines the permissions kube-router requires to interact with Kubernetes API resources. It grants access to list, get, watch, and update operations for various API groups including networking.k8s.io, extensions, coordination.k8s.io, core API (services/status), and discovery.k8s.io.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-router
rules:
  - apiGroups:
    - "networking.k8s.io"
    resources:
      - networkpolicies
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - extensions
    resources:
      - networkpolicies
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "coordination.k8s.io"
    resources:
      - leases
    verbs:
      - get
      - create
      - update
  - apiGroups:
      - ""
    resources:
      - services/status
    verbs:
      - update
  - apiGroups:
      - "discovery.k8s.io"
    resources:
      - endpointslices
    verbs:
      - get
      - list
      - watch
```

--------------------------------

### Configure BGP Peer Passwords via CLI Flag

Source: https://www.kube-router.io/docs/bgp

Configures BGP peer passwords using command-line flags. Passwords must be base64 encoded. This example shows one peer with a password and another without, indicated by a trailing comma.

```bash
--peer-router-ips="192.168.1.99,192.168.1.100"
--peer-router-asns="65000,65000"
--peer-router-passwords="U2VjdXJlUGFzc3dvcmQK,"
```

--------------------------------

### Custom CNI Configuration with Multiple Pod Ranges

Source: https://www.kube-router.io/docs/ipv6

Example of a CNI configuration file that accepts multiple pod ranges, supporting both IPv4 and IPv6 subnets. This configuration is used when kube-router needs to manage multiple IP families for pod networking.

```json
{
  "cniVersion": "0.3.0",
  "name": "mynet",
  "plugins": [
    {
      "bridge": "kube-bridge",
      "ipam": {
        "ranges": [
          [
            {
              "subnet": "10.242.0.0/24"
            }
          ],
          [
            {
              "subnet": "2001:db8:42:1000::/64"
            }
          ]
        ],
        "type": "host-local"
      },
      "isDefaultGateway": true,
      "mtu": 9001,
      "name": "kubernetes",
      "type": "bridge"
    }
  ]
}

```

--------------------------------

### Deploy Kube-router with All Features (Bash)

Source: https://www.kube-router.io/docs/generic

Deploys Kube-router with full functionality including pod/service networking, network policy firewall, and IPVS/LVS service proxy. It fetches the configuration from a GitHub URL, substitutes variables, and applies it using kubectl.

```bash
CLUSTERCIDR=10.32.0.0/12 \
APISERVER=https://cluster01.int.domain.com:6443 \
sh -c 'curl -s https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/generic-kuberouter-all-features.yaml | \
sed -e "s;%APISERVER%;$APISERVER;g" -e "s;%CLUSTERCIDR%;$CLUSTERCIDR;g"' | \
kubectl apply -f -

```

--------------------------------

### Remove Kube-proxy Daemonset (Kubectl)

Source: https://www.kube-router.io/docs/generic

Removes the kube-proxy daemonset from the `kube-system` namespace. This is a necessary step before deploying Kube-router if kube-proxy was previously installed, to avoid conflicts.

```kubectl
kubectl -n kube-system delete ds kube-proxy

```

--------------------------------

### Create and Push Git Tag for Release

Source: https://www.kube-router.io/docs/RELEASE

This sequence of Git commands creates a new tag locally and then pushes that tag to the remote repository, signifying a new release point. Ensure the remote name is correct.

```shell
git tag <tag_name>
git push origin <tag_name>
```

--------------------------------

### Update Dependencies with Go Command

Source: https://www.kube-router.io/docs/RELEASE

This command lists Go modules that have available updates. It uses `go list` with specific formatting to identify indirect dependencies that might also need updating.

```shell
go list -mod=mod -u -m -f '{{.}}{{if .Indirect}} IAMINDIRECT{{end}}' all | grep -v IAMINDIRECT
```

--------------------------------

### Cherry-Pick and Tag for Patch Release

Source: https://www.kube-router.io/docs/RELEASE

This is a multi-step Git process for creating a patch release. It involves checking out the master branch, identifying commits, switching to the release branch, cherry-picking specific commits, and then tagging and pushing the new release.

```shell
git checkout master
git log --color --pretty=format:'%h - %s (%cr) <%an>' --abbrev-commit --decorate
git checkout <release_branch>
git cherry-pick <commit_hash_from_above_log>
git tag <tag_name>
git push origin <tag_name>
```

--------------------------------

### Display IPVS Services and Real Servers

Source: https://www.kube-router.io/docs/pod-toolbox

The `ipvsadm -ln` command displays information about IP Virtual Server (IPVS) services and their associated real servers. This is useful for understanding load balancing configurations within the cluster. It requires the ipvsadm tool to be available.

```bash
ipvsadm -ln
```

--------------------------------

### Generate Changelog Syntax with Git Log

Source: https://www.kube-router.io/docs/RELEASE

This Git command retrieves the commit history between two tags, formatting it for use in changelogs. It provides a concise list of commits with their hash, subject, and author.

```shell
git log --format='* %h - %s `< %an>`' <tag>..<tag>
```

--------------------------------

### Kubernetes ClusterRole for Kube-router

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-hostport

Defines the permissions granted to the kube-router. It allows 'get', 'list', and 'watch' operations on 'endpointslices' resources within the 'discovery.k8s.io' API group. This is essential for kube-router to monitor network endpoint changes.

```yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-router
rules:
  - apiGroups:
      - "discovery.k8s.io"
    resources:
      - endpointslices
    verbs:
      - get
      - list
      - watch
```

--------------------------------

### Push Changes and Initiate Pull Request (Bash)

Source: https://www.kube-router.io/docs/CONTRIBUTING

Commands to push the local feature branch to your forked repository and then initiate a pull request on GitHub. This makes your contributions available for review.

```bash
$ git push origin master
$ git push origin feature_x
```

--------------------------------

### Update GitHub Actions with Dependabot

Source: https://www.kube-router.io/docs/RELEASE

This command uses the `dependabot` tool to automatically update GitHub Actions configurations within the cloudnativelabs/kube-router repository.

```shell
dependabot update github_actions cloudnativelabs/kube-router
```

--------------------------------

### Retrieve Kube-router Service and Node Annotations

Source: https://www.kube-router.io/docs/upgrading

These commands help in inspecting current service and node annotations across all namespaces, facilitating the migration of annotation values to new keys during the upgrade process.

```bash
kubectl describe services --all-namespaces |grep -E '^(Name:|Annotations:)'
```

```bash
kubectl describe nodes |grep -E '^(Name:|Annotations:)'
```

--------------------------------

### Cleanup kube-proxy using Containerd

Source: https://www.kube-router.io/docs/kubeadm

Utilizes the `ctr` command to pull the kube-proxy image and then run it with the `--cleanup` flag for containerd environments. This removes kube-proxy's iptables configurations.

```containerd
ctr images pull k8s.gcr.io/kube-proxy-amd64:v1.28.2
ctr run --rm --privileged --net-host --mount type=bind,src=/lib/modules,dst=/lib/modules,options=rbind:ro \
    registry.k8s.io/kube-proxy-amd64:v1.28.2 kube-proxy-cleanup kube-proxy --cleanup
```

--------------------------------

### Get BGP Neighbor and Route Information

Source: https://www.kube-router.io/docs/pod-toolbox

The `gobgp` command retrieves Border Gateway Protocol (BGP) information, including neighbor status and route details. This is essential for diagnosing BGP peering and routing issues. It can query the local node or a specified remote node.

```bash
gobgp
```

--------------------------------

### Kube-router Controller Structure - Go

Source: https://www.kube-router.io/docs/architecture

This Go code snippet illustrates the typical structure of a Kube-router controller. It includes a continuous run loop for periodic synchronization, an OnUpdate function triggered by API object changes, and Sync and Cleanup functions for state reconciliation and resource management. This pattern is fundamental to how Kube-router operates.

```go
func Run() {
    for {
        Sync() // control loop that runs for ever and perfom sync at periodic interval
    }
}

func OnUpdate() {
    Sync() // on receiving update of a watched API object (namespace, node, pod, network policy etc)
}

Sync() {
    //re-concile any state changes
}

Cleanup() {
    // cleanup any changes (to iptables, ipvs, network etc) done to the system
}

```

--------------------------------

### Configure MTU in CNI Config (JSON)

Source: https://www.kube-router.io/docs/user-guide

This JSON snippet shows how to manually set the MTU for pod interfaces within the kube-router CNI configuration file (`10-kuberouter.conflist`). This is used when `auto-mtu` is set to false. Ensure the `kube-bridge` MTU is also set manually to avoid fragmentation.

```json
{
   "cniVersion":"0.3.0",
   "name":"mynet",
   "plugins":[
      {
         "name":"kubernetes",
         "type":"bridge",
         "mtu": 1400,
         "bridge":"kube-bridge",
         "isDefaultGateway":true,
         "ipam":{
            "type":"host-local"
         }
      }
   ]
}
```

--------------------------------

### Annotate Node for AS Path Prepending (AS) in Kube-Router

Source: https://www.kube-router.io/docs/bgp

This command annotates a Kubernetes node to specify the AS number to be used for AS path prepending. This is a traffic shaping mechanism used by Kube-Router. It requires kubectl.

```bash
kubectl annotate node <kube-node> "kube-router.io/path-prepend.as=65000"
```

--------------------------------

### Configure Kube-router Metrics Port and Path

Source: https://www.kube-router.io/docs/metrics

Enables Prometheus metrics by specifying a port greater than 0 and an optional path. Older versions have fixed defaults.

```bash
--metrics-path string "/metrics"
--metrics-port uint16 0
```

--------------------------------

### Configure BGP Peer Passwords via Node Annotations

Source: https://www.kube-router.io/docs/bgp

Configures BGP peer passwords using Kubernetes node annotations. Passwords must be base64 encoded. Similar to the CLI flag, this example sets a password for the first peer and none for the second.

```bash
kubectl annotate node <kube-node> "kube-router.io/peer.ips=192.168.1.99,192.168.1.100"
kubectl annotate node <kube-node> "kube-router.io/peer.asns=65000,65000"
kubectl annotate node <kube-node> "kube-router.io/peer.passwords=U2VjdXJlUGFzc3dvcmQK,"
```

--------------------------------

### Annotate Node for AS Path Prepending (Repeat Count) in Kube-Router

Source: https://www.kube-router.io/docs/bgp

This command annotates a Kubernetes node to specify the number of times an AS path should be prepended. This is used for traffic shaping with Kube-Router. It requires kubectl.

```bash
kubectl annotate node <kube-node> "kube-router.io/path-prepend.repeat-n=5"
```

--------------------------------

### Kube-router DaemonSet Deployment

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-dsr

This DaemonSet deploys kube-router on each node of the Kubernetes cluster. It configures kube-router to run the router, firewall, and service proxy components. Key configurations include BGP graceful restart, runtime endpoint, and various volume mounts for network and system access.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-router
      tier: node
  template:
    metadata:
      labels:
        k8s-app: kube-router
        tier: node
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: kube-router
      serviceAccount: kube-router
      containers:
      - name: kube-router
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        args:
        - --run-router=true
        - --run-firewall=true
        - --run-service-proxy=true
        - --bgp-graceful-restart=true
        - --kubeconfig=/var/lib/kube-router/kubeconfig
        - --runtime-endpoint=unix:///run/containerd/containerd.sock
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBE_ROUTER_CNI_CONF_FILE
          value: /etc/cni/net.d/10-kuberouter.conflist
        livenessProbe:
          httpGet:
            path: /healthz
            port: 20244
          initialDelaySeconds: 10
          periodSeconds: 3
        resources:
          requests:
            cpu: 250m
            memory: 250Mi
        securityContext:
          privileged: true
        volumeMounts:
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kubeconfig
          mountPath: /var/lib/kube-router
          readOnly: true
        - name: run-containerd
          mountPath: /var/run/containerd/
          readOnly: true
        - name: rt-tables
          mountPath: /etc/iproute2/rt_tables
          readOnly: false
        - name: xtables-lock
          mountPath: /run/xtables.lock
          readOnly: false
      initContainers:
      - name: install-cni
        image: docker.io/cloudnativelabs/kube-router
        imagePullPolicy: Always
        command:
        - /bin/sh
        - -c
        - set -e -x;
          if [ ! -f /etc/cni/net.d/10-kuberouter.conflist ]; then
            if [ -f /etc/cni/net.d/*.conf ]; then
              rm -f /etc/cni/net.d/*.conf;
            fi;
            TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
            cp /etc/kube-router/cni-conf.json ${TMP};
            mv ${TMP} /etc/cni/net.d/10-kuberouter.conflist;
          fi;
          if [ -x /usr/local/bin/cni-install ]; then
            /usr/local/bin/cni-install;
          fi;
        volumeMounts:
        - name: cni-conf-dir
          mountPath: /etc/cni/net.d
        - name: kube-router-cfg
          mountPath: /etc/kube-router
        - name: host-opt
          mountPath: /opt
      hostNetwork: true
      hostIPC: true
      hostPID: true
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: cni-conf-dir
        hostPath:
          path: /etc/cni/net.d
      # To mount the hosts containerd socket.
      # We must mount the whole directory as the socket file might change due to containerd restarts
      - name: run-containerd
        hostPath:
          path: /var/run/containerd/
      # To mount the hosts iproute2 configuration
      - name: rt-tables
        hostPath:
          path: /etc/iproute2/rt_tables
      - name: kube-router-cfg
        configMap:
          name: kube-router-cfg
      - name: kubeconfig
        configMap:
          name: kube-proxy
          items:
          - key: kubeconfig.conf
            path: kubeconfig
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: host-opt
        hostPath:
          path: /opt
```

--------------------------------

### Configure BGP Peer Passwords via File

Source: https://www.kube-router.io/docs/bgp

Specifies a file containing base64 encoded BGP peer passwords. The file format mirrors command-line and annotation syntax, with a comma separating entries. Only passwords should be present in the file.

```bash
--peer-router-passwords-file="/etc/kube-router/bgp-passwords.conf"
```

```text
U2VjdXJlUGFzc3dvcmQK,
```

--------------------------------

### Kube-router Cluster Role Permissions (Kubernetes RBAC)

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-hostport

This ClusterRole defines the permissions required by kube-router to manage network resources within the Kubernetes cluster. It includes rules for listing, getting, and watching various API resources like namespaces, pods, services, nodes, endpoints, network policies, and leases for coordination. It also allows updating service status.

```yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-router
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    resources:
      - namespaces
      - pods
      - services
      - nodes
      - endpoints
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - "networking.k8s.io"
    resources:
      - networkpolicies
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - extensions
    resources:
      - networkpolicies
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "coordination.k8s.io"
    resources:
      - leases
    verbs:
      - get
      - create
      - update
  - apiGroups:
      - ""
    resources:
      - services/status
    verbs:
      - update
  -
```

--------------------------------

### Create and Commit Changes to Feature Branch (Bash)

Source: https://www.kube-router.io/docs/CONTRIBUTING

Steps to create a new feature branch, make code modifications, stage changes, and commit them with a descriptive message. This is a standard Git workflow for developing new features or fixes.

```bash
$ git checkout -b feature_x
   (make your changes)
$ git status
$ git add .
$ git commit -a -m "descriptive commit message for your changes"
```

--------------------------------

### List Kube-router Pods and Nodes

Source: https://www.kube-router.io/docs/pod-toolbox

This command lists all kube-router pods along with the nodes they are running on. This is useful for identifying which node to investigate further. It uses kubectl to query pod information.

```bash
kubectl -n kube-system get pods -l k8s-app=kube-router -o wide
```

--------------------------------

### Cleanup Kube-proxy iptables Rules (Containerd)

Source: https://www.kube-router.io/docs/generic

Cleans up iptables rules left by kube-proxy using `ctr` commands with Containerd. It pulls the necessary image and runs a container with privileged access and host module mounts for cleanup.

```containerd
ctr images pull k8s.gcr.io/kube-proxy-amd64:v1.28.2
ctr run --rm --privileged --net-host --mount type=bind,src=/lib/modules,dst=/lib/modules,options=rbind:ro \
    registry.k8s.io/kube-proxy-amd64:v1.28.2 kube-proxy-cleanup kube-proxy --cleanup

```

--------------------------------

### Cleanup kube-proxy using Docker

Source: https://www.kube-router.io/docs/kubeadm

Executes a kube-proxy container with the `--cleanup` flag to remove any iptables rules or configurations previously set by kube-proxy. Requires privileged access and host networking.

```docker
docker run --privileged -v /lib/modules:/lib/modules --net=host registry.k8s.io/kube-proxy-amd64:v1.28.2 kube-proxy --cleanup
```

--------------------------------

### Cleanup kube-proxy using CRI-O

Source: https://www.kube-router.io/docs/kubeadm

Employs the `crictl` command to pull the kube-proxy image and execute it with the `--cleanup` flag in a CRI-O runtime environment. This action removes existing kube-proxy iptables configurations.

```cri-o
crictl pull registry.k8s.io/kube-proxy-amd64:v1.28.2
crictl run --rm --privileged --net-host --mount type=bind,src=/lib/modules,dst=/lib/modules,options=rbind:ro
    registry.k8s.io/kube-proxy-amd64:v1.28.2 kube-proxy-cleanup kube-proxy --cleanup
```

--------------------------------

### Kube-router DaemonSet for Containerd DSR

Source: https://www.kube-router.io/docs/dsr

This Kubernetes DaemonSet manifest snippet shows the specific configurations required for Kube-router to function in DSR mode with Containerd. It highlights the necessary volume mounts for the Containerd socket and rt_tables, as well as the `--runtime-endpoint` argument for the kube-router container.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-router
spec:
  template:
    spec:
    ...
      volumes:
      - name: containerd-sock
        hostPath:
          path: /run/containerd/
      - name: rt-tables
        hostPath:
          path: /etc/iproute2/rt_tables
      ...
      containers:
      - name: kube-router
        args:
        - --runtime-endpoint=unix:///run/containerd/containerd.sock
        ...
        volumeMounts:
        - name: containerd-sock
          mountPath: /run/containerd/
          readOnly: true
        - name: rt-tables
          mountPath: /etc/iproute2/rt_tables
          readOnly: false
...
```

--------------------------------

### Display IPVS Cumulative Traffic Statistics

Source: https://www.kube-router.io/docs/pod-toolbox

The `ipvsadm -ln --stats` command displays cumulative traffic statistics for IPVS services. This provides historical traffic data for analysis. Requires the ipvsadm tool.

```bash
ipvsadm -ln --stats
```

--------------------------------

### Display IPVS Traffic Rate Information

Source: https://www.kube-router.io/docs/pod-toolbox

The `ipvsadm -ln --rate` command shows the current traffic rate for IPVS services. This helps in identifying services with high or low traffic volumes. Requires the ipvsadm tool.

```bash
ipvsadm -ln --rate
```

--------------------------------

### Kubernetes ClusterRoleBinding for Kube-router

Source: https://www.kube-router.io/docs/manifests/kubeadm-kuberouter-all-features-hostport

Binds the 'kube-router' ClusterRole to the 'kube-router' ServiceAccount in the 'kube-system' namespace. This grants the permissions defined in the ClusterRole to the kube-router pod.

```yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kube-router
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-router
subjects:
- kind: ServiceAccount
  name: kube-router
  namespace: kube-system
```

--------------------------------

### Configure Kube-controller-manager for Kube-router

Source: https://www.kube-router.io/docs/generic

Configures the kube-controller-manager to allocate pod CIDRs for Kube-router. This involves setting the `--allocate-node-cidrs` and `--cluster-cidr` flags, and optionally `--service-cluster-ip-range`.

```bash
--allocate-node-cidrs=true
--cluster-cidr=10.32.0.0/12
--service-cluster-ip-range=10.50.0.0/22

```

--------------------------------

### Enable Override Next Hop for Advertised Routes

Source: https://www.kube-router.io/docs/bgp

Enables the `--override-nexthop` flag, which leverages GoBGP's next-hop-self functionality. This automatically selects the appropriate next hop when advertising routes, especially useful for nodes with multiple interfaces connecting to external peers.

```bash
--override-nexthop=true
```

--------------------------------

### Cleanup Kube-proxy iptables Rules (Docker)

Source: https://www.kube-router.io/docs/generic

Cleans up iptables rules potentially left behind by kube-proxy using a Docker container. This command requires privileged access and mounts host modules to ensure proper cleanup.

```docker
docker run --privileged -v /lib/modules:/lib/modules --net=host registry.k8s.io/kube-proxy-amd64:v1.28.2 kube-proxy --cleanup

```

--------------------------------

### Add New Kube-router Annotation for Service

Source: https://www.kube-router.io/docs/upgrading

This command demonstrates how to add a new annotation to a service, specifically for updating Kube-router configurations like hairpin mode, as part of the v0.0.17 upgrade.

```bash
kubectl annotate service hairpin-service "kube-router.io/service.hairpin="

```

--------------------------------

### Enabling Dual-Stack in kube-router Configuration

Source: https://www.kube-router.io/docs/ipv6

This snippet illustrates the command-line options required to enable IPv6 and dual-stack support in kube-router. It includes enabling both IPv4 and IPv6, alongside configuring service cluster IP ranges for both address families.

```bash
--enable-ipv4=true
--enable-ipv6=true
--service-cluster-ip-range=10.96.0.0/16,2001:db8:42:1::/112
--service-external-ip-range=10.96.0.0/16,2001:db8:42:1::/112
```
Tokens:
Raw