SuperEdge

https://superedge.io/docs
Admin
SuperEdge is an open-source container management system for edge computing that extends native...
Tokens:41,599
Snippets:218
Trust Score:6.5
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Install and configure Haproxy

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Install Haproxy on the load balancing machine and create the configuration file /etc/haproxy/haproxy.cfg. Replace <master VIP> with the user's own VIP in the backend server line.

```bash
# yum install -y haproxy
# cat << EOF >/etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
frontend  main *:5000
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    acl url_static       path_end       -i .jpg .gif .png .css .js

    use_backend static          if url_static
    default_backend             app

frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server  master-0  <master VIP>:6443 check # Here replace the master VIP with the user's own VIP
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check
backend app
    balance     roundrobin
    server  app1 127.0.0.1:5001 check
    server  app2 127.0.0.1:5002 check
    server  app3 127.0.0.1:5003 check
    server  app4 127.0.0.1:5004 check
EOF

```

--------------------------------

### Start TCP server for local debugging

Source: https://superedge.io/docs/components/tunnel

Starts the TCP server for local debugging. It initializes the configuration, modules, TCP module (InitTcp), and stream module (stream.InitStream) simultaneously, then loads modules and shuts down.

```go
func Test_TcpServer(t *testing.T) {
	err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml")
	if err != nil {
		t.Errorf("failed to initialize stream server configuration file err = %v", err)
		return
	}
	model.InitModules(util.CLOUD)
	InitTcp()
	stream.InitStream(util.CLOUD)
	model.LoadModules(util.CLOUD)
	model.ShutDown()
}
```

--------------------------------

### Install and configure Keepalived

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Install Keepalived on both master nodes and create /etc/keepalived/keepalived.conf. Replace <master VIP>, <master Public IP>, and <Public IP of other master nodes> with actual values. Note that the public IPs are opposite on the two masters.

```bash
# yum install -y keepalived
# cat << EOF >/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   smtp_connect_timeout 30
   router_id LVS_DEVEL_EDGE_1
}
vrrp_script checkhaproxy{
script "/etc/keepalived/do_sth.sh"
interval 5
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    nopreempt
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass aaa
    }
    virtual_ipaddress {
        <master VIP> # Here replace the master VIP with the user's own VIP
    }
    unicast_src_ip <master Public IP>
    unicast_peer {
      <Public IP of other master nodes>
    }
notify_master "/etc/keepalived/notify_action.sh MASTER"
notify_backup "/etc/keepalived/notify_action.sh BACKUP"
notify_fault "/etc/keepalived/notify_action.sh FAULT"
notify_stop "/etc/keepalived/notify_action.sh STOP"
garp_master_delay 1
garp_master_refresh 5
   track_interface {
     eth0
   }
   track_script {
     checkhaproxy
   }
}
EOF

```

--------------------------------

### Start TCP client for local debugging

Source: https://superedge.io/docs/components/tunnel

Starts the TCP client for local debugging. It sets the node name environment variable, initializes the configuration, modules, TCP module (InitTcp), and stream module (stream.InitStream) simultaneously, then loads modules and shuts down.

```go
func Test_TcpClient(t *testing.T) {
	os.Setenv(util.NODE_NAME_ENV, "node1")
	err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml")
	if err != nil {
		t.Errorf("failed to initialize stream client configuration file err = %v", err)
		return
	}
	model.InitModules(util.EDGE)
	InitTcp()
	stream.InitStream(util.EDGE)
	model.LoadModules(util.EDGE)
	model.ShutDown()
}
```

--------------------------------

### Initialize edge Kubernetes master node with edgeadm init

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Use this command to initialize an edge Kubernetes master node. Set --install-pkg-path to the static installation package path (local or URL), --apiserver-cert-extra-sans to a reachable master IP/domain, and optionally adjust --image-repository. --enable-edge=true is default; set false to install native Kubernetes. On success, output shows join instructions.

```bash
./edgeadm init --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --install-pkg-path ./kube-linux-*.tar.gz --apiserver-cert-extra-sans=<Master public IP> --apiserver-advertise-address=<master Intranet IP> --enable-edge=true -v=6

```

```bash
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
    --install-pkg-path <Path of edgeadm kube-* install package>

```

--------------------------------

### Start stream client for local debugging

Source: https://superedge.io/docs/components/tunnel

Starts the stream client for local debugging. It sets the node name environment variable (NODE_NAME), initializes the configuration, modules, and stream module, registers a custom handler, and then runs a goroutine that sends a StreamMsg every 10 seconds. The node name is loaded from the NODE_NAME environment variable.

```go
func Test_StreamClient(t *testing.T) {
	os.Setenv(util.NODE_NAME_ENV, "node1")
	err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml")
	if err != nil {
		t.Errorf("failed to initialize stream client configuration file err = %v", err)
		return
	}
	model.InitModules(util.EDGE)
	InitStream(util.EDGE)
	model.LoadModules(util.EDGE)
	context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler)
	go func() {
		running := true
		for running {
			node := context.GetContext().GetNode(os.Getenv(util.NODE_NAME_ENV))
			if node != nil {
				node.Send2Node(&proto.StreamMsg{
					Node:     os.Getenv(util.NODE_NAME_ENV),
					Category: util.STREAM,
					Type:     util.MODULE_DEBUG,
					Topic:    uuid.NewV4().String(),
					Data:     []byte{'c'},
				})
			}
			time.Sleep(10 * time.Second)
		}
	}()
	model.ShutDown()

}
```

```text
Set the node name environment variable->Load configuration file (conf.InitConf)->Initialization module (model.InitMoudule)->Initialize the stream module (InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown)
```

--------------------------------

### Start stream server for local debugging

Source: https://superedge.io/docs/components/tunnel

Starts the stream server for local debugging. It initializes the configuration, modules, and stream module, registers a custom handler (StreamDebugHandler) for cloud-side messaging, and then shuts down. The configuration file is generated by calling the config_test test method Test_Config, with config_path and main_path set appropriately.

```go
func Test_StreamServer(t *testing.T) {
	err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml")
	if err != nil {
		t.Errorf("failed to initialize stream server configuration file err = %v", err)
		return
	}
	model.InitModules(util.CLOUD)
	InitStream(util.CLOUD)
	model.LoadModules(util.CLOUD)
	context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler)
	model.ShutDown()

}
```

```text
Load configuration file(conf.InitConf)->Initialize the module(model.InitMoudule)->Initialize the stream module(InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown)
```

--------------------------------

### Download and extract edgeadm static installation package

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Downloads the edgeadm installation package for the specified architecture and version, extracts it, and runs the edgeadm command. Choose the package based on your node's CPU architecture (amd64 or arm64). The package is about 200M.

```bash
arch=amd64 version=v0.3.0 && rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version && ./edgeadm

```

--------------------------------

### Initialize high-availability edge Kubernetes master with edgeadm init

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Run this command on one of the master nodes to initialize the cluster. The parameters are the same as in '3. Use edgeadm to install edge Kubernetes cluster' and kubeadm. On success, the output includes the 'edgeadm join' command needed to add master and edge nodes later.

```bash
./edgeadm init --control-plane-endpoint <Master VIP> --upload-certs --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans=<Domain or Public/Intranet IP of Master node> --install-pkg-path <edgeadm Kube-*Static installation package address/FTP path> -v=6

```

--------------------------------

### Generate lite-apiserver key and certificate using Kubernetes CA

Source: https://superedge.io/docs/installation/install-manually

Commands to generate lite-apiserver key and certificate. First, get the ClusterIP of the 'kubernetes' service. Then generate a private key, create a configuration file with subjectAltName including the ClusterIP, generate a CSR, and finally sign it with the cluster's ca.crt and ca.key.

```bash
#get service 'kubernetes' ClusterIP
$ kubectl get service kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.10.0.1    <none>        443/TCP   23d

#Generate lite-apiserver.key
$ openssl genrsa -out lite-apiserver.key 2048

#create lite-apiserver.csr
$ cat << EOF >lite_apiserver.conf
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
CN = lite-apiserver
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
IP.1 = 127.0.0.1
IP.2 = 10.10.0.1 # please change the value to Kubernetes's Cluster IP
EOF

$ openssl req -new -key lite-apiserver.key -subj "/CN=lite-apiserver" -config lite-apiserver.conf -out lite-apiserver.csr

#generating lite-apiserver.crt
openssl x509 -req -in lite-apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 5000 -extensions v3_req -extfile lite-apiserver.conf -out lite-apiserver.crt

```

--------------------------------

### make build

Source: https://superedge.io/docs/contributionguidelines

Builds the project.

```bash
make build
```

--------------------------------

### make manifest.multiarch

Source: https://superedge.io/docs/contributionguidelines

Pushes multi-architecture manifests to a registry.

```bash
make manifest.multiarch IMAGES="application-grid-wrapper application-grid-controller" PLATFORMS="linux_amd64 linux_arm64" REGISTRY_PREFIX="docker.io/superedge" VERSION="v2.0.0"
```

--------------------------------

### Kubernetes static installation package structure

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Directory structure of the Kubernetes static installation package for kube-v1.18.2 on arm64. It contains binaries (conntrack, kubectl, kubelet, lite-apiserver), CNI plugins, and Docker 19.03 container runtime.

```text
kube-linux-arm64-v1.18.2.tar.gz ## Kubernetes static installation package for kube-v1.18.2
├── bin                         ## Binary directory
│   ├── conntrack               ## Binary file for connection tracking
│   ├── kubectl                 ## kubectl for kube-v1.18.2
│   ├── kubelet                 ## kubelet for kube-v1.18.2
│   └── lite-apiserver          ## The corresponding version of lite-apiserver
├── cni                         ## cin configuration
│   └── cni-plugins-linux-v0.8.3.tar.gz ## CNI plug-in binary compression package of v0.8.3
└── container                   ## Container runtime directory
    └── docker-19.03-linux-arm64.tar.gz ## Docker 19.03 arm64 system installation script and installation package

```

--------------------------------

### make - format, build and test

Source: https://superedge.io/docs/contributionguidelines

Runs the default make target to format, build, and test the project.

```bash
make
```

--------------------------------

### Get base64 encoded persistent connection certifications

Source: https://superedge.io/docs/installation/install-manually

Outputs the BASE64 encoded values for TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt.

```bash
# generate TunnelPersistentConnectionServerKey
cat tunnel_persistent_connectiong_server.key | base64 --wrap=0
#generate TunnelPersistentConnectionServerCrt
cat tunnel_persistent_connectiong_server.crt | base64 --wrap=0

```

--------------------------------

### make build.multiarch

Source: https://superedge.io/docs/contributionguidelines

Builds commands for multiple architectures using the PLATFORMS variable.

```bash
make build.multiarch BINS="lite-apiserver edgeadm" PLATFORMS="linux_amd64 linux_arm64"
```

--------------------------------

### Use edgeadm change with modified manifests

Source: https://superedge.io/docs/installation/install-via-edgeadm

After modifying the yamls in ./manifests/, use this command to deploy the changes. For example, after changing the edge-health image to edge-health:0.1.0.

```bash
[root@master01 ~]# edgeadm change -m ./manifests/

```

--------------------------------

### make image.multiarch

Source: https://superedge.io/docs/contributionguidelines

Builds multi-architecture Docker images with version specification.

```bash
make image.multiarch IMAGES="application-grid-controller application-grid-wrapper" PLATFORMS="linux_amd64 linux_arm64" VERSION="v2.0.0"
```

--------------------------------

### Verify ServiceGrid deployment with kubectl

Source: https://superedge.io/docs/components/servicegroup

Shows the expected output of kubectl get commands to verify the ServiceGrid, StatefulSets, and Services. The Service is a ClusterIP with IP 192.168.21.99.

```bash
[~]## kubectl get ssg
NAME                   AGE
statefulsetgrid-demo   21h

[~]## kubectl get statefulset
NAME                          READY   AGE
statefulsetgrid-demo-zone-0   3/3     21h
statefulsetgrid-demo-zone-1   3/3     21h
statefulsetgrid-demo-zone-2   3/3     21h

[~]## kubectl get svc
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes             ClusterIP   192.168.0.1     <none>        443/TCP   22h
servicegrid-demo-svc   ClusterIP   192.168.21.99   <none>        80/TCP    21h

```

--------------------------------

### Point kube-apiserver DNS to tunnel-coredns

Source: https://superedge.io/docs/installation/install-manually

Gets the tunnel-coredns Cluster IP and shows how to configure kube-apiserver's dnsConfig to use it. The output shows the service details.

```bash
#Get tunnel-coredns's Cluster IP
$ kubectl get service tunnel-coredns -n edge-system
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
tunnel-coredns   ClusterIP   10.10.47.74   <none>        53/UDP,53/TCP,9153/TCP   140m
#Replace kube-apierver's DNS nameservers with tunnel-coredns's Cluster IP
...
dnsConfig:
    nameservers:
    - 10.10.47.74 #tunnel-cloud's CLUSTER IP；  
...

```

--------------------------------

### make image

Source: https://superedge.io/docs/contributionguidelines

Builds Docker images for specified components.

```bash
make image IMAGES="application-grid-controller application-grid-wrapper"
```

--------------------------------

### ApplicationGrid Wrapper Sample Configuration (serviceGroup/application-grid-wrapper.yaml)

Source: https://superedge.io/docs/reference/servicegroup/application-grid-wrapper

This YAML defines the full deployment for the ApplicationGrid wrapper: a ServiceAccount, RBAC rules, a ConfigMap with a kubeconfig, and a DaemonSet. The DaemonSet runs the wrapper with a kubeconfig pointing to 127.0.0.1:51003 and binds to 127.0.0.1:51006. It requires a node selector for linux and tolerates the master taint.

```yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: application-grid-wrapper
  namespace: edge-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: edge:application-grid-wrapper
rules:
  - apiGroups:
      - ""
    resources:
      - endpoints
      - services
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
      - events.k8s.io
    resources:
      - events
    verbs:
      - create
      - patch
      - update
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: edge:application-grid-wrapper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edge:application-grid-wrapper
subjects:
  - kind: ServiceAccount
    name: application-grid-wrapper
    namespace: edge-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: application-grid-wrapper
  namespace: edge-system
data:
  kubeconfig.conf: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:51003
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    kind: Config
    preferences: {}
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token    

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: application-grid-wrapper
    addonmanager.kubernetes.io/mode: Reconcile
  name: application-grid-wrapper
  namespace: edge-system
spec:
  selector:
    matchLabels:
      k8s-app: application-grid-wrapper
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
  template:
    metadata:
      labels:
        k8s-app: application-grid-wrapper
    spec:
      serviceAccount: application-grid-wrapper
      serviceAccountName: application-grid-wrapper
      priorityClassName: system-node-critical
      hostNetwork: true
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/os: linux # TODO select edge node
      tolerations:
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: application-grid-wrapper
          image: superedge/application-grid-wrapper:v0.3.0
          imagePullPolicy: IfNotPresent
          command:
            - /usr/local/bin/application-grid-wrapper
            - --kubeconfig=/var/lib/application-grid-wrapper/kubeconfig.conf
            - --bind-address=127.0.0.1:51006
            - --hostname=$(NODE_NAME)
            - --notify-channel-size=10000
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          resources:
            limits:
              cpu: 50m
              memory: 100Mi
            requests:
              cpu: 10m
              memory: 20Mi
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /var/lib/application-grid-wrapper
              name: application-grid-wrapper
      volumes:
        - configMap:
            defaultMode: 420
            name: application-grid-wrapper
          name: application-grid-wrapper
        - hostPath:
            path: /var/tmp
            type: Directory
          name: host-var-tmp

```

--------------------------------

### Successful edgeadm join output

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Expected output when the new master successfully joins the cluster. It includes instructions for setting up kubectl and running 'kubectl get nodes' to verify the node joined.

```text
This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

```

--------------------------------

### Success message after edgeadm join

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Expected output after a successful edgeadm join operation. The node is added to the cluster and the control-plane node can verify it with 'kubectl get nodes'. On failure, an error message is returned and the join is interrupted.

```text
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

```

--------------------------------

### Generate Kubelet client key and certificate for tunnel-edge authentication

Source: https://superedge.io/docs/installation/install-manually

OpenSSL commands to create a private key and self-signed certificate for tunnel-edge to authenticate with Kubelet. The certificate is signed using the Kubernetes cluster's ca.crt and ca.key, typically found at /etc/kubernetes/pki in kubeadm installs.

```bash
# private key
openssl genrsa -des3 -out kubelet_client.key 1024
# generate csr
openssl req -new -key kubelet_client.key -out kubelet_client.csr

# Generate Self Signed certificate（Notice: it is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki）
openssl ca -in kubelet-client.csr -out kubelet-client.crt -cert ca.crt -keyfile ca.key

```

--------------------------------

### Configure tunnel-edge parameters in deployment/tunnel-edge.yaml

Source: https://superedge.io/docs/installation/install-manually

Lists the required parameters to set in the tunnel-edge deployment file: MasterIP, TunnelCloudEdgeToken, TunnelPersistentConnectionPort, KubernetesCaCert (base64), KubeletClientKey and KubeletClientCrt.

```yaml
MasterIP:  #Normal Kubernetes master node's IP or domain(currently, only one IP address or domain is supported)
TunnelCloudEdgeToken:  #Fill in the same token as "TunnelCloudEdgeToken" in Tunnel-cloud
TunnelPersistentConnectionPort:  #Tunnel-cloud's Persistent connection server Port
KubernetesCaCert:  #kube-apiserver's ca.crt(base64 encoded)
KubeletClientKey:  #Kubelet client key for Tunnel-edge to access Kubelet
KubeletClientCrt:  #Kubelet client cert for Tunnel-edge to access Kubelet

```

--------------------------------

### Successful edgeadm init output

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Expected output when the cluster initializes successfully. It includes instructions for setting up kubectl and the 'edgeadm join' commands for adding control-plane and worker nodes. The certificate-key is sensitive and uploaded certs are deleted after two hours.

```text
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \
    --control-plane --certificate-key xxxxxxxxxx
    --install-pkg-path <Path of edgeadm kube-* install package>

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"edgeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \
    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx  
    --install-pkg-path <Path of edgeadm kube-* install package>

```

--------------------------------

### DeploymentGrid canary deployment YAML

Source: https://superedge.io/docs/components/servicegroup

Example DeploymentGrid manifest using templatePool and templates for canary deployment. The templatePool defines test1 (image 2.2) and test2 (image 2.3). The templates field assigns zone1 to test1 and zone2 to test2. Other NodeGroups use the default template test1.

```yaml
apiVersion: superedge.io/v1
kind: DeploymentGrid
metadata:
  name: deploymentgrid-demo
  namespace: default
spec:
  defaultTemplateName: test1 ## (Optional) Default to "default".
  autoDeleteUnusedTemplate: false ## (Optional). Default to false.
  gridUniqKey: zone
  template:
    ...... ## Omit workload spec. If defaultTemplateName is set to default, this template will be used. Otherwise, it will be ignored.
  templatePool: ## Defines workload templates for NodeUnits
    test1:
      replicas: 2
      selector:
        matchLabels:
          appGrid: echo
      strategy: {}
      template:
        metadata:
          creationTimestamp: null
          labels:
            appGrid: echo
        spec:
          containers:
          - image: superedge/echoserver:2.2 ## Old release
            name: echo
            ## Omit container spec
    test2:
      replicas: 3
      selector:
        matchLabels:
          appGrid: echo
      strategy: {}
      template:
        metadata:
          creationTimestamp: null
          labels:
            appGrid: echo
        spec:
          containers:
          - image: superedge/echoserver:2.3  ## New release
            name: echo
            ## Omit container spec
  templates: ## Assigns workload templates to NodeUnits.
    ## <NodeUnit_name>: <template_name_from_templatePool>
    zone1: test1
    zone2: test2

```

--------------------------------

### Deploy Application Grid Controller on master nodes

Source: https://superedge.io/docs/installation/install-manually

Command to apply the application-grid-controller.yaml on master nodes.

```bash
$ kubectl apply -f deployment/application-grid-controller.yaml

```

--------------------------------

### statefulset-grid-daemon.yaml

Source: https://superedge.io/docs/reference/servicegroup/statefulset-grid-daemon

Complete sample configuration for the statefulset-grid daemon. It defines a ServiceAccount, RBAC rules, a ConfigMap with a kubeconfig pointing to https://127.0.0.1:51003, and a DaemonSet that runs the daemon with hostNetwork and privileged access. The DaemonSet uses the image superedge/statefulset-grid-daemon:v0.3.0 and mounts the ConfigMap as a volume.

```yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: statefulset-grid-daemon
  namespace: edge-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: statefulset-grid-daemon
  namespace: edge-system
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - services
      - pods
    verbs:
      - "*"
  - apiGroups:
      - extensions
      - apps
    resources:
      - statefulsets
    verbs:
      - "*"
  - apiGroups:
      - superedge.io
    resources:
      - statefulsetgrids
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: statefulset-grid-daemon
  namespace: edge-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: statefulset-grid-daemon
subjects:
  - kind: ServiceAccount
    name: statefulset-grid-daemon
    namespace: edge-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: statefulset-grid-daemon
  namespace: edge-system
data:
  kubeconfig.conf: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:51003
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    kind: Config
    preferences: {}
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token    
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: statefulset-grid-daemon
    addonmanager.kubernetes.io/mode: Reconcile
  name: statefulset-grid-daemon
  namespace: edge-system
spec:
  selector:
    matchLabels:
      k8s-app: statefulset-grid-daemon
  template:
    metadata:
      labels:
        k8s-app: statefulset-grid-daemon
    spec:
      serviceAccount: statefulset-grid-daemon
      serviceAccountName: statefulset-grid-daemon
      priorityClassName: system-node-critical
      hostNetwork: true
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/os: linux # TODO select edge node
      tolerations:
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: statefulset-grid-daemon
          image: superedge/statefulset-grid-daemon:v0.3.0
          imagePullPolicy: IfNotPresent
          command:
            - /usr/local/bin/statefulset-grid-daemon
            - --hostname=$(NODE_NAME)
            - --kubeconfig=/var/lib/statefulset-grid-daemon/kubeconfig.conf
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          resources:
            limits:
              cpu: 50m
              memory: 100Mi
            requests:
              cpu: 10m
              memory: 20Mi
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /data/edge
              name: hosts
            - mountPath: /var/lib/statefulset-grid-daemon
              name: statefulset-grid-daemon
      volumes:
        - configMap:
            defaultMode: 420
            name: statefulset-grid-daemon
          name: statefulset-grid-daemon
        - hostPath:
            path: /data/edge
            type: DirectoryOrCreate
          name: hosts

```

--------------------------------

### Access echoserver and verify pod info

Source: https://superedge.io/docs/components/lite-apiserver

Commands to list pods and curl the echoserver, showing the pod name, namespace, and IP. The output confirms the echoserver is running and accessible.

```bash
$ kubectl get pods -owide
NAME                        READY   STATUS    RESTARTS   AGE     IP         NODE           NOMINATED NODE   READINESS GATES
lite-demo-c7b458ddc-6lpnx   1/1     Running   0          4m23s   10.0.6.2   ecm-q5h6hhd   <none>           <none>

$ curl http://10.0.6.2:8080 | grep pod
	pod name:	lite-demo-c7b458ddc-6lpnx
	pod namespace:	default
	pod IP:	10.0.6.2
```

--------------------------------

### Join edge node with edgeadm join

Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster

Executes the edgeadm join command on an edge node to join it to a Kubernetes cluster. Specify the master node's address, token, CA cert hash, and the path to the static installation package. Setting --enable-edge=true (default) deploys edge capability components, while false joins as a native Kubernetes node.

```bash
./edgeadm join <Master public/Intranet IP or domain>:Port --token xxxx \
     --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
     --install-pkg-path <edgeadm Kube-*Static installation package address/FTP path>
     --enable-edge=true

```

--------------------------------

### Deploy StatefulSetGrid YAML

Source: https://superedge.io/docs/components/servicegroup

Use this YAML to create a StatefulSetGrid custom resource. The manifest defines a grid with key 'zone', a template for the statefulset with 3 replicas, and an echoserver container. The serviceName field must be set to the service to be created.

```yaml
apiVersion: superedge.io/v1
kind: StatefulSetGrid
metadata:
  name: statefulsetgrid-demo
  namespace: default
spec:
  gridUniqKey: zone
  template:
    selector:
      matchLabels:
        appGrid: echo
    serviceName: "servicegrid-demo-svc"
    replicas: 3
    template:
      metadata:
        labels:
          appGrid: echo
      spec:
        terminationGracePeriodSeconds: 10
        containers:
        - image: superedge/echoserver:2.2
          name: echo
          ports:
          - containerPort: 8080
            protocol: TCP
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          resources: {}

```

--------------------------------

### Create /etc/kubernetes/edge/tls.json for lite-apiserver

Source: https://superedge.io/docs/installation/install-manually

JSON configuration file specifying the key and certificate paths for lite-apiserver to authenticate with kubelet. Uses kubelet-client-current.pem for both key and cert.

```json
[
    {
        "key":"/var/lib/kubelet/pki/kubelet-client-current.pem",
        "cert":"/var/lib/kubelet/pki/kubelet-client-current.pem"
    }
]

```
Tokens:
Raw