### Install and configure Haproxy Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Install Haproxy on the load balancing machine and create the configuration file /etc/haproxy/haproxy.cfg. Replace with the user's own VIP in the backend server line. ```bash # yum install -y haproxy # cat << EOF >/etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend main *:5000 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js use_backend static if url_static default_backend app frontend kubernetes-apiserver mode tcp bind *:16443 option tcplog default_backend kubernetes-apiserver backend kubernetes-apiserver mode tcp balance roundrobin server master-0 :6443 check # Here replace the master VIP with the user's own VIP backend static balance roundrobin server static 127.0.0.1:4331 check backend app balance roundrobin server app1 127.0.0.1:5001 check server app2 127.0.0.1:5002 check server app3 127.0.0.1:5003 check server app4 127.0.0.1:5004 check EOF ``` -------------------------------- ### Start TCP server for local debugging Source: https://superedge.io/docs/components/tunnel Starts the TCP server for local debugging. It initializes the configuration, modules, TCP module (InitTcp), and stream module (stream.InitStream) simultaneously, then loads modules and shuts down. ```go func Test_TcpServer(t *testing.T) { err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml") if err != nil { t.Errorf("failed to initialize stream server configuration file err = %v", err) return } model.InitModules(util.CLOUD) InitTcp() stream.InitStream(util.CLOUD) model.LoadModules(util.CLOUD) model.ShutDown() } ``` -------------------------------- ### Install and configure Keepalived Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Install Keepalived on both master nodes and create /etc/keepalived/keepalived.conf. Replace , , and with actual values. Note that the public IPs are opposite on the two masters. ```bash # yum install -y keepalived # cat << EOF >/etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { smtp_connect_timeout 30 router_id LVS_DEVEL_EDGE_1 } vrrp_script checkhaproxy{ script "/etc/keepalived/do_sth.sh" interval 5 } vrrp_instance VI_1 { state BACKUP interface eth0 nopreempt virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass aaa } virtual_ipaddress { # Here replace the master VIP with the user's own VIP } unicast_src_ip unicast_peer { } notify_master "/etc/keepalived/notify_action.sh MASTER" notify_backup "/etc/keepalived/notify_action.sh BACKUP" notify_fault "/etc/keepalived/notify_action.sh FAULT" notify_stop "/etc/keepalived/notify_action.sh STOP" garp_master_delay 1 garp_master_refresh 5 track_interface { eth0 } track_script { checkhaproxy } } EOF ``` -------------------------------- ### Start TCP client for local debugging Source: https://superedge.io/docs/components/tunnel Starts the TCP client for local debugging. It sets the node name environment variable, initializes the configuration, modules, TCP module (InitTcp), and stream module (stream.InitStream) simultaneously, then loads modules and shuts down. ```go func Test_TcpClient(t *testing.T) { os.Setenv(util.NODE_NAME_ENV, "node1") err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml") if err != nil { t.Errorf("failed to initialize stream client configuration file err = %v", err) return } model.InitModules(util.EDGE) InitTcp() stream.InitStream(util.EDGE) model.LoadModules(util.EDGE) model.ShutDown() } ``` -------------------------------- ### Initialize edge Kubernetes master node with edgeadm init Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Use this command to initialize an edge Kubernetes master node. Set --install-pkg-path to the static installation package path (local or URL), --apiserver-cert-extra-sans to a reachable master IP/domain, and optionally adjust --image-repository. --enable-edge=true is default; set false to install native Kubernetes. On success, output shows join instructions. ```bash ./edgeadm init --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --install-pkg-path ./kube-linux-*.tar.gz --apiserver-cert-extra-sans= --apiserver-advertise-address= --enable-edge=true -v=6 ``` ```bash Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxx --install-pkg-path ``` -------------------------------- ### Start stream client for local debugging Source: https://superedge.io/docs/components/tunnel Starts the stream client for local debugging. It sets the node name environment variable (NODE_NAME), initializes the configuration, modules, and stream module, registers a custom handler, and then runs a goroutine that sends a StreamMsg every 10 seconds. The node name is loaded from the NODE_NAME environment variable. ```go func Test_StreamClient(t *testing.T) { os.Setenv(util.NODE_NAME_ENV, "node1") err := conf.InitConf(util.EDGE, "../../../../conf/edge_mode.toml") if err != nil { t.Errorf("failed to initialize stream client configuration file err = %v", err) return } model.InitModules(util.EDGE) InitStream(util.EDGE) model.LoadModules(util.EDGE) context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler) go func() { running := true for running { node := context.GetContext().GetNode(os.Getenv(util.NODE_NAME_ENV)) if node != nil { node.Send2Node(&proto.StreamMsg{ Node: os.Getenv(util.NODE_NAME_ENV), Category: util.STREAM, Type: util.MODULE_DEBUG, Topic: uuid.NewV4().String(), Data: []byte{'c'}, }) } time.Sleep(10 * time.Second) } }() model.ShutDown() } ``` ```text Set the node name environment variable->Load configuration file (conf.InitConf)->Initialization module (model.InitMoudule)->Initialize the stream module (InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown) ``` -------------------------------- ### Start stream server for local debugging Source: https://superedge.io/docs/components/tunnel Starts the stream server for local debugging. It initializes the configuration, modules, and stream module, registers a custom handler (StreamDebugHandler) for cloud-side messaging, and then shuts down. The configuration file is generated by calling the config_test test method Test_Config, with config_path and main_path set appropriately. ```go func Test_StreamServer(t *testing.T) { err := conf.InitConf(util.CLOUD, "../../../../conf/cloud_mode.toml") if err != nil { t.Errorf("failed to initialize stream server configuration file err = %v", err) return } model.InitModules(util.CLOUD) InitStream(util.CLOUD) model.LoadModules(util.CLOUD) context.GetContext().RegisterHandler(util.MODULE_DEBUG, util.STREAM, StreamDebugHandler) model.ShutDown() } ``` ```text Load configuration file(conf.InitConf)->Initialize the module(model.InitMoudule)->Initialize the stream module(InitStream)->Load the initialized module->Register a custom handler (StreamDebugHandler)->Shut down the module (model.ShutDown) ``` -------------------------------- ### Download and extract edgeadm static installation package Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Downloads the edgeadm installation package for the specified architecture and version, extracts it, and runs the edgeadm command. Choose the package based on your node's CPU architecture (amd64 or arm64). The package is about 200M. ```bash arch=amd64 version=v0.3.0 && rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version && ./edgeadm ``` -------------------------------- ### Initialize high-availability edge Kubernetes master with edgeadm init Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Run this command on one of the master nodes to initialize the cluster. The parameters are the same as in '3. Use edgeadm to install edge Kubernetes cluster' and kubeadm. On success, the output includes the 'edgeadm join' command needed to add master and edge nodes later. ```bash ./edgeadm init --control-plane-endpoint --upload-certs --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans= --install-pkg-path -v=6 ``` -------------------------------- ### Kubernetes static installation package structure Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Directory structure of the Kubernetes static installation package for kube-v1.18.2 on arm64. It contains binaries (conntrack, kubectl, kubelet, lite-apiserver), CNI plugins, and Docker 19.03 container runtime. ```text kube-linux-arm64-v1.18.2.tar.gz ## Kubernetes static installation package for kube-v1.18.2 ├── bin ## Binary directory │ ├── conntrack ## Binary file for connection tracking │ ├── kubectl ## kubectl for kube-v1.18.2 │ ├── kubelet ## kubelet for kube-v1.18.2 │ └── lite-apiserver ## The corresponding version of lite-apiserver ├── cni ## cin configuration │ └── cni-plugins-linux-v0.8.3.tar.gz ## CNI plug-in binary compression package of v0.8.3 └── container ## Container runtime directory └── docker-19.03-linux-arm64.tar.gz ## Docker 19.03 arm64 system installation script and installation package ``` -------------------------------- ### Generate lite-apiserver key and certificate using Kubernetes CA Source: https://superedge.io/docs/installation/install-manually Commands to generate lite-apiserver key and certificate. First, get the ClusterIP of the 'kubernetes' service. Then generate a private key, create a configuration file with subjectAltName including the ClusterIP, generate a CSR, and finally sign it with the cluster's ca.crt and ca.key. ```bash #get service 'kubernetes' ClusterIP $ kubectl get service kubernetes NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.10.0.1 443/TCP 23d #Generate lite-apiserver.key $ openssl genrsa -out lite-apiserver.key 2048 #create lite-apiserver.csr $ cat << EOF >lite_apiserver.conf [req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] CN = lite-apiserver [v3_req] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = localhost IP.1 = 127.0.0.1 IP.2 = 10.10.0.1 # please change the value to Kubernetes's Cluster IP EOF $ openssl req -new -key lite-apiserver.key -subj "/CN=lite-apiserver" -config lite-apiserver.conf -out lite-apiserver.csr #generating lite-apiserver.crt openssl x509 -req -in lite-apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 5000 -extensions v3_req -extfile lite-apiserver.conf -out lite-apiserver.crt ``` -------------------------------- ### Successful edgeadm join output Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Expected output when the new master successfully joins the cluster. It includes instructions for setting up kubectl and running 'kubectl get nodes' to verify the node joined. ```text This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane (master) label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster. ``` -------------------------------- ### DeploymentGrid canary deployment YAML Source: https://superedge.io/docs/components/servicegroup Example DeploymentGrid manifest using templatePool and templates for canary deployment. The templatePool defines test1 (image 2.2) and test2 (image 2.3). The templates field assigns zone1 to test1 and zone2 to test2. Other NodeGroups use the default template test1. ```yaml apiVersion: superedge.io/v1 kind: DeploymentGrid metadata: name: deploymentgrid-demo namespace: default spec: defaultTemplateName: test1 ## (Optional) Default to "default". autoDeleteUnusedTemplate: false ## (Optional). Default to false. gridUniqKey: zone template: ...... ## Omit workload spec. If defaultTemplateName is set to default, this template will be used. Otherwise, it will be ignored. templatePool: ## Defines workload templates for NodeUnits test1: replicas: 2 selector: matchLabels: appGrid: echo strategy: {} template: metadata: creationTimestamp: null labels: appGrid: echo spec: containers: - image: superedge/echoserver:2.2 ## Old release name: echo ## Omit container spec test2: replicas: 3 selector: matchLabels: appGrid: echo strategy: {} template: metadata: creationTimestamp: null labels: appGrid: echo spec: containers: - image: superedge/echoserver:2.3 ## New release name: echo ## Omit container spec templates: ## Assigns workload templates to NodeUnits. ## : zone1: test1 zone2: test2 ``` -------------------------------- ### Join edge node with edgeadm join Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Executes the edgeadm join command on an edge node to join it to a Kubernetes cluster. Specify the master node's address, token, CA cert hash, and the path to the static installation package. Setting --enable-edge=true (default) deploys edge capability components, while false joins as a native Kubernetes node. ```bash ./edgeadm join :Port --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxx --install-pkg-path --enable-edge=true ``` -------------------------------- ### make build Source: https://superedge.io/docs/contributionguidelines Builds the project. ```bash make build ``` -------------------------------- ### make manifest.multiarch Source: https://superedge.io/docs/contributionguidelines Pushes multi-architecture manifests to a registry. ```bash make manifest.multiarch IMAGES="application-grid-wrapper application-grid-controller" PLATFORMS="linux_amd64 linux_arm64" REGISTRY_PREFIX="docker.io/superedge" VERSION="v2.0.0" ``` -------------------------------- ### make - format, build and test Source: https://superedge.io/docs/contributionguidelines Runs the default make target to format, build, and test the project. ```bash make ``` -------------------------------- ### Get base64 encoded persistent connection certifications Source: https://superedge.io/docs/installation/install-manually Outputs the BASE64 encoded values for TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt. ```bash # generate TunnelPersistentConnectionServerKey cat tunnel_persistent_connectiong_server.key | base64 --wrap=0 #generate TunnelPersistentConnectionServerCrt cat tunnel_persistent_connectiong_server.crt | base64 --wrap=0 ``` -------------------------------- ### make build.multiarch Source: https://superedge.io/docs/contributionguidelines Builds commands for multiple architectures using the PLATFORMS variable. ```bash make build.multiarch BINS="lite-apiserver edgeadm" PLATFORMS="linux_amd64 linux_arm64" ``` -------------------------------- ### Use edgeadm change with modified manifests Source: https://superedge.io/docs/installation/install-via-edgeadm After modifying the yamls in ./manifests/, use this command to deploy the changes. For example, after changing the edge-health image to edge-health:0.1.0. ```bash [root@master01 ~]# edgeadm change -m ./manifests/ ``` -------------------------------- ### make image.multiarch Source: https://superedge.io/docs/contributionguidelines Builds multi-architecture Docker images with version specification. ```bash make image.multiarch IMAGES="application-grid-controller application-grid-wrapper" PLATFORMS="linux_amd64 linux_arm64" VERSION="v2.0.0" ``` -------------------------------- ### Verify ServiceGrid deployment with kubectl Source: https://superedge.io/docs/components/servicegroup Shows the expected output of kubectl get commands to verify the ServiceGrid, StatefulSets, and Services. The Service is a ClusterIP with IP 192.168.21.99. ```bash [~]## kubectl get ssg NAME AGE statefulsetgrid-demo 21h [~]## kubectl get statefulset NAME READY AGE statefulsetgrid-demo-zone-0 3/3 21h statefulsetgrid-demo-zone-1 3/3 21h statefulsetgrid-demo-zone-2 3/3 21h [~]## kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 192.168.0.1 443/TCP 22h servicegrid-demo-svc ClusterIP 192.168.21.99 80/TCP 21h ``` -------------------------------- ### Point kube-apiserver DNS to tunnel-coredns Source: https://superedge.io/docs/installation/install-manually Gets the tunnel-coredns Cluster IP and shows how to configure kube-apiserver's dnsConfig to use it. The output shows the service details. ```bash #Get tunnel-coredns's Cluster IP $ kubectl get service tunnel-coredns -n edge-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE tunnel-coredns ClusterIP 10.10.47.74 53/UDP,53/TCP,9153/TCP 140m #Replace kube-apierver's DNS nameservers with tunnel-coredns's Cluster IP ... dnsConfig: nameservers: - 10.10.47.74 #tunnel-cloud's CLUSTER IP； ... ``` -------------------------------- ### make image Source: https://superedge.io/docs/contributionguidelines Builds Docker images for specified components. ```bash make image IMAGES="application-grid-controller application-grid-wrapper" ``` -------------------------------- ### ApplicationGrid Wrapper Sample Configuration (serviceGroup/application-grid-wrapper.yaml) Source: https://superedge.io/docs/reference/servicegroup/application-grid-wrapper This YAML defines the full deployment for the ApplicationGrid wrapper: a ServiceAccount, RBAC rules, a ConfigMap with a kubeconfig, and a DaemonSet. The DaemonSet runs the wrapper with a kubeconfig pointing to 127.0.0.1:51003 and binds to 127.0.0.1:51006. It requires a node selector for linux and tolerates the master taint. ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: application-grid-wrapper namespace: edge-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: edge:application-grid-wrapper rules: - apiGroups: - "" resources: - endpoints - services verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - list - watch - apiGroups: - "" - events.k8s.io resources: - events verbs: - create - patch - update - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: edge:application-grid-wrapper roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: edge:application-grid-wrapper subjects: - kind: ServiceAccount name: application-grid-wrapper namespace: edge-system --- apiVersion: v1 kind: ConfigMap metadata: name: application-grid-wrapper namespace: edge-system data: kubeconfig.conf: | apiVersion: v1 clusters: - cluster: certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt server: https://127.0.0.1:51003 name: default contexts: - context: cluster: default namespace: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: k8s-app: application-grid-wrapper addonmanager.kubernetes.io/mode: Reconcile name: application-grid-wrapper namespace: edge-system spec: selector: matchLabels: k8s-app: application-grid-wrapper updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 10% template: metadata: labels: k8s-app: application-grid-wrapper spec: serviceAccount: application-grid-wrapper serviceAccountName: application-grid-wrapper priorityClassName: system-node-critical hostNetwork: true restartPolicy: Always nodeSelector: kubernetes.io/os: linux # TODO select edge node tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" containers: - name: application-grid-wrapper image: superedge/application-grid-wrapper:v0.3.0 imagePullPolicy: IfNotPresent command: - /usr/local/bin/application-grid-wrapper - --kubeconfig=/var/lib/application-grid-wrapper/kubeconfig.conf - --bind-address=127.0.0.1:51006 - --hostname=$(NODE_NAME) - --notify-channel-size=10000 env: - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName resources: limits: cpu: 50m memory: 100Mi requests: cpu: 10m memory: 20Mi securityContext: privileged: true volumeMounts: - mountPath: /var/lib/application-grid-wrapper name: application-grid-wrapper volumes: - configMap: defaultMode: 420 name: application-grid-wrapper name: application-grid-wrapper - hostPath: path: /var/tmp type: Directory name: host-var-tmp ``` -------------------------------- ### Success message after edgeadm join Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Expected output after a successful edgeadm join operation. The node is added to the cluster and the control-plane node can verify it with 'kubectl get nodes'. On failure, an error message is returned and the join is interrupted. ```text This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` -------------------------------- ### Generate Kubelet client key and certificate for tunnel-edge authentication Source: https://superedge.io/docs/installation/install-manually OpenSSL commands to create a private key and self-signed certificate for tunnel-edge to authenticate with Kubelet. The certificate is signed using the Kubernetes cluster's ca.crt and ca.key, typically found at /etc/kubernetes/pki in kubeadm installs. ```bash # private key openssl genrsa -des3 -out kubelet_client.key 1024 # generate csr openssl req -new -key kubelet_client.key -out kubelet_client.csr # Generate Self Signed certificate（Notice: it is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki） openssl ca -in kubelet-client.csr -out kubelet-client.crt -cert ca.crt -keyfile ca.key ``` -------------------------------- ### Configure tunnel-edge parameters in deployment/tunnel-edge.yaml Source: https://superedge.io/docs/installation/install-manually Lists the required parameters to set in the tunnel-edge deployment file: MasterIP, TunnelCloudEdgeToken, TunnelPersistentConnectionPort, KubernetesCaCert (base64), KubeletClientKey and KubeletClientCrt. ```yaml MasterIP: #Normal Kubernetes master node's IP or domain(currently, only one IP address or domain is supported) TunnelCloudEdgeToken: #Fill in the same token as "TunnelCloudEdgeToken" in Tunnel-cloud TunnelPersistentConnectionPort: #Tunnel-cloud's Persistent connection server Port KubernetesCaCert: #kube-apiserver's ca.crt(base64 encoded) KubeletClientKey: #Kubelet client key for Tunnel-edge to access Kubelet KubeletClientCrt: #Kubelet client cert for Tunnel-edge to access Kubelet ``` -------------------------------- ### Successful edgeadm init output Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Expected output when the cluster initializes successfully. It includes instructions for setting up kubectl and the 'edgeadm join' commands for adding control-plane and worker nodes. The certificate-key is sensitive and uploaded certs are deleted after two hours. ```text Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \ --control-plane --certificate-key xxxxxxxxxx --install-pkg-path Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "edgeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxx --install-pkg-path ``` -------------------------------- ### Deploy Application Grid Controller on master nodes Source: https://superedge.io/docs/installation/install-manually Command to apply the application-grid-controller.yaml on master nodes. ```bash $ kubectl apply -f deployment/application-grid-controller.yaml ``` -------------------------------- ### statefulset-grid-daemon.yaml Source: https://superedge.io/docs/reference/servicegroup/statefulset-grid-daemon Complete sample configuration for the statefulset-grid daemon. It defines a ServiceAccount, RBAC rules, a ConfigMap with a kubeconfig pointing to https://127.0.0.1:51003, and a DaemonSet that runs the daemon with hostNetwork and privileged access. The DaemonSet uses the image superedge/statefulset-grid-daemon:v0.3.0 and mounts the ConfigMap as a volume. ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: statefulset-grid-daemon namespace: edge-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: statefulset-grid-daemon namespace: edge-system rules: - apiGroups: - "" resources: - nodes - services - pods verbs: - "*" - apiGroups: - extensions - apps resources: - statefulsets verbs: - "*" - apiGroups: - superedge.io resources: - statefulsetgrids verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: statefulset-grid-daemon namespace: edge-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: statefulset-grid-daemon subjects: - kind: ServiceAccount name: statefulset-grid-daemon namespace: edge-system --- apiVersion: v1 kind: ConfigMap metadata: name: statefulset-grid-daemon namespace: edge-system data: kubeconfig.conf: | apiVersion: v1 clusters: - cluster: certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt server: https://127.0.0.1:51003 name: default contexts: - context: cluster: default namespace: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: k8s-app: statefulset-grid-daemon addonmanager.kubernetes.io/mode: Reconcile name: statefulset-grid-daemon namespace: edge-system spec: selector: matchLabels: k8s-app: statefulset-grid-daemon template: metadata: labels: k8s-app: statefulset-grid-daemon spec: serviceAccount: statefulset-grid-daemon serviceAccountName: statefulset-grid-daemon priorityClassName: system-node-critical hostNetwork: true restartPolicy: Always nodeSelector: kubernetes.io/os: linux # TODO select edge node tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" containers: - name: statefulset-grid-daemon image: superedge/statefulset-grid-daemon:v0.3.0 imagePullPolicy: IfNotPresent command: - /usr/local/bin/statefulset-grid-daemon - --hostname=$(NODE_NAME) - --kubeconfig=/var/lib/statefulset-grid-daemon/kubeconfig.conf env: - name: NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName resources: limits: cpu: 50m memory: 100Mi requests: cpu: 10m memory: 20Mi securityContext: privileged: true volumeMounts: - mountPath: /data/edge name: hosts - mountPath: /var/lib/statefulset-grid-daemon name: statefulset-grid-daemon volumes: - configMap: defaultMode: 420 name: statefulset-grid-daemon name: statefulset-grid-daemon - hostPath: path: /data/edge type: DirectoryOrCreate name: hosts ``` -------------------------------- ### Access echoserver and verify pod info Source: https://superedge.io/docs/components/lite-apiserver Commands to list pods and curl the echoserver, showing the pod name, namespace, and IP. The output confirms the echoserver is running and accessible. ```bash $ kubectl get pods -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES lite-demo-c7b458ddc-6lpnx 1/1 Running 0 4m23s 10.0.6.2 ecm-q5h6hhd $ curl http://10.0.6.2:8080 | grep pod pod name: lite-demo-c7b458ddc-6lpnx pod namespace: default pod IP: 10.0.6.2 ``` -------------------------------- ### Deploy StatefulSetGrid YAML Source: https://superedge.io/docs/components/servicegroup Use this YAML to create a StatefulSetGrid custom resource. The manifest defines a grid with key 'zone', a template for the statefulset with 3 replicas, and an echoserver container. The serviceName field must be set to the service to be created. ```yaml apiVersion: superedge.io/v1 kind: StatefulSetGrid metadata: name: statefulsetgrid-demo namespace: default spec: gridUniqKey: zone template: selector: matchLabels: appGrid: echo serviceName: "servicegrid-demo-svc" replicas: 3 template: metadata: labels: appGrid: echo spec: terminationGracePeriodSeconds: 10 containers: - image: superedge/echoserver:2.2 name: echo ports: - containerPort: 8080 protocol: TCP env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP resources: {} ``` -------------------------------- ### Create /etc/kubernetes/edge/tls.json for lite-apiserver Source: https://superedge.io/docs/installation/install-manually JSON configuration file specifying the key and certificate paths for lite-apiserver to authenticate with kubelet. Uses kubelet-client-current.pem for both key and cert. ```json [ { "key":"/var/lib/kubelet/pki/kubelet-client-current.pem", "cert":"/var/lib/kubelet/pki/kubelet-client-current.pem" } ] ``` -------------------------------- ### Deploy application grid wrapper on edge worker nodes Source: https://superedge.io/docs/installation/install-manually Command to apply the application-grid-wrapper.yaml on edge worker nodes. ```bash $ kubectl apply -f deployment/application-grid-wrapper.yaml ``` -------------------------------- ### Deploy tunnel-cloud Source: https://superedge.io/docs/installation/install-manually Deploys tunnel-cloud on master nodes. Run this command on the master node. ```bash $ kubectl apply -f deployment/tunnel-cloud.yaml ``` -------------------------------- ### Join master node with edgeadm join Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Execute this command on another master to join it to the cluster. The command is saved from the output of './edgeadm init'. ```bash ./edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \ --control-plane --certificate-key xxxxxxxxxx \ --install-pkg-path ``` -------------------------------- ### make clean Source: https://superedge.io/docs/contributionguidelines Cleans build artifacts. ```bash make clean ``` -------------------------------- ### Save default SuperEdge yamls with edgeadm manifests Source: https://superedge.io/docs/installation/install-via-edgeadm Run edgeadm manifests to save all default SuperEdge yamls into the ./manifests/ directory. ```bash [root@master01 ~]# edgeadm manifests ``` -------------------------------- ### make build BINS Source: https://superedge.io/docs/contributionguidelines Builds specific commands by specifying the BINS variable. ```bash make build BINS="lite-apiserver edgeadm" ``` -------------------------------- ### Set master kube-config file for non-root users Source: https://superedge.io/docs/installation/how-to-bootstrap-an-edge-kubernetes-cluster Run these commands as a regular user to enable kubectl access. They are also part of the edgeadm init output. ```bash mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` -------------------------------- ### edgeadm change with custom kubeconfig and CA files Source: https://superedge.io/docs/installation/install-via-edgeadm Use this command to specify custom kubeconfig and CA certificate/key files. The priority order is: --kubeconfig > Env KUBECONFIG > ~/.kube/config. ```bash [root@master01 ~]# edgeadm change --kubeconfig [kubeconfig file] --ca.cert [cluster ca certificate file] --ca.key [cluster ca key file] ``` -------------------------------- ### Deploy edge-health admission and webhook Source: https://superedge.io/docs/installation/install-manually Deploy the edge-health admission controller and webhook on master nodes. The webhook certificates are pre-populated; replace the caBundle in edge-health-webhook.yaml and the server.crt/server.key in the validate-admission-control-server-certs Secret of edge-health-admission.yaml with your own certificates. ```bash $ kubectl apply -f deployment/edge-health-admission.yaml $ kubectl apply -f deployment/edge-health-webhook.yaml ``` -------------------------------- ### Verify ServiceGroup deployment with kubectl Source: https://superedge.io/docs/components/servicegroup Commands to verify the DeploymentGrid, deployments, and services created for the ServiceGroup. The output shows the deployments for each zone and the service cluster IP. ```bash [~]## kubectl get dg NAME AGE deploymentgrid-demo 85s [~]## kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE deploymentgrid-demo-zone-0 2/2 2 2 85s deploymentgrid-demo-zone-1 2/2 2 2 85s deploymentgrid-demo-zone-2 2/2 2 2 85s [~]## kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 172.19.0.1 443/TCP 87m servicegrid-demo-svc ClusterIP 172.19.0.177 80/TCP 80s ``` -------------------------------- ### Deploy DeploymentGrid for stateless ServiceGroup Source: https://superedge.io/docs/components/servicegroup Defines a DeploymentGrid custom resource that deploys the echo-service to multiple node groups. The gridUniqKey field is set to 'zone' to match the node labels. The template includes environment variables for node name, pod name, namespace, and IP. ```yaml apiVersion: superedge.io/v1 kind: DeploymentGrid metadata: name: deploymentgrid-demo namespace: default spec: gridUniqKey: zone template: replicas: 2 selector: matchLabels: appGrid: echo strategy: {} template: metadata: creationTimestamp: null labels: appGrid: echo spec: containers: - image: superedge/echoserver:2.2 name: echo ports: - containerPort: 8080 protocol: TCP env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP resources: {} ``` -------------------------------- ### Configure tunnel-edge with tunnel-edge-conf ConfigMap Source: https://superedge.io/docs/components/tunnel Creates a ConfigMap named tunnel-edge-conf in the edge-system namespace. It embeds a tunnel_edge.toml configuration for the tunnel-edge, which includes stream client settings (token, cert, dns, servername, logport, channelzaddr) and HTTPS client settings (cert and key paths). The token and certificate paths must be present and valid for the edge to connect to tunnel-cloud. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: tunnel-edge-conf namespace: edge-system data: tunnel_edge.toml: | [mode] [mode.edge] [mode.edge.stream] # stream module [mode.edge.stream.client] # gRPC client component token = "6ff2a1ea0f1611eb9896362096106d9d" # Authentication token for access to tunnel-cloud cert = "../../conf/certs/ca.crt" # The ca certificate of the server-side certificate of the gRPC server of tunnel-cloud is used to verify the server-side certificate dns = "localhost" # The IP or domain name signed by the gRPC server certificate of tunnel-cloud servername = "localhost:9000" # The IP and port of gRPC server of tunnel-cloud logport = 7000 # The listening port of the http server for log and health check,use (curl -X PUT http://podip:logport/debug/flags/v -d "8") to set the log level channelzaddr = "0.0.0.0:5000" # The listening address of the gRPC channlez server, used to obtain the debugging information of gRPC [mode.edge.https] cert= "../../conf/certs/kubelet-client.crt" # The client certificate of the HTTPS server forwarded by tunnel-cloud proxy key= "../../conf/certs/kubelet-client.key" # The private key of the client side of the HTTPS server forwarded by the tunnel-cloud proxy ``` -------------------------------- ### edgeadm manifests with custom output directory Source: https://superedge.io/docs/installation/install-via-edgeadm Use this command to specify an output directory for the generated yamls. ```bash [root@master01 ~]# edgeadm manifests -m output_dir ``` -------------------------------- ### Deploy tunnel-coredns Source: https://superedge.io/docs/installation/install-manually Deploys the tunnel CoreDNS on master nodes. Run this command on the master node. ```bash $ kubectl apply -f deployment/tunnel-coredns.yaml ```