ito-archived/research/KubeEdge.md

## 概述

为开源产品，构建在Kubernetes之上，并将本机容器化应用程序编排和设备管理扩展到边缘主机。

![KubeEdge Architecture](imgs/KubeEdge/kubeedge_arch.png)

优势：

+ 原生支持kubernetes
+ 云边缘可靠协作
+ 边缘自治
+ 边缘设备管理：通过CRD实现的Kubernetes本地api管理边缘设备。
+ 极轻量边缘代理 （EdgeCore）

组件：

+ Edged  运行在边缘节点的代理，管理容器化应用程序
+ EdgeHub 管理和云服务交互的web socket客户端。包括同步云端资源到边缘，上报边端主机和设备状态变更到云。
+ CloudHub 在云端运行的websocket服务，监听边端的事件
+ EdgeController： Kubernetes Controller的扩展，管理边缘node和pod元数据
+ EventBus： Mqtt的客户端用于连接Mqtt服务(mosquitto)，提供到其他组件的订阅和发布。
+ DeviceTwin：存储设备状态、同步设备状态到云端。并提供应用的查询接口
+ MetaManager：Edged和edgehub之间的消息处理器，它还负责在轻量级数据库（SQLite）中存储/检索元数据。

- [ServiceBus](https://kubeedge.io/en/docs/architecture/edge/servicebus): a HTTP client to interact with HTTP servers (REST), offering HTTP client capabilities to components of cloud to reach HTTP servers running at edge.



<img src="imgs/KubeEdge/avatar_hu6dd63758e9c8a7a1da773a2d4193fd13_641557_250x250_fill_box_center_2.png" height=50 align="left">

官网：https://kubeedge.io/en/docs/setup/local/

<img src="imgs/KubeEdge/image-20210709140821500.png" height=50 align="left">

Github: https://github.com/kubeedge/kubeedge



| 术语 | 描述 |
| ---- | ---- |
|      |      |

## 安装

[下载](https://github.com/kubeedge/kubeedge/releases)

+ [keadm-v1.7.1-linux-amd64.tar.gz](F:\H\edge) keadm安装文件
+ [kubeedge-v1.7.1-linux-amd64.tar.gz](F:\H\edge)  二进制文件安装
+ [edgesite-v1.7.1-linux-amd64.tar](F:\H\edge)



### [通过Keadm来安装](https://kubeedge.io/en/docs/setup/keadm/)

keadm 用来安装 KubeEdge 的云和端组件。

#### 云端

需要提前安装kubernetes。

```sh
keadm init --advertise-address="10.8.30.38"
```

将在服务器上安装 cloudcore，生成证书并安装CRDs。

获取token，将在添加边缘节点时使用。

```sh
 keadm gettoken
```



#### 边端

```sh
keadm join --cloudcore-ipport=10.8.30.38:10000 --token=xxxx
```

将在客户端安装edgecore和mqtt。



### [通过二进制文件安装](https://kubeedge.io/en/docs/setup/local/)

或者参考https://www.cnblogs.com/kkbill/p/12600541.html

安装环境**准备**：

`Cloud:10.8.30.38`：kubernetes

`Edge:10.8.30.35` ： golang docker mqtt 

| 系统   | ubuntu 16.04         |
| :----- | -------------------- |
| golang | go1.10.4 linux/amd64 |
| docker | 19.03.12             |
| k8s    | 1.18                 |



准备安装KubeEdnge1.7.1：

将安装文件 `[kubeedge-v1.7.1-linux-amd64.tar.gz] 拷贝到服务器和边缘节点

#### 云端

创建 CRDs

```shell
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_device.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_devicemodel.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/objectsync_v1alpha1.yaml
```

创建config文件

```sh
cloudcore --minconfig > cloudcore.yaml
```

[配置](https://kubeedge.io/en/docs/setup/config/#configuration-cloud-side-kubeedge-master)

```yaml
apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
kubeAPIConfig:
  kubeConfig: /root/.kube/config
  master: ""
modules:
  cloudHub:
    advertiseAddress:
    - 10.8.30.38
    enable: true
    https:
      address: 0.0.0.0
      enable: true
      port: 10002
    nodeLimit: 1000
    tlsCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key
    tlsCertFile: /etc/kubeedge/certs/edge.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key
    unixsocket:
      address: unix:///var/lib/kubeedge/kubeedge.sock
      enable: true
    websocket:
      address: 0.0.0.0
      enable: true
      port: 10000
  router:
    address: 0.0.0.0
    enable: true
    port: 9443
    restTimeout: 60

```

运行

```sh
cloudcore --config cloudcore.yaml
```

生成证书

```sh
 wget https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/tools/certgen.sh
chmod +x certgen.sh
./certgen.sh genCertAndKey edge
 
root@node38:/home/anxin/edge# ls /etc/kubeedge
ca  certs  crds

拷贝到边缘节点：
 scp /etc/kubeedge/ca/rootCA.crt root@node35:/etc/kubeedge/ca/
 scp /etc/kubeedge/certs/edge.crt root@node35:/etc/kubeedge/certs/
 scp /etc/kubeedge/certs/edge.key root@node35:/etc/kubeedge/certs/
```



获取token （将用于edge配置）

```sh
kubectl get secret -nkubeedge tokensecret -o json # 找到token 加入到edgecore.yml中
 
# kubectl get secret -nkubeedge default-token-pdrnq -o=jsonpath='{.data.tokendata}' | base64 -d
# sed -i -e "s|token: .*|token: ${token}|g" edgecore.yaml
```





#### 边端

>  需要提前安装docker



初始化配置

```
./edgecore --minconfig > edgecore.yaml
./edgecore --defaultconfig > edgecore.yaml
```



将云端token配置到edgecore.yaml

```yaml
# With --minconfig , you can easily used this configurations as reference.
# It's useful to users who are new to KubeEdge, and you can modify/create your own configs accordingly. 
# This configuration is suitable for beginners.

apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
  dataSource: /var/lib/kubeedge/edgecore.db
kind: EdgeCore
modules:
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://10.8.30.38:10002
    tlsCaFile: /etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    token: 5e630e7bf11b00a77a513233e067788e12d36179be1d4b83d804bf77c463781e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjYyNDE2ODd9.oRE1mzeRVC7ohsA5KF9QP_EvxE83BSTjP6fuZuVL2Yk
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 10.8.30.38:10000
      writeDeadline: 15
  edged:
    #cgroupDriver: cgroupfs 通过docker info查看本机安装的cgroup-driver
    cgroupDriver: systemd
    cgroupRoot: ""
    cgroupsPerQOS: true
    clusterDNS: ""
    clusterDomain: ""
    devicePluginEnabled: false
    dockerAddress: unix:///var/run/docker.sock
    enable: true
    registerNode: true
    gpuPluginEnabled: false
    hostnameOverride: node35
    nodeIP: 10.8.30.35
    podSandboxImage: kubeedge/pause:3.1
    remoteImageEndpoint: unix:///var/run/dockershim.sock
    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    runtimeType: docker
  eventBus:
    enable: true
    mqttMode: 2
    mqttQOS: 0
    mqttRetain: false
    mqttServerExternal: tcp://127.0.0.1:1883
    mqttServerInternal: tcp://127.0.0.1:1884
```



启动

```shell
edgecore --config edgecore.yaml
```

启动后 docker ps:

![image-20210713140207317](imgs/KubeEdge/image-20210713140207317.png)



在cloud端查看

```sh
root@node38:/home/anxin# kubectl get no 
NAME     STATUS   ROLES        AGE    VERSION
node35   Ready    agent,edge   334d   v1.19.3-kubeedge-v1.7.1
node37   Ready    <none>       334d   v1.18.6
node38   Ready    master       334d   v1.18.6
```





##### 常见报错

1. Error: token credentials are in the wrong format

   ```sh
   配置的token格式不对，从cloud端拿到的是base64，需要取解密后字符串、
   ```

2.  cgroup-driver

   需要通过docker info命令查看自己安装的docker的cgroup-driver，然后修改配置文件中的cgroup-driver

3. 无法访问平台侧的证书文件
   F0709 14:54:31.063677    3153 certmanager.go:92] Error: failed to get edge certificate from the cloudcore, error: Get "https://10.8.30.38:10002/edge.crt": x509: cannot validate certificate for 10.8.30.38 because it doesn't contain any IP SANs

```shell
I0709 15:27:42.098408   14300 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret
I0709 15:27:42.103401   14300 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret
```

**解决方法**：

删除原来的配置

```shell
kubectl delete secret casecret -nkubeedge
kubectl delete secret cloudcoresecret -nkubeedge
# both cloud and edge side
mv /etc/kubeedge/ca  /etc/kubeedge/ca.bak 
mv  /etc/kubeedge/certs /etc/kubeedge/certs.bak
```



执行： ./certgen.sh genCertAndKey edge

新问题：

`Can't load /root/.rnd`  (第一次生成的时候可能就错在这里)

```shell
cd /root

openssl rand -writerand .rnd
```



### 在Raspberry Pi 上安装

#### 安装Lite系统

https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-05-28/2021-05-07-raspios-buster-armhf-lite.zip

设置网络

```shell
interface eth0 
static ip_address=10.8.30.197/24 
static routers=10.8.30.1 
static domain_name_servers=114.114.114.114 223.5.5.5 223.6.6.6 

#设置密码
passwd pi

PasswordAuthentication yes

systemctl enable ssh
```

安装mosquitto

```shell
wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key

sudo apt-key add mosquitto-repo.gpg.key

# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-jessie.list
# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-stretch.list
sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-buster.list 

apt-get update

apt-get install mosquitto

systemctl status mosquitto
```



#### [安装docker](https://peppe8o.com/setup-a-docker-environment-with-raspberry-pi-os-lite-and-portainer/)

<img src="imgs/KubeEdge/raspberry-pi-docker-portainer-featured-image.jpg" width="100" align="left"/>

```shell
sudo apt-get update && sudo apt-get upgrade

curl -fsSL https://get.docker.com -o get-docker.sh

sudo sh get-docker.sh

sudo docker version

sudo docker run hello-world

sudo usermod -aG docker pi

```





#### [安装containerd](https://docs.docker.com/engine/install/debian/)

[Setting different container runtime with CRI](https://docs.kubeedge.io/en/docs/advanced/cri/)



修改`/etc/apt/sources.list`

```shell
deb http://mirrors.163.com/raspbian/raspbian/ buster main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi
```

安装

```shell
 sudo apt-get update
 sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
    
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

 echo \
  "deb [arch=armhf signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  
 sudo apt-get update
 sudo apt-get install containerd.io
```



```shell
# Configure containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# Restart containerd
systemctl restart containerd
```



安装golang

```shell
sudo apt update
sudo apt full-upgrade

wget https://dl.google.com/go/go1.16.6.linux-armv6l.tar.gz -O go.tar.gz

```





#### 配置运行EdgeCore

配置EdgeCore.yaml支持containerd.。 默认cgroup 配置是 cgroupfs。

```yaml
remoteRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
remoteImageEndpoint: unix:///var/run/containerd/containerd.sock
runtimeRequestTimeout: 2
podSandboxImage: k8s.gcr.io/pause:3.2
runtimeType: remote
```

配置

```yaml

```

启动：

![image-20210721161810471](imgs/KubeEdge/image-20210721161810471.png)

问题

1. Cgroup subsystem not mounted:[memory]

   ```shell
   Following Cgroup subsystem not mounted: [memory]
   ```

   https://zhuanlan.zhihu.com/p/384443481

   ```shell
   #解决问题
   #修改/boot/cmdline.txt
   sudo vim /boot/cmdline.txt
   cgroup_enable=memory cgroup_memory=1
   添加在同一行的最后面,接着内容后空格后添加, 注意:不要换行添加
   
   #重启机器配置生效
   reboot
   ```

2. 



## 配置

### 云端

设置边缘端自动注册 modules.edged.registerNode=true。 或者使用[手动注册](https://kubeedge.io/en/docs/setup/config/)。



### 边端

#### 设置基础镜像：

```sh
modules.edged.podSandboxImage
```

检查机器架构：

```shell
getconf LONG_BIT
```

+ `kubeedge/pause-arm:3.1` **for** arm arch
+ `kubeedge/pause-arm64:3.1` **for** arm64 arch 
+ `kubeedge/pause:3.1` **for** x86 arch

#### 容器运行时

```yaml
runtimeType: docker
```

or

```yaml
runtimeType: remote
```

#### MQTT模式

MQTT用于DeviceTwin和Devices之间的通信，支持3中MQTT模式：

0 internalMqttMode 内部MQTT代理使能

1 bothMqttMonde 内部和外部MQTT代理均使能

2 externalMqttMode 外部代理使能





## EdgeMesh

云端和边缘之间发生网络问题，集成EdgeMesh 支持DNS访问

```shell
grep hosts /etc/nsswitch.conf
hosts:          dns file mdns4_minimal # 确保dns在第一位
```

**IP Forward**设置

```shell
$ sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
$ sudo sysctl -p
#check
$ sudo sysctl -p | grep ip_forward
net.ipv4.ip_forward = 1
```

部署一个nginx例子

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          hostPort: 8050
```

`kubeclt apply -f pod-nginx.yaml`

![image-20210713171944426](imgs/KubeEdge/image-20210713171944426.png)

按照官网例子访问容器ip curl 10.0.35.3:8050，无法访问，直接访问http://10.8.30.35:8050 可以。

创建一个服务：

```yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
    - name: http-0
      port: 18050
      protocol: TCP
      targetPort: 80
```

![image-20210713174342754](imgs/KubeEdge/image-20210713174342754.png)

在边缘端访问： `<service_name>.<service_namespace>.svc.<cluster>.<local>:<port>`

curl http://nginx-svc.default.svc.cluster.local:18050



## 开发指南

### Device Model 设备原型

KubeEdge支持通过Kubernetes CRDS，以及Device Mapper进行设备管理

```yaml
apiVersion: devices.kubeedge.io/v1alpha2
kind: DeviceModel
metadata:
 name: sensor-tag-model
 namespace: default
spec:
 properties:
  - name: temperature
    description: temperature in degree celsius
    type:
     int:
      accessMode: ReadWrite
      maximum: 100
      unit: degree celsius
  - name: temperature-enable
    description: enable data collection of temperature sensor
    type:
      string:
        accessMode: ReadWrite
        defaultValue: 'OFF'
```

### Device Instance

```yaml
apiVersion: devices.kubeedge.io/v1alpha2
kind: Device
metadata:
  name: sensor-tag-instance-01
  labels:
    description: TISimplelinkSensorTag
    manufacturer: TexasInstruments
    model: CC2650
spec:
  deviceModelRef:
    name: sensor-tag-model
  protocol:
    modbus:
      slaveID: 1
    common:
      com:
        serialPort: '1'
        baudRate: 115200
        dataBits: 8
        parity: even
        stopBits: 1
      customizedValues: # 自定义属性值
        def1: def1-val
        def2: def2-val
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: ''
        operator: In
        values:
        - node1
  propertyVisitors:
    - propertyName: temperature
      modbus:
        register: CoilRegister
        offset: 2
        limit: 1
        scale: 1
        isSwap: true
        isRegisterSwap: true
    - propertyName: temperature-enable
      modbus:
        register: DiscreteInputRegister
        offset: 3
        limit: 1
        scale: 1.0
        isSwap: true
        isRegisterSwap: true
    - propertyName: temperature # 自定义协议写法
      collectCycle: 500000000
      reportCycle: 1000000000
      customizedProtocol:
        protocolName: MY-TEST-PROTOCOL
        configData:
          def1: def1-val
          def2: def2-val
          def3:
            innerDef1: idef-val
status:
  twins:
    - propertyName: temperature
      reported:
        metadata:
          timestamp: '1550049403598'
          type: int
        value: '10'
      desired:
        metadata:
          timestamp: '1550049403598'
          type: int
        value: '15'
data: # 定义通过mappers上报到边缘端MQTT代理的一系列时序属性
  dataTopic: "$ke/events/device/+/data/update"
  dataProperties: 
    - propertyName: pressure
      metadata:
        type: int
    - propertyName: temperature
      metadata:
        type: int
```

### Device Mapper

是连接和控制设备的应用程序，包含功能：

1) 扫描和连接设备

2) 报告设备孪生属性（ twin-attributes）的真实状态

3) 转换预期状态到真实状态

4) 采集遥感数据

5) 将设备的读取转换成KubeEdge可以接受的格式

6) 安排设备上的动作(schedule)

7) 检查设备的健康状态

![img](imgs/KubeEdge/mapper-design.png)

### 创建步骤

1. kubectl apply -f <path to device model yaml>
2. kubectl apply -f <path to device instance yaml>
3. 执行基于指定协议的mapper程序
4. 改变设备实例yaml中的状态段,应用改变将影响边缘端，通过设备控制和设备孪生组件（DeviceController和DeviceTwin）
5. 边缘端的mapper进程上报设备孪生的数据，并通过DeviceController同步回云端。



### Router Manager 路由管理

通过CRDs和Router模块，基于mqtt，实现云边消息通信。 控制用户数据的传输，一次传输的大小不超过12M。

使用场景：

+ 通过云端api发布自定义消息，最终到达终端mqtt代理
+ 边缘发布消息到mqtt代理，最终通过RESTApi返回云端
+ 通过云端api发布自定义消息，最终调用边缘rest api传送消息

#### 规则的定义

RuleEndpoint 和 Rule Definition:

+ RuleEndpoint 定义消息从哪里来的、或到哪里去，包括3中类型
  + rest
  + eventbus (MQTT)
  + servicebus (边缘端的Rest Api应用)
+ Rule 定义从源Endpoint到目标Endpoint消息传输的方式，包含3中类型：
  + rest->eventbus :  cloud api --> edge mqtt
  + eventbus->rest: edge mqtt --> cloud api
  + rest->servicebus: cloud api --> user's app

RuleEndpoint定义

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ruleendpoints.rules.kubeedge.io
spec:
  group: rules.kubeedge.io
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                ruleEndpointType:
                  type: rest/eventbus/servicebus
              required:
                - ruleEndpointType
  scope: Namespaced
  names:
    plural: ruleendpoints
    singular: ruleendpoint
    kind: RuleEndpoint
    shortNames:
      - re
```

Rule定义：

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: rules.rules.kubeedge.io
spec:
  group: rules.kubeedge.io
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                source:
                  type: string
                  value: my-rest
                sourceResource:
                  description: {"path":"/a/b"} 或 {"topic":"<user define string>","node_name":"edge-node"}
                  type: object
                  additionalProperties:
                    type: string
                target:
                  type: string
                  value: my-eventbus
                targetResource:
                  description: {"topic":"/test"}
                  type: object
                  additionalProperties:
                    type: string
              required:
                - source
                - sourceResource
                - target
                - targetResource
            status:
              type: object
              properties:
                successMessages:
                  type: integer
                failMessages:
                  type: integer
                errors:
                  items:
                    type: string
                  type: array
  scope: Namespaced
  names:
    plural: rules
    singular: rule
    kind: Rule
```



### [发消息的步骤](https://docs.kubeedge.io/en/docs/developer/custom_message_deliver/)

以rest->eventbus 模式举例

1. 云端配置 router模块使能

2. 创建ruleEndpoint

3. 创建rule

   ```yaml
   apiVersion: rules.kubeedge.io/v1
   kind: Rule
   metadata:
     name: my-rule
     labels:
       description: test
   spec:
     source: "my-rest"
     sourceResource: {"path":"/test"}
     target: "my-eventbus"
     targetResource: {"topic":"test"}
   ```

4. 云端调用api接口发送消息

   ```shell
   POST
   http://{cloudcore_ip}:9443/{node_name}/{namespace}/{path}
   ```

   

5. 边缘订阅

   ```shell
   mosquitto_sub -t 'test' -d
   ```



MORE：

人工智能、数字孪生、边缘计算、量子计算、沉浸式技术、区块链、智能空间、5G