## 概述 为开源产品，构建在Kubernetes之上，并将本机容器化应用程序编排和设备管理扩展到边缘主机。 ![KubeEdge Architecture](imgs/KubeEdge/kubeedge_arch.png) 优势： + 原生支持kubernetes + 云边缘可靠协作 + 边缘自治 + 边缘设备管理：通过CRD实现的Kubernetes本地api管理边缘设备。 + 极轻量边缘代理 （EdgeCore） 组件： + Edged 运行在边缘节点的代理，管理容器化应用程序 + EdgeHub 管理和云服务交互的web socket客户端。包括同步云端资源到边缘，上报边端主机和设备状态变更到云。 + CloudHub 在云端运行的websocket服务，监听边端的事件 + EdgeController： Kubernetes Controller的扩展，管理边缘node和pod元数据 + EventBus： Mqtt的客户端用于连接Mqtt服务(mosquitto)，提供到其他组件的订阅和发布。 + DeviceTwin：存储设备状态、同步设备状态到云端。并提供应用的查询接口 + MetaManager：Edged和edgehub之间的消息处理器，它还负责在轻量级数据库（SQLite）中存储/检索元数据。 - [ServiceBus](https://kubeedge.io/en/docs/architecture/edge/servicebus): a HTTP client to interact with HTTP servers (REST), offering HTTP client capabilities to components of cloud to reach HTTP servers running at edge. 官网：https://kubeedge.io/en/docs/setup/local/ Github: https://github.com/kubeedge/kubeedge | 术语 | 描述 | | ---- | ---- | | | | ## 安装 [下载](https://github.com/kubeedge/kubeedge/releases) + [keadm-v1.7.1-linux-amd64.tar.gz](F:\H\edge) keadm安装文件 + [kubeedge-v1.7.1-linux-amd64.tar.gz](F:\H\edge) 二进制文件安装 + [edgesite-v1.7.1-linux-amd64.tar](F:\H\edge) ### [通过Keadm来安装](https://kubeedge.io/en/docs/setup/keadm/) keadm 用来安装 KubeEdge 的云和端组件。 #### 云端 需要提前安装kubernetes。 ```sh keadm init --advertise-address="10.8.30.38" ``` 将在服务器上安装 cloudcore，生成证书并安装CRDs。 获取token，将在添加边缘节点时使用。 ```sh keadm gettoken ``` #### 边端 ```sh keadm join --cloudcore-ipport=10.8.30.38:10000 --token=xxxx ``` 将在客户端安装edgecore和mqtt。 ### [通过二进制文件安装](https://kubeedge.io/en/docs/setup/local/) 或者参考https://www.cnblogs.com/kkbill/p/12600541.html 安装环境**准备**： `Cloud:10.8.30.38`：kubernetes `Edge:10.8.30.35` ： golang docker mqtt | 系统 | ubuntu 16.04 | | :----- | -------------------- | | golang | go1.10.4 linux/amd64 | | docker | 19.03.12 | | k8s | 1.18 | 准备安装KubeEdnge1.7.1： 将安装文件 `[kubeedge-v1.7.1-linux-amd64.tar.gz] 拷贝到服务器和边缘节点 #### 云端 创建 CRDs ```shell kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_device.yaml kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_devicemodel.yaml kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/objectsync_v1alpha1.yaml ``` 创建config文件 ```sh cloudcore --minconfig > cloudcore.yaml ``` [配置](https://kubeedge.io/en/docs/setup/config/#configuration-cloud-side-kubeedge-master) ```yaml apiVersion: cloudcore.config.kubeedge.io/v1alpha1 kind: CloudCore kubeAPIConfig: kubeConfig: /root/.kube/config master: "" modules: cloudHub: advertiseAddress: - 10.8.30.38 enable: true https: address: 0.0.0.0 enable: true port: 10002 nodeLimit: 1000 tlsCAFile: /etc/kubeedge/ca/rootCA.crt tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key tlsCertFile: /etc/kubeedge/certs/edge.crt tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key unixsocket: address: unix:///var/lib/kubeedge/kubeedge.sock enable: true websocket: address: 0.0.0.0 enable: true port: 10000 router: address: 0.0.0.0 enable: true port: 9443 restTimeout: 60 ``` 运行 ```sh cloudcore --config cloudcore.yaml ``` 生成证书 ```sh wget https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/tools/certgen.sh chmod +x certgen.sh ./certgen.sh genCertAndKey edge root@node38:/home/anxin/edge# ls /etc/kubeedge ca certs crds 拷贝到边缘节点： scp /etc/kubeedge/ca/rootCA.crt root@node35:/etc/kubeedge/ca/ scp /etc/kubeedge/certs/edge.crt root@node35:/etc/kubeedge/certs/ scp /etc/kubeedge/certs/edge.key root@node35:/etc/kubeedge/certs/ ``` 获取token （将用于edge配置） ```sh kubectl get secret -nkubeedge tokensecret -o json # 找到token 加入到edgecore.yml中 # kubectl get secret -nkubeedge default-token-pdrnq -o=jsonpath='{.data.tokendata}' | base64 -d # sed -i -e "s|token: .*|token: ${token}|g" edgecore.yaml ``` #### 边端 > 需要提前安装docker 初始化配置 ``` ./edgecore --minconfig > edgecore.yaml ./edgecore --defaultconfig > edgecore.yaml ``` 将云端token配置到edgecore.yaml ```yaml # With --minconfig , you can easily used this configurations as reference. # It's useful to users who are new to KubeEdge, and you can modify/create your own configs accordingly. # This configuration is suitable for beginners. apiVersion: edgecore.config.kubeedge.io/v1alpha1 database: dataSource: /var/lib/kubeedge/edgecore.db kind: EdgeCore modules: edgeHub: enable: true heartbeat: 15 httpServer: https://10.8.30.38:10002 tlsCaFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/server.crt tlsPrivateKeyFile: /etc/kubeedge/certs/server.key token: 5e630e7bf11b00a77a513233e067788e12d36179be1d4b83d804bf77c463781e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjYyNDE2ODd9.oRE1mzeRVC7ohsA5KF9QP_EvxE83BSTjP6fuZuVL2Yk websocket: enable: true handshakeTimeout: 30 readDeadline: 15 server: 10.8.30.38:10000 writeDeadline: 15 edged: #cgroupDriver: cgroupfs 通过docker info查看本机安装的cgroup-driver cgroupDriver: systemd cgroupRoot: "" cgroupsPerQOS: true clusterDNS: "" clusterDomain: "" devicePluginEnabled: false dockerAddress: unix:///var/run/docker.sock enable: true registerNode: true gpuPluginEnabled: false hostnameOverride: node35 nodeIP: 10.8.30.35 podSandboxImage: kubeedge/pause:3.1 remoteImageEndpoint: unix:///var/run/dockershim.sock remoteRuntimeEndpoint: unix:///var/run/dockershim.sock runtimeType: docker eventBus: enable: true mqttMode: 2 mqttQOS: 0 mqttRetain: false mqttServerExternal: tcp://127.0.0.1:1883 mqttServerInternal: tcp://127.0.0.1:1884 ``` 启动 ```shell edgecore --config edgecore.yaml ``` 启动后 docker ps: ![image-20210713140207317](imgs/KubeEdge/image-20210713140207317.png) 在cloud端查看 ```sh root@node38:/home/anxin# kubectl get no NAME STATUS ROLES AGE VERSION node35 Ready agent,edge 334d v1.19.3-kubeedge-v1.7.1 node37 Ready 334d v1.18.6 node38 Ready master 334d v1.18.6 ``` ##### 常见报错 1. Error: token credentials are in the wrong format ```sh 配置的token格式不对，从cloud端拿到的是base64，需要取解密后字符串、 ``` 2. cgroup-driver 需要通过docker info命令查看自己安装的docker的cgroup-driver，然后修改配置文件中的cgroup-driver 3. 无法访问平台侧的证书文件 F0709 14:54:31.063677 3153 certmanager.go:92] Error: failed to get edge certificate from the cloudcore, error: Get "https://10.8.30.38:10002/edge.crt": x509: cannot validate certificate for 10.8.30.38 because it doesn't contain any IP SANs ```shell I0709 15:27:42.098408 14300 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret I0709 15:27:42.103401 14300 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret ``` **解决方法**： 删除原来的配置 ```shell kubectl delete secret casecret -nkubeedge kubectl delete secret cloudcoresecret -nkubeedge # both cloud and edge side mv /etc/kubeedge/ca /etc/kubeedge/ca.bak mv /etc/kubeedge/certs /etc/kubeedge/certs.bak ``` 执行： ./certgen.sh genCertAndKey edge 新问题： `Can't load /root/.rnd` (第一次生成的时候可能就错在这里) ```shell cd /root openssl rand -writerand .rnd ``` ### 在Raspberry Pi 上安装 #### 安装Lite系统 https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-05-28/2021-05-07-raspios-buster-armhf-lite.zip 设置网络 ```shell interface eth0 static ip_address=10.8.30.197/24 static routers=10.8.30.1 static domain_name_servers=114.114.114.114 223.5.5.5 223.6.6.6 #设置密码 passwd pi PasswordAuthentication yes systemctl enable ssh ``` 安装mosquitto ```shell wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key sudo apt-key add mosquitto-repo.gpg.key # sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-jessie.list # sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-stretch.list sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-buster.list apt-get update apt-get install mosquitto systemctl status mosquitto ``` #### [安装docker](https://peppe8o.com/setup-a-docker-environment-with-raspberry-pi-os-lite-and-portainer/) ```shell sudo apt-get update && sudo apt-get upgrade curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo docker version sudo docker run hello-world sudo usermod -aG docker pi ``` #### [安装containerd](https://docs.docker.com/engine/install/debian/) [Setting different container runtime with CRI](https://docs.kubeedge.io/en/docs/advanced/cri/) 修改`/etc/apt/sources.list` ```shell deb http://mirrors.163.com/raspbian/raspbian/ buster main contrib non-free rpi # Uncomment line below then 'apt-get update' to enable 'apt-get source' #deb-src http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi ``` 安装 ```shell sudo apt-get update sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg \ lsb-release curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo \ "deb [arch=armhf signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install containerd.io ``` ```shell # Configure containerd mkdir -p /etc/containerd containerd config default > /etc/containerd/config.toml # Restart containerd systemctl restart containerd ``` 安装golang ```shell sudo apt update sudo apt full-upgrade wget https://dl.google.com/go/go1.16.6.linux-armv6l.tar.gz -O go.tar.gz ``` #### 配置运行EdgeCore 配置EdgeCore.yaml支持containerd.。 默认cgroup 配置是 cgroupfs。 ```yaml remoteRuntimeEndpoint: unix:///var/run/containerd/containerd.sock remoteImageEndpoint: unix:///var/run/containerd/containerd.sock runtimeRequestTimeout: 2 podSandboxImage: k8s.gcr.io/pause:3.2 runtimeType: remote ``` 配置 ```yaml ``` 启动： ![image-20210721161810471](imgs/KubeEdge/image-20210721161810471.png) 问题 1. Cgroup subsystem not mounted:[memory] ```shell Following Cgroup subsystem not mounted: [memory] ``` https://zhuanlan.zhihu.com/p/384443481 ```shell #解决问题 #修改/boot/cmdline.txt sudo vim /boot/cmdline.txt cgroup_enable=memory cgroup_memory=1 添加在同一行的最后面,接着内容后空格后添加, 注意:不要换行添加 #重启机器配置生效 reboot ``` 2. ## 配置 ### 云端 设置边缘端自动注册 modules.edged.registerNode=true。 或者使用[手动注册](https://kubeedge.io/en/docs/setup/config/)。 ### 边端 #### 设置基础镜像： ```sh modules.edged.podSandboxImage ``` 检查机器架构： ```shell getconf LONG_BIT ``` + `kubeedge/pause-arm:3.1` **for** arm arch + `kubeedge/pause-arm64:3.1` **for** arm64 arch + `kubeedge/pause:3.1` **for** x86 arch #### 容器运行时 ```yaml runtimeType: docker ``` or ```yaml runtimeType: remote ``` #### MQTT模式 MQTT用于DeviceTwin和Devices之间的通信，支持3中MQTT模式： 0 internalMqttMode 内部MQTT代理使能 1 bothMqttMonde 内部和外部MQTT代理均使能 2 externalMqttMode 外部代理使能 ## EdgeMesh 云端和边缘之间发生网络问题，集成EdgeMesh 支持DNS访问 ```shell grep hosts /etc/nsswitch.conf hosts: dns file mdns4_minimal # 确保dns在第一位 ``` **IP Forward**设置 ```shell $ sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf $ sudo sysctl -p #check $ sudo sysctl -p | grep ip_forward net.ipv4.ip_forward = 1 ``` 部署一个nginx例子 ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 hostPort: 8050 ``` `kubeclt apply -f pod-nginx.yaml` ![image-20210713171944426](imgs/KubeEdge/image-20210713171944426.png) 按照官网例子访问容器ip curl 10.0.35.3:8050，无法访问，直接访问http://10.8.30.35:8050 可以。 创建一个服务： ```yaml apiVersion: v1 kind: Service metadata: name: nginx-svc namespace: default spec: clusterIP: None selector: app: nginx ports: - name: http-0 port: 18050 protocol: TCP targetPort: 80 ``` ![image-20210713174342754](imgs/KubeEdge/image-20210713174342754.png) 在边缘端访问： `..svc..:` curl http://nginx-svc.default.svc.cluster.local:18050 ## 开发指南 ### Device Model 设备原型 KubeEdge支持通过Kubernetes CRDS，以及Device Mapper进行设备管理 ```yaml apiVersion: devices.kubeedge.io/v1alpha2 kind: DeviceModel metadata: name: sensor-tag-model namespace: default spec: properties: - name: temperature description: temperature in degree celsius type: int: accessMode: ReadWrite maximum: 100 unit: degree celsius - name: temperature-enable description: enable data collection of temperature sensor type: string: accessMode: ReadWrite defaultValue: 'OFF' ``` ### Device Instance ```yaml apiVersion: devices.kubeedge.io/v1alpha2 kind: Device metadata: name: sensor-tag-instance-01 labels: description: TISimplelinkSensorTag manufacturer: TexasInstruments model: CC2650 spec: deviceModelRef: name: sensor-tag-model protocol: modbus: slaveID: 1 common: com: serialPort: '1' baudRate: 115200 dataBits: 8 parity: even stopBits: 1 customizedValues: # 自定义属性值 def1: def1-val def2: def2-val nodeSelector: nodeSelectorTerms: - matchExpressions: - key: '' operator: In values: - node1 propertyVisitors: - propertyName: temperature modbus: register: CoilRegister offset: 2 limit: 1 scale: 1 isSwap: true isRegisterSwap: true - propertyName: temperature-enable modbus: register: DiscreteInputRegister offset: 3 limit: 1 scale: 1.0 isSwap: true isRegisterSwap: true - propertyName: temperature # 自定义协议写法 collectCycle: 500000000 reportCycle: 1000000000 customizedProtocol: protocolName: MY-TEST-PROTOCOL configData: def1: def1-val def2: def2-val def3: innerDef1: idef-val status: twins: - propertyName: temperature reported: metadata: timestamp: '1550049403598' type: int value: '10' desired: metadata: timestamp: '1550049403598' type: int value: '15' data: # 定义通过mappers上报到边缘端MQTT代理的一系列时序属性 dataTopic: "$ke/events/device/+/data/update" dataProperties: - propertyName: pressure metadata: type: int - propertyName: temperature metadata: type: int ``` ### Device Mapper 是连接和控制设备的应用程序，包含功能： 1) 扫描和连接设备 2) 报告设备孪生属性（ twin-attributes）的真实状态 3) 转换预期状态到真实状态 4) 采集遥感数据 5) 将设备的读取转换成KubeEdge可以接受的格式 6) 安排设备上的动作(schedule) 7) 检查设备的健康状态 ![img](imgs/KubeEdge/mapper-design.png) ### 创建步骤 1. kubectl apply -f 2. kubectl apply -f 3. 执行基于指定协议的mapper程序 4. 改变设备实例yaml中的状态段,应用改变将影响边缘端，通过设备控制和设备孪生组件（DeviceController和DeviceTwin） 5. 边缘端的mapper进程上报设备孪生的数据，并通过DeviceController同步回云端。 ### Router Manager 路由管理 通过CRDs和Router模块，基于mqtt，实现云边消息通信。 控制用户数据的传输，一次传输的大小不超过12M。 使用场景： + 通过云端api发布自定义消息，最终到达终端mqtt代理 + 边缘发布消息到mqtt代理，最终通过RESTApi返回云端 + 通过云端api发布自定义消息，最终调用边缘rest api传送消息 #### 规则的定义 RuleEndpoint 和 Rule Definition: + RuleEndpoint 定义消息从哪里来的、或到哪里去，包括3中类型 + rest + eventbus (MQTT) + servicebus (边缘端的Rest Api应用) + Rule 定义从源Endpoint到目标Endpoint消息传输的方式，包含3中类型： + rest->eventbus : cloud api --> edge mqtt + eventbus->rest: edge mqtt --> cloud api + rest->servicebus: cloud api --> user's app RuleEndpoint定义 ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: ruleendpoints.rules.kubeedge.io spec: group: rules.kubeedge.io versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: ruleEndpointType: type: rest/eventbus/servicebus required: - ruleEndpointType scope: Namespaced names: plural: ruleendpoints singular: ruleendpoint kind: RuleEndpoint shortNames: - re ``` Rule定义： ```yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: rules.rules.kubeedge.io spec: group: rules.kubeedge.io versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: source: type: string value: my-rest sourceResource: description: {"path":"/a/b"} 或 {"topic":"","node_name":"edge-node"} type: object additionalProperties: type: string target: type: string value: my-eventbus targetResource: description: {"topic":"/test"} type: object additionalProperties: type: string required: - source - sourceResource - target - targetResource status: type: object properties: successMessages: type: integer failMessages: type: integer errors: items: type: string type: array scope: Namespaced names: plural: rules singular: rule kind: Rule ``` ### [发消息的步骤](https://docs.kubeedge.io/en/docs/developer/custom_message_deliver/) 以rest->eventbus 模式举例 1. 云端配置 router模块使能 2. 创建ruleEndpoint 3. 创建rule ```yaml apiVersion: rules.kubeedge.io/v1 kind: Rule metadata: name: my-rule labels: description: test spec: source: "my-rest" sourceResource: {"path":"/test"} target: "my-eventbus" targetResource: {"topic":"test"} ``` 4. 云端调用api接口发送消息 ```shell POST http://{cloudcore_ip}:9443/{node_name}/{namespace}/{path} ``` 5. 边缘订阅 ```shell mosquitto_sub -t 'test' -d ``` MORE： 人工智能、数字孪生、边缘计算、量子计算、沉浸式技术、区块链、智能空间、5G