21 KiB
概述
为开源产品,构建在Kubernetes之上,并将本机容器化应用程序编排和设备管理扩展到边缘主机。
优势:
- 原生支持kubernetes
- 云边缘可靠协作
- 边缘自治
- 边缘设备管理:通过CRD实现的Kubernetes本地api管理边缘设备。
- 极轻量边缘代理 (EdgeCore)
组件:
- Edged 运行在边缘节点的代理,管理容器化应用程序
- EdgeHub 管理和云服务交互的web socket客户端。包括同步云端资源到边缘,上报边端主机和设备状态变更到云。
- CloudHub 在云端运行的websocket服务,监听边端的事件
- EdgeController: Kubernetes Controller的扩展,管理边缘node和pod元数据
- EventBus: Mqtt的客户端用于连接Mqtt服务(mosquitto),提供到其他组件的订阅和发布。
- DeviceTwin:存储设备状态、同步设备状态到云端。并提供应用的查询接口
- MetaManager:Edged和edgehub之间的消息处理器,它还负责在轻量级数据库(SQLite)中存储/检索元数据。
- ServiceBus: a HTTP client to interact with HTTP servers (REST), offering HTTP client capabilities to components of cloud to reach HTTP servers running at edge.
官网:https://kubeedge.io/en/docs/setup/local/
Github: https://github.com/kubeedge/kubeedge
术语 | 描述 |
---|---|
安装
- keadm-v1.7.1-linux-amd64.tar.gz keadm安装文件
- kubeedge-v1.7.1-linux-amd64.tar.gz 二进制文件安装
- edgesite-v1.7.1-linux-amd64.tar
通过Keadm来安装
keadm 用来安装 KubeEdge 的云和端组件。
云端
需要提前安装kubernetes。
keadm init --advertise-address="10.8.30.38"
将在服务器上安装 cloudcore,生成证书并安装CRDs。
获取token,将在添加边缘节点时使用。
keadm gettoken
边端
keadm join --cloudcore-ipport=10.8.30.38:10000 --token=xxxx
将在客户端安装edgecore和mqtt。
通过二进制文件安装
或者参考https://www.cnblogs.com/kkbill/p/12600541.html
安装环境准备:
Cloud:10.8.30.38
:kubernetes
Edge:10.8.30.35
: golang docker mqtt
系统 | ubuntu 16.04 |
---|---|
golang | go1.10.4 linux/amd64 |
docker | 19.03.12 |
k8s | 1.18 |
准备安装KubeEdnge1.7.1:
将安装文件 `[kubeedge-v1.7.1-linux-amd64.tar.gz] 拷贝到服务器和边缘节点
云端
创建 CRDs
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_device.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_devicemodel.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/objectsync_v1alpha1.yaml
创建config文件
cloudcore --minconfig > cloudcore.yaml
apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
kubeAPIConfig:
kubeConfig: /root/.kube/config
master: ""
modules:
cloudHub:
advertiseAddress:
- 10.8.30.38
enable: true
https:
address: 0.0.0.0
enable: true
port: 10002
nodeLimit: 1000
tlsCAFile: /etc/kubeedge/ca/rootCA.crt
tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key
tlsCertFile: /etc/kubeedge/certs/edge.crt
tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key
unixsocket:
address: unix:///var/lib/kubeedge/kubeedge.sock
enable: true
websocket:
address: 0.0.0.0
enable: true
port: 10000
router:
address: 0.0.0.0
enable: true
port: 9443
restTimeout: 60
运行
cloudcore --config cloudcore.yaml
生成证书
wget https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/tools/certgen.sh
chmod +x certgen.sh
./certgen.sh genCertAndKey edge
root@node38:/home/anxin/edge# ls /etc/kubeedge
ca certs crds
拷贝到边缘节点:
scp /etc/kubeedge/ca/rootCA.crt root@node35:/etc/kubeedge/ca/
scp /etc/kubeedge/certs/edge.crt root@node35:/etc/kubeedge/certs/
scp /etc/kubeedge/certs/edge.key root@node35:/etc/kubeedge/certs/
获取token (将用于edge配置)
kubectl get secret -nkubeedge tokensecret -o json # 找到token 加入到edgecore.yml中
# kubectl get secret -nkubeedge default-token-pdrnq -o=jsonpath='{.data.tokendata}' | base64 -d
# sed -i -e "s|token: .*|token: ${token}|g" edgecore.yaml
边端
需要提前安装docker
初始化配置
./edgecore --minconfig > edgecore.yaml
./edgecore --defaultconfig > edgecore.yaml
将云端token配置到edgecore.yaml
# With --minconfig , you can easily used this configurations as reference.
# It's useful to users who are new to KubeEdge, and you can modify/create your own configs accordingly.
# This configuration is suitable for beginners.
apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
dataSource: /var/lib/kubeedge/edgecore.db
kind: EdgeCore
modules:
edgeHub:
enable: true
heartbeat: 15
httpServer: https://10.8.30.38:10002
tlsCaFile: /etc/kubeedge/ca/rootCA.crt
tlsCertFile: /etc/kubeedge/certs/server.crt
tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
token: 5e630e7bf11b00a77a513233e067788e12d36179be1d4b83d804bf77c463781e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjYyNDE2ODd9.oRE1mzeRVC7ohsA5KF9QP_EvxE83BSTjP6fuZuVL2Yk
websocket:
enable: true
handshakeTimeout: 30
readDeadline: 15
server: 10.8.30.38:10000
writeDeadline: 15
edged:
#cgroupDriver: cgroupfs 通过docker info查看本机安装的cgroup-driver
cgroupDriver: systemd
cgroupRoot: ""
cgroupsPerQOS: true
clusterDNS: ""
clusterDomain: ""
devicePluginEnabled: false
dockerAddress: unix:///var/run/docker.sock
enable: true
registerNode: true
gpuPluginEnabled: false
hostnameOverride: node35
nodeIP: 10.8.30.35
podSandboxImage: kubeedge/pause:3.1
remoteImageEndpoint: unix:///var/run/dockershim.sock
remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
runtimeType: docker
eventBus:
enable: true
mqttMode: 2
mqttQOS: 0
mqttRetain: false
mqttServerExternal: tcp://127.0.0.1:1883
mqttServerInternal: tcp://127.0.0.1:1884
启动
edgecore --config edgecore.yaml
启动后 docker ps:
在cloud端查看
root@node38:/home/anxin# kubectl get no
NAME STATUS ROLES AGE VERSION
node35 Ready agent,edge 334d v1.19.3-kubeedge-v1.7.1
node37 Ready <none> 334d v1.18.6
node38 Ready master 334d v1.18.6
常见报错
-
Error: token credentials are in the wrong format
配置的token格式不对,从cloud端拿到的是base64,需要取解密后字符串、
-
cgroup-driver
需要通过docker info命令查看自己安装的docker的cgroup-driver,然后修改配置文件中的cgroup-driver
- 无法访问平台侧的证书文件 F0709 14:54:31.063677 3153 certmanager.go:92] Error: failed to get edge certificate from the cloudcore, error: Get "https://10.8.30.38:10002/edge.crt": x509: cannot validate certificate for 10.8.30.38 because it doesn't contain any IP SANs
I0709 15:27:42.098408 14300 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret
I0709 15:27:42.103401 14300 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret
解决方法:
删除原来的配置
kubectl delete secret casecret -nkubeedge
kubectl delete secret cloudcoresecret -nkubeedge
# both cloud and edge side
mv /etc/kubeedge/ca /etc/kubeedge/ca.bak
mv /etc/kubeedge/certs /etc/kubeedge/certs.bak
执行: ./certgen.sh genCertAndKey edge
新问题:
Can't load /root/.rnd
(第一次生成的时候可能就错在这里)
cd /root
openssl rand -writerand .rnd
在Raspberry Pi 上安装
安装Lite系统
设置网络
interface eth0
static ip_address=10.8.30.197/24
static routers=10.8.30.1
static domain_name_servers=114.114.114.114 223.5.5.5 223.6.6.6
#设置密码
passwd pi
PasswordAuthentication yes
systemctl enable ssh
安装mosquitto
wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-jessie.list
# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-stretch.list
sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-buster.list
apt-get update
apt-get install mosquitto
systemctl status mosquitto
安装docker
sudo apt-get update && sudo apt-get upgrade
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo docker version
sudo docker run hello-world
sudo usermod -aG docker pi
安装containerd
Setting different container runtime with CRI
修改/etc/apt/sources.list
deb http://mirrors.163.com/raspbian/raspbian/ buster main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi
安装
sudo apt-get update
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg \
lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=armhf signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install containerd.io
# Configure containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# Restart containerd
systemctl restart containerd
安装golang
sudo apt update
sudo apt full-upgrade
wget https://dl.google.com/go/go1.16.6.linux-armv6l.tar.gz -O go.tar.gz
配置运行EdgeCore
配置EdgeCore.yaml支持containerd.。 默认cgroup 配置是 cgroupfs。
remoteRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
remoteImageEndpoint: unix:///var/run/containerd/containerd.sock
runtimeRequestTimeout: 2
podSandboxImage: k8s.gcr.io/pause:3.2
runtimeType: remote
配置
启动:
问题
-
Cgroup subsystem not mounted:[memory]
Following Cgroup subsystem not mounted: [memory]
https://zhuanlan.zhihu.com/p/384443481
#解决问题 #修改/boot/cmdline.txt sudo vim /boot/cmdline.txt cgroup_enable=memory cgroup_memory=1 添加在同一行的最后面,接着内容后空格后添加, 注意:不要换行添加 #重启机器配置生效 reboot
配置
云端
设置边缘端自动注册 modules.edged.registerNode=true。 或者使用手动注册。
边端
设置基础镜像:
modules.edged.podSandboxImage
检查机器架构:
getconf LONG_BIT
kubeedge/pause-arm:3.1
for arm archkubeedge/pause-arm64:3.1
for arm64 archkubeedge/pause:3.1
for x86 arch
容器运行时
runtimeType: docker
or
runtimeType: remote
MQTT模式
MQTT用于DeviceTwin和Devices之间的通信,支持3中MQTT模式:
0 internalMqttMode 内部MQTT代理使能
1 bothMqttMonde 内部和外部MQTT代理均使能
2 externalMqttMode 外部代理使能
EdgeMesh
云端和边缘之间发生网络问题,集成EdgeMesh 支持DNS访问
grep hosts /etc/nsswitch.conf
hosts: dns file mdns4_minimal # 确保dns在第一位
IP Forward设置
$ sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
$ sudo sysctl -p
#check
$ sudo sysctl -p | grep ip_forward
net.ipv4.ip_forward = 1
部署一个nginx例子
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
hostPort: 8050
kubeclt apply -f pod-nginx.yaml
按照官网例子访问容器ip curl 10.0.35.3:8050,无法访问,直接访问http://10.8.30.35:8050 可以。
创建一个服务:
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
namespace: default
spec:
clusterIP: None
selector:
app: nginx
ports:
- name: http-0
port: 18050
protocol: TCP
targetPort: 80
在边缘端访问: <service_name>.<service_namespace>.svc.<cluster>.<local>:<port>
curl http://nginx-svc.default.svc.cluster.local:18050
开发指南
Device Model 设备原型
KubeEdge支持通过Kubernetes CRDS,以及Device Mapper进行设备管理
apiVersion: devices.kubeedge.io/v1alpha2
kind: DeviceModel
metadata:
name: sensor-tag-model
namespace: default
spec:
properties:
- name: temperature
description: temperature in degree celsius
type:
int:
accessMode: ReadWrite
maximum: 100
unit: degree celsius
- name: temperature-enable
description: enable data collection of temperature sensor
type:
string:
accessMode: ReadWrite
defaultValue: 'OFF'
Device Instance
apiVersion: devices.kubeedge.io/v1alpha2
kind: Device
metadata:
name: sensor-tag-instance-01
labels:
description: TISimplelinkSensorTag
manufacturer: TexasInstruments
model: CC2650
spec:
deviceModelRef:
name: sensor-tag-model
protocol:
modbus:
slaveID: 1
common:
com:
serialPort: '1'
baudRate: 115200
dataBits: 8
parity: even
stopBits: 1
customizedValues: # 自定义属性值
def1: def1-val
def2: def2-val
nodeSelector:
nodeSelectorTerms:
- matchExpressions:
- key: ''
operator: In
values:
- node1
propertyVisitors:
- propertyName: temperature
modbus:
register: CoilRegister
offset: 2
limit: 1
scale: 1
isSwap: true
isRegisterSwap: true
- propertyName: temperature-enable
modbus:
register: DiscreteInputRegister
offset: 3
limit: 1
scale: 1.0
isSwap: true
isRegisterSwap: true
- propertyName: temperature # 自定义协议写法
collectCycle: 500000000
reportCycle: 1000000000
customizedProtocol:
protocolName: MY-TEST-PROTOCOL
configData:
def1: def1-val
def2: def2-val
def3:
innerDef1: idef-val
status:
twins:
- propertyName: temperature
reported:
metadata:
timestamp: '1550049403598'
type: int
value: '10'
desired:
metadata:
timestamp: '1550049403598'
type: int
value: '15'
data: # 定义通过mappers上报到边缘端MQTT代理的一系列时序属性
dataTopic: "$ke/events/device/+/data/update"
dataProperties:
- propertyName: pressure
metadata:
type: int
- propertyName: temperature
metadata:
type: int
Device Mapper
是连接和控制设备的应用程序,包含功能:
-
扫描和连接设备
-
报告设备孪生属性( twin-attributes)的真实状态
-
转换预期状态到真实状态
-
采集遥感数据
-
将设备的读取转换成KubeEdge可以接受的格式
-
安排设备上的动作(schedule)
-
检查设备的健康状态
创建步骤
- kubectl apply -f
- kubectl apply -f
- 执行基于指定协议的mapper程序
- 改变设备实例yaml中的状态段,应用改变将影响边缘端,通过设备控制和设备孪生组件(DeviceController和DeviceTwin)
- 边缘端的mapper进程上报设备孪生的数据,并通过DeviceController同步回云端。
Router Manager 路由管理
通过CRDs和Router模块,基于mqtt,实现云边消息通信。 控制用户数据的传输,一次传输的大小不超过12M。
使用场景:
- 通过云端api发布自定义消息,最终到达终端mqtt代理
- 边缘发布消息到mqtt代理,最终通过RESTApi返回云端
- 通过云端api发布自定义消息,最终调用边缘rest api传送消息
规则的定义
RuleEndpoint 和 Rule Definition:
- RuleEndpoint 定义消息从哪里来的、或到哪里去,包括3中类型
- rest
- eventbus (MQTT)
- servicebus (边缘端的Rest Api应用)
- Rule 定义从源Endpoint到目标Endpoint消息传输的方式,包含3中类型:
- rest->eventbus : cloud api --> edge mqtt
- eventbus->rest: edge mqtt --> cloud api
- rest->servicebus: cloud api --> user's app
RuleEndpoint定义
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ruleendpoints.rules.kubeedge.io
spec:
group: rules.kubeedge.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
ruleEndpointType:
type: rest/eventbus/servicebus
required:
- ruleEndpointType
scope: Namespaced
names:
plural: ruleendpoints
singular: ruleendpoint
kind: RuleEndpoint
shortNames:
- re
Rule定义:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: rules.rules.kubeedge.io
spec:
group: rules.kubeedge.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
source:
type: string
value: my-rest
sourceResource:
description: {"path":"/a/b"} 或 {"topic":"<user define string>","node_name":"edge-node"}
type: object
additionalProperties:
type: string
target:
type: string
value: my-eventbus
targetResource:
description: {"topic":"/test"}
type: object
additionalProperties:
type: string
required:
- source
- sourceResource
- target
- targetResource
status:
type: object
properties:
successMessages:
type: integer
failMessages:
type: integer
errors:
items:
type: string
type: array
scope: Namespaced
names:
plural: rules
singular: rule
kind: Rule
发消息的步骤
以rest->eventbus 模式举例
-
云端配置 router模块使能
-
创建ruleEndpoint
-
创建rule
apiVersion: rules.kubeedge.io/v1 kind: Rule metadata: name: my-rule labels: description: test spec: source: "my-rest" sourceResource: {"path":"/test"} target: "my-eventbus" targetResource: {"topic":"test"}
-
云端调用api接口发送消息
POST http://{cloudcore_ip}:9443/{node_name}/{namespace}/{path}
-
边缘订阅
mosquitto_sub -t 'test' -d
MORE:
人工智能、数字孪生、边缘计算、量子计算、沉浸式技术、区块链、智能空间、5G