ito-archived

kubernetes linux devops

21 KiB

Raw Blame History

概述

为开源产品，构建在Kubernetes之上，并将本机容器化应用程序编排和设备管理扩展到边缘主机。

优势：

原生支持kubernetes
云边缘可靠协作
边缘自治
边缘设备管理：通过CRD实现的Kubernetes本地api管理边缘设备。
极轻量边缘代理（EdgeCore）

组件：

Edged 运行在边缘节点的代理，管理容器化应用程序
EdgeHub 管理和云服务交互的web socket客户端。包括同步云端资源到边缘，上报边端主机和设备状态变更到云。
CloudHub 在云端运行的websocket服务，监听边端的事件
EdgeController： Kubernetes Controller的扩展，管理边缘node和pod元数据
EventBus： Mqtt的客户端用于连接Mqtt服务(mosquitto)，提供到其他组件的订阅和发布。
DeviceTwin：存储设备状态、同步设备状态到云端。并提供应用的查询接口
MetaManager：Edged和edgehub之间的消息处理器，它还负责在轻量级数据库（SQLite）中存储/检索元数据。

ServiceBus: a HTTP client to interact with HTTP servers (REST), offering HTTP client capabilities to components of cloud to reach HTTP servers running at edge.

官网：https://kubeedge.io/en/docs/setup/local/

Github: https://github.com/kubeedge/kubeedge

术语	描述

安装

下载

keadm-v1.7.1-linux-amd64.tar.gz keadm安装文件
kubeedge-v1.7.1-linux-amd64.tar.gz 二进制文件安装
edgesite-v1.7.1-linux-amd64.tar

通过Keadm来安装

keadm 用来安装 KubeEdge 的云和端组件。

云端

需要提前安装kubernetes。

keadm init --advertise-address="10.8.30.38"

将在服务器上安装 cloudcore，生成证书并安装CRDs。

获取token，将在添加边缘节点时使用。

 keadm gettoken

边端

keadm join --cloudcore-ipport=10.8.30.38:10000 --token=xxxx

将在客户端安装edgecore和mqtt。

通过二进制文件安装

或者参考https://www.cnblogs.com/kkbill/p/12600541.html

安装环境准备：

Cloud:10.8.30.38：kubernetes

Edge:10.8.30.35 ： golang docker mqtt

系统	ubuntu 16.04
golang	go1.10.4 linux/amd64
docker	19.03.12
k8s	1.18

准备安装KubeEdnge1.7.1：

将安装文件 `[kubeedge-v1.7.1-linux-amd64.tar.gz] 拷贝到服务器和边缘节点

云端

创建 CRDs

kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_device.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/devices/devices_v1alpha2_devicemodel.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/cluster_objectsync_v1alpha1.yaml
kubectl apply -f https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/crds/reliablesyncs/objectsync_v1alpha1.yaml

创建config文件

cloudcore --minconfig > cloudcore.yaml

配置

apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
kubeAPIConfig:
  kubeConfig: /root/.kube/config
  master: ""
modules:
  cloudHub:
    advertiseAddress:
    - 10.8.30.38
    enable: true
    https:
      address: 0.0.0.0
      enable: true
      port: 10002
    nodeLimit: 1000
    tlsCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsCAKeyFile: /etc/kubeedge/ca/rootCA.key
    tlsCertFile: /etc/kubeedge/certs/edge.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key
    unixsocket:
      address: unix:///var/lib/kubeedge/kubeedge.sock
      enable: true
    websocket:
      address: 0.0.0.0
      enable: true
      port: 10000
  router:
    address: 0.0.0.0
    enable: true
    port: 9443
    restTimeout: 60

运行

cloudcore --config cloudcore.yaml

生成证书

 wget https://raw.githubusercontent.com/kubeedge/kubeedge/master/build/tools/certgen.sh
chmod +x certgen.sh
./certgen.sh genCertAndKey edge
 
root@node38:/home/anxin/edge# ls /etc/kubeedge
ca  certs  crds

拷贝到边缘节点：
 scp /etc/kubeedge/ca/rootCA.crt root@node35:/etc/kubeedge/ca/
 scp /etc/kubeedge/certs/edge.crt root@node35:/etc/kubeedge/certs/
 scp /etc/kubeedge/certs/edge.key root@node35:/etc/kubeedge/certs/

获取token （将用于edge配置）

kubectl get secret -nkubeedge tokensecret -o json # 找到token 加入到edgecore.yml中
 
# kubectl get secret -nkubeedge default-token-pdrnq -o=jsonpath='{.data.tokendata}' | base64 -d
# sed -i -e "s|token: .*|token: ${token}|g" edgecore.yaml

边端

需要提前安装docker

初始化配置

./edgecore --minconfig > edgecore.yaml
./edgecore --defaultconfig > edgecore.yaml

将云端token配置到edgecore.yaml

# With --minconfig , you can easily used this configurations as reference.
# It's useful to users who are new to KubeEdge, and you can modify/create your own configs accordingly. 
# This configuration is suitable for beginners.

apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
  dataSource: /var/lib/kubeedge/edgecore.db
kind: EdgeCore
modules:
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://10.8.30.38:10002
    tlsCaFile: /etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    token: 5e630e7bf11b00a77a513233e067788e12d36179be1d4b83d804bf77c463781e.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2MjYyNDE2ODd9.oRE1mzeRVC7ohsA5KF9QP_EvxE83BSTjP6fuZuVL2Yk
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 10.8.30.38:10000
      writeDeadline: 15
  edged:
    #cgroupDriver: cgroupfs 通过docker info查看本机安装的cgroup-driver
    cgroupDriver: systemd
    cgroupRoot: ""
    cgroupsPerQOS: true
    clusterDNS: ""
    clusterDomain: ""
    devicePluginEnabled: false
    dockerAddress: unix:///var/run/docker.sock
    enable: true
    registerNode: true
    gpuPluginEnabled: false
    hostnameOverride: node35
    nodeIP: 10.8.30.35
    podSandboxImage: kubeedge/pause:3.1
    remoteImageEndpoint: unix:///var/run/dockershim.sock
    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    runtimeType: docker
  eventBus:
    enable: true
    mqttMode: 2
    mqttQOS: 0
    mqttRetain: false
    mqttServerExternal: tcp://127.0.0.1:1883
    mqttServerInternal: tcp://127.0.0.1:1884

启动

edgecore --config edgecore.yaml

启动后 docker ps:

在cloud端查看

root@node38:/home/anxin# kubectl get no 
NAME     STATUS   ROLES        AGE    VERSION
node35   Ready    agent,edge   334d   v1.19.3-kubeedge-v1.7.1
node37   Ready    <none>       334d   v1.18.6
node38   Ready    master       334d   v1.18.6

常见报错

Error: token credentials are in the wrong format

配置的token格式不对，从cloud端拿到的是base64，需要取解密后字符串、

cgroup-driver

需要通过docker info命令查看自己安装的docker的cgroup-driver，然后修改配置文件中的cgroup-driver

无法访问平台侧的证书文件 F0709 14:54:31.063677 3153 certmanager.go:92] Error: failed to get edge certificate from the cloudcore, error: Get "https://10.8.30.38:10002/edge.crt": x509: cannot validate certificate for 10.8.30.38 because it doesn't contain any IP SANs

I0709 15:27:42.098408   14300 server.go:243] Ca and CaKey don't exist in local directory, and will read from the secret
I0709 15:27:42.103401   14300 server.go:288] CloudCoreCert and key don't exist in local directory, and will read from the secret

解决方法：

删除原来的配置

kubectl delete secret casecret -nkubeedge
kubectl delete secret cloudcoresecret -nkubeedge
# both cloud and edge side
mv /etc/kubeedge/ca  /etc/kubeedge/ca.bak 
mv  /etc/kubeedge/certs /etc/kubeedge/certs.bak

执行： ./certgen.sh genCertAndKey edge

新问题：

Can't load /root/.rnd (第一次生成的时候可能就错在这里)

cd /root

openssl rand -writerand .rnd

在Raspberry Pi 上安装

安装Lite系统

https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-05-28/2021-05-07-raspios-buster-armhf-lite.zip

设置网络

interface eth0 
static ip_address=10.8.30.197/24 
static routers=10.8.30.1 
static domain_name_servers=114.114.114.114 223.5.5.5 223.6.6.6 

#设置密码
passwd pi

PasswordAuthentication yes

systemctl enable ssh

安装mosquitto

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key

sudo apt-key add mosquitto-repo.gpg.key

# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-jessie.list
# sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-stretch.list
sudo wget -P /etc/apt/sources.list.d/ http://repo.mosquitto.org/debian/mosquitto-buster.list 

apt-get update

apt-get install mosquitto

systemctl status mosquitto

安装docker

sudo apt-get update && sudo apt-get upgrade

curl -fsSL https://get.docker.com -o get-docker.sh

sudo sh get-docker.sh

sudo docker version

sudo docker run hello-world

sudo usermod -aG docker pi

安装containerd

Setting different container runtime with CRI

修改/etc/apt/sources.list

deb http://mirrors.163.com/raspbian/raspbian/ buster main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
#deb-src http://raspbian.raspberrypi.org/raspbian/ buster main contrib non-free rpi

安装

 sudo apt-get update
 sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
    
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

 echo \
  "deb [arch=armhf signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  
 sudo apt-get update
 sudo apt-get install containerd.io

# Configure containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# Restart containerd
systemctl restart containerd

安装golang

sudo apt update
sudo apt full-upgrade

wget https://dl.google.com/go/go1.16.6.linux-armv6l.tar.gz -O go.tar.gz

配置运行EdgeCore

配置EdgeCore.yaml支持containerd.。默认cgroup 配置是 cgroupfs。

remoteRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
remoteImageEndpoint: unix:///var/run/containerd/containerd.sock
runtimeRequestTimeout: 2
podSandboxImage: k8s.gcr.io/pause:3.2
runtimeType: remote

配置

启动：

问题

Cgroup subsystem not mounted:[memory]

Following Cgroup subsystem not mounted: [memory]

https://zhuanlan.zhihu.com/p/384443481

#解决问题
#修改/boot/cmdline.txt
sudo vim /boot/cmdline.txt
cgroup_enable=memory cgroup_memory=1
添加在同一行的最后面,接着内容后空格后添加, 注意:不要换行添加

#重启机器配置生效
reboot

配置

云端

设置边缘端自动注册 modules.edged.registerNode=true。或者使用手动注册。

边端

设置基础镜像：

modules.edged.podSandboxImage

检查机器架构：

getconf LONG_BIT

kubeedge/pause-arm:3.1 for arm arch
kubeedge/pause-arm64:3.1 for arm64 arch
kubeedge/pause:3.1 for x86 arch

容器运行时

runtimeType: docker

runtimeType: remote

MQTT模式

MQTT用于DeviceTwin和Devices之间的通信，支持3中MQTT模式：

0 internalMqttMode 内部MQTT代理使能

1 bothMqttMonde 内部和外部MQTT代理均使能

2 externalMqttMode 外部代理使能

EdgeMesh

云端和边缘之间发生网络问题，集成EdgeMesh 支持DNS访问

grep hosts /etc/nsswitch.conf
hosts:          dns file mdns4_minimal # 确保dns在第一位

IP Forward设置

$ sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
$ sudo sysctl -p
#check
$ sudo sysctl -p | grep ip_forward
net.ipv4.ip_forward = 1

部署一个nginx例子

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          hostPort: 8050

kubeclt apply -f pod-nginx.yaml

按照官网例子访问容器ip curl 10.0.35.3:8050，无法访问，直接访问http://10.8.30.35:8050 可以。

创建一个服务：

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: default
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
    - name: http-0
      port: 18050
      protocol: TCP
      targetPort: 80

在边缘端访问： <service_name>.<service_namespace>.svc.<cluster>.<local>:<port>

curl http://nginx-svc.default.svc.cluster.local:18050

开发指南

Device Model 设备原型

KubeEdge支持通过Kubernetes CRDS，以及Device Mapper进行设备管理

apiVersion: devices.kubeedge.io/v1alpha2
kind: DeviceModel
metadata:
 name: sensor-tag-model
 namespace: default
spec:
 properties:
  - name: temperature
    description: temperature in degree celsius
    type:
     int:
      accessMode: ReadWrite
      maximum: 100
      unit: degree celsius
  - name: temperature-enable
    description: enable data collection of temperature sensor
    type:
      string:
        accessMode: ReadWrite
        defaultValue: 'OFF'

Device Instance

apiVersion: devices.kubeedge.io/v1alpha2
kind: Device
metadata:
  name: sensor-tag-instance-01
  labels:
    description: TISimplelinkSensorTag
    manufacturer: TexasInstruments
    model: CC2650
spec:
  deviceModelRef:
    name: sensor-tag-model
  protocol:
    modbus:
      slaveID: 1
    common:
      com:
        serialPort: '1'
        baudRate: 115200
        dataBits: 8
        parity: even
        stopBits: 1
      customizedValues: # 自定义属性值
        def1: def1-val
        def2: def2-val
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: ''
        operator: In
        values:
        - node1
  propertyVisitors:
    - propertyName: temperature
      modbus:
        register: CoilRegister
        offset: 2
        limit: 1
        scale: 1
        isSwap: true
        isRegisterSwap: true
    - propertyName: temperature-enable
      modbus:
        register: DiscreteInputRegister
        offset: 3
        limit: 1
        scale: 1.0
        isSwap: true
        isRegisterSwap: true
    - propertyName: temperature # 自定义协议写法
      collectCycle: 500000000
      reportCycle: 1000000000
      customizedProtocol:
        protocolName: MY-TEST-PROTOCOL
        configData:
          def1: def1-val
          def2: def2-val
          def3:
            innerDef1: idef-val
status:
  twins:
    - propertyName: temperature
      reported:
        metadata:
          timestamp: '1550049403598'
          type: int
        value: '10'
      desired:
        metadata:
          timestamp: '1550049403598'
          type: int
        value: '15'
data: # 定义通过mappers上报到边缘端MQTT代理的一系列时序属性
  dataTopic: "$ke/events/device/+/data/update"
  dataProperties: 
    - propertyName: pressure
      metadata:
        type: int
    - propertyName: temperature
      metadata:
        type: int

Device Mapper

是连接和控制设备的应用程序，包含功能：

扫描和连接设备
报告设备孪生属性（ twin-attributes）的真实状态
转换预期状态到真实状态
采集遥感数据
将设备的读取转换成KubeEdge可以接受的格式
安排设备上的动作(schedule)
检查设备的健康状态

创建步骤

kubectl apply -f
kubectl apply -f
执行基于指定协议的mapper程序
改变设备实例yaml中的状态段,应用改变将影响边缘端，通过设备控制和设备孪生组件（DeviceController和DeviceTwin）
边缘端的mapper进程上报设备孪生的数据，并通过DeviceController同步回云端。

Router Manager 路由管理

通过CRDs和Router模块，基于mqtt，实现云边消息通信。控制用户数据的传输，一次传输的大小不超过12M。

使用场景：

通过云端api发布自定义消息，最终到达终端mqtt代理
边缘发布消息到mqtt代理，最终通过RESTApi返回云端
通过云端api发布自定义消息，最终调用边缘rest api传送消息

规则的定义

RuleEndpoint 和 Rule Definition:

RuleEndpoint 定义消息从哪里来的、或到哪里去，包括3中类型
- rest
- eventbus (MQTT)
- servicebus (边缘端的Rest Api应用)
Rule 定义从源Endpoint到目标Endpoint消息传输的方式，包含3中类型：
- rest->eventbus : cloud api --> edge mqtt
- eventbus->rest: edge mqtt --> cloud api
- rest->servicebus: cloud api --> user's app

RuleEndpoint定义

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ruleendpoints.rules.kubeedge.io
spec:
  group: rules.kubeedge.io
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                ruleEndpointType:
                  type: rest/eventbus/servicebus
              required:
                - ruleEndpointType
  scope: Namespaced
  names:
    plural: ruleendpoints
    singular: ruleendpoint
    kind: RuleEndpoint
    shortNames:
      - re

Rule定义：

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: rules.rules.kubeedge.io
spec:
  group: rules.kubeedge.io
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                source:
                  type: string
                  value: my-rest
                sourceResource:
                  description: {"path":"/a/b"} 或 {"topic":"<user define string>","node_name":"edge-node"}
                  type: object
                  additionalProperties:
                    type: string
                target:
                  type: string
                  value: my-eventbus
                targetResource:
                  description: {"topic":"/test"}
                  type: object
                  additionalProperties:
                    type: string
              required:
                - source
                - sourceResource
                - target
                - targetResource
            status:
              type: object
              properties:
                successMessages:
                  type: integer
                failMessages:
                  type: integer
                errors:
                  items:
                    type: string
                  type: array
  scope: Namespaced
  names:
    plural: rules
    singular: rule
    kind: Rule

发消息的步骤

以rest->eventbus 模式举例

云端配置 router模块使能
创建ruleEndpoint

创建rule

apiVersion: rules.kubeedge.io/v1
kind: Rule
metadata:
  name: my-rule
  labels:
    description: test
spec:
  source: "my-rest"
  sourceResource: {"path":"/test"}
  target: "my-eventbus"
  targetResource: {"topic":"test"}

云端调用api接口发送消息

POST
http://{cloudcore_ip}:9443/{node_name}/{namespace}/{path}

边缘订阅
```
mosquitto_sub -t 'test' -d
```

人工智能、数字孪生、边缘计算、量子计算、沉浸式技术、区块链、智能空间、5G

21 KiB Raw Blame History

概述

安装