kata container与kubernetes集成的几种方式

作者: ywhu | 来源:发表于2018-06-04 20:27 被阅读1404次

概览

kata container简介
cri-containerd plugin + containerd方式
cri-o方式

kata container简介

kata containers是由OpenStack基金会管理，但独立于OpenStack项目之外的容器项目。它是一个可以使用容器镜像以超轻量级虚机的形式创建容器的运行时工具。 kata containers整合了Intel的 Clear Containers 和 Hyper.sh 的 runV，能够支持不同平台的硬件（x86-64，arm等），并符合OCI(Open Container Initiative)规范，同时还可以兼容k8s的 CRI（Container Runtime Interface）接口规范。目前项目包含几个配套组件，即Runtime，Agent， Proxy，Shim，Kernel等。目前Kata Containers的运行时还没有整合，即Clear containers 和 runV 还在独立的组织内。

runtime：符合OCI规范的容器运行时命令工具，主要用来创建轻量级虚机，并通过agent控制虚拟机内容器的生命周期。目前kata containers还没有一个统一的运行时工具，用户可以选择 clear containers和runV中的其中之一。
agent：运行在虚机中的一个运行时代理组件，主要用来执行runtime传给他的指令，在虚机内管理容器的生命周期。
shim：以对接docker为例，这里的shim相当于是containerd-shim的一个适配，用来处理容器进程的stdio和signals。shim可以将containerd-shim发来的数据流（如stdin）传给proxy，然后转交给agent，也可以将agent经由proxy发过来的数据流（stdout/stderr）传给containerdshim，同时也可以传输signal。

kernel：kernel其实比较好理解，就是提供一个轻量化虚机的linux内核，根据不同的需要，提供几个内核选择，最小的内核仅有4M多

kata和传统容器对比

kata and docker

kata in kubernetes

OCI部分即为kata,也可以是符合oci标准的运行时容器。

以下环境安装默认都是overlayfs ,centos 建议版本7.4，7.5版本默认f_type=0,不支持D_TYPE参数，会导致containerd和crio启动失败

kata + cri-containerd plugin + containerd

环境版本

kubeadm      v1.10.3
kubenetes    v1.10.3
crictl       v1.0.0-beta.0
containerd   v1.1.0
kata-runtime 1.0.0

kubeadm安装

https://www.jianshu.com/p/3ec8945a864f

此处只需安装一个主节点即可，kata容器所在虚拟机均为从节点

镜像版本

k8s.gcr.io/kube-proxy-amd64               v1.10.3 
k8s.gcr.io/kube-apiserver-amd64           v1.10.3 
k8s.gcr.io/kube-scheduler-amd64           v1.10.3 
k8s.gcr.io/kube-controller-manager-amd64  v1.10.3 
k8s.gcr.io/etcd-amd64                     3.1.12  
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.8  
k8s.gcr.io/k8s-dns-sidecar-amd64          1.14.8  
k8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.8  
k8s.gcr.io/pause-amd64                    3.1     
quay.io/coreos/flannel                    v0.10.0-amd64

kata 安装

sudo yum -y install yum-utils

source /etc/os-release

sudo -E VERSION_ID=$VERSION_ID yum-config-manager --add-repo \
"http://download.opensuse.org/repositories/home:/katacontainers:/release/CentOS_${VERSION_ID}/home:katacontainers:release.repo"

sudo -E yum -y install kata-runtime kata-proxy kata-shim

安装完成后测试

[root@localhost tmp]# kata-runtime -v
kata-runtime  : 1.0.0
   commit   : a3d11b73ac4f1d5c5ae6451d83fc4b1fca385098
   OCI specs: 1.0.1

cri-containerd 安装

wget https://github.com/containerd/containerd/releases/download/v1.1.0/containerd-1.1.0.linux-amd64.tar.gz

sudo tar -C / -xzf  containerd-1.1.0.linux-amd64.tar.gz

sudo systemctl start containerd

sudo systemctl status containerd

containerd config default #生成默认配置，位置/etc/containerd/config.toml

crictl 安装

确保有go环境

go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

export PATH=$PATH:$GOPATH/bin

配置修改

crictl配置

[root@localhost bin]# cat /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

containerd配置

[root@localhost bin]# cat /etc/containerd/config.toml 
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = ""
    stream_server_port = "10010"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause-amd64:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      ## default_runtime为默认runtime
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      ## untrusted_workload_runtime当pod参数带有
      ## io.kubernetes.cri.untrusted-workload: "true"时生效
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://9c56nwew.mirror.aliyuncs.com", "https://registry-1.docker.io"]
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

各参数配置含义：https://github.com/containerd/cri/blob/release/1.0/docs/config.md

kubelet配置

[root@localhost bin]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意该行配置，修改kubelet启动使用containerd
Environment="KUBELET_EXTRA_ARGS=--runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

关于镜像下载的问题

由于默认镜像都是 k8s.gcr.io上的无法访问，所以最好有docker环境给镜像打tag，然后使用

ctr cri load xxx.tar

cri-o + kata

kata安装

同上

cri-o安装

yum install -y \
  btrfs-progs-devel \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  golang-github-cpuguy83-go-md2man \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  ostree-devel \
  pkgconfig \
  runc \
  skopeo-containers

git clone https://github.com/kubernetes-incubator/cri-o # or your fork
cd cri-o

make install.tools
make
sudo make install

##第一次安装时生成初始化配置文件
make install.config

make install.tools时可能有些package下载不下来，建议直接在git clone到go path下

配置

修改crio配置

vim /etc/crio/crio.conf

主要修改crio.runtime的配置

[crio.runtime]
runtime = "/usr/bin/runc"
runtime_untrusted_workload = "/usr/bin/kata-runtime"
default_workload_trust = "trusted"

kubelet配置

[root@localhost crio]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意该行配置，修改kubelet启动使用crio
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

关于镜像

cri-o缺少像containerd那样的镜像管理工具，个人采用的方式时修改主节点kube-proxy和kube-flannel-ds这两个DaemonSet的镜像为本地个人上传的阿里云的镜像，才能启动成功。

[root@kata-node1 tmp]# kubectl get ds -n kube-system
NAME              DESIRED   CURRENT   READY UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-flannel-ds   3            3        3       3             3    beta.kubernetes.io/arch=amd64   4h
kube-proxy        3            3        3       3             3       <none>                    5h              

kubectl edit kube-flannel-ds -n kube-system

加入集群

环境安装完成后各节点使用

kubeadm join 172.20.95.208:6443 --token ss60t6.fb1hymd6qxf5tbx7 --discovery-token-ca-cert-hash sha256:ea3a8f578071f73e7a5aab2e783f9d1cce22d4cf70faad24b8eb9090a4ef6b3b --ignore-preflight-errors all

具体命令使用主节点初始化完成后的kubeadm join命令，只需加上--ignore-preflight-errors all即可

测试

[root@kata-node1 tmp]# cat nginx-untrusted-1.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted-1
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx-1
    image: nginx


kubecrl create -f  nginx-untrusted-1.yaml

kata container与kubernetes集成的几种方式

概览

kata container简介

kata和传统容器对比

kata and docker

kata in kubernetes

以下环境安装默认都是overlayfs ,centos 建议版本7.4，7.5版本默认f_type=0,不支持D_TYPE参数，会导致containerd和crio启动失败

kata + cri-containerd plugin + containerd

环境版本

kubeadm安装

kata 安装

cri-containerd 安装

crictl 安装

配置修改

crictl配置

containerd配置

kubelet配置

关于镜像下载的问题

cri-o + kata

kata安装

cri-o安装

配置

修改crio配置

kubelet配置

关于镜像

加入集群

测试

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

OpenStack

云计算丶Docker丶K8s