helm

背景

上一篇文章简单介绍了Kubernetes 单机版的开发测试环境minikube的搭建和环境配置,在很长的一段时间里面我也没有搭建了,今天突然下了一个最新版,发现启动启动不了,于是又乖乖minikube 退回到了0.25.2版本,由于最近都在使用Helm与Swift进行项目的部署发布,本文简单记录步骤.

概念

Helm

Helm是 k8s 集群部署和管理应用的一个包管理工具,使用Chart文件配置项目,隐藏了 k8s 本身的一些基本概念,具有版本回滚,方便升级的一些特性.
Helm分为客户端和服务端,服务端tiller运行在 k8s 集群中,用户管理和部署,更新 k8s集群中的应用,客户端有 Kubernetes-helm的客户端(命令行)工具,用于和服务端Tiller进行连接和通讯.

Swift

因为为了方便在代码中间调用Helm的一些接口不是特别方便,例如tiller都是走grpc协议等,所以社区的小伙伴们开发了这么一个代理(Swift),封装成为restful 的 http的接口形式,方便各种语言和tiller进行通信和操作.

搭建

本文 环境:
kubernetes 1.9.4
minikube 0.25.2
helm 2.12.1
swift 0.10.0
swift的版本需要和 helm 版本对应,参考https://github.com/appscode/swift

启动 minikube 环境

minikube start  --kubernetes-version v1.9.4  --extra-config=apiserver.Authorization.Mode=RBAC

• 必须开启 RBAC 的权限的模式,否则后续安装会有很多问题,比如默认的模式没有cluster-admin的集群权限,无法安装swift等.
• 如果用 helm 安装 swift 遇到这样的问题,就是这个原因.

Error: release my-searchlight failed: clusterroles.rbac.authorization.k8s.io "my-searchlight" is forbidden: attempt to grant extra privileges: [PolicyRule{APIGroups:["apiextensions.k8s.io"], Resources:["customresourcedefinitions"], Verbs:["*"]} PolicyRule{APIGroups:["extensions"], Resources:["thirdpartyresources"], Verbs:["*"]} PolicyRule{APIGroups:["monitoring.appscode.com"], Resources:["*"], Verbs:["*"]} PolicyRule{APIGroups:["storage.k8s.io"], Resources:["*"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["componentstatuses"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["componentstatuses"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["pods/exec"], Verbs:["create"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["create"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["list"]}] user=&{system:serviceaccount:kube-system:tilleruser a61ce1ed-0a6d-11e9-babc-0800274b952b [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]

• 用 k8s 1.9.4的原因在于,1.10.0没有默认的kube-dns模块了,有些不方便,而 swift 和 tiller 通信依赖 dns.
• 和上文一样配置一下代理,便于获取一些墙外的资源.

Helm 客户端安装

brew install kubernetes-helm
helm version 

Kube-dns安装

• 等待kube-dns 安装完成

kubectl get deployment -n kube-system --watch

• 创建 serviceaccount 账户

 kubectl create serviceaccount kube-dns -n kube-system

• 为kube-dns的 deployment设置关联关系

kubectl patch deployment kube-dns -n kube-system -p '{"spec":{"template":{"spec":{"serviceAccountName":"kube-dns"}}}}'

• 等待 dns 相关 pods 启动完成

kubectl get pods -n kube-system --watch

dns

部署 Tiller

• Tiller 作为 Helm 的服务端,版本和客户端保持一致.
• 部署 Tiller 之前需要为 tiller 赋予 k8s 的操作权限,创建流程如下

$ kubectl create serviceaccount tiller --namespace kube-system
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller
$ helm version --short
Client: v2.12.1+g02a47c7
Server: v2.12.1+g02a47c7

• 等待 tiller 部署完成

  
  tiller

部署 Swift

• 先为 helm 添加个 appscode 的仓库

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm repo list
NAME        URL
stable      https://kubernetes-charts.storage.googleapis.com
local       http://127.0.0.1:8879/charts
appscode    https://charts.appscode.com/stable/

• helm中搜索 swift

$ helm search swift
NAME            CHART VERSION   APP VERSION DESCRIPTION
appscode/swift  0.10.0          0.10.0      Swift by AppsCode - Ajax friendly Helm Tiller Proxy
stable/swift    0.6.3           0.7.3       DEPRECATED swift by AppsCode - Ajax friendly Helm Tiller ...

• 安装 0.10.0

helm install appscode/swift --name my-swift

• 检查是否安装完成

kubectl get pods --all-namespaces -l app=swift --watch

swift

测试

• 获取service 查看一下 ip 和端口

$ kubectl get svc swift-my-swift
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                          AGE
swift-my-swift   NodePort   10.107.55.194   <none>        9855:31743/TCP,50055:32523/TCP,56790:30092/TCP   3h58m

• minikube ssh后进入 minikube 访问,成功返回,说明已经 ok

$ curl  http://10.107.55.194:9855/tiller/v2/version/json
{"Version":{"sem_ver":"v2.12.1","git_commit":"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e","git_tree_state":"clean"}}

• 上面 service 中其实默认的 Type 应该是Cluster IP,为了宿主机能够访问,改成了NodePort,方法:

kubectl patch svc swift -n kube-system -p '{"spec":{"type":"NodePort"}}'

• 宿主机试一下

$ minikube ip
192.168.99.100

$ curl http://192.168.99.100:31743/tiller/v2/version/json
{"Version":{"sem_ver":"v2.12.1","git_commit":"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e","git_tree_state":"clean"}}%

• dashborad查看

  
  dashboard

大功告成!

ps:刚开始启动minikube 只用了minikube start折腾了半天.....

如何测试kube-dns是否在工作,参考官方案例
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

相关的错误

1. 无法解析内部 dns

I1228 09:25:08.241821       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:08.243483       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:25:28.242336       1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:25:28.242368       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:28.242629       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:25:53.349620       1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:25:53.349990       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:53.350133       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:26:32.635786       1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:26:32.635949       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:26:32.636553       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:27:12.647474       1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup tiller-deploy.kube-system.svc on 10.96.0.10:53: read udp 172.17.0.4:58290->10.96.0.10:53: i/o timeout". Reconnecting...
I1228 09:27:12.647527       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:27:44.000042       1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:28:08.235280       1 clientconn.go:830] Failed to dial tiller-deploy.kube-system.svc:44134: grpc: the connection is closing; please retry.

2. 发现 minikube 起来后,根本没有kube-dns和kube-dashborad,但是minikube addons list中这两个却是enabled的状态,参考解决方案https://github.com/kubernetes/minikube/issues/2619
   原理就是手动装了kube-dns组件让其正常工作.

参考

[1] https://github.com/appscode/swift/blob/master/docs/setup/rbac.md
[2] https://appscode.com/products/swift/0.10.0/guides/api/