istio(1.6.0)集群服务与 VM (虚拟机)服务进行 M

作者: 陈谭军 | 来源:发表于2020-07-05 13:09 被阅读0次

istio(1.6.0)集群服务与 VM (虚拟机)服务进行 M
XDH_LESSON6
黑苹果安装
Ubuntu搭建Hadoop
Istio 在阿里云容器服务的部署及流量治理实践
Istio 在阿里云容器服务的部署及流量治理实践
大数据||hadoop分布式集群安装
ZooKeeper集群搭建
Istio是什么？
VM上搭建iOS开发环境

使用 istio ingress-gateway将 VM或裸机主机集成到部署在 Kubernetes上的 Istio Mesh中.

前提条件

Kubernetes集群 + Istio集群 + VM

kubectl get nodes


NAME        STATUS  ROLES    AGE  VERSION

k8s-master  Ready    master  42d  v1.17.0

k8s-node01  Ready      42d  v1.17.0

k8s-node02  Ready      42d  v1.17.0

kubectl version


Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

kubectl get svc,pods -n istio-system


NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP  PORT(S)                                                                                                                  AGE

service/istio-ingressgateway  NodePort    10.32.0.178          15021:30658/TCP,80:30039/TCP,443:32501/TCP,15443:30149/TCP,15011:31708/TCP,15012:30605/TCP,8060:32204/TCP,853:32658/TCP  37h

service/istiod                NodePort    10.32.0.98            15010:30383/TCP,15012:31794/TCP,443:30403/TCP,15014:30095/TCP,53:30828/UDP,853:31315/TCP                                  37h

service/prometheus            ClusterIP  10.32.0.109          9090/TCP                                                                                                                  37h

NAME                                        READY  STATUS    RESTARTS  AGE

pod/istio-ingressgateway-569669bb67-c9rgn  1/1    Running  0          37h

pod/istiod-788cf6c878-gxqzg                1/1    Running  0          37h

pod/prometheus-79878ff5fd-jppb4            2/2    Running  0          37h

istioctl version


client version: 1.6.0

control plane version: 1.6.0

data plane version: 1.6.0 (3 proxies)

测试步骤

在 /home/k8s-master/vm-test目录下载 Istio 1.6.0 release包.
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.0 sh -
设置 istioctl到环境变量中.

cd istio-1.6.0 && export PATH=$PWD/bin:$PATH

设置测试步骤所需的环境变量


ISTIO_DIR="/home/k8s-master/vm-test/istio-1.6.0"

CLUSTER_NAME="cluster150"

SERVICE_NAMESPACE="vm"

WORK_DIR="/home/k8s-master/vm-test/cluster-vm"

创建工作目录

mkdir -p "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"

创建用于虚拟机和 Istio控制平面通信认证的证书

cd "${WORK_DIR}"
make -f "${ISTIO_DIR}"/samples/certs/Makefile NAME="${CLUSTER_NAME}" NAMESPACE="${SERVICE_NAMESPACE}" "${CLUSTER_NAME}"-certs-wl

Istio 1.6.0 GitHub源码库中不存在这个 Makefile文件，branch 1.6.0没有这个文件，tag 1.6.0中存在这个文件

注意：Istio 1.6.0以上，使用 make -f "${ISTIO_DIR}"/tools/certs/Makefile NAME="${CLUSTER_NAME}" NAMESPACE="${SERVICE_NAMESPACE}" "${CLUSTER_NAME}"-certs-wl

在集群中注册所需的 secret


kubectl create namespace istio-system

kubectl create secret generic cacerts -n istio-system \

--from-file="${WORK_DIR}"/"${CLUSTER_NAME}"/ca-cert.pem \

--from-file="${WORK_DIR}"/"${CLUSTER_NAME}"/ca-key.pem \

--from-file="${WORK_DIR}"/"${CLUSTER_NAME}"/root-cert.pem \

--from-file="${WORK_DIR}"/"${CLUSTER_NAME}"/cert-chain.pem

创建安装 istio集群 yaml文件


cat < "${WORK_DIR}"/vmintegration.yaml

apiVersion: install.istio.io/v1alpha1

metadata:

namespace: istio-system

name: example-istiocontrolplane

kind: IstioOperator

spec:

values:

global:

meshExpansion:

enabled: true

EOF

安装 istio集群

    istioctl install -f "${WORK_DIR}"/vmintegration.yaml

集群安装成功后,编辑 istiod, istio-ingressgateway svc type类型为 NodePort

kubectl edit svc istiod -n istio-system更换 type: NodePort

kubectl edit svc istio-ingressgateway -n istio-system更换 type: NodePort

1.png

准备复制到 VM机器的文件


cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/ca-cert.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/

cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/key.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/

cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/root-cert.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/

cp -a "${WORK_DIR}"/"${CLUSTER_NAME}"/workload-cert-chain.pem "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/cert-chain.pem

准备 cluster.env文件


ISTIO_SERVICE_CIDR=$(echo '{"apiVersion":"v1","kind":"Service","metadata":{"name":"tst"},"spec":{"clusterIP":"1.1.1.1","ports":[{"port":443}]}}' | kubectl apply -f - 2>&1 | sed 's/.*valid IPs is //')

touch "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/cluster.env

echo ISTIO_SERVICE_CIDR=$ISTIO_SERVICE_CIDR > "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/cluster.env

echo "ISTIO_INBOUND_PORTS=3306,8080" >> "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/cluster.env


NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP  PORT(S)                                                                                                                  AGE

service/istio-ingressgateway  NodePort    10.32.0.178          15021:30658/TCP,80:30039/TCP,443:32501/TCP,15443:30149/TCP,15011:31708/TCP,15012:30605/TCP,8060:32204/TCP,853:32658/TCP  37h

service/istiod                NodePort    10.32.0.98            15010:30383/TCP,15012:31794/TCP,443:30403/TCP,15014:30095/TCP,53:30828/UDP,853:31315/TCP                                  37h

service/prometheus            ClusterIP  10.32.0.109          9090/TCP

ISTIO_PILOT_PORT对应于 svc istiod中的 15012:31794 ISTIO_PILOT_PORT在这里是 31794

echo ISTIO_PILOT_PORT=31794 >> "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/cluster.env

创建 vm所需的 hosts文件


  192.168.140.150 是主机的 IP 地址

touch "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/hosts-addendum

echo "192.168.140.150 istiod.istio-system.svc" > "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}"/hosts-addendum

复制必要的文件到 VM机器


scp -r "${WORK_DIR}"/"${CLUSTER_NAME}"/"${SERVICE_NAMESPACE}" k8s-develop@192.168.17.140:/home/k8s-develop/vm-test

在 VM执行以下操作

在 /home/k8s-develop/vm-test目录下执行以下操作：


sudo apt -y update

sudo apt -y upgrade

curl -LO https://storage.googleapis.com/istio-release/releases/1.6.0/deb/istio-sidecar.deb

sudo dpkg -i istio-sidecar.deb

sudo mkdir -p /etc/certs

sudo cp {root-cert.pem,cert-chain.pem,key.pem} /etc/certs

sudo cp cluster.env /var/lib/istio/envoy

sudo sh -c 'cat hosts-addendum >> /etc/hosts'

sudo chown -R istio-proxy /etc/certs /var/lib/istio/envoy

sudo systemctl start istio

其中


root@ubuntu:/home/k8s-develop/vm-test# cat vm/cluster.env

ISTIO_SERVICE_CIDR=10.32.0.0/24

ISTIO_INBOUND_PORTS=3306,8080

ISTIO_PILOT_PORT=31794


root@ubuntu:/home/k8s-develop/vm-test# cat /etc/hosts

127.0.0.1  localhost

127.0.1.1  ubuntu

# The following lines are desirable for IPv6 capable hosts

::1    ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

192.168.17.150 istiod.istio-system.svc


root@ubuntu:/home/k8s-develop/vm-test# sudo systemctl status istio

● istio.service - istio-sidecar: The Istio sidecar

Loaded: loaded (/lib/systemd/system/istio.service; disabled; vendor preset: enabled)

Active: active (running) since Fri 2020-07-03 07:37:01 PDT; 1 day 13h ago

Docs: http://istio.io/

Main PID: 56788 (su)

Tasks: 0 (limit: 4632)

CGroup: /system.slice/istio.service

              ‣56788 su -s /bin/bash -c INSTANCE_IP=192.168.17.140 POD_NAME=ubuntu POD_NAMESPACE=default exec /usr/local/bin/pilot-agent proxy  2> /var/log/istio/istio.err.

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: -A ISTIO_OUTPUT -m owner --gid-owner 0 -j RETURN

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: -A ISTIO_OUTPUT -d 10.32.0.0/24 -j ISTIO_REDIRECT

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: -A ISTIO_OUTPUT -j RETURN

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: COMMIT

Jul 03 07:37:01 ubuntu istio-start.sh[56788]: # Completed on Fri Jul  3 07:37:01 2020

Jul 03 07:37:01 ubuntu su[56788]: Successful su for istio-proxy by root

Jul 03 07:37:01 ubuntu su[56788]: + ??? root:istio-proxy

Jul 03 07:37:01 ubuntu su[56788]: pam_unix(su:session): session opened for user istio-proxy by (uid=0)

在 VM执行以下命令，启动一个服务

python -m SimpleHTTPServer 8080


k8s-develop@ubuntu:~/vm-test/vm$ ls

ca-cert.pem  cert-chain.pem  cluster.env  hosts-addendum  key.pem  root-cert.pem

k8s-develop@ubuntu:~/vm-test/vm$ python -m SimpleHTTPServer 8080

Serving HTTP on 0.0.0.0 port 8080 ...

在 istio集群中添加 vm刚刚启动的服务

    istioctl experimental add-to-mesh external-service vmhttp ${VM_IP} http:8080 -n ${SERVICE_NAMESPACE}

istioctl experimental add-to-mesh external-service vmhttp 192.168.17.140 http:8080 -n vm


root@k8s-master:/home/k8s-master/vm-test# kubectl get se,svc -n vm

NAME                                                    HOSTS                          LOCATION        RESOLUTION  AGE

serviceentry.networking.istio.io/mesh-expansion-vmhttp  [vmhttp.vm.svc.cluster.local]  MESH_INTERNAL  STATIC      37h

NAME            TYPE        CLUSTER-IP    EXTERNAL-IP  PORT(S)    AGE

service/vmhttp  ClusterIP  10.32.0.157          8080/TCP  37h

在 istio集群中启动 sleep示例服务

    kubectl apply -f samples/sleep/sleep.yaml

kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl vmhttp.vm.svc.cluster.local:8080


sleep-f8cbf5b76-w8vd8root@k8s-master:/home/k8s-master/vm-test# kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl vmhttp.vm.svc.cluster.local:8080

Directory listing for /

Directory listing for /

[ca-cert.pem](https://www.jianshu.com/ca-cert.pem)

[cert-chain.pem](https://www.jianshu.com/cert-chain.pem)

[cluster.env](https://www.jianshu.com/cluster.env)

[hosts-addendum](https://www.jianshu.com/hosts-addendum)

[key.pem](https://www.jianshu.com/key.pem)

[root-cert.pem](https://www.jianshu.com/root-cert.pem)

2.png

3.png

istiod日志

5.png

vm envoy日志

4.png

问题解疑

之前在某个测试集群中发现，证书认证过期的问题

istiod 日志如下：

2020-07-03T01:53:13.005120Z    info  No certificates specified, skipping K8S DNS certificate controller

2020-07-03T01:53:13.007660Z    info  CRD controller watching namespaces ""

2020-07-03T01:53:13.060350Z    info  Ingress controller watching namespaces ""

2020-07-03T01:53:13.112044Z    info  Skipping IngressClass, resource not supported

2020-07-03T01:53:13.112409Z    warn  Config Store &{0xc0009ac6c0 cluster.local 0xc000956580 0xc000d59440 0xc000a56000 [] [] } cannot track distribution in aggregate: this SetLedger operation is not supported by kube ingress controller

2020-07-03T01:53:13.112446Z    info  Adding Kubernetes registry adapter

2020-07-03T01:53:13.112458Z    info  Service controller watching namespace "" for services, endpoints, nodes and pods, refresh 1m0s

2020-07-03T01:53:13.112621Z    info  JWT policy is first-party-jwt

2020-07-03T01:53:13.112728Z    info  Use local CA certificate

2020-07-03T01:53:13.113287Z    error  failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z)

Error: failed to create discovery service: failed to create CA: failed to create an istiod CA: failed to create CA KeyCertBundle (cannot verify the cert with the provided root chain and cert pool with error: x509: certificate has expired or is not yet valid: current time 2020-07-03T01:53:13Z is before 2020-07-03T02:00:31Z

有证书认证问题的集群的时区是date Fri Jul  3 22:57:47 CST 2020

没有遇到证书认证问题的集群的时区是date Fri Jul  3 07:57:53 PDT 2020

可能是由于集群节点之间的 date 日期不同步的原因导致的

https://github.com/istio/istio/issues/25108

https://github.com/istio/istio/issues/23108

参考文献

https://istio.io/latest/docs/setup/install/virtual-machine/

https://istio.io/latest/docs/examples/virtual-machines/multi-network/

istio(1.6.0)集群服务与 VM (虚拟机)服务进行 M
使用 istio ingress-gateway将 VM或裸机主机集成到部署在 Kubernetes上的 Isti...
XDH_LESSON6
服务器集群搭建、管理与快速部署四台ubuntu虚拟机搭建集群服务器。什么叫服务器集群?服务器集群就是指将很多服务器...
黑苹果安装
1.先安装VM虚拟机2.停止VM服务操作:此电脑上右击->管理->服务和应用程序->服务->按名称排序->将vm开...
Ubuntu搭建Hadoop
环境服务器(虚拟机)： vm-master 10.211.55.23 vm-slave1 10.211....
Istio 在阿里云容器服务的部署及流量治理实践
目标在阿里云容器服务 Kubernetes 集群上部署 Istio 服务网格实践灰度发布、故障注入、熔断等 I...
Istio 在阿里云容器服务的部署及流量治理实践
目标在阿里云容器服务 Kubernetes 集群上部署 Istio 服务网格实践灰度发布、故障注入、熔断等 I...
大数据||hadoop分布式集群安装
集群前先计划虚拟机，看文章大数据||Hadoop分布式部署虚拟机基于伪分布式环境安装进行展开规划机器与服务（★★...
ZooKeeper集群搭建
Hadoop集群基础配置 hadoop伪分布式系统配置VM安装Linux虚拟机环境VM克隆Linux虚拟机集群配置...
Istio是什么？
本文源自《云原生服务网关Istio——原理、实践、架构与源码分析》 1. Istio是什么？ Istio 是一个与...
VM上搭建iOS开发环境
安装虚拟机VM 10 安装unlock130 先把VM的相关服务停止进入unlock-all-v130\wind...