1.排查应用 Pod 的 DNS 配置

#查找应用
$kubectl get pod  |grep flink
deployment-flink-jobmanager-c695cf9d-rgtbh             1/1     Running   0          2d23h
deployment-flink-taskmanager-7c7bbcd4db-5qv9k          1/1     Running   0          2d23h
deployment-flink-taskmanager-7c7bbcd4db-hnpm5          1/1     Running   0          2d23h

#查看应用中的DNS配置
$kubectl exec -it deployment-flink-jobmanager-c695cf9d-rgtbh -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

#查看DNS地址是否正确
$kubectl get svc -nkube-system |grep 10.96.0.10
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   32d

2.排查 DNS 是否启用 Service 及其后端 Pod 是否正常运行

#查找dns的service
$kubectl get svc -nkube-system -owide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   32d   k8s-app=kube-dns

#查看 service 详情
$kubectl describe svc -nkube-system kube-dns
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP Families:       <none>
IP:                10.96.0.10
IPs:               10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.244.0.34:53,10.244.0.66:53,10.244.0.8:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.244.0.34:53,10.244.0.66:53,10.244.0.8:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         10.244.0.34:9153,10.244.0.66:9153,10.244.0.8:9153
Session Affinity:  None
Events:            <none>

#查看endpoint对应的pod状态
$kubectl get pod -nkube-system -owide |grep 10.244.0.34
coredns-659f5bbffd-w5vzw                   1/1     Running   0          2d   10.244.0.34     master-0002    <none>           <none>
$kubectl get pod -nkube-system -owide |grep 10.244.0.66
coredns-659f5bbffd-qrzl8                   1/1     Running   0         2d   10.244.0.66     master-0003    <none>           <none>
$kubectl get pod -nkube-system -owide |grep 10.244.0.8
coredns-659f5bbffd-rfr79                   1/1     Running   0          2d   10.244.0.8      master-0001    <none>           <none>

3.如何在 CoreDNS 配置文件添加访问日志

CoreDNS 配置文件是 Corefile，通过添加 log 插件可以打印访问日志，而 Corefile 是保存在 ConfigMap 中。

$kubectl edit configmap -nkube-system coredns

修改如下 ConfigMap，添加 log：

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }    

ConfigMap 保存后需要等待 1 到 2 分钟生效到 CoreDNS 的 Pod 中。如果配置生效，则 CoreDNS 将在日志看到：

.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
[INFO] Reloading complete

在应用 Pod 中测试目标地址的连通性：

#查询目标地址
$kubectl get svc |grep datachannel
datatest                               ClusterIP   10.96.0.37    <none>        8080/TCP                                         29d

#测试连通性
$kubectl exec -it deployment-flink-jobmanager-c695cf9d-rgtbh -- curl -vi datatest:8080
*   Trying 10.96.0.37:8080...
* Connected to datachannel (10.96.0.37) port 8092 (#0)
> GET / HTTP/1.1
> Host: datachannel:8080
> User-Agent: curl/7.69.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400
HTTP/1.1 400
< Content-Type: text/plain;charset=UTF-8
Content-Type: text/plain;charset=UTF-8
< Connection: close
Connection: close

<
Bad Request
This combination of host and port requires TLS.
* Closing connection 0

#DNS的关键日志
kubectl logs -nkube-system coredns-659f5bbffd-rfr79 |grep datatest
[INFO] 10.244.2.252:36293 - 17111 "AAAA IN datatest.default.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000184943s
[INFO] 10.244.2.252:36293 - 32461 "A IN datatest.default.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000144745s