研究 k8s pod 状态变化

作者: Yellowtail | 来源:发表于2020-09-23 16:35 被阅读0次

研究 k8s pod 状态变化
kubernetes pod status 监控
K8S 集群定时清理 Evicted 状态的 Pod
kubernetes service 介绍
k8s滚动更新时删除pod的规则
使用 docker 部署 spring boot 并接入 sky
k8s快速入门-概念
k8s 网络基础
k8s中service和endpoint
Kubernetes 运维小记：node 为系统保留最低资源

背景

最近打算使用 k8s api server 的 api 来监控 pod 状态变更事件，所以顺带研究了一下 pod 状态变化

k8s api

k8s 的这个 list-watch 很牛逼，把 HTTP/1.1 玩出花来了
知乎

使用 curl

curl -v -k  https://172.18.xxxx:6443/api/v1/namespaces/backend/pods?watch=true  -H "Authorization: Bearer eyJxxxxxx"

变化

我的这个pod 副本数量为1

逐个分析

0x1 ADDED 新pod

{
    "type": "ADDED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-598b9fd477-r8qvm"
        },
        "status": {
            "phase": "Pending",
            "qosClass": "Burstable"
        }
    }
}

创建新的 pod ，名字是 cayenne-598b9fd477-r8qvm

0x2 MODIFIED 新pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-598b9fd477-r8qvm"      
        },      
        "status": {
            "phase": "Pending",
            "conditions": [{
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

状态更新

PodScheduled 意思是把pod 调度到节点node上

pod 状态为 Pending
容器已调度到节点上了，此时正在做一些比如 pull docker image 之类的事情

0x3 MODIFIED 新pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-598b9fd477-r8qvm",         
        },      
        "status": {
            "phase": "Pending",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }, {
                    "type": "Ready",
                    "status": "False",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "ContainersReady",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                }
            ],
            "hostIP": "172.18.237.190",
            "startTime": "2020-09-23T07:15:34Z",
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "waiting": {
                            "reason": "ContainerCreating"
                        }
                    },
                    "lastState": {},
                    "ready": false,
                    "restartCount": 0,
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

状态更新

PodScheduled: true pod 已调度到节点上
Initialized: true 所有的 init containers 启动成功
Ready: false pod里面的容器，不是所有的都就绪了
ContainersReady: false 这个pod还没有准备好接收service的请求，此时也不参与service的负载均衡

此时容器的状态是 waiting
reason: ContainerCreating 说明当前处于容器启动过程

pod 此时状态是 Pending
容器已调度到节点上
init containers 全部启动完成
容器状态为 waiting，此时正在创建容器

0x4 MODIFIED 新pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-598b9fd477-r8qvm",     
        },      
        "status": {
            "phase": "Running",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }, {
                    "type": "Ready",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "ContainersReady",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }
            ],
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "running": {
                            "startedAt": "2020-09-23T07:15:40Z"
                        }
                    },
                    "ready": false,
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

容器状态为 running，创建完成，正处于启动过程
pod 此时状态是 Running，但是没有就绪

0x5 MODIFIED 新pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-598b9fd477-r8qvm", 
        },      
        "status": {
            "phase": "Running",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }, {
                    "type": "Ready",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:50Z"
                }, {
                    "type": "ContainersReady",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:50Z"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:34Z"
                }
            ],
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "running": {
                            "startedAt": "2020-09-23T07:15:40Z"
                        }
                    },
                    "lastState": {},
                    "ready": true,
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

容器状态为 running，创建完成，启动完成，通过了就绪检查
pod 此时状态是 Running，且已经就绪，可以开始参与服务请求

0x6 MODIFIED 老pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-6cdf75c8b7-rtbgg", 
        },      
        "status": {
            "phase": "Running",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }, {
                    "type": "Ready",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:14:04Z"
                }, {
                    "type": "ContainersReady",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:14:04Z"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }
            ],
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "running": {
                            "startedAt": "2020-09-23T07:13:53Z"
                        }
                    },
                    "ready": true,
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

老 pod 此时状态是 Running， 开始准备变化了

0x7 MODIFIED 老pod

{
    "type": "MODIFIED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-6cdf75c8b7-rtbgg", 
        },      
        "status": {
            "phase": "Pending",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }, {
                    "type": "Ready",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:52Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "ContainersReady",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:52Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }
            ],
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "waiting": {
                            "reason": "ContainerCreating"
                        }
                    },
                    "ready": false, 
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

老 pod 此时状态是 Pending
Ready 状态变为 false, 不参与服务请求了
容器状态变为 waiting， 原因是 ContainerCreating （这里貌似不太对，应该是容器销毁吧）

0x8 MODIFIED 老pod

和0x7 一样，没看出什么区别

0x8 DELETED 老pod

{
    "type": "DELETED",
    "object": {
        "kind": "Pod",
        "apiVersion": "v1",
        "metadata": {
            "name": "cayenne-6cdf75c8b7-rtbgg",     
        },
        "status": {
            "phase": "Pending",
            "conditions": [{
                    "type": "Initialized",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }, {
                    "type": "Ready",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:52Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "ContainersReady",
                    "status": "False",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:15:52Z",
                    "reason": "ContainersNotReady",
                    "message": "containers with unready status: [cayenne]"
                }, {
                    "type": "PodScheduled",
                    "status": "True",
                    "lastProbeTime": null,
                    "lastTransitionTime": "2020-09-23T07:13:52Z"
                }
            ],
            "containerStatuses": [{
                    "name": "cayenne",
                    "state": {
                        "waiting": {
                            "reason": "ContainerCreating"
                        }
                    },
                    "lastState": {},
                    "ready": false
                }
            ],
            "qosClass": "Burstable"
        }
    }
}

分析

此时准备删除pod，删除之前pod 的状态是  Pending
``

## 参考
[监听所有命名空间](https://kubernetes.cn/docs/reference/generated/kubernetes-api/v1.19/#list-all-namespaces-pod-v1-core)
[监听一个命名空间](https://kubernetes.cn/docs/reference/generated/kubernetes-api/v1.19/#list-pod-v1-core)

[知乎](https://zhuanlan.zhihu.com/p/59660536)

研究 k8s pod 状态变化
背景最近打算使用 k8s api server 的 api 来监控 pod 状态变更事件，所以顺带研究了一下 ...
kubernetes pod status 监控
dashboard 监控项：各个k8s集群所有pod not running的状态，监控pod的CrashLoo...
K8S 集群定时清理 Evicted 状态的 Pod
问题现象看到 k8s 集群中有 Evicted 状态的 pod，没有被清理排查过程可以看到 pod 的状态是...
kubernetes service 介绍
一、问题首先，我们思考这样一个问题：访问k8s集群中的pod，客户端需要知道pod地址，需要感知pod的状态...
k8s滚动更新时删除pod的规则
k8s 在弹性缩容或滚动更新时默认的pod删除规则如下： 1 如果pod没分配到节点，先被删除2 如果pod的状态...
使用 docker 部署 spring boot 并接入 sky
一、概述最近在研究skywalking，打算使用k8s部署 skywalking 并将 pod 中的应用接入 s...
k8s快速入门-概念
k8s快速入门 k8s的四组基本概念 Pod/Pod控制器 Name/Namespace Lable/Label选...
k8s 网络基础
k8s 网络基础 pod 重点关注containerPort，他与Pod IP组成了endpoint，Pod IP...
k8s中service和endpoint
K8s提供了service对象来访问pod。为什么没有直接访问pod？一是k8s中pod不是持久性的，宕机重建将...
Kubernetes 运维小记：node 为系统保留最低资源
今天优化了 k8s 生产集群的部署，为系统（linux 与 k8s系统pod）保留最低资源，以免应用 pod 负载...