pod容器日志管理

作者: 微凉哇 | 来源:发表于2021-11-20 13:24 被阅读0次

通过本文，你将了解kubelet是如何管理pod内的容器日志的。

定义pod日志目录

kubelet定义pod日志路径为：

/var/log/pods/<pod namespace>_<pod name>_<pod uid>

通过一个例子来验证上面的路径规则

default命名空间下存在一个名为stakater-reloader-598f958967-ddkl7的pod

$ kubectl get pod
NAME                                 READY   STATUS    RESTARTS   AGE
stakater-reloader-598f958967-ddkl7   1/1     Running   7          117d

获取其uid（metadata.uid字段）

$ kubectl get pod stakater-reloader-598f958967-ddkl7 -o yaml|grep uid
          k:{"uid":"ea8b9161-bee3-4bf6-a4e3-65fe256e3771"}:
            f:uid: {}
    uid: ea8b9161-bee3-4bf6-a4e3-65fe256e3771
  uid: 2681cb7f-daa4-4620-bb60-1d449709181c

跟据上面信息拼装，我们获取了该pod的日志目录

$ ls /var/log/pods/default_stakater-reloader-598f958967-ddkl7_2681cb7f-daa4-4620-bb60-1d449709181c
stakater-reloader

证明上面规则正确

关于源码的实现

kubernetes/pkg/kubelet/kuberuntime/kuberuntime_sandbox.go

$ func (m *kubeGenericRuntimeManager) generatePodSandboxConfig(pod *v1.Pod, attempt uint32) (*runtimeapi.PodSandboxConfig, error) {
...
    logDir := BuildPodLogsDirectory(pod.Namespace, pod.Name, pod.UID)
    podSandboxConfig.LogDirectory = logDir
...
}

pod日志目录结构

思考一: pod日志目录下日志文件生成规则

上文我们了解到，kubelet定义pod日志路径为：

/var/log/pods/<pod namespace>_<pod name>_<pod uid>

我们观察下其路径结构：

$ ls -R /var/log/pods/default_stakater-reloader-598f958967-ddkl7_2681cb7f-daa4-4620-bb60-1d449709181c
/var/log/pods/default_stakater-reloader-598f958967-ddkl7_2681cb7f-daa4-4620-bb60-1d449709181c:
stakater-reloader

/var/log/pods/default_stakater-reloader-598f958967-ddkl7_2681cb7f-daa4-4620-bb60-1d449709181c/stakater-reloader:
1.log  6.log  7.log

其中stakater-reloader是根据容器名称生成的目录
其中1.log 6.log 7.log是根据容器重启次数生成日志文件，格式为: <容器重启次数>.log

我们看下源码实现:

kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go

...
func (m *kubeGenericRuntimeManager) startContainer(podSandboxID string, podSandboxConfig *runtimeapi.PodSandboxConfig, spec *startSpec, pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, podIP string, podIPs []string) (string, error) {
    ...
    // 1. 生成容器配置
    // 其中日志目录规则为：
    containerConfig, cleanupAction, err := m.generateContainerConfig(container, pod, restartCount, podIP, imageRef, podIPs, target)
    ...
    // 2. 拼接容器日志目录
    // 其中podSandboxConfig.LogDirectory = /var/log/pods/<pod namespace>_<pod name>_<pod uid>
    // containerConfig.LogPath = <container name>/容器重启次数.log 
    containerLog := filepath.Join(podSandboxConfig.LogDirectory, containerConfig.LogPath)
    // only create legacy symlink if containerLog path exists (or the error is not IsNotExist).
    // Because if containerLog path does not exist, only dandling legacySymlink is created.
    // This dangling legacySymlink is later removed by container gc, so it does not make sense
    // to create it in the first place. it happens when journald logging driver is used with docker.
    // 3. 建立链接
    if _, err := m.osInterface.Stat(containerLog); !os.IsNotExist(err) {
        if err := m.osInterface.Symlink(containerLog, legacySymlink); err != nil {
            klog.Errorf("Failed to create legacy symbolic link %q to container %q log %q: %v",
            legacySymlink, containerID, containerLog, err)
        }
    }
...
}

kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go

func (m *kubeGenericRuntimeManager) generateContainerConfig(container *v1.Container, pod *v1.Pod, restartCount int, podIP, imageRef string, podIPs []string, nsTarget *kubecontainer.ContainerID) (*runtimeapi.ContainerConfig, func(), error) {
    opts, cleanupAction, err := m.runtimeHelper.GenerateRunContainerOptions(pod, container, podIP, podIPs)
...
    // 拼接容器日志文件路径
    containerLogsPath := buildContainerLogsPath(container.Name, restartCount)
...
}

其中restartCount重启次数这个值我们可以通过以下方式获取:

$ kubectl get pod stakater-reloader-598f958967-ddkl7 -o yaml
...
status:
...
  containerStatuses:
  - containerID: docker://c73e273fd2886df631739b03fd71a8216054549b11c651a7c8c38ce902342332
...
    restartCount: 7
...

思考二: 日志文件软链接规则

通过观察我们可以发现，这个目录下的日志文件是软链接类型文件:

$ ls -l /var/log/pods/default_stakater-reloader-598f958967-ddkl7_2681cb7f-daa4-4620-bb60-1d449709181c/stakater-reloader
lrwxrwxrwx 1 root root 165 Jul 31 18:24 1.log -> /var/lib/docker/containers/cc75d79b3aef49f739b03f5fa7379f75c69944cbeb0a4555d3833a26ebfe06b4/cc75d79b3aef49f739b03f5fa7379f75c69944cbeb0a4555d3833a26ebfe06b4-json.log
lrwxrwxrwx 1 root root 165 Oct 23 21:56 6.log -> /var/lib/docker/containers/e9bc97e0cdd1e4258f17bb18db976ac371860103b861529cb690005ada97d153/e9bc97e0cdd1e4258f17bb18db976ac371860103b861529cb690005ada97d153-json.log
lrwxrwxrwx 1 root root 165 Nov  6 16:08 7.log -> /var/lib/docker/containers/c73e273fd2886df631739b03fd71a8216054549b11c651a7c8c38ce902342332/c73e273fd2886df631739b03fd71a8216054549b11c651a7c8c38ce902342332-json.log

其实是kubelet在启动容器后，生成的链接。

通过样例，我们不难发现链接匹配的规则如下：(适用于docker运行时，且为json日志插件)

/var/log/pods/<pod namespace>_<pod name>_<pod uid>/<容器名称>/重启重启次数.log
指向
/var/lib/docker/containers/<容器id>/<容器id>-json.log

其中容器id我们可以通过以下方式获取

$ kubectl get pod stakater-reloader-598f958967-zbn2g -o yaml|grep -e "- containerID"
  - containerID: docker://fed0bdb0183c8bcfd1c91090d96e3c594c588de94f89f68ff24a8fe7f940e50e

源码实现

日志文件的软链接设置逻辑实际发生于容器启动后：

/var/log/pods/<pod namespace>_<pod name>_<pod uid>/<容器名称>/重启重启次数.log
指向
/var/lib/docker/containers/<容器id>/<容器id>-json.log

kubelet启动容器核心源码：kubernetes/pkg/kubelet/dockershim/docker_container.go

func (ds *dockerService) StartContainer(_ context.Context, r *runtimeapi.StartContainerRequest) (*runtimeapi.StartContainerResponse, error) {
    err := ds.client.StartContainer(r.ContainerId)

    // Create container log symlink for all containers (including failed ones).
    if linkError := ds.createContainerLogSymlink(r.ContainerId); linkError != nil {
        // Do not stop the container if we failed to create symlink because:
        //   1. This is not a critical failure.
        //   2. We don't have enough information to properly stop container here.
        // Kubelet will surface this error to user via an event.
        return nil, linkError
    }

    if err != nil {
        err = transformStartContainerError(err)
        return nil, fmt.Errorf("failed to start container %q: %v", r.ContainerId, err)
    }

    return &runtimeapi.StartContainerResponse{}, nil
}

其中ds.createContainerLogSymlink(r.ContainerId)便是创建日志文件软链接的逻辑，我们接下来对其进行深入分析

容器日志链接创建流程解析

创建容器日志软链接，主要分为三个步骤:

根据容器ID，解析容器的真实日志路径，对应返回值realPath

$ docker inspect cff000d9fc5e -f "{{ .LogPath }}"
/var/lib/docker/containers/cff000d9fc5edad5a8b042b8879fedd1d7de978b928c522a53c11b3217c220df/cff000d9fc5edad5a8b042b8879fedd1d7de978b928c522a53c11b3217c220df-json.log

根据容器ID，解析容器的pod日志路径标签，对应返回值path

$ docker inspect cff000d9fc5e -f '{{ index .Config.Labels "io.kubernetes.container.logpath" }}'
/var/log/pods/istio-system_istio-ingressgateway-c694cfd-cgk2n_c6ea97b6-d858-4c4c-8d83-17477599aebf/istio-proxy/4.log

根据前两步获取的path与realPath创建文件链接，等同于

$ ln -s $realPath $path

源码解析其实现

其中path, realPath, err := ds.getContainerLogPath(containerID)为获取容器信息逻辑，关于返回值解析:

path: 容器的Config.Labels["io.kubernetes.container.logpath"]字段，该容器所属pod日志路径（实质为真实日志路径的软链接）
realPath: 该容器的真实日志路径

kubernetes/pkg/kubelet/dockershim/docker_container.go源码

// createContainerLogSymlink creates the symlink for docker container log.
func (ds *dockerService) createContainerLogSymlink(containerID string) error {
    path, realPath, err := ds.getContainerLogPath(containerID)
    if err != nil {
        return fmt.Errorf("failed to get container %q log path: %v", containerID, err)
    }

    if path == "" {
        klog.V(5).Infof("Container %s log path isn't specified, will not create the symlink", containerID)
        return nil
    }

    if realPath != "" {
        // Only create the symlink when container log path is specified and log file exists.
        // Delete possibly existing file first
        if err = ds.os.Remove(path); err == nil {
            klog.Warningf("Deleted previously existing symlink file: %q", path)
        }
        if err = ds.os.Symlink(realPath, path); err != nil {
            return fmt.Errorf("failed to create symbolic link %q to the container log file %q for container %q: %v",
                path, realPath, containerID, err)
        }
    } else {
        supported, err := ds.IsCRISupportedLogDriver()
        if err != nil {
            klog.Warningf("Failed to check supported logging driver by CRI: %v", err)
            return nil
        }

        if supported {
            klog.Warningf("Cannot create symbolic link because container log file doesn't exist!")
        } else {
            klog.V(5).Infof("Unsupported logging driver by CRI")
        }
    }

    return nil
}

其中函数getContainerLogPath()源码如下:

func (ds *dockerService) getContainerLogPath(containerID string) (string, string, error) {
    info, err := ds.client.InspectContainer(containerID)
    if err != nil {
        return "", "", fmt.Errorf("failed to inspect container %q: %v", containerID, err)
    }
    return info.Config.Labels[containerLogPathLabelKey], info.LogPath, nil
}

其中函数createContainerLogSymlink()源码如下:

func (ds *dockerService) getContainerLogPath(containerID string) (string, string, error) {
    info, err := ds.client.InspectContainer(containerID)
    if err != nil {
        return "", "", fmt.Errorf("failed to inspect container %q: %v", containerID, err)
    }
    return info.Config.Labels[containerLogPathLabelKey], info.LogPath, nil
}

ds.os.Symlink(realPath, path)为调用系统接口，创建日志文件软链接

总结

通过上述分析，我们可以得出以下结论（docker运行时下）:

k8s下容器日志目录为：/var/log/pods/<pod namespace>_<pod name>_<pod uid>/<容器名称>/重启重启次数.log，并且是文件

/var/lib/docker/containers/<容器id>/<容器id>-json.log的链接。

pod会按容器的重启次数对应保留日志，具体保留个数应该与GC策略有关

pod容器日志管理

定义pod日志目录

pod日志目录结构

思考一: pod日志目录下日志文件生成规则

思考二: 日志文件软链接规则

总结

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

kubelet源码解析