Loki 问题 - filetarget.fsnotify.Ne

问题描述

level=info ts=2021-01-21T01:56:05.301932147Z caller=server.go:225 http=[::]:9080 grpc=[::]:41303 msg="server listening on addresses"
level=info ts=2021-01-21T01:56:05.302278747Z caller=main.go:108 msg="Starting Promtail" version="(version=2.0.0, branch=HEAD, revision=6978ee5d7)"
level=info ts=2021-01-21T01:56:10.30212042Z caller=filetargetmanager.go:261 msg="Adding target" key="{log_from=\"static_pods\"}"
level=error ts=2021-01-21T01:56:10.302202345Z caller=filetargetmanager.go:265 msg="Failed to create target" key="{log_from=\"static_pods\"}" error="filetarget.fsnotify.NewWatcher: too many open files"

解决办法

# 在宿主机执行
sudo sysctl -w fs.inotify.max_user_instances=512

参考文档

• https://github.com/grafana/loki/issues/1153
• https://unix.stackexchange.com/questions/444998/how-to-set-and-understand-fs-notify-max-user-watches