k8s 容器资源可见性配置-lxcfs
<a name="609h1"></a>
问题
使用k8s容器化后，对内存 CPU 资源限制后，在容器内查看资源，显示的和容器所在宿主机信息一致，无法看到限制后的内存情况
<a name="UOYMO"></a>
目标
实现资源可见性<br />比如 resources.requests.memory: "1024Mi"<br />那么在容器内查看内存，执行free -m 后显示的内存也为1024
<a name="iKTq8"></a>
1.安装依赖
在k8s所有节点执行以下命令，安装依赖包<br />yum -y install fuse-devel fuse fuse-libs
yum -y install https://copr-be.cloud.fedoraproject.org/results/ganto/lxd/epel-7-x86_64/00486278-lxcfs/lxcfs-2.0.5-3.el7.centos.x86_64.rpm
<a name="J4mlh"></a>
2.安装lcxfs
安装lxcfs有如下两种方案：
<a name="GL3Gg"></a>
1）.直接在所有k8s节点上安装lxcfs 包，并启动进程
yum -y install https://copr-be.cloud.fedoraproject.org/results/ganto/lxd/epel-7-x86_64/00486278-lxcfs/lxcfs-2.0.5-3.el7.centos.x86_64.rpm
systemctl start lxcfs
systemctl enable lxcfs
<a name="c8iGO"></a>
2）.使用k8s Daemonsets部署lxcfs，让其在每一台节点上运行lxcfs 进程
apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
name: lxcfs
labels:
app: lxcfs
spec:
selector:
matchLabels:
app: lxcfs
template:
metadata:
labels:
app: lxcfs
spec:
hostPID: true
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: lxcfs
image: registry.cn-hangzhou.aliyuncs.com/denverdino/lxcfs:2.0.8-1
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- name: cgroup
mountPath: /sys/fs/cgroup
- name: lxcfs
mountPath: /var/lib/lxcfs
mountPropagation: Bidirectional
- name: usr-local
mountPath: /usr/local
volumes:
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: usr-local
hostPath:
path: /usr/local
- name: lxcfs
hostPath:
path: /var/lib/lxcfs
type: DirectoryOrCreate
将以上内容保存为lxcfs-dadaemontset.yaml 文件<br />kubectl apply -f ``lxcfs-dadaemontset.yaml<br />以上两种方式二选一
<a name="AtwEM"></a>
3.使用lxcfs
使用lxcfs会有三种方式，分别是 直接挂载，PodPreset， Initializer
<a name="Q3gos"></a>
方案1:直接挂载
在创建pod的时候，将lxcfs相应文件直接挂载到pod即可<br />相关文件：<br />/var/lib/lxcfs/proc/cpuinfo<br />/var/lib/lxcfs/proc/meminfo<br />/var/lib/lxcfs/proc/diskstats<br />/var/lib/lxcfs/proc/stat<br />/var/lib/lxcfs/proc/swaps<br />/var/lib/lxcfs/proc/uptime
案例：
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
"initializer.kubernetes.io/lxcfs": "true"
labels:
app: web
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: httpd:2.4.32
volumeMounts:
- name: cpuinfo
mountPath: /proc/cpuinfo
- name: meminfo
mountPath: /proc/meminfo
- name: diskstats
mountPath: /proc/diskstats
- name: stat
mountPath: /proc/stat
- name: swaps
mountPath: /proc/swaps
- name: uptime
mountPath: /proc/uptime
imagePullPolicy: Always
resources:
requests:
memory: "1024Mi"
cpu: "100m"
limits:
memory: "1024Mi"
cpu: "100m"
volumes:
- name: cpuinfo
hostPath:
path: /var/lib/lxcfs/proc/cpuinfo
type: File
- name: meminfo
hostPath:
path: /var/lib/lxcfs/proc/meminfo
type: File
- name: diskstats
hostPath:
path: /var/lib/lxcfs/proc/diskstats
type: File
- name: stat
hostPath:
path: /var/lib/lxcfs/proc/stat
type: File
- name: swaps
hostPath:
path: /var/lib/lxcfs/proc/swaps
type: File
- name: uptime
hostPath:
path: /var/lib/lxcfs/proc/uptime
type: File
<a name="YquMZ"></a>
方案2:使用PodPreset 注入
使用该方案，需要开启kube-apiserver 相关参数<br />--enable-admission-plugins=PodPreset,XXXX,XXXX<br />--runtime-config=settings.k8s.io/v1alpha1<br />PodPreset 会根据<br />然后创建PodPreset spec.selector 来选择pod并注入<br />以下案例就是PodPreset 会将该Namespace 下所有包含标签inject-lxcfs: "true" 的pod 进行 注入volumeMounts，volumes
apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: inject-lxcfs
spec:
selector:
matchLabels:
inject-lxcfs: "true"
volumeMounts:
- name: cpuinfo
mountPath: /proc/cpuinfo
- name: meminfo
mountPath: /proc/meminfo
- name: diskstats
mountPath: /proc/diskstats
- name: stat
mountPath: /proc/stat
- name: swaps
mountPath: /proc/swaps
- name: uptime
mountPath: /proc/uptime
volumes:
- name: cpuinfo
hostPath:
path: /var/lib/lxcfs/proc/cpuinfo
type: File
- name: meminfo
hostPath:
path: /var/lib/lxcfs/proc/meminfo
type: File
- name: diskstats
hostPath:
path: /var/lib/lxcfs/proc/diskstats
type: File
- name: stat
hostPath:
path: /var/lib/lxcfs/proc/stat
type: File
- name: swaps
hostPath:
path: /var/lib/lxcfs/proc/swaps
type: File
- name: uptime
hostPath:
path: /var/lib/lxcfs/proc/uptime
type: File
创建Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
"kubernetes.io/inject-lxcfs": "true"
labels:
app: lxcfs-test
name: lxcfs-test
spec:
replicas: 1
selector:
matchLabels:
app: lxcfs-test
template:
metadata:
labels:
app: lxcfs-test
inject-lxcfs: "true"
spec:
containers:
- name: lxcfs-test
image: httpd:2.4.32
imagePullPolicy: Always
resources:
requests:
memory: "1024Mi"
cpu: "100m"
limits:
memory: "1024Mi"
cpu: "100m"
接下来查看pod 就会有相应的内容注入<br />kubectl get pod lxcfs-test-xxxxx-xxx -o yaml
<a name="pZznT"></a>
4.验证
根据上面部署的httpd 将内存限制为1024m，cpu限制为0.1<br />因此效果如下
<a name="8syiW"></a>
内存
image.png
shancangchen@localhost:~/hellobike/code/k8s$ kubectl exec -it lxcfs-test-b9d549ddf-vqhwt bash
root@lxcfs-test-b9d549ddf-vqhwt:/usr/local/apache2# free -m
total used free shared buffers cached
Mem: 1024 13 1010 3 0 0
-/+ buffers/cache: 13 1010
Swap: 0 0 0
root@lxcfs-test-b9d549ddf-vqhwt:/usr/local/apache2#
<a name="m3khy"></a>
CPU
image.png
<a name="DhBfT"></a>
5.参考连接
https://yq.aliyun.com/articles/566208<br />https://kubernetes.io/docs/tasks/inject-data-application/podpreset/<br />https://kubernetes.io/docs/concepts/workloads/pods/podpreset/<br />https://github.com/lijiaocn/lxcfs-initializer
网友评论