Mac Docker - Prometheus QPS Hpa

作者: 国服最坑开发 | 来源:发表于2022-03-17 15:07 被阅读0次

top

0x00 TLDR;

在本机K8s环境验证基于QPS实施 hpa 的功能 . 顺带看一下qps 指标如何使用.

k8s 版本: v1.22.4

前提: 有一个干净的Mac Docker Desktop, 启用 Kubernetes.

# kubectl get po -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-77tph                 1/1     Running   0          91s
kube-system   coredns-78fcd69978-x7gf8                 1/1     Running   0          91s
kube-system   etcd-docker-desktop                      1/1     Running   0          97s
kube-system   kube-apiserver-docker-desktop            1/1     Running   0          96s
kube-system   kube-controller-manager-docker-desktop   1/1     Running   0          95s
kube-system   kube-proxy-j2fgt                         1/1     Running   0          92s
kube-system   kube-scheduler-docker-desktop            1/1     Running   0          94s
kube-system   storage-provisioner                      1/1     Running   0          65s
kube-system   vpnkit-controller                        1/1     Running   0          65s

0x01 安装 ingress nginx

参考: https://kubernetes.github.io/ingress-nginx/deploy/#docker-for-mac

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

查看部署结果

 k get po -A | grep nginx
ingress-nginx   ingress-nginx-controller-cb87575f5-sxjgs   1/1     Running   0          9m20s

注意: 默认的系统端口 80, 是无法访问到集群里面的. 需要转到系统其他端口.

k get svc -A
NAMESPACE       NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                           ClusterIP      10.96.0.1        <none>        443/TCP                      18m
ingress-nginx   ingress-nginx-controller             LoadBalancer   10.108.140.103   localhost     80:30725/TCP,443:32615/TCP   15m
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      10.107.145.159   <none>        443/TCP                      15m
kube-system     kube-dns                             ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       18m

# 转发到主机 8000端口
k port-forward service/ingress-nginx-controller -n ingress-nginx 8000:80

请求默认 nginx-ingress 响应: 404, 表示成功.

curl localhost:8000
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

0x02 关于 Prometheus 在K8s中使用

为了实现自定义metrics提取功能 , 需要三个条件:

安装 kube-prometheus : 在K8s中部署promtheus 全家桶
pod里的服务支持 /metrics 接口
安装 prometheus-adapter: 实现 k8s 监控格式向 prometheus 格式转换

0x03 部署 kube-prometheus

git clone https://github.com/coreos/kube-prometheus.git --depth=1
cd kube-prometheus
kubectl create -f manifests/setup
kubectl create -f manifests

这里有个小坑: node-expoter-pod 运行异常. 这个问题不是本篇主要方向, 直接砍掉.
有缘以后再回来补上.

k delete -f manifests/nodeExporter-daemonset.yaml

查看Pod状态:

# k get po  -n monitoring
NAME                                   READY   STATUS    RESTARTS        AGE
alertmanager-main-0                    2/2     Running   2 (2m25s ago)   9m29s
alertmanager-main-1                    2/2     Running   2 (2m25s ago)   9m29s
alertmanager-main-2                    2/2     Running   2 (2m25s ago)   9m29s
blackbox-exporter-69894767d5-5r684     3/3     Running   3 (2m25s ago)   13m
grafana-7bb5967c6-5n289                1/1     Running   1 (2m25s ago)   13m
kube-state-metrics-5d6885d89-kdvcm     3/3     Running   3 (2m25s ago)   13m
prometheus-adapter-6cf5d8bfcf-56rbl    1/1     Running   1 (2m25s ago)   13m
prometheus-adapter-6cf5d8bfcf-9tq8d    1/1     Running   2 (2m4s ago)    13m
prometheus-k8s-0                       2/2     Running   0               9m28s
prometheus-k8s-1                       2/2     Running   0               9m28s
prometheus-operator-7f58778b57-5xbdb   2/2     Running   2 (2m25s ago)   13m

暴露一下 grafana 和 prometheus-server的主机访问

k port-forward service/grafana -n monitoring 3000
k port-forward service/prometheus-k8s -n monitoring 9090

网页打开:
grafana : http://localhost:3000, admin/admin
prometheus: http://localhost:9090

验证一下 top po 指令:

k top po -A
NAMESPACE       NAME                                       CPU(cores)   MEMORY(bytes)
ingress-nginx   ingress-nginx-controller-cb87575f5-sxjgs   2m           156Mi
kube-system     coredns-78fcd69978-77tph                   2m           18Mi
kube-system     coredns-78fcd69978-x7gf8                   3m           19Mi
kube-system     etcd-docker-desktop                        20m          75Mi
kube-system     kube-apiserver-docker-desktop              49m          330Mi
kube-system     kube-controller-manager-docker-desktop     22m          51Mi
kube-system     kube-proxy-j2fgt                           0m           15Mi
kube-system     kube-scheduler-docker-desktop              2m           19Mi
kube-system     storage-provisioner                        1m           6Mi
kube-system     vpnkit-controller                          0m           8Mi
monitoring      alertmanager-main-0                        2m           32Mi
monitoring      alertmanager-main-1                        2m           33Mi
monitoring      alertmanager-main-2                        4m           29Mi
monitoring      blackbox-exporter-69894767d5-5r684         0m           21Mi
monitoring      grafana-7bb5967c6-5n289                    10m          58Mi
monitoring      kube-state-metrics-5d6885d89-kdvcm         3m           33Mi
monitoring      prometheus-adapter-6cf5d8bfcf-56rbl        2m           25Mi
monitoring      prometheus-adapter-6cf5d8bfcf-9tq8d        4m           23Mi
monitoring      prometheus-k8s-0                           21m          180Mi
monitoring      prometheus-k8s-1                           18m          189Mi
monitoring      prometheus-operator-7f58778b57-5xbdb       0m           35Mi

0x04 部署测试程序

参考: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/walkthrough.md
部署应用

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: sample-app
  name: sample-app
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP

暴露访问接口

k get po
NAME                          READY   STATUS    RESTARTS   AGE
sample-app-7cfb596f98-hxwqj   1/1     Running   0          4m25s
# 直接把Pod端口暴露出来
k port-forward sample-app-7cfb596f98-hxwqj 8010:8080

# 访问 验证
curl localhost:8010/metrics
# HELP http_requests_total The amount of requests served by the server in total
# TYPE http_requests_total counter
http_requests_total 2

此时, 测试项目已支持 /metrics接口.

部署HPA:

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    # point the HPA at the sample application
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  # autoscale between 1 and 10 replicas
  minReplicas: 1
  maxReplicas: 10
  metrics:
  # use a "Pods" metric, which takes the average of the
  # given metric across all pods controlled by the autoscaling target
  - type: Pods
    pods:
      # use the metric that you used above: pods/http_requests
      metricName: http_requests
      # target 500 milli-requests per second,
      # which is 1 request every two seconds
      targetAverageValue: 500m

此时查看日志, 发现会有错误:

k logs -f kube-controller-manager-docker-desktop -n kube-system

I0315 09:47:07.652833 1 event.go:291] "Event occurred" object="default/sample-app" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get pods metric value: unable to get metric http_requests: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered"

意思是, hpa 找不到 http_requests 这个指标.
这就扯出下一个话题: 部署 adapter

0x05 部署 prometheus-adapter

参考: https://www.cnblogs.com/dudu/p/12146344.html

git clone https://github.com/DirectXMan12/k8s-prometheus-adapter.git --depth=1
cd k8s-prometheus-adapter

# 创建证书
k create ns custom-metrics
export PURPOSE=serving
openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE}.key -out ${PURPOSE}.crt -subj "/CN=ca"
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "${PURPOSE}-ca-config.json"
kubectl -n custom-metrics create secret generic cm-adapter-serving-certs --from-file=./serving.crt --from-file=./serving.key 


# 部署
cd deploy/manifests
# 更新 custom-metrics-apiservice.yaml: 
# apiregistration.k8s.io/v1beta1  -->  apiregistration.k8s.io/v1

# 更新 custom-metrics-apiserver-deployment.yaml
# 镜像修改:  image: directxman12/k8s-prometheus-adapter:v0.8.4
# URL修改: --prometheus-url=http://prometheus.prom.svc:9090/  --> --prometheus-url=http://prometheus-k8s.monitoring.svc:9090/

k apply -f .

clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
deployment.apps/custom-metrics-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
serviceaccount/custom-metrics-apiserver created
service/custom-metrics-apiserver created
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
apiservice.apiregistration.k8s.io/v1beta2.custom.metrics.k8s.io created
apiservice.apiregistration.k8s.io/v1beta1.external.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
configmap/adapter-config created
clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created

安装 monitor, 这一步才是自定义指标的关键

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  endpoints:
  - port: http

0x05 验证HPA

前面几步如果正常进行的话, 可以先通过api 进行确认:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests?selector=app=sample-app" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "sample-app-7cfb596f98-87hms",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests",
      "timestamp": "2022-03-16T01:09:49Z",
      "value": "66m",
      "selector": null
    }
   ...
}

看到上面的输出, 表示可以adapter可以正常输出 http_requests 指标.

接下来, 多请求几次接口: http://localhost:8010/metrics
查看 hpa 变化

k get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   66m/500m   1         10        4          15h

看到po 增加到4个.
停止请求, 等待一段时间 , 查看 hpa, 回落到 1个副本

k get hpa
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   66m/500m   1         10        1          15h

到此, 基于QPS的 HPA 实验完毕

0x06 自定义规则: http_requests_total 和 http_requests 的关系

我们来查看一下, /metrics 接口只暴露了 http_requests_total, 那么这个 http_requets是怎么来的.

先查看一下下面的结果 :

k get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "sample-app-7cfb596f98-hxwqj",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests",
      "timestamp": "2022-03-16T02:44:08Z",
      "value": "66m",
      "selector": null
    }
  ]
}

表示测试 pod 支持 "metricName": "http_requests"
可是这货从哪来的 ?

进一步跟一下 : https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config-walkthrough.md

配置adapter规则大致分四步:

The adapter considers metrics in the following ways:
First, it discovers the metrics available (Discovery)
Then, it figures out which Kubernetes resources each metric is associated with (Association)
Then, it figures out how it should expose them to the custom metrics API (Naming)
Finally, it figures out how it should query Prometheus to get the actual numbers (Querying)

参考一段例子: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/sample-config.yaml

# this rule matches cumulative non-cAdvisor metrics
- seriesQuery: '{namespace!="",__name__!="^container_.*"}'
  name: {matches: "^(.*)_total$"}
  resources:
    # specify an a generic mapping between resources and labels.  This
    # is a template, like the `metricsQuery` template, except with the `.Group`
    # and `.Resource` strings available.  It will also be used to match labels,
    # so avoid using template functions which truncate the group or resource.
    # Group will be converted to a form acceptible for use as a label automatically.
    template: "<<.Resource>>"
    # if we wanted to, we could also specify overrides here
  metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)"

{matches: "^(.*)_total$"} 这里仿佛解释了 http_requests_total 和 http_requests 的关系了.
相当于对(.*)_total 这类计数型指标, 统一转换成名称为 \1 , 值为概要数据.

0x07 Spring工程自定义指标

大致过程:

pod 已经有/metrics 输出
通过 prometheus 进行表达式验证

官方指引: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md
Spring Demo 参考: https://tanzu.vmware.com/developer/guides/spring-prometheus/

编写测试用Spring项目:

pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

application.yaml

spring:
  application:
    name: java-demo
management:
  server:
    port: 8611
  endpoints:
    web:
      exposure:
        include: "*"
      base-path: /metrics
  metrics:
    tags:
      application: ${spring.application.name}
server:
  port: 8888

测试类

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;

@RestController
@SpringBootApplication
public class QpsDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(QpsDemoApplication.class, args);
    }

    Counter visitCounter;

    public QpsDemoApplication(MeterRegistry registry) {
        visitCounter = Counter.builder("mvc_access").description("http access count").register(registry);
    }

    @GetMapping({"/", "/index"})
    public String index() {
        visitCounter.increment();
        return "running count .. " + visitCounter.count();
    }
}

本机运行检测指标接口:

curl -s http://localhost:8611/metrics/prometheus | grep access
# HELP mvc_access_total http access count
# TYPE mvc_access_total counter
mvc_access_total 0.0

运行业务接口:

curl -s http://localhost:8888/
running count .. 2.0

注意 mvc_access_total 这个名称的来自 Java 代码

编译镜像

Dockerfile

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
COPY target/*.jar app.jar
ENTRYPOINT [ "java", "-jar", "/app.jar" ]

编译

# package
mvn clean package -DskipTests -U
# 编译镜像,存到本机 Docker 仓库
docker build . -t mvc_access:0.1

部署应用

---
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mvc-demo
  namespace: default
  labels:
    app: mvc-demo
spec:
  replicas: 1
  template:
    metadata:
      name: mvc-demo
      labels:
        app: mvc-demo
    spec:
      containers:
        - name: mvc-demo
          image: mvc_access:0.1
          imagePullPolicy: IfNotPresent
          args:
            - --server.port=8080
      restartPolicy: Always
  selector:
    matchLabels:
      app: mvc-demo
---
apiVersion: v1
kind: Service
metadata:
  name: mvc-demo
  namespace: default
  labels:
    app: mvc-demo  
spec:
  selector:
    app: mvc-demo
  ports:
    - port: 8080
      targetPort: 8080
      name: http
    - port: 8611
      targetPort: 8611
      name: probe
  type: ClusterIP

暴露并验证两种接口

k port-forward mvc-demo-58d6b877b4-n6245 8088:8080
k port-forward mvc-demo-58d6b877b4-n6245 8611

# 业务接口
curl -s http://localhost:8088/index
running count .. 1.0

# 指标接口
curl -s http://localhost:8611/metrics/prometheus | grep access
# HELP mvc_access_total http access count
# TYPE mvc_access_total counter
mvc_access_total 1.0

查看 api 指标收集:

# 目前没有 mvc 相关指标
k get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq . | grep mvc

为了让prometheus发现这些目标, 来部署一下 ServiceMonitor

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  name: mvc-demo
  labels:
    app: mvc-demo
spec:
  selector:
    matchLabels:
      app: mvc-demo
  endpoints:
    - port: probe
      path: "/metrics/prometheus"

接下来打开PrometheusUI http://localhost:9090/targets , 查看serviceMonitor状态.
这里有个小坑, target active 一直没出来, 查看k8s日志, 查明是权限问题.

k logs -f prometheus-k8s-0 -n monitoring
ts=2022-03-17T01:19:52.747Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.22.5/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

参考: https://github.com/prometheus-operator/kube-prometheus/issues/483#issuecomment-610427646
更新Role角色权限:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.32.1
  name: prometheus-k8s
  namespace: monitoring
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - services
  - pods
  - endpoints
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /metrics/prometheus
  verbs:
  - get

其实, 这一步应该在安装 kube-protmethes时就准备好.

注意: - nonResourceURLs: 定义了默认提取信息的路径, 可以和客户端约定好, 就少写配置了.

然后, 再来刷新一下 PrometheusUI: http://localhost:9090/targets

service monitor targets

Nice, 说明Pod原始的数据, 已经上报给Promtheus了, 我们通过PromQL来确认一下: http://localhost:9090/graph

输入我们的metrics名称: mvc_access

PromQL

Cool ~, 到这里, 我们在代码里定义的指标名称和说明, 已经上报到Promtheus数据库, 而且支持查询.
这里就有两个后续了, 一种常规思路就是通过 Grafana 编写 PromQL 去显示这些自定义指标数据.
但我们的目标是, 基于这些指标, 去创建HPA 能识别的规则.

0x08 编写 adapter rule 文件

上面聊到HPA 基于自定义指标(QPS) 来伸缩, 但是上一步的指标是 Prometheus的数值, hpa 是k8s的核心功能 ,它们协作的话, 需要 adapter rule .

官方例子:

rules:
- seriesQuery: '{__name__=~"^container_.*",container!="POD",namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"},
      pod: {resource: "pod"},
  name:
    matches: "^container_(.*)_seconds_total$"
  metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)"

写rule 之前呢, 先了解一下 PromQL的入门用法.
首先, 给Pod改成两个实例.

指标查询

对比一下例子的第一行, 不难发现, seriesQuery 对标的就是 Promtheus指标的 tag .
我们可以试一下效果:
输入 {application="mvc-demo"}, 会把关联tag 的所有指标输出来.

query

参考 https://prometheus.io/docs/prometheus/latest/querying/functions/#rate
解释一下
rate(): rate用来计算两个间隔时间内发生的变化率（一段时间内平均每秒的增量）。

rate
rate(mvc_access_total[1m]) 返回每个Pod, 一分钟内每秒请求次数

所以, 接下来再来一个sum 就是表示一个业务类型的合计QPS:

sum

直接更新rule 配置文件, 参考: https://github.com/ntman4real/nginxmets#to-configure-the-custom-metrics-rule-and-cusotmizong-the-name-to-myrequests

# k edit cm adapter-config -n custom-metrics
apiVersion: v1
data:
  config.yaml: |
    rules:
    - seriesQuery: 'mvc_access_total'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: ""
        as: "mvc_qps"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container!="POD"}[2m])) by (<<.GroupBy>>)

重启 metrics-server, 使其读取rule configmap 配置

# k get po -A | grep metrics-api
custom-metrics   custom-metrics-apiserver-54bdbb4ff8-tx6bb   1/1     Running   0               7m20s
k get po custom-metrics-apiserver-54bdbb4ff8-tx6bb -n custom-metrics -o yaml  | k replace --force -f -

提交后, 验证一下 custom api

#k get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq .  | grep qps
      "name": "namespaces/mvc_qps",
      "name": "pods/mvc_qps",

看到上面的信息, 表示已经为我们的Pod添加了 mvc_qps 指标.

创建 hpa, 1000m 表示: service每秒请求数为1时, 就先触发伸缩.

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: mvc-demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mvc-demo
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metricName: mvc_qps
        targetAverageValue: 1000m

多请求几次外部接口
查看 hpa 状态

k get hpa -w
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
mvc-demo     Deployment/mvc-demo     1189m/1    2         10        2          100s
sample-app   Deployment/sample-app   66m/500m   1         10        1          45h
mvc-demo     Deployment/mvc-demo     274m/1     2         10        3          2m
mvc-demo     Deployment/mvc-demo     318m/1     2         10        3          3m
mvc-demo     Deployment/mvc-demo     240m/1     2         10        3          4m
mvc-demo     Deployment/mvc-demo     18m/1      2         10        3          5m
mvc-demo     Deployment/mvc-demo     0/1        2         10        3          6m
mvc-demo     Deployment/mvc-demo     0/1        2         10        2          7m

看到 replicas 副本数配置生效.

到这里, 淦了3天自定义 qps HPA任务终于达成.

继续学习:
https://github.com/kubernetes-sigs/prometheus-adapter
https://www.cnblogs.com/dudu/p/12146344.html
https://www.cnblogs.com/charlieroro/p/11898521.html

参考:
http://www.xuyasong.com/?p=1520

What's Next

上面的实现方式代码侵入性太强, 不适合生产.
找时间查找其他方式.

Mac Docker - Prometheus QPS Hpa

0x00 TLDR;

0x01 安装 ingress nginx

0x02 关于 Prometheus 在K8s中使用

0x03 部署 kube-prometheus

0x04 部署测试程序

0x05 部署 prometheus-adapter

0x05 验证HPA

0x06 自定义规则: http_requests_total 和 http_requests 的关系

0x07 Spring工程自定义指标

0x08 编写 adapter rule 文件

What's Next

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读

DevOps