1.利用 Grafana 视图定位 GC

从 SaaS业务监 到视图进程监控，下列选择微服务 servicesA。

① 查看视图 JVM Heap，选择 G1 Old Gen

如下图可知，6 个实例在 11 点左右进行了 GC，回收了部分老年代。

image-20220420153732819.png

② 选择其中实例，确定 GC 的具体时间

如下图所示，选择实例 10.244.1.117，确定 GC 回收老年代的时间范围 10:52~10:53

image-20220420153809951.png image-20220420153923081.png

2.进入容器查看 JVM 参数和 GC 日志

① 查看 JVM 参数

如下所示，利用 G1 作为垃圾收集器，Heap 的最大和最小内存分别是总内存的 80% 和 25%，GC 的停顿时间是 200 ms，最大堆外内存是 512mb，GC 的日志在 logs 目录下。

[root@master-0001 ~]# kubectl get pod |grep servicesA
servicesA-6bb575dc44-2rcpf                 1/1     Running   0          19h
servicesA-6bb575dc44-2x9l7                 1/1     Running   0          19h
servicesA-6bb575dc44-j4t5k                 1/1     Running   0          19h
[root@stage-master-0001 ~]# kubectl exec -it servicesA-6bb575dc44-2rcpf bash
bash-5.0$ ps -ef|grep java
paas         72      1 30 Apr19 ?        05:55:49 java -XX:+UseG1GC -XX:MaxRAMPercentage=80.0 -XX:MinRAMPercentage=25.0 -XX:MaxGCPauseMillis=200 -XX:MaxDirectMemorySize=512m -verbose:gc -Xloggc:/opt/../logs/gc-servicesA.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1024k -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -Dcctm.envid=172

① 查看 GC 日志

上述可知，GC 回收老年代的时间范围 10:52~10:53，根据 GC 日志可以知道总共有 5 次 mixed GC，把老年代的内存从 1194.3M 下降到 399.1M

image-20220420160323972.png

GC 日志详解：

取最后一次 mixed GC 的日志，关键信息是 [Eden: 120.0M(120.0M)->0.0B(119.0M) Survivors: 6144.0K->7168.0K Heap: 560.1M(2536.0M)->399.1M(2536.0M)]，即 Eden 内存从 120.0M 下降到 0， Survivors 内存从 6144.0K 到 7168.0K，而 Heap 即 Old 从 560.1M 下降到 399.1M

2022-04-20T02:53:07.176+0000: 54685.006: [GC pause (G1 Evacuation Pause) (mixed)GC Start: NUMA region verification (id: matched/mismatched/total): 0: 0/0/2536  
GC End: NUMA region verification (id: matched/mismatched/total): 0: 0/0/2536  
, 0.1532921 secs]
   [Parallel Time: 149.1 ms, GC Workers: 2]
      [GC Worker Start (ms): Min: 54685006.9, Avg: 54685006.9, Max: 54685006.9, Diff: 0.0]
      [Ext Root Scanning (ms): Min: 4.5, Avg: 4.6, Max: 4.7, Diff: 0.2, Sum: 9.2]
      [Update RS (ms): Min: 16.1, Avg: 16.1, Max: 16.1, Diff: 0.0, Sum: 32.2]
         [Processed Buffers: Min: 69, Avg: 76.0, Max: 83, Diff: 14, Sum: 152]
      [Scan RS (ms): Min: 102.8, Avg: 107.2, Max: 111.5, Diff: 8.7, Sum: 214.4]
      [Code Root Scanning (ms): Min: 0.1, Avg: 4.2, Max: 8.4, Diff: 8.3, Sum: 8.5]
      [Object Copy (ms): Min: 16.6, Avg: 16.9, Max: 17.2, Diff: 0.6, Sum: 33.8]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 2]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 149.1, Avg: 149.1, Max: 149.1, Diff: 0.0, Sum: 298.2]
      [GC Worker End (ms): Min: 54685156.0, Avg: 54685156.0, Max: 54685156.0, Diff: 0.0]
   [Code Root Fixup: 0.3 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.4 ms]
   [Other: 3.5 ms]
      [Choose CSet: 0.7 ms]
      [Ref Proc: 0.2 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.3 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.0 ms]
      [Free CSet: 1.4 ms]
   [Eden: 120.0M(120.0M)->0.0B(119.0M) Survivors: 6144.0K->7168.0K Heap: 560.1M(2536.0M)->399.1M(2536.0M)]
Heap after GC invocations=1616 (full 0):
 garbage-first heap   total 2596864K, used 408716K [0x0000000700000000, 0x0000000700104f40, 0x00000007c0000000)
  region size 1024K, 7 young (7168K), 7 survivors (7168K)
 Metaspace       used 180608K, capacity 191473K, committed 193872K, reserved 1218560K
  class space    used 22269K, capacity 23691K, committed 24188K, reserved 1048576K
}
 [Times: user=0.29 sys=0.00, real=0.16 secs] 
{Heap before GC invocations=1616 (full 0):
 garbage-first heap   total 2596864K, used 530572K [0x0000000700000000, 0x0000000700104f40, 0x00000007c0000000)
  region size 1024K, 126 young (129024K), 7 survivors (7168K)
 Metaspace       used 180608K, capacity 191473K, committed 193872K, reserved 1218560K
  class space    used 22269K, capacity 23691K, committed 24188K, reserved 1048576K