背景：docker内运行的Java服务不像宿主机直接运行那么方便查询问题，今天记录下，首先镜像内有些命令是不支持的，比如tomcat镜像没有ls、top等命令（jdk镜像默认有这些命令），不方便查找问题，tomcat的openjdk版本是有的，如tomcat:8.5.49-jdk8-openjdk
参考
碰到系统CPU飙高和频繁GC，你会怎么排查？
Linux使用jstat命令查看jvm的GC情况
小知识
1 根据容器名查看对应宿主机的进程id
docker inspect -f '{{.State.Pid}}' competent_lichterman
2 查看gc情况，9是进程id ，每5s自动刷新一次。
jstat -gc 9 5000
full gc的次数太多说明程序有问题，fgc 期间会导致服务不可用，后面有fgc的时间
图片.png
3 查看jvm线程的堆栈信息。查看进程id为1的线程信息，包含iot代码字样的附近30行内容，一般程序出问题，这里会定位到有问题的代码部分
jstack 1 | grep "iot" -A 30
导出线程信息到文本
jstack -l <pid> >jvm.txt
4 查看整个JVM内存状态
jmap -heap [pid]
查看JVM堆中对象详细占用情况
jmap -histo [pid]
导出整个JVM 中内存信息
jmap -dump:format=b,file=文件名 [pid]
jmap -dump:live,format=b,file=d:/20170417.dump PID
[root@01384fe8ffb3 /]# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 2092957696 (1996.0MB)
NewSize = 44040192 (42.0MB)
MaxNewSize = 697303040 (665.0MB)
OldSize = 88080384 (84.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 490733568 (468.0MB)
used = 24698304 (23.55413818359375MB)
free = 466035264 (444.44586181640625MB)
5.032935509314904% used
From Space:
capacity = 6815744 (6.5MB)
used = 6531296 (6.228729248046875MB)
free = 284448 (0.271270751953125MB)
95.82660381610577% used
To Space:
capacity = 14155776 (13.5MB)
used = 0 (0.0MB)
free = 14155776 (13.5MB)
0.0% used
PS Old Generation
capacity = 150470656 (143.5MB)
used = 50911384 (48.552879333496094MB)
free = 99559272 (94.9471206665039MB)
33.83475911741888% used
35304 interned Strings occupying 4184288 bytes.
[root@01384fe8ffb3 /]#
问题排查思路
1通过docker stats 查看服务cpu、内存占用情况，定位有问题的服务
图片.png
2 进入容器top命令查看cpu占用最多的进程id，一般jdk镜像java进程pid都是1
图片.png
3 查看进程中各线程占用情况
top -Hp 1
图片.png
4 jstack -F 线程id ：即可查看某线程正在运行情况(不加F会报错！！！)
[root@826d95fa3a92 /]# jstack -F 167
Attaching to process ID 167, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11
Deadlock Detection:
No deadlocks found.
Thread 436: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:93)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:86)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLo
5导出堆内存数据
jmap -dump:format=b,file=myjvm.dump 1
网友评论