plumbr整理

• JMX
  • Architecture of the JMX Technology
    • Instrumentation
      • MBeans
    • JMX agent
      • directly controls resources and makes them available to remote management applications
      • MBean server
    • Remote management
      • JMX technology instrumentation can be accessed in many different ways
      • The MBean server relies on protocol adaptors and connectors to make a JMX agent accessible
  • JMX 3层架构图
• JConsole
  • based-on JMX(rmi adapter), used to connect MBean Server to manage remote MBeans
• 1.publishing or consuming services over RMI
• 2.third party libraries or utilities can still open RMI endpoints
  • One such common culprit is for example JMX, which, if attached to remotely, will use RMI underneath to publish the data
• the JVM periodically launches full GC to make sure that locally unused objects are also not taking up space on the other end
• When you check the old generation consumption, there is often no pressure to the memory as there is plenty of free space in the old generation, but full GC is triggered, stopping the application threads.
• This behavior of removing remote references via System.gc() is triggered by the sun.rmi.transport.ObjectTable class requesting garbage collection to be run periodically as specified in the sun.misc.GC.requestLatency() method.
• java -Dsun.rmi.dgc.server.gcInterval=9223372036854775807L -Dsun.rmi.dgc.client.gcInterval=9223372036854775807L com.yourcompany.YourApplication(两个参数默认都是1h, 最佳实践是两个参数都设置上由JVM选择)
• This sets the period after which System.gc() is run to Long.MAX_VALUE; for all practical matters, this equals eternity.
• -XX:+DisableExplicitGC禁用System.gc()不推荐

• SetTag
  • jvmtiError SetTag(jvmtiEnv* env, jobject object, jlong tag)
  • Set the tag associated with an object. The tag is a long value typically used to store a unique identifier or pointer to object information. The tag is visible with GetTag.
• Get Tag
  • jvmtiError GetTag(jvmtiEnv* env, jobject object, jlong* tag_ptr)
  • Retrieve the tag associated with an object. The tag is a long value typically used to store a unique identifier or pointer to object information. The tag is set with SetTag. Objects for which no tags have been set return a tag value of zero.
• Whenever the application is run alongside with a Java Agent (-javaagent), there is a chance that the agent can tag the objects in the heap using JVMTI tagging. Agents can use tagging for various reasons that are not in the scope of this handbook, but there is a GC-related performance issue that can start affecting the latency and throughput of your application if tagging is applied to a large subset of objects inside the heap.
• The problem is hidden in the native code where JvmtiTagMap::do_weak_oops iterates over all the tags during each garbage collection event and performs a number of not-so-cheap operations for all of them. To make things worse, this operation is performed sequentially and is not parallelized.
• To check whether or not a particular agent can be the reason for extended GC pauses, you would need to turn on the diagnostic option of –XX:+TraceJVMTIObjectTagging. Enabling the trace will allow you to get an estimate of how much native memory the tag map consumes and how much time the heap walks take.
• If you are not the author of the agent yourself, fixing the problem is often out of your reach. Apart from contacting the vendor of a particular agent you cannot do much. In case you do end up in a situation like this, recommend that the vendor clean up the tags that are no longer needed.
• 实例
  • 花名有孚-中文翻译篇
• Humongous Allocations
  • 使用G1垃圾收集器才有
• Conclusion
  • Therefore, there is no real silver bullet approach to tuning the JVM to match the performance goals you have to fulfill. What we have tried to do here is walk you through some common (and not so common) examples to give you a general idea of how problems like these can be approached. Coupled with the tooling overview and with a solid understanding of how the GC works, you have all the chances of successfully tuning garbage collection to boost the performance of your application.

• Another class of issues affecting GC is linked to the use of non-strong references in the application. While this may help to avoid an unwanted OutOfMemoryError in many cases, heavy usage of such references may significantly impact the way garbage collection affects the performance of your application.
• Weak Reference
  • Whenever GC discovers that an object is weakly reachable, that is, the last remaining reference to the object is a weak reference, it is put onto the corresponding ReferenceQueue,
  • One may then poll this reference queue and perform the associated cleanup activities. A typical example for such cleanup would be the removal of the now missing key from the cache.
  • The trick here is that at this point you can still create new strong references to the object, so before it can be, at last, finalized and reclaimed, GC has to check again that it really is okay to do this. Thus, the weakly referenced objects are not reclaimed for an extra GC cycle.
  • Many caching solutions build the implementations using weak referencing
• Soft Reference
  • you should bear in mind that soft references are collected much less eagerly than the weak ones. The exact point at which it happens is not specified and depends on the implementation of the JVM. Typically the collection of soft references happens only as a last ditch effort before running out of memory. What it implies is that you might find yourself in situations where you face either more frequent or longer full GC pauses than expected, since there are more objects resting in the old generation.
• Phantom Reference
  • do manual memory management
  • In order to ensure that a reclaimable object remains so, the referent of a phantom reference may not be retrieved: The get method of a phantom reference always returns null.
  • Unlike soft and weak references, phantom references are not automatically cleared by the garbage collector as they are enqueued. An object that is reachable via phantom references will remain so until all such references are cleared or themselves become unreachable.
  • That is right, we have to manually clear() up phantom references or risk facing a situation where the JVM starts dying with an OutOfMemoryError. The reason why the Phantom references are there in the first place is that this is the only way to find out when an object has actually become unreachable via the usual means. Unlike with soft or weak references, you cannot resurrect a phantom-reachable object.

• 在使用cms算法下，如果开启参数UseAdaptiveSizePolicy，则每次minor gc后会重新计算eden，from和to的大小，计算过程依据的是gc过程统计的一些数据，计算后的eden+from+to不会超过Xmx，同时from和to一般是不相等(初始化的时候from和to是相等的), 会导致MBean使用时抛出异常Memory pool not found, 可以先设置 –XX:-UseAdaptiveSizePolicy来workaround

• Parallel Scavenge收集器有一个参数-XX:+UseAdaptiveSizePolicy。当这个参数打开之后，就不需要手工指定新生代的大小、Eden与Survivor区的比例、晋升老年代对象年龄等细节参数了，虚拟机会根据当前系统的运行情况收集性能监控信息，动态调整这些参数以提供最合适的停顿时间或者最大的吞吐量，这种调节方式称为GC自适应的调节策略（GC Ergonomics）

• -Xmx24m -XX:NewSize=16m -XX:MaxTenuringThreshold=1
• starting from using the object as keys in a weak hash map and ending with allocation profiling.
• The root cause, of course, lies with the weak references. Before we added them, the objects created by the application were dying just before being promoted to the old generation. But with the addition, they are now sticking around for an extra GC round so that the appropriate cleanup can be done on them. Like before, a simple solution would be to increase the size of the young generation by specifying -Xmx64m -XX:NewSize=32m: The objects are now once again reclaimed during minor garbage collection.
• demo analysis
  • https://github.com/gvsmirnov/java-perv/blob/master/labs-8/src/main/java/ru/gvsmirnov/perv/labs/gc/WeakReferences.java
  • This example shows how having weak references pointing to objects May result in more frequent Full GC pauses

• The situation is even worse when soft references are used as seen in the next demo application. The softly-reachable objects are not reclaimed until the application risks getting an OutOfMemoryError. Replacing weak references with soft references in the demo application immediately surfaces many more Full GC events:
• https://github.com/gvsmirnov/java-perv/blob/master/labs-8/src/main/java/ru/gvsmirnov/perv/labs/gc/SoftReferences.java

• https://github.com/gvsmirnov/java-perv/blob/master/labs-8/src/main/java/ru/gvsmirnov/perv/labs/gc/PhantomReferences.java
• 如果不手动queue.poll的话, 不仅仅可能导致fullgc还可能导致OOM
• One must exercise extreme caution when using phantom references and always clear up the phantom reachable objects in a timely manner. Failing to do so will likely end up with an OutOfMemoryError. And trust us when we say that it is quite easy to fail at this: one unexpected exception in the thread that processes the reference queue, and you will have a dead application at your hand.

• -XX:+PrintReferenceGC
  • 打开时可能看到日志
    • 2.173: [Full GC (Ergonomics) 2.234: [SoftReference, 0 refs, 0.0000151 secs]2.234: [WeakReference, 2648 refs, 0.0001714 secs]2.234: [FinalReference, 1 refs, 0.0000037 secs]2.234: [PhantomReference, 0 refs, 0 refs, 0.0000039 secs]2.234: [JNI Weak Reference, 0.0000027 secs]...
• As always, this information should only be analyzed when you have identified that GC is having impact to either the throughput or latency of your application. In such case you may want to check these sections of the logs. Normally, the number of references cleared during each GC cycle is quite low, in many cases exactly zero. If this is not the case, however, and the application is spending a significant period of time clearing references, or just a lot of them are being cleared, then further investigation is required.

• Weak references – if the problem is triggered by increased consumption of a specific memory pool, an increase in the corresponding pool (and possibly the total heap along with it) can help you out. As seen in the example section, increasing the total heap and young generation sizes alleviated the pain.
• Phantom references – make sure you are actually clearing the references. It is easy to dismiss certain corner cases and have the clearing thread to not being able to keep up with the pace the queue is filled or to stop clearing the queue altogether, putting a lot of pressure to GC and creating a risk of ending up with an OutOfMemoryError.
• Soft references – when soft references are identified as the source of the problem, the only real way to alleviate the pressure is to change the application’s intrinsic logic.

• promote rate
  • Notice that you can extract this information only from minor GC pauses. Full GC pauses do not expose the promotion rate as the change in the old generation usage in GC logs also includes objects cleaned by the major GC.
• Why Should I Care?
  • Similarly to the allocation rate, the main impact of the promotion rate is the change of frequency in GC pauses. But as opposed to the allocation rate that affects the frequency of minor GC events, the promotion rate affects the frequency of major GC events.
• https://github.com/gvsmirnov/java-perv/blob/master/labs-8/src/main/java/ru/gvsmirnov/perv/labs/gc/PrematurePromotion.java

工具

• JMX API
  • You can access this API either programmatically from your own application running inside the very same JVM, or using JMX clients.
  • Two of the most popular JMX clients are JConsole and JVisualVM (with a corresponding plugin installed).
  • All the JMX clients run as a separate application connecting to the target JVM. The target JVM can be either local to the client if running both in the same machine, or remote. For the remote connections from client, JVM has to explicitly allow remote JMX connections. This can be achieved by setting a specific system property to the port where you wish to enable the JMX RMI connection to arrive：java -Dcom.sun.management.jmxremote.port=5432 com.yourcompanyYourApp
  • After connecting your JMX client to the JVM of interest and navigating to the MBeans list, select MBeans under the node “java.lang/GarbageCollector”
  • In my experience this information is not enough to make any conclusions about the efficiency of the garbage collector.
  • This approach can rarely be used as we will see in the next sections, which give better ways of getting beneficial insight into garbage collection activities.
• JVisualVM
  • JVisualVM adds extra information to the basic JMX client functionality via a separate plugin called “VisualGC”. It provides a real-time view into GC events and the occupancy of different memory regions inside JVM.
  • The most common use-case for the Visual GC plugin is the monitoring of the locally running application, when an application developer or a performance specialist wants an easy way to get visual information about the general behavior of the GC during a test run of the application.
  • When compared with pure JMX tools, the VisualGC add-on to JVisualVM does offer slightly better insight to the JVM, so when you have only these two tools in your toolkit, pick the VisualGC plug-in. If you can use any other solutions referred to in this chapter, read on. Alternative options can give you more information and better insight. There is however a particular use-case discussed in the “Profilers” section where JVisualVM is suitable for – namely allocation profiling, so by no means we are demoting the tool in general, just for the particular use case.

References

• 从零开始玩转JMX(一)——简介和Standard MBean
• UseAdaptiveSizePolicy 选项引起Memory pool not found异常