Subject: Re: GC performance results
Date: Tuesday 6th November 2007 19:20:36 UTC (over 11 years ago)
Charles, How did you figure out the application pause times with the current settings? Are you going by CMS initial mark and remark times? I'd like to take a look at our production logs to see what kind of pause time we're seeing to determine whether UseMembar is necessary for us as well. Kyong At 10:00 AM -0500 11/6/07, Charles Hedrick wrote: >For the last several weeks, we've been seeing regular slowdowns in >Sakai. They can list for several minutes. Usually Sakai recovers. > >I now believe that the primary cause is the GC. When I went back and >looked at catalina.out, I found that the GC was pausing all >processes for periods as long as 5 minutes. Note that we run with a >fairly large JVM, 13 GB total, with 3 GB in new. It's pretty clear >that we don't need that much space. > >We've spent the last week trying various configurations. I now recommend > >XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseMembar > >The primary cause of the very long GCs seems to be a bug: >http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278 > >UseMembar works around the bug. The bug seems to have been >introduced in Java 5, after initial release, when significant >changes to the memory model were introduced. UseMembar apparently >returns to the older behavior. Sun claims that the problem was fixed >in 1.5u13. However others report that it wasn't fixed, and our own >tests verify that. More fixes are scheduled for 1.5u14. The same >issues are present in Java 6; a fix is scheduled for a future >version of 6 as well. The workaround could have a performance >penalty, although we don't think it's significant. Most reports have >been with Linux, however we are using Solaris (on x64). > >While the reports I've seen refer to CMS (the concurrent mark-sweep >GC, i.e. XX:+UseConcMarkSweepGC), our tests showed it with the older >UseParallelGC (which is the default in most cases). Without >UseMembar we saw a few cases of GCs taking several minutes even with >UseParallelGC. > >With UseMembar, CMS typically pauses processes for less than 1 >second for minor GCs and 2 seconds for full GCs. However I've seen >up to 2 and 3 seconds now and then. Even with UseMembar, >UseParallelGC can pause up to 45 sec for a GC. WIthout it, pauses >can be much longer. These results seem to say that we should use CMS >with UseMembar. > >If you are using JForum, a GC that pauses processes more than a few >seconds will cause that JVM to fall out of sync with the others in >JGroups. Typically it will resync, but this may take quite some >time, and we saw one situation where it never happened. If for some >reason you can't use the parameters I recommend, you will probably >want to increase the timeouts in the cache configuration for JForums >to be longer than the longest pause caused by GC. > >Our current setting is > >JAVA_OPTS=" -d64 -Xmx13000m -Xms13000m -Xmn3g >-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 >-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 >-XX:MaxPermSize=512m -XX:PermSize=64m -XX:+UseMembar " > >There are additional options you can use with CMS. I have no >objection to them; we just haven't tried them yet. > >Our options result in a 10 GB old area. This is certainly larger >than needed. I prefer to keep a very large new with 2.3.1. Because >of how the 2.3.1 chat and content are implemented, we need new to be >several times the largest file we will ever manipulate, and for chat >in large courses it needs to be very big. > > >---------------------- >This automatic notification message was sent by Sakai Collab >(https://collab.sakaiproject.org/portal) from the DG: Development >(a.k.a. sakai-dev) site. >You can modify how you receive notifications at My Workspace > Preferences. -- Kyong Kim Instructional Multimedia & Web Programmer Distance Learning Foothill College 12345 El Monte Rd Los Altos Hills, CA 94022 650-949-7091 ---------------------- This automatic notification message was sent by Sakai Collab (https://collab.sakaiproject.org/portal) from the DG: Development (a.k.a. sakai-dev) site. You can modify how you receive notifications at My Workspace > Preferences.