Rectangle 27 0

java Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?


And you are not using the native Tomcat functionaliy (tomcat.apache.org/tomcat-6.0-doc/apr.html)? Unless you are keeping references to a lot of open FileChannel objects (and this would cause other problems, like reaching the max number of open files allowed), FileChannel usage alone is not really an explanation of your applications behaviour.

No JNI, but the application does rely heavily on java.nio.FileChannel to send data from disk to NIC...

No, I'm not. a) is it is flaky on 64-bit Linux (or was last time I checked), b) I had issues with in an a 3rd party jar, and c) I really don't have that many connections per second to worry about connector performance.

The current garbage collector in Java is well known for not releasing allocated memory, although the memory is not required anymore. It's quite strange however, that your RSS size increases to >3GB although your heap size is limited to 640MB. Are you using any native code in your application or are you having the native performance optimization pack for Tomcat enabled? In that case, you may of course have a native memory leak in your code or in Tomcat.

With Java 6u14, Sun introduced the new "Garbage-First" garbage collector, which is able to release memory back to the operating system if it's not required anymore. It's still categorized as experimental and not enabled by default, but if it is a feasible option for you, I would try to upgrade to the newest Java 6 release and enable the new garbage collector with the command line arguments "-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC". It might solve your problem.

Note
Rectangle 27 0

java Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?


+1 - although this should only be an issue for the files that are being actively accessed

An argument against this would be that a) the RSS graph has an amazingly regular, straight slope, and b) the FileChannel usage is tied to how busy the application is, which fluctuates wildly hour-to-hour, day-to-day. I would expect to see a correlation.

Interesting. I do use the java.nio.FileChannel allot...must investigate...

Just an idea: NIO buffers are placed outside the JVM.

Note
Rectangle 27 0

java Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?


[stu@server ~]$ ps -p 12720 -o pcpu,cutime,cstime,cmin_flt,cmaj_flt,rss,size,vsize
%CPU - - - -  RSS SZ  VSZ
28.8 - - - - 3262316 1333832 8725584
ps -p 7429 -o pcpu,cutime,cstime,cmin_flt,cmaj_flt,rss,size,vsize

As you can seem by the ps output (edited into your answer), the 3GB number is accurate. (I started graphing over time after noticing large numbers from top and ps on long running instances.) It is what surprised me--something seems wrong if my RSS is 5 times the heap size. Hence this SO question.

I noticed that you mentioned having hundreds to low thousands of threads. The stacks for these threads will also add to the RSS, although it shouldn't be much. Assuming that the threads have a shallow call depth (typical for app-server handler threads), each should only consume a page or two of physical memory, even though there's a half-meg commit charge for each.

In your case, I don't think it's an issue. The graph appears to show 3 meg consumed, not 3 gig as you write in the text. That's really small, and is unlikely to be causing paging.

Normally, VSZ is slightly > SZ, and very much > RSS. This output indicates a very unusual situation.

RSS represents pages that are actively in use -- for Java, it's primarily the live objects in the heap, and the internal data structures in the JVM. There's not much that you can do to reduce its size except use fewer objects or do less processing.

RSS represents the number of pages resident in RAM -- the pages that are actively accessed. With Java, the garbage collector will periodically walk the entire object graph. If this object graph occupies most of the heap space, then the collector will touch every page in the heap, requiring all of those pages to become memory-resident. The GC is very good about compacting the heap after each major collection, so if you're running with a partial heap, there most of the pages should not need to be in RAM.

RSS, as noted, is the resident set size: the pages in physical memory. SZ holds the number of pages writable by the process (the commit charge); the manpage describes this value as "very rough". VSZ holds the size of the virtual memory map for the process: writable pages plus shared pages.

Regarding the 3M RSS size - yeah, that seemed too low for a Tomcat process (I checked my box, and have one at 89M that hasn't been active for a while). However, I don't necessarily expect it to be > heap size, and I certainly don't expect it to be almost 5 times heap size (you use -Xmx640) -- it should at worst be heap size + some per-app constant.

So what else is happening in your system? Is it a situation where you have lots of Tomcat servers, each consuming 3M of RSS? You're throwing in a lot of GC flags, do they indicate the process is spending most of its time in GC? Do you have a database running on the same machine?

The GC times look OK. I continue to monitor them. Like I said, the io wait time is increasing. And I can see that the system file cache shrinks to a very small number, compared to when the JVM is not sucking up huge swaths of real memory.

The RSS value is 3GB, not 3MB. The graph is in Kilo Bytes. 3 mega of KB = 3GB. I will update question for clarity. (Besides, one would logically expect real memory to be > java heap size. 3MB is a fraction of 400MB.)

Which causes me to suspect your numbers. So, rather than a graph over time, please run the following to get a snapshot (replace 7429 by whatever process ID you're using):

Note
Rectangle 27 0

java Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?


-Xss?

About JNI, you wrote that the application wasn't using JNI but... What type of JDBC driver are you using? Could it be a type 2, and leaking? It's very unlikely though as you said database usage was low.

About excessive threads creation, each thread gets its own stack which may be quite large. The stack size actually depends on the VM, OS and architecture e.g. for JRockit it's 256K on Linux x64, I didn't find the reference in Sun's documentation for Sun's VM. This impacts directly the thread memory (thread memory = thread stack size * number of threads). And if you create and destroy lots of thread, the memory is probably not reused.

Anyway, thanks for the ideas. It does look like I'll find the issue in the 'real memory graph', which is a whole new world for me.

JNI: Good point, I have no idea what type it is. But I'm not inclined to investigate just yet because, as you noted, my app is not DB intensive.

JVM uses more memory than just the heap. For example Java methods, thread stacks and native handles are allocated in memory separate from the heap, as well as JVM internal data structures.

Threads: My apps threads, specifically the threads that handle an HTTP connection, are frequently long lived: tens of seconds, minutes, or potentially longer. The number of connections my server can process at at once (the number of HTTP streams) is a linear function of how many threads I can have. In every day usage, like in the scope of above graphs, the number of threads varied between 50 and 700. An unusual application, yea.

To be honest, hundreds to low thousands of threads seems enormous to me. That said, if you really need that much threads, the thread stack size can be configured via the -Xss option. This may reduce the memory consumption. But I don't think this will solve the whole problem. I tend to think that there is a leak somewhere when I look at the real memory graph.

What can I do to keep the JVM's real memory consumption in check?

Why is this happening? What is going on "under the hood"?

Note