Overview

Why 35GB Heap is Less Than 32GB – Java JVM Memory Oddities

3 Comments

When I run the following Java program, which consumes all free memory with artificial data structures, something interesting happens. Something very similar happens to all real applications:
https://gist.github.com/CodingFabian/8708393

[node04]  ~ ➜ jdk1.7.0_51/bin/java -Xms31g -Xmx31g -Xmn50m Memory
Total Memory (in bytes): 33279705088
Free Memory (in bytes): 33278908064
Max Memory (in bytes): 33279705088
Elements created and added to LinkedList: 587889429

The parameters cause the JVM heap to be 31GB big, and the space used for object allocation only 50MB. Lets try a higher max memory setting of 32GB and see what happens:

[node04]  ~ ➜ jdk1.7.0_51/bin/java -Xms32g -Xmx32g -Xmn50m Memory
Total Memory (in bytes): 34353446912
Free Memory (in bytes): 34352649800
Max Memory (in bytes): 34353446912
Elements created and added to LinkedList: 385481085

What is happening here? With more memory, we can create less elements? Oops!

Oop = Ordinary object pointer

Yes, Oops are actually the reason. An oop is the memory address of an object.
In the 32 bit world, this can go from 0 to 2^32-1. Which happens to be 4G, which means every byte of a 4GB heap can be addressed.
To manage larger amounts of memory the JVM actually needs to run with 64 bit addresses, which can manage 2^64-1 bytes which are about 18.5 Exabyte (a lot).

What this means is that for a twice as big address, a much larger memory area can be addressed. But there are not that many applications that really need 18.5 Exabyte of memory (let alone there are no servers which have so much memory). When people switched from a 32bit JVM to a 64bit JVM they noticed the loss of available memory due to larger addresses and complained. The JVM developers invented a feature called CompressedOops to fight it.
Because the JVM memory layout uses a 8byte addressing scheme, which means that objects can be at address 0, 8, 16, 24… but not at 2, 7 or any other non multiple of 8, compressed oops just address the virtual positions 0, 1, 2, 3 instead of the real ones 0, 8, 16, 24. To get from the compressed address to the real one, the JVM needs just to left shift it 3 times. Easy.

But, as you might imagine, this fails for heaps larger 32GB. The trick would actually work on a 32bit VM and enable addressing 8 times the memory which is 4×8 = 32GB. But sadly the 32 bit operating systems can’t do that.
As soon as the JVM needs to manage 32GB+ heaps, it cannot do the CompressedOops trick anymore. All object addresses become 64 bit. When going from 31GB, to 32GB in my example, about 200 million less objects can fit into the heap. Yuk.
To accommodate the same amount of objects one would need at least a 48GB heap:

[node04]  ~ ➜ jdk1.7.0_51/bin/java -Xms48g -Xmx48g -Xmn50m Memory
Total Memory (in bytes): 51533316096
Free Memory (in bytes): 51532518984
Max Memory (in bytes): 51533316096
Elements created and added to LinkedList: 578932717

We can turn the compression off, which shows us how much it actually helps us:

[node04]  ~ ➜ jdk1.7.0_51/bin/java -XX:-UseCompressedOops -Xms31g -Xmx31g -Xmn50m Memory
Total Memory (in bytes): 33279705088
Free Memory (in bytes): 33278907976
Max Memory (in bytes): 33279705088
Elements created and added to LinkedList: 373400520

As you can see the difference is dramatic. My demo application, as well as most real life applications consist of a large amount of references. Due to that most applications will be unable to utilize the memory between 32GB and mid 40’s GB. Bummer!

Appendix

Kommentare

  • SK

    Hi Fabian,

    Thanks for sharing this post – I wanted to clarify one thing. Where you list 35GB in your post, its followed by an entry with the heap size at 32 GB.
    Did you set it back to 32GB in your test to demonstrate the effect of compressedoops being unavailable at sizes starting at 32 GB, instead of using 35GB?

  • Fabian Lange

    Hi SK,
    Indeed I did change the test setup. Because the effect was already showing at 32GB compared to 31 GB. However later in the post, you will see that only at 48GB you “gain” memory, which is quite surprising.
    Ill fix the text! Thanks for asking

    Fabian

  • Philip Dufault

    30. March 2015 von Philip Dufault

    Thanks Fabian, this explains something I saw at work. Great post.

Comment

Your email address will not be published. Required fields are marked *