When I run the following Java program, which consumes all free memory with artificial data structures, something interesting happens. Something very similar happens to all real applications:
[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms31g -Xmx31g -Xmn50m Memory Total Memory (in bytes): 33279705088 Free Memory (in bytes): 33278908064 Max Memory (in bytes): 33279705088 Elements created and added to LinkedList: 587889429
The parameters cause the JVM heap to be 31GB big, and the space used for object allocation only 50MB. Lets try a higher max memory setting of 32GB and see what happens:
[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms32g -Xmx32g -Xmn50m Memory Total Memory (in bytes): 34353446912 Free Memory (in bytes): 34352649800 Max Memory (in bytes): 34353446912 Elements created and added to LinkedList: 385481085
What is happening here? With more memory, we can create less elements? Oops!
Oop = Ordinary object pointer
Yes, Oops are actually the reason. An oop is the memory address of an object.
In the 32 bit world, this can go from 0 to 2^32-1. Which happens to be 4G, which means every byte of a 4GB heap can be addressed.
To manage larger amounts of memory the JVM actually needs to run with 64 bit addresses, which can manage 2^64-1 bytes which are about 18.5 Exabyte (a lot).
What this means is that for a twice as big address, a much larger memory area can be addressed. But there are not that many applications that really need 18.5 Exabyte of memory (let alone there are no servers which have so much memory). When people switched from a 32bit JVM to a 64bit JVM they noticed the loss of available memory due to larger addresses and complained. The JVM developers invented a feature called CompressedOops to fight it.
Because the JVM memory layout uses a 8byte addressing scheme, which means that objects can be at address 0, 8, 16, 24… but not at 2, 7 or any other non multiple of 8, compressed oops just address the virtual positions 0, 1, 2, 3 instead of the real ones 0, 8, 16, 24. To get from the compressed address to the real one, the JVM needs just to left shift it 3 times. Easy.
But, as you might imagine, this fails for heaps larger 32GB. The trick would actually work on a 32bit VM and enable addressing 8 times the memory which is 4×8 = 32GB. But sadly the 32 bit operating systems can’t do that.
As soon as the JVM needs to manage 32GB+ heaps, it cannot do the CompressedOops trick anymore. All object addresses become 64 bit. When going from 31GB, to 32GB in my example, about 200 million less objects can fit into the heap. Yuk.
To accommodate the same amount of objects one would need at least a 48GB heap:
[node04] ~ ➜ jdk1.7.0_51/bin/java -Xms48g -Xmx48g -Xmn50m Memory Total Memory (in bytes): 51533316096 Free Memory (in bytes): 51532518984 Max Memory (in bytes): 51533316096 Elements created and added to LinkedList: 578932717
We can turn the compression off, which shows us how much it actually helps us:
[node04] ~ ➜ jdk1.7.0_51/bin/java -XX:-UseCompressedOops -Xms31g -Xmx31g -Xmn50m Memory Total Memory (in bytes): 33279705088 Free Memory (in bytes): 33278907976 Max Memory (in bytes): 33279705088 Elements created and added to LinkedList: 373400520
As you can see the difference is dramatic. My demo application, as well as most real life applications consist of a large amount of references. Due to that most applications will be unable to utilize the memory between 32GB and mid 40’s GB. Bummer!
- Compressed Oops explained by Oracle: https://wikis.oracle.com/display/HotSpotInternals/CompressedOops
- Server used is a Dell Power Edge 510 24 core (2xE5645), 128GB, running Debian Wheezy and JDK 7.51, part of our centerdevice.de infrastructure.