Java application VMs allocate thread stacks in native memory. Native memory is outside of the heap. Thus, the -Xms and -Xms VM arguments have no effect on it. The -Xss VM argument is used to determine the thread stack size. The default is operating system and VM version dependent. An application can exhaust the native heap with thread allocations and still have plenty of heap.
One way to know that there is a native memory issue is when an OutOfMemoryError with the message 'unable to create new native thread' is thrown either by GemFire or the application. The error must contain the 'unable to create new native thread' message and not the 'Java heap space' message. See the Heap Memory Issue page for details on that issue. An example is shown below.
Another way to check whether there is a native memory issue is to use either the gemfire stats command or vsd to display the number of threads contained in a given GemFire statistics archive. The vmStats category shows the number of threads in the VM at any time.
Use the gemfire stats command to display the VMStats threads value contained in given GemFire statistics archive as shown below.
Use vsd to graph the VMStats threads value contained in a given GemFire statistics archive as shown below.
You can dump all the live threads of a running VM using kill -3 <pid>. This does not kill the VM. Instead it signals it to dump the current state of all of its live threads. An example is shown below:
In simple terms, native memory is the difference between the physical RAM on the machine and the heap size of the VM(s). It is actually even less than that since the operating system also uses some of this memory. Given this, there are several ways to eliminate a native memory issue:
- Reduce the thread stack size of the VM using -Xss. Something like -Xss256m or -Xss384m should be sufficient.
- Reduce the max heap size of the VM using -Xmx. This will provide a greater difference between the physical RAM and the heap and thus more native memory.
- Add RAM to the machine