Thursday, January 3, 2013

Linux Specific Tuning "HugePages"



When you are addressing large amounts of memory this quickly adds up to lots of memory pages.

The mechanism allows hugepages Linux kernel can take advantage of the ability of multiple sizes of pages like "HugeTLB". Default memory page size is typically 4KB for x86/x86_64 architecture, the block of memory that the system allocates for recording data are 4KB and this could be checked on OS by the PAGE_SIZE(getconf -a). Use a larger page addressing allows for less fragmentation and reduces the overhead of system management pages and tables.

Note that the application must be prepared to work with this type of feature. Normally hugepages allow allocation blocks 2048KB (2 MB) to 262144KB (256 MB) depending on the hardware architecture.

− Large memory page support usually starts with 2MB memory pages, and can be as large as 256MB on some architectures.
− All the major JVM's support large memory pages on Linux, but its a little trickier to setup than one would think.
− Besides the system overhead of mapping so many memory pages, large memory pages on Linux cannot be swapped to disk.

>> Enabling featuring "Large Page Support" JVM
The Sun JVM requires the following option, passed on the command-line, to use large pages :
-XX:+UseLargePages

Next, you set the following in "/etc/sysctl.conf"


kernel.shmmax = n
Where "n" is equal to the number of bytes of the maximum shared memory segment allowed on the system. You should set it at least to the size of the largest heap size you want to use for the JVM, or alternatively you can set it to the total amount of memory in the system.

vm.nr_hugepages = n
Where "n" is equal to the number of large pages. You will need to look up the large page size in /proc/meminfo.

vm.huge_tlb_shm_group = gid
Where gid is a shared group id for the users you want to have access to the large pages.

In /etc/security/limits.conf         <------------------
<username> soft memlock n
<username> hard memlock n


Large Page Support Example
I have a server with 4GB of memory and will allocate 2GB to be shared by the JVM
Page size is 2MB (2048 KB), as shown in "/proc/meminfo" or "getconf -a | grep -i page_size"

Hugepagesize:
2048 KB


Here's /etc/sysctl.conf        <------------------

# Change maximum shared memory segment size to 4GB
kernel.shmmax="4000000000" # 4GB

# Add the gid to the hugetlb_shm_group to give access to the users
vm.hugetlb_shm_group="504" # id of jboss group

# Dedicating 2GB of RAM to JVM (2MB * 1024)
vm.nr_hugepages="1024"

Here is /etc/security/limits.conf    <------------------

jboss soft memlock 2097152
jboss hard memlock 2097152 # (nr_hugepages * HUGE PAGE SIZE) = (1024 * 2048(2MB)) = 2097152

No comments:

Post a Comment