Sorry, you have been blocked
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
What can I do to resolve this?
You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
Cloudflare Ray ID: 81942ee88e3cb353 • Your IP: Click to reveal 45.84.122.38 • Performance & security by Cloudflare
Linux Huge Pages and Web Performance
One aspect where your server can be optimized for performance is making its memory management more efficient.
By default, programs operate on memory using small chunks of data. When large blocks of memory have to be accessed and written in small chunks, things are getting slower.
Huge Pages mechanism enables programs to work with memory using larger chunks. And thus it is faster.
There are basically two types of Huge Pages available.
1. Explicit Huge Pages (Hugetlbfs)
First is explicit huge pages (Hugetlbfs). These require programs to be compiled with support for them.
2. Transparent Huge Pages (THP)
Second is transparent huge pages. These must be enabled by the kernel settings. Even the programs which are not aware of huge pages can leverage them if transparent huge pages are enabled.
THP are enabled by default starting from CentOS / RedHat 6.
Nice explanation on explicit (old) vs transparent huge pages is here.
However, I have seens some performance enthusiasts skip the lines and going as far as saying:
… automatically enables Hugepages support if you have a Linux kernel that supports it + CentOS 7 and are not using redis server. If you use redis server, hugepages support is disabled for best redis server performance and memory usage.
Obviously without giving any distinction to end users about transparent vs explicit huge pages. These things can be enabled and disabled independently of each other. Not to say a careful approach needs to be taken while adjusting configuration values for each.
So what should we really do and what should you disable or keep? It depends on the situation, but in most cases, you would never want to disable explicit huge pages!
Redis and Huge Pages
Redis only has problem with transparent huge pages. So if you must use Redis, disable THP.
PHP and Huge Pages
PHP opcache extension can actually benefit from huge pages. PHP 7, by default, is compiled with support for explicit huge pages. In this case you may want to configure few things for smooth operation.
Your specific PHP opcache version might have been compiled without support for huge pages. Run this to confirm:
Empty output means that you’re out of luck as PHP was compiled without —enable-huge-code-pages switch (or old enough to not support it).
Result like this:
…means that huge pages are supported and enabled in Opcache.
If you want to enable huge pages support for Opcache (provided it was compiled with it), add to your php.ini the following:
Next thing to know is that the number of huge pages is controlled via kernel setting /proc/sys/vm/nr_hugepages . What's the proper setting value?
Suppose that you want to allocate 256M of RAM to PHP Opcache (YMMV). Pages that are used as huge pages are reserved inside the kernel and cannot be used for other purposes. So we don’t want to allocate too many explicit huge pages.
Let’s find out the size of a huge page on your system:
grep "Hugepagesize:" /proc/meminfo
Hugepagesize: 2048 kB
So each huge page is equal 2MB and we need a total of 256 / 2 = 128 huge pages allocated by kernel. Edit /etc/sysctl.conf and add
Apply changes with:
Or better yet, reboot the server so that the kernel can reallocate things properly. (You may get too few huge pages reserved unless rebooted).
Observe your huge pages being used with:
AnonHugePages: 0 kB
HugePages_Total: 128
HugePages_Free: 95
HugePages_Rsvd: 32
HugePages_Surp: 0
Hugepagesize: 2048 kB
Huge pages что это
HugePages is a feature integrated into the Linux kernel. For Oracle Database, using HugePages reduces the operating system maintenance of page states and increases Translation Lookaside Buffer (TLB) hit ratio.
- Check if the kernel supports HugePages:
-
Create a text file hugepages_settings.sh
Before you run this script, ensure that all the applications that use hugepages run.
If you cannot set your HugePages allocation using nr_hugepages , then your available memory may be fragmented. Restart your server for the Hugepages allocation to take effect.
Huge Pages and PostgreSQL
When talking to customers, sometimes I get the question: How should PostgreSQL installations deal with huge pages and large memory allocations? In particular, experienced Oracle DBA’s are interested in the details behind PostgreSQL and Huge Pages on Linux, so I’ll try to explain it in a bit more detail in the following blog post.
What am I talking about? What are Huge Pages?
For people not familiar with the meaning of huge pages, here is a short overview of what they are and which problems they try to solve.
On any modern system, applications don’t use physical memory directly. Instead, they use a virtual memory address model to make it easier to handle memory allocation and avoid the complexity of computing and mapping physical addresses into the application memory space. The virtual memory address model, however, requires the computer to translate virtual addresses into their corresponding physical memory addresses as soon as any application reads or writes to or from memory. This mapping is organized in the so-called page tables, a hierarchically-organized lookup table.
The TLB (Translation Lookaside Buffer) and Performance
Memory allocations on any common operating system are done by allocating single or multiple pages of available memory. These pages have a specific size, mostly depending on the operating system. On Linux, the default page size used by memory allocation are 4kB. So someone can imagine that allocating very large amounts of memory requires a high number of pages to be held in the page table, in order to map them to their corresponding physical addresses. Looking up virtual addresses requires multiple memory accesses, which are very costly when compared to CPU speed. Thus, modern CPUs maintain a so-called Translation Lookaside Buffer (TLB), which is basically a cache that speeds up translation between virtual and physical memory.
The size of a TLB is constrained to a few thousand entries, and especially on systems with high memory-specific memory workload, very soon becomes a bottleneck. If, for example, a database like PostgreSQL uses a shared buffer pool to implement Inter Process Communication (IPC), block layer access, and other functionality over it, this very quickly puts a high amount of pressure on the TLB, since the number of virtual address mappings is limited. Especially when using dozens of gigabytes for the shared buffer pool, such instances will suffer from performance hits because of this, depending on their workload.
The solution: Huge Pages
On modern operating systems like Linux this problem can be mitigated by using Huge Pages. Since page tables are organized hierarchically, they enable us to summarize allocations in much larger pages than the default. The size of huge pages are architecture-dependent, on x86 systems we can usually expect 2MB or 1GB sizes, IBM POWER allows 64kB, 16MB and 16 GB.
Lookups for specific virtual addresses of memory allocations are then much faster and hopefully more independent of the entries found in the TLB.
On x86, when configuring huge pages, the 2MB page size is the default. You can easily get your system current settings via /proc/meminfo :
HugePages_Total says no huge pages are configured at the moment, accordingly HugePages_Rsvd , HugePages_Free and HugePages_Surp are equal to zero. Hugepagesize shows the currently configured default size for huge pages in the kernel. When configuring huge pages later, we will revisit the value of these settings.
PostgreSQL usage of huge pages
PostgreSQL uses huge page allocations only for its shared buffer pool. Dynamic memory allocation for e.g. work_mem , maintenance_work_mem , autovacuum_work_mem doesn’t use them. Also, dynamic shared memory segments used to transfer data and communication between parallel query workers don’t request them automatically, with one exception, which I’ll describe in the next paragraph about configuring PostgreSQL for huge pages on Linux. It’s also necessary to have shared buffers allocated in so-called mmap’ed segments, which are controlled by the shared_memory_type parameter. This is the default setting on Linux. System V shared memory is only used by PostgreSQL to initialize a very small shared memory area; the buffer pool itself is allocated via mmap .
Configure your system
Huge Pages aren’t configured on recent Linux Distributions out of the box. There are usually two ways to teach your Linux system to provide them to your applications:
– Transparent Huge Pages (THP)
– Explicitely configured pools of huge pages for applications
The first possibility via THP isn’t recommended for PostgreSQL on Linux, since it doesn’t allocate huge pages directly but provides them to applications as they need them. THP are swappable and considered a slower option compared to explicit configured huge pages. Other operating systems like FreeBSD provide better support in this case.
An easy example
So we are going to focus on explicit huge pages support for PostgreSQL. In the following sections, examples are all done on a Rocky Linux 9 system, but should apply to all other Linux systems accordingly. To configure explicit huge pages, we need to know the computed size of shared memory and additional shared allocations in advance. As of PostgreSQL 15, this is very easy (but unfortunately requires a PostgreSQL instance to be down). With the PostgreSQL 15 Community RPM installation from yum.postgresql.org it looks as follows:
Again, the above code doesn’t start PostgreSQL, but calculates the value of shared_memory_size_in_huge_pages and prints the result to the terminal. This also gives us some flexibility to calculate this value according to specific configuration settings we can provide to the postgres binary as command line options. The example above specifies —shared-buffers as a command line option, overriding any other setting in the configuration files (like postgresql.conf ).
If PostgreSQL cannot allocate the requested amount of shared buffers with huge pages, it falls back to normal page size allocations. This is controlled by the configuration parameter huge_pages , which can have the following values:
- try : Try to allocate huge pages, fall back to normal page allocation if not possible (the default setting currently)
- off : Never try to request huge page allocation
- on : Force shared buffer pool allocations with huge pages
If we have finally found our settings to start with, we need to reserve the required pool of huge pages by the kernel. As user root and using the default huge page size, this can be done during runtime by the following command:
The memory settings then can be used to confirm what we have in mind:
Pools for specific huge pages must be reserved as a continuous memory area, so the kernel needs a memory area which is available for the reservation. If that’s not possible, far less memory might be available to the pool. In this case, the HugePages_Total number is lower than we requested. Experimenting with values that are too large can also drive a system into an out-of-memory situation, rendering it unavailable in the worst case. So I recommend carefully checking memory usage if the system hosts other applications or users.
Setting the number of pages like this, though, is not persistent across system reboots. To survive them, the easiest way is to put a configuration into /etc/sysctl.d/ with the right settings and reboot:
PostgreSQL is then configured to request huge pages. In this example, we adjust the postgresql.conf configuration file to force PostgreSQL to perform allocations accordingly:
PostgreSQL must be started (or restarted) and should allocate its shared buffer pool via the default huge page size. /proc/meminfo changes its figures. Now it looks like it does here in the following output:
Note the HugePages_Rsvd figure: It reflects the number of pages reserved due to the allocations requested. You can use the pmap tool with the PID of the postmaster process to get the allocated memory mapped buffer pool segment:
This output shows the address where the memory mapped buffer pool area is mapped, its size and permissions. The (deleted) indicates this memory mapped allocation is already unlinked, which is a common trick to make it disappear as soon as the process owning it exits.
Dynamically allocated shared memory and huge pages
In the section on how PostgreSQL uses huge pages, we owe an explanation to the exception mentioned there. In general, dynamic shared memory allocations are done on purpose, for example when parallel query workers process a parallelized query. These dynamically allocated memory areas aren’t done by mmap by default, instead PostgreSQL uses the posix API via shm_open() on Linux by default. In this case, allocations don’t use huge pages. However, when having parallel queries, memory pressure can arise there as well, especially when large amount of data needs to be processed.
PostgreSQL starting with version 14 provides the configuration parameter min_dynamic_shared_memory . When configured, the specified value defines the additional space in the shared memory created at server startup that should be used for parallel query workers. Since it is allocated along with the buffer pool, it also uses huge pages for allocation. We can extend the example above and re-calculate the setting with 2GB of additional shared memory for use for parallel queries:
Again, the PostgreSQL instance needs to be stopped to execute the command successfully. Compared to the previous example, the number for huge pages to be reserved has now increased by an additional 2GB.
What about PostgreSQL versions older than 15?
These versions are a little bit more difficult to handle. Particularly calculations on how many huge pages need to be reserved are much trickier. The best solution so far is just trying to start PostgreSQL manually with huge_pages=on and some debugging parameters. In this case PostgreSQL prints the following error message if it is not able to allocate the requested shared memory size:
The interesting thing is the HINT part of the message, telling us that PostgreSQL wants to allocate 21965570048 bytes of shared memory. This value can be used to calculate the number of pages ourselves. The problem with this number is that this is more than we requested when specifying shared buffers to 20GB. This is because PostgreSQL also needs additional shared memory for other internal processes, like wal_buffers or max_locks_per_transactions. Extension might require additional shared
memory, too. PostgreSQL internally uses a formula that corresponds approximately to the following
Standard PostgreSQL installations are always compiled with BLKSZ equal to 8192, specifying the requested huge page size gives then the following result:
Configuration with gigantic huge pages
Things get even more complicated if you want PostgreSQL to use a page size of 1GB. Since this is not the default on Linux, configuration requires more steps and more careful thought in such a case. This is left to the next blog post, where I’ll show you how to deal with such a requirement.
Find out more about allocating memory in Hans-Jürgen Schönig’s blog, fixing track activity query size in PostgreSQL conf.
In order to receive regular updates on important changes in PostgreSQL, subscribe to our newsletter, or follow us on Twitter, Facebook, or LinkedIn.
Bernd Helmle
Bernd Helmle works as a Senior Database Engineer at CYBERTEC. He’s been using and contributing to PostgreSQL since 2000, and has helped enterprises implement PostgreSQL since 2006. He develops and maintains Informix FDW, and has written several patches for PostgreSQL core. He holds a Diplom (FH) in Computer Science from the University of Aalen. In his spare time he is a movie maniac and a cat can opener.