10

We have a bunch of general purpose Centos 7 LAMP stack servers that run apache with underlying MariaDB databases.

From the RedHat Performance Tuning docs for RHEL 6:

Trasparent Hugepages (THP) hides much of the complexity in using huge pages from system administrators and developers. As the goal of THP is improving performance, its developers (both from the community and Red Hat) have tested and optimized THP across a wide range of systems, configurations, applications, and workloads. This allows the default settings of THP to improve the performance of most system configurations. However, THP is not recommended for database workloads.

Which sounds great, up until the database section. In fact, some of the MariaDB options such as the TokuDB engine actually require THP to be disabled.

A lot of the guides I found for disabling THP instruct changing the default configuration from transparent_hugepages= always to never.

However, from the RedHat Performance Tuning docs for RHEL 7:

To prevent applications from allocating more memory resources than necessary, you can disable huge pages system-wide and only enable them inside MADV_HUGEPAGE madvise regions

If I have read this right, transparent_hugepages= madvise seems like a much better option, allowing processes that benefit from THP to use them and those that don't to not.

I did find one article suggesting this approach.

Why is this not the default option?

Are there any situations where using transparent_hugepages= never is wiser than madvise?

Arth
  • 353
  • 5
  • 14

1 Answers1

9

It's the difference between opt-in (madvise) and opt-out (always). Applications that are sensitive to THP can opt-out even under transparent_hugepages=always. For example, jemalloc-using applications are notably sensitive to it (redis, go rt, rust rt) and jemalloc offers the option at compile time as part of the allocator configuration to use MADV_NOHUGEPAGE. Starting with kernel 3.15 (RHEL7 ships with 3.10), they can opt out on a per-process basis using prctl(PR_SET_THP_DISABLE).

The default always should usually be fine because data locality is a very common property for applications making large allocations. Databases are the notable exception where they suffer for accessing small chunks across a high number of large allocations (and they can disable THP).

The kernel documentation for transparent hugepages suggests that transparent_hugepage=never is more of a debugging option than a recommended solution. It is only preferable to transparent_hugepage=madvise when the software is buggy; ie. when the developer is using MADV_HUGEPAGE incorrectly.

Andrew Domaszek
  • 5,103
  • 1
  • 14
  • 26
  • Thank you for your input, I think I'll leave our servers on `always` for now and benchmark going to `madvise` in the future if necessary. I did find one [forum post](https://groups.google.com/forum/#!topic/tokudb-dev/_1YNBMlHftU) suggesting that `madvise` is only available/sufficient for TokuDB as of RHEL 7, which explains some of the older advice out there. – Arth Feb 08 '18 at 11:23
  • Mongodb does not consider their code buggy ... and their arguments for disabling THP are compelling. I wonder how old that kernel doc is and whether is has had the benefit of viewpoints arrived at later in time. – Rondo Jul 18 '18 at 00:38
  • It makes sense to turn off transparent hugepages for mongodb. Mongodb probably works fine with THP=madvise set. I updated the doc reference to the latest admin guide (2018); that particular part hasn't changed. – Andrew Domaszek Jul 18 '18 at 15:45