​

环境

  • Red Hat Enterprise Linux (RHEL) 6
  • transparent hugepages (THP)
  • tuned
  • ktune

问题

  • Unable to disable transparent hugepages (THP) even after appending "transparent_hugepage=never" to kernel command line in /boot/grub/grub.conf file.
    Eg:

​​Raw​​

# grep -i never /boot/grub/grub.conf 
    kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=a216d1e5-884f-4e5c-859a-6e2e2530d486 rhgb quiet transparent_hugepage=never

# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] never

# grep -i AnonHugePages /proc/meminfo 
AnonHugePages:    206848 kB

决议

Create a customized tuned profile with disabled THP

With this resolution we will create a customized version of the currently running profile. The customized version will disable THP.

Find out which profile is active, create a copy. In the following example we currently use the ​​throughput-performance​​ profile:

​​Raw​​

$ tuned-adm  active
Current active profile: throughput-performance
Service tuned: enabled, running
Service ktune: enabled, running
$ cd /etc/tune-profiles/
$ cp -r throughput-performance throughput-performance-no-thp

We will now disable THP in the new profile and activate the new profile:

​​Raw​​

$ sed -ie 's,set_transparent_hugepages always,set_transparent_hugepages never,' \
      /etc/tune-profiles/throughput-performance-no-thp/ktune.sh
$ grep set_transparent_hugepages /etc/tune-profiles/throughput-performance-no-thp/ktune.sh
        set_transparent_hugepages never
$ tuned-adm profile throughput-performance-no-thp
$ cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always [never]

We will also have to add the ​​transparent_hugepage=never​​ parameter to the kernel line ​​/etc/grub.conf​​ to ensure that when the system is booted applications already do not request THP, for example:

​​Raw​​

kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=a216d1e5-884f-4e5c-859a-6e2e2530d486 rhgb quiet transparent_hugepage=never

The system should then be rebooted.

Alternative Resolution: Disable ​​tuned​​ and ​​ktune​​ services.

This resolution will disable the tuned/ktune services.

Eg:

​​Raw​​

# service tuned stop
# chkconfig tuned off
# service ktune stop
# chkconfig ktune off

OR

​​Raw​​

# tuned-adm off

Note: The ​​tuned-adm​​ command will revert all your settings to what they were before tuned started and disable the tuning services from running at boot.

  • Append "transparent_hugepage=never" to kernel command line in /boot/grub/grub.conf file.
    Eg:

​​Raw​​

kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=a216d1e5-884f-4e5c-859a-6e2e2530d486 rhgb quiet transparent_hugepage=never

  • Reboot the server for changes to take effect.
    Eg:

​​Raw​​

# reboot

根源

  • The ​​ktune​​ service enables ​​transparent hugepages​​ (THP) by default for all profiles.

​​Raw​​

# cat /etc/tune-profiles/enterprise-storage/ktune.sh 
#!/bin/sh

. /etc/tune-profiles/functions

start() {
    set_cpu_governor performance
    set_transparent_hugepages always  <<<----
    disable_disk_barriers
    multiply_disk_readahead 4

    return 0
}

stop() {
    restore_cpu_governor
    restore_transparent_hugepages
    enable_disk_barriers
    restore_disk_readahead

    return 0
}

process $@

  • Any time a tuned profile is enabled, whether at bootup via an init.d service, or manually at the command line, transparent hugepages (THP) gets re-enabled.

诊断步骤

  • Verify ktune and tuned services:

​​Raw​​

# chkconfig --list |egrep -i "ktune|tuned"
ktune           0:off   1:off   2:off   3:on    4:on    5:on    6:off
tuned           0:off   1:off   2:on    3:on    4:on    5:on    6:off

  • When ​​transparent_hugepage=never​​ is appended to ​​/boot/grub/grub.conf​​ and the system is rebooted, the ​​/sys/kernel/mm/transparent_hugepage/defrag​​ option may still be enabled (ie: set to ​​always​​). This can safely be ignored as THP is disabled, and THP defrag will not come into effect:

​​Raw​​

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always [never]

cat /sys/kernel/mm/transparent_hugepage/defrag
[always] never