SGA bigger than than the amount of HugePages configured (Linux – 11.2.0.3)

I just learned something new yesterday when demoing large page use on Linux during my AOT seminar.

I had 512 x 2MB hugepages configured in Linux ( 1024 MB ). So I set the USE_LARGE_PAGES = TRUE (it actually is the default anyway in 11.2.0.2+). This allows the use of large pages (it doesn’t force, the ONLY option would force the use of hugepages, otherwise the instance wouldn’t start up). Anyway, the previous behavior with hugepages was, that if Oracle was not able to allocate the entire SGA from the hugepages area, it would silently allocate the entire SGA from small pages. It was all or nothing. But to my surprise, when I set my SGA_MAX_SIZE bigger than the amount of allocated hugepages in my testing, the instance started up and the hugepages got allocated too!

It’s just that the remaining part was allocated as small pages, as mentioned in the alert log entry below (and in the latest documentation too – see the link above):

Thu Oct 24 20:58:47 2013
ALTER SYSTEM SET sga_max_size='1200M' SCOPE=SPFILE;
Thu Oct 24 20:58:54 2013
Shutting down instance (abort)
License high water mark = 19
USER (ospid: 18166): terminating the instance
Instance terminated by USER, pid = 18166
Thu Oct 24 20:58:55 2013
Instance shutdown complete
Thu Oct 24 20:59:52 2013
Adjusting the default value of parameter parallel_max_servers
from 160 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 1024 MB (85%)

Large Pages used by this instance: 512 (1024 MB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 16 MB)
Large Pages configured system wide = 512 (1024 MB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total Shared Global Region size is 1202 MB. For optimal performance,
  prior to the next instance restart increase the number
  of unused Large Pages by atleast 89 2048 KB Large Pages (178 MB)
  system wide to get 100% of the Shared
  Global Region allocated with Large pages
***********************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0

The ipcs -m command confirmed this (multiple separate shared memory segments had been created).

Note that despite what documentation says, there’s a 4th option, called AUTO for the USE_LARGE_PAGES parameter too (in 11.2.0.3+ I think), which can now ask the OS to increase the number of hugepages when instance starts up – but I would always try to pre-allocate the right number of hugepages from start (ideally right after the OS reboot – via sysctl.conf), to reduce any overhead potential kernel CPU usage spikes due to search and defragmentation effort for “building” large consecutive pages.

Marting Bach has written about the AUTO option here.

Note that this year’s only Advanced Oracle Troubleshooting class takes place in the end of April/May 2014, so sign up now if you plan to attend this year!

This entry was posted in Linux, Oracle. Bookmark the permalink.

12 Responses to SGA bigger than than the amount of HugePages configured (Linux – 11.2.0.3)

  1. Tanel,
    I was looking for an easy way to query the database to find out if the current instance is in hugepages or not. I run 79 to 80 instances on each of my Exadata nodes. Using ipcs is not practical with that many instances, so I cooked up the following sql block. I’d like an opinion on it’s worthiness.

    declare
    v_alert_time timestamp;
    v_inst_start_time timestamp;
    begin
    select originating_timestamp
    into v_alert_time
    from
    (select originating_timestamp
    from x$dbgalertext
    where message_text like ‘%Huge Pages allocation failed%’
    order by originating_timestamp desc
    )
    where rownum=1;
    select to_timestamp(startup_time) into v_inst_start_time from v$instance;
    if v_alert_time >= v_inst_start_time then
    raise_application_error (-20001, ‘Instance already in hugepages.’) ;
    else
    raise_application_error (-20002, ‘Instance NOT in hugepages.’);
    end if;
    end;
    /

  2. Hi Tanel,
    nice one :-))

    I also blogged about the different behaviors in section “Myth 2 – Huge pages are difficult to manage in highly consolidated and critical environments” (and other huge pages stuff) here: http://tinyurl.com/p75moxh

    In my experience (especially in highly consolidated and virtualized database environments) huge pages can make a performance difference by logical I/O throughput for example. TLB implementation for VMware is described here (page 3) for example: http://www.vmware.com/files/pdf/techpaper/vsp_41_perf_memory_mgmt.pdf

    Regards
    Stefan

  3. Yep…that’s a blast from the past indeed:

    “Insufficient hugepages is an even more difficult situation when booting with _enable_NUMA_support=TRUE as partial hugepages backing is possible.”
    http://kevinclosson.wordpress.com/category/use_large_pages/

  4. Ashish Shanbhag says:

    $ORACLE_HOME/bin/sysresv should also get you the information. Loop through instances with -l flag for each instance. One behaviour we have seen after 11.2.0.3 is Oracle by default is allocating 3 segments per instance instead of a single. Further research on metalink revealed Doc ID 1399908.1 which confirms it. You can match sysresv with ipcs output to get your totals too.

  5. Jared says:

    UUOC alarm just detected a useless use of cat
    Just use ‘grep Huge /proc/meminfo’

  6. Tanel was this test done on RHEL6/OEL6?
    if yes oracle could use Transparent hugepages which is not always good.
    ALERT: Disable Transparent HugePages on SLES11, RHEL6, OL6 and UEK2 Kernels (Doc ID 1557478.1)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>