Using Process Memory Matrix script for calculating Oracle process memory usage on Solaris

I just published a new script and article about calculating the real Oracle process memory usage on Solaris.

The problem with V$PROCESS* views (and the V$SESSTAT) is that they will tell you what Oracle thinks it’s using, not the real amount of memory used. There will be discrepancies due how memory is actually allocated in OS, libraries out of Oracle’s control, the static memory areas inside Oracle binary and of course bugs.

I was working on one of such problems and decided to make my life easier by writing the script. It’s not so much about calculating the exact figures (they will never be 100% correct), but more about presenting the memory usage data in a better and more convenient fashion.

The script is called procmm and stands for Process Memory Matrix as it shows the memory usage in a matrix grid.

Here’s an example output to show what I’m talking about:

oracle@solaris02:~/research/memory$ ./procmm.sh -t `pgrep -f ora_.*SOL102`

-- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )
-- All numbers are shown in kilobytes

Total PIDs 17, working: .................

PID            SEGMENT_TYPE      VIRTUAL          RSS         ANON       LOCKED    SWAP_RSVD
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                       lib       389844       388796        13180            0        17816
0                    oracle      1629064      1628908         3336            0        42012
0            ism_shmid=0x1d      6963336      6963336            0      6963336            0
0             hc_SOL102.dat           48           48            0            0            0
0                      anon        32936        15936        15452            0        32868
0                     stack         1660         1628         1592            0         1660
0                      heap        37004        18016        16844            0        37004
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                 TOTAL(kB)      9053892      9016668        50404      6963336       131360

And here’s the full article and in there a link to the script:

Comments are welcome here, as I haven’t set up commenting on my other site yet…

This entry was posted in Uncategorized. Bookmark the permalink.

36 Responses to Using Process Memory Matrix script for calculating Oracle process memory usage on Solaris

  1. Tanel Poder says:

    The link was broken, fixed now.

  2. Paul Janda says:

    I’ve tried to write that script several times (we’ve been on Solaris for 6 years). You are a much better programmer than me. Do you have no children and/or are you a genious?

  3. Tanel Poder says:

    No and no ;)

    I don’t have kids and not a genious either, but I am interested in this stuff a lot so when I’m not working, my vacation consists of writing which either are useful or just cool ;-)

  4. Tanel Poder says:

    NB! There may be some problems with awk apparently, I’m looking into that!

  5. Tanel Poder says:

    Ok, the bug should be fixed now!

  6. Hi Tanel,

    Any chance the script could include the physical memory utilization as seen from mdb? I know mdb needs to run as root, but an option would be good to correlate between process and kernel.

    ——————
    # mdb -k
    Loading modules: [ unix krtld genunix random sgsbbc ip usba ipc md ptm ]
    > ::memstat
    Page Summary Pages MB %Tot
    ———— —————- —————- —-
    Kernel 665350 5198 8%
    Anon 1421201 11103 18%
    Exec and libs 13614 106 0%
    Page cache 131717 1029 2%
    Free (cachelist) 280369 2190 3%
    Free (freelist) 5564191 43470 69%

    Total 8076442 63097

    ————————–

    Thanks
    Krishna

  7. Yong Huang says:

    Krishna, Tanel’s script is about a process’s memory, not the system’s, which is what your mdb command returns. So it makes no sense to do the correlation.

  8. JC Dauchy says:

    Hello Tanel,

    Nice useful tool, will you port it to AIX for procmap ?
    Thanks

  9. Tanel Poder says:

    Hi Krishna,

    I think DBAs (who have root access) can easily run this themselves (I don’t want to make my script to su root or anything).

    Anyone who want to run this automatically, can use this, if they’re allowed to run sudo mdb..

    echo ::memstat | sudo mdb -k

    memstat is CPU intensive though as it walks through lots of pagetable entries, so kstat -n system_pages would be a better start.

  10. Tanel Poder says:

    @JC Dauchy
    JC Dauchy,

    I don’t have access to an AIX box right now, but I checked AIX procmap manpage:

    http://www.ibm.com/developerworks/systems/articles/AIX5.2PerfTools.html

    The procmap example output doesn’t show the RSS and ANON figures like Solaris pmap (and Linux /proc/ /smaps do). So, it looks like in AIX 5.2 at least the procmap only reports the virtual memory segment size (reserved memory size), not the actual resident set size and the allocation size.
    I didn’t see any command line options for procmap either which would show this.

    If you have access to AIX 5.3 or AIX 6.x right now, you could check whether they have added some new options to procmap?

  11. Tanel Poder says:

    @Yong Huang
    Hi Yong,

    Well it would make sense if you run procmm with -t option across many processes, like this:

    procmm -t `pgrep oracle`

    With -t option procmm shows the total sum figures for all processes in the process list passed in to it.

  12. JC Dauchy says:

    I have access to both AIX 6.1 and 5.3, I’ll check all options to see what is availble. THanks for the reply.

  13. an oracle dba says:

    Hi JC Dauchy / Tanel

    AIX, ps would present the memory usage,
    please refer the oracle script
    $EMDROOT/sysman/admin/scripts/topProcs.pl

    Best Regards.

  14. Tanel Poder says:

    @an oracle dba
    Hi,

    Ps wouldn’t probably work on AIX either.

    The ps utilities have a shortcoming, they usually report the SGA as part of RSS for every process. That’s why tools like pmap are needed to peek inside the process address space for breaking the memory usage down by segment type.

  15. Taral says:

    Hello Tanel,

    Still i think there is a bug for awk. I tried to run this

    $ ./procmm.sh 1336

    — procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )
    — All numbers are shown in kilobytes

    awk: syntax error near line 5
    awk: illegal statement near line 5

  16. Tanel Poder says:

    @Taral
    Hi Taral, which Solaris version are you running on? (uname -a)

  17. Taral says:

    @Tanel Poder
    Hello Tanel,

    I am using solaris 10 on sun virtualbox
    $ uname -a
    SunOS unknown 5.10 Generic_142901-04 i86pc i386 i86pc

  18. laurent DEMARET says:

    Hi Tanel,
    I’m having the same error Taral is having on Solaris 10 running on a M9000.
    Do we need to use GNU awk?

    Regards,

  19. JC Dauchy says:

    Hello Tanel,

    Sorry for the delay, i check the options on procmm, but no option is available to get
    the RSS and ANON info.

    But i guess what i can understand (i am not an expert at all, just rrying to understand for the moment), the scripts “memdetails.sh” from IBM available in the following package :

    ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/

    Gives the following results : http://blog.krystalia.net/post/2008/01/27/aix-description-dtaille-de-l-utilisation-mmoire-avec-memdetailssh.html

    Using svmon options, not sure you’ll have time to check out the script to grab the important information.

    Thanks anyway for providing the script for linux and Solaris Users.
    JC

  20. Tanel Poder says:

    @laurent DEMARET
    Laurent you’re right, I had replaced awk with GNU awk in my test box for working around another problem in past. That’s why in my box everything worked. I don’t have time to test right now but if you replace awk with “nawk”, does the script still error out?

  21. mdinh says:

    FYI:

    > uname -a
    SunOS proddb01 5.10 Generic_137137-09 sun4u sparc SUNW,SPARC-Enterprise

    Replacing awk with nawk works.

  22. Jule says:

    I did run this script on Solaris 5.9(5.9 Generic_122300-13 sun4u sparc SUNW) with ‘nawk’ because the script was throwing “awk: syntax error near line 5
    awk: illegal statement near line 5″ otherwise. For some reason I’m not getting ism_shmid(or dism for he same matter) segment in the script output. I did test with a process id for smon and I also tried for the whole instance , like -t `pgrep -f ora_.*DBTEST1`

  23. carey holman says:

    @Taral I had the same problem. Try nawk instead of awk.

  24. Tanel Poder says:

    I’ve fixed the AWK problem in the script (by replacing awk with nawk), so it should work now…

  25. maclean says:

    till aix 6.1 , procmap command do not show anon segment and stack detail,
    Maybe workaround way is utilizing “svmon -P” , but it reports in another style format.

  26. Tanel Poder says:

    @maclean
    Thanks, do you have an example output of svmon -P? (believe it or not, but I don’t have an AIX box in my garage ;-)

    By the way, I’ve updated the script and fixed the earlier problems with it (which caused AWK to fail)

  27. Max says:

    Can anyone please explain the following output from “pmap” and “procmm.sh”, where the number for “Anon” from “pmap” is more likely “SWAP_RSVD” from “procmm.sh”?

    # pmap -x 20211
    20211: oraclelpiweb (LOCAL=NO)
    Address Kbytes RSS Anon Locked Mode Mapped File
    0000000100000000 103696 103392 – – r-x– oracle
    0000000106642000 832 560 208 – rwx– oracle

    FFFFFFFF7FFE8000 96 96 96 – rw— [ stack ]
    —————- ———- ———- ———- ———-
    total Kb 847512 837128 2736 716816

    # ./procmm.sh 20211

    — procmm.sh: Process Memory Matrix v1.02 by Tanel Poder ( http://tech.e2sn.com )
    — All numbers are shown in kilobytes

    PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD
    —— ——————– ———— ———— ———— ———— ————
    20211 stack 96 96 96 0 96
    20211 anon 1048 960 816 0 1040
    20211 lib 23064 14392 944 0 1512
    —— ——————– ———— ———— ———— ———— ————
    20211 TOTAL(kB) 24208 15448 1856 0 2648

  28. Tanel Poder says:

    @Max
    Can you send me the full output of both commands?

    As which user are you running this command? (is the oracle user or a member of dba group)?

    procmm should report more output than the current (for example the shared mem segments etc).

  29. Max says:

    Hi Tanel,

    The original process has logged out. Here is another example when both commands ran immediately one after another:

    >pmap -x 18918
    18918: oraclelpiweb (LOCAL=NO)
    Address Kbytes RSS Anon Locked Mode Mapped File
    0000000100000000 103696 103624 – – r-x– oracle
    0000000106642000 832 656 208 – rwx– oracle
    0000000106712000 1960 912 888 – rwx– [ heap ]
    0000000380000000 188416 188416 – 188416 rwxsR [ ism shmid=0x6a00002c ]
    00000003C0000000 172032 172032 – 172032 rwxsR [ ism shmid=0x50000032 ]
    0000000400000000 180224 180224 – 180224 rwxsR [ ism shmid=0x6c000038 ]
    0000000440000000 176128 176128 – 176128 rwxsR [ ism shmid=0xf000042 ]
    0000000480000000 16 16 – 16 rwxsR [ ism shmid=0x7b000043 ]
    FFFFFFFF7BA60000 64 64 64 – rw— [ anon ]
    FFFFFFFF7BAC0000 64 40 40 – rw— [ anon ]
    FFFFFFFF7BAD0000 64 64 64 – rw— [ anon ]
    FFFFFFFF7BAE0000 64 40 40 – rw— [ anon ]
    FFFFFFFF7BB00000 16 16 – – r-x– liblgrp.so.1
    FFFFFFFF7BC04000 8 8 8 – rwx– liblgrp.so.1
    FFFFFFFF7BD00000 56 56 – – r-x– libmd.so.1
    FFFFFFFF7BE0E000 8 8 8 – rwx– libmd.so.1
    FFFFFFFF7BF00000 640 472 – – r-x– libm.so.2
    FFFFFFFF7C09E000 40 32 24 – rwx– libm.so.2
    FFFFFFFF7C100000 32 32 – – r-x– librt.so.1
    FFFFFFFF7C208000 8 8 8 – rwx– librt.so.1
    FFFFFFFF7C300000 32 32 – – r-x– libaio.so.1
    FFFFFFFF7C408000 8 8 8 – rwx– libaio.so.1
    FFFFFFFF7C500000 1264 896 – – r-x– libc.so.1
    FFFFFFFF7C700000 64 32 32 – rwx– [ anon ]
    FFFFFFFF7C73C000 72 72 72 – rwx– libc.so.1
    FFFFFFFF7C74E000 8 – – – rwx– libc.so.1
    FFFFFFFF7C800000 8 8 – – r-x– libdl.so.1
    FFFFFFFF7C902000 8 8 8 – rwx– libdl.so.1
    FFFFFFFF7CA00000 32 32 – – r-x– libgen.so.1
    FFFFFFFF7CB08000 8 8 8 – rwx– libgen.so.1
    FFFFFFFF7CC00000 56 56 – – r-x– libsocket.so.1
    FFFFFFFF7CD00000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7CD0E000 16 16 16 – rwx– libsocket.so.1
    FFFFFFFF7CE00000 688 448 – – r-x– libnsl.so.1
    FFFFFFFF7CF00000 64 16 16 – rw— [ anon ]
    FFFFFFFF7CF10000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CF20000 64 64 56 – rw— [ anon ]
    FFFFFFFF7CF30000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CF40000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CF50000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CF60000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CF70000 64 64 64 – rw— [ anon ]
    FFFFFFFF7CFAC000 64 64 64 – rwx– libnsl.so.1
    FFFFFFFF7CFBC000 32 32 24 – rwx– libnsl.so.1
    FFFFFFFF7D000000 8 8 – – r-x– libkstat.so.1
    FFFFFFFF7D102000 8 8 8 – rwx– libkstat.so.1
    FFFFFFFF7D200000 2184 408 – – r-x– libnnz10.so
    FFFFFFFF7D500000 64 64 64 – rwx– [ anon ]
    FFFFFFFF7D520000 240 216 200 – rwx– libnnz10.so
    FFFFFFFF7D55C000 8 – – – rwx– libnnz10.so
    FFFFFFFF7D600000 72 72 – – r-x– libdbcfg10.so
    FFFFFFFF7D700000 24 16 16 – rwx– [ anon ]
    FFFFFFFF7D710000 8 8 8 – rwx– libdbcfg10.so
    FFFFFFFF7D770000 64 64 56 – rw— [ anon ]
    FFFFFFFF7D780000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D790000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D7A0000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D7B0000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D7C0000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D7D0000 64 64 64 – rw— [ anon ]
    FFFFFFFF7D7E0000 64 32 32 – rw— [ anon ]
    FFFFFFFF7D800000 9272 8208 – – r-x– libjox10.so
    FFFFFFFF7E200000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7E20C000 560 560 448 – rwx– libjox10.so
    FFFFFFFF7E300000 1072 112 – – r-x– libclsra10.so
    FFFFFFFF7E500000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7E50A000 56 40 40 – rwx– libclsra10.so
    FFFFFFFF7E518000 8 – – – rwx– libclsra10.so
    FFFFFFFF7E600000 1072 192 – – r-x– libocrutl10.so
    FFFFFFFF7E800000 8 8 – – r-x– libc_psr.so.1
    FFFFFFFF7E80A000 56 48 48 – rwx– libocrutl10.so
    FFFFFFFF7E818000 8 – – – rwx– libocrutl10.so
    FFFFFFFF7E900000 1376 192 – – r-x– libocrb10.so
    FFFFFFFF7EB00000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7EB56000 64 56 56 – rwx– libocrb10.so
    FFFFFFFF7EB66000 8 – – – rwx– libocrb10.so
    FFFFFFFF7EC00000 1536 256 – – r-x– libocr10.so
    FFFFFFFF7EE00000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7EE7E000 64 56 56 – rwx– libocr10.so
    FFFFFFFF7EE8E000 8 – – – rwx– libocr10.so
    FFFFFFFF7EF00000 8 8 – – r-x– libskgxn2.so
    FFFFFFFF7F000000 8 8 8 – rwx– libskgxn2.so
    FFFFFFFF7F100000 1752 792 – – r-x– libhasgen10.so
    FFFFFFFF7F300000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7F3B4000 72 64 64 – rwx– libhasgen10.so
    FFFFFFFF7F3C6000 16 8 8 – rwx– libhasgen10.so
    FFFFFFFF7F400000 144 96 – – r-x– libskgxp10.so
    FFFFFFFF7F500000 8 8 8 – rwx– [ anon ]
    FFFFFFFF7F522000 16 16 16 – rwx– libskgxp10.so
    FFFFFFFF7F600000 216 216 – – r-x– ld.so.1
    FFFFFFFF7F700000 8 8 – – r-x– libsched.so.1
    FFFFFFFF7F736000 16 16 16 – rwx– ld.so.1
    FFFFFFFF7F73A000 8 8 8 – rwx– ld.so.1
    FFFFFFFF7F7B8000 8 8 – – rwxs- [ anon ]
    FFFFFFFF7FFDE000 136 136 136 – rw— [ stack ]
    —————- ———- ———- ———- ———-
    total Kb 848000 837472 3768 716816
    >./procmm.sh 18918

    — procmm.sh: Process Memory Matrix v1.02 by Tanel Poder ( http://tech.e2sn.com )
    — All numbers are shown in kilobytes

    PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD
    —— ——————– ———— ———— ———— ———— ————
    18918 stack 136 136 136 0 136
    18918 anon 1496 1328 1304 0 1488
    18918 lib 23064 14000 1232 0 1512
    —— ——————– ———— ———— ———— ———— ————
    18918 TOTAL(kB) 24696 15464 2672 0 3136

    Thanks,
    Max

  30. Max says:

    Also when man pmap, it says:

    Anonymous Memory (Anon)

    The number of pages, counted by using the system page
    size, of anonymous memory associated with the specified
    mapping. Anonymous memory shared with other address
    spaces is not included, unless the -a option is speci-
    fied.

    Anonymous memory is reported for the process heap,
    stack, for ‘copy on write’ pages with mappings mapped
    with MAP_PRIVATE (see mmap(2)).

    So “Anon” in pmap seems showing the number of pages, not in kb.

  31. Roberts says:

    Hi,
    I tried to use script on Solaris, but did not succeded with -t option. As it does not show summary for me. Without -t option script seems to be working.
    Roberts

    uname -a
    SunOS 5.10 Generic_142900-08 sun4v sparc SUNW,Sun-Blade-T6320

    bash-3.00$ sudo -u oraclet1 ./procmm.sh -t `pgrep -f ora`

    — procmm.sh: Process Memory Matrix v1.03 by Tanel Poder ( http://tech.e2sn.com )
    — All numbers are shown in kilobytes

    Total PIDs 19, working: ……………….

    PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD
    —— ——————– ———— ———— ———— ———— ————
    0 stack 1984 1984 1984 0 1984
    0 tnslsnr 1688 1296 0 0 96
    0 anon 256 224 96 0 248
    0 lib 15544 12800 40 0 1096
    0 heap 1584 1520 600 0 1584
    —— ——————– ———— ———— ———— ———— ————
    0 TOTAL(kB) 21056 17824 2720 0 5008

    — Note that in Total (-t) calculation mode it makes sense to look into ANON and SWAP_RSVD
    — totals only as other numbers may be heavily “doublecounted” due to overlaps of shared mappings

  32. MikeD says:

    @Max, the pmap man page seems to contradict itself. In one of the examples it states that for an Anon value of 56, “each additional bourne shell uses approximately 56Kbytes of additional physical memory.” Also the totals row is labelled total Kb, so that suggests the Anon is really in Kb.

    @Tanel, good work on this. My question is this (hopefully not a stupid one). Are the virtual addresses unique to each process or do they represent all of the system’s virtual memory? When using the -t option with multiple pids, I have seen that the segment “type” can vary for the same address across different pids (one may be anon, one may be a lib, etc) and they can also have different allocations for the same segment. It looks like the join of the -x and the -S files will wipe out duplicate rows with the same virtual address. I’m wondering if that is an accurate way to deal with the duplicates or not.

  33. Ganapathi says:

    Tanel,
    We have a requirement of how much memory utilized by oracle in Linux,AIX,HP-UX. I have tried with pmap command in linux and HP-UX.
    Can you guide us ?

  34. Andres Ramirez says:

    Hi, you can compile this program in C to find oracle process memory usage

    http://ejemplosprogramacionc.blogspot.com/2013/06/memoria-de-procesos-oracle-sobre.html

  35. Mike Buel says:

    Hello Tanel,
    Question on your results:
    RSS = 9016668 , since it is in K,
    is it 9,016 GB RAM?

    • Tanel Poder says:

      It’s 50404 KB (the ANON column sum). Read the tech.e2sn article I pointed to, I explain how to read it (many other columns are “double”-counted due to how shared memory works).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>