Using Process Memory Matrix script for calculating Oracle process memory usage on Solaris
I just published a new script and article about calculating the real Oracle process memory usage on Solaris.
The problem with V$PROCESS* views (and the V$SESSTAT) is that they will tell you what Oracle thinks it’s using, not the real amount of memory used. There will be discrepancies due how memory is actually allocated in OS, libraries out of Oracle’s control, the static memory areas inside Oracle binary and of course bugs.
I was working on one of such problems and decided to make my life easier by writing the script. It’s not so much about calculating the exact figures (they will never be 100% correct), but more about presenting the memory usage data in a better and more convenient fashion.
The script is called procmm and stands for Process Memory Matrix as it shows the memory usage in a matrix grid.
Here’s an example output to show what I’m talking about:
oracle@solaris02:~/research/memory$ ./procmm.sh -t `pgrep -f ora_.*SOL102` -- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com ) -- All numbers are shown in kilobytes Total PIDs 17, working: ................. PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD ------ -------------------- ------------ ------------ ------------ ------------ ------------ 0 lib 389844 388796 13180 0 17816 0 oracle 1629064 1628908 3336 0 42012 0 ism_shmid=0x1d 6963336 6963336 0 6963336 0 0 hc_SOL102.dat 48 48 0 0 0 0 anon 32936 15936 15452 0 32868 0 stack 1660 1628 1592 0 1660 0 heap 37004 18016 16844 0 37004 ------ -------------------- ------------ ------------ ------------ ------------ ------------ 0 TOTAL(kB) 9053892 9016668 50404 6963336 131360
And here’s the full article and in there a link to the script:
Comments are welcome here, as I haven’t set up commenting on my other site yet…
Trackbacks
Comments
The link was broken, fixed now.
I’ve tried to write that script several times (we’ve been on Solaris for 6 years). You are a much better programmer than me. Do you have no children and/or are you a genious?
No and no ;)
I don’t have kids and not a genious either, but I am interested in this stuff a lot so when I’m not working, my vacation consists of writing which either are useful or just cool ;-)
NB! There may be some problems with awk apparently, I’m looking into that!
Ok, the bug should be fixed now!
Hi Tanel,
Any chance the script could include the physical memory utilization as seen from mdb? I know mdb needs to run as root, but an option would be good to correlate between process and kernel.
——————
# mdb -k
Loading modules: [ unix krtld genunix random sgsbbc ip usba ipc md ptm ]
> ::memstat
Page Summary Pages MB %Tot
———— —————- —————- —-
Kernel 665350 5198 8%
Anon 1421201 11103 18%
Exec and libs 13614 106 0%
Page cache 131717 1029 2%
Free (cachelist) 280369 2190 3%
Free (freelist) 5564191 43470 69%
Total 8076442 63097
————————–
Thanks
Krishna
Krishna, Tanel’s script is about a process’s memory, not the system’s, which is what your mdb command returns. So it makes no sense to do the correlation.
Hello Tanel,
Nice useful tool, will you port it to AIX for procmap ?
Thanks
Hi Krishna,
I think DBAs (who have root access) can easily run this themselves (I don’t want to make my script to su root or anything).
Anyone who want to run this automatically, can use this, if they’re allowed to run sudo mdb..
echo ::memstat | sudo mdb -k
memstat is CPU intensive though as it walks through lots of pagetable entries, so kstat -n system_pages would be a better start.
@JC Dauchy
JC Dauchy,
I don’t have access to an AIX box right now, but I checked AIX procmap manpage:
http://www.ibm.com/developerworks/systems/articles/AIX5.2PerfTools.html
The procmap example output doesn’t show the RSS and ANON figures like Solaris pmap (and Linux /proc//smaps do). So, it looks like in AIX 5.2 at least the procmap only reports the virtual memory segment size (reserved memory size), not the actual resident set size and the allocation size.
I didn’t see any command line options for procmap either which would show this.
If you have access to AIX 5.3 or AIX 6.x right now, you could check whether they have added some new options to procmap?
@Yong Huang
Hi Yong,
Well it would make sense if you run procmm with -t option across many processes, like this:
procmm -t `pgrep oracle`
With -t option procmm shows the total sum figures for all processes in the process list passed in to it.
I have access to both AIX 6.1 and 5.3, I’ll check all options to see what is availble. THanks for the reply.
Hi JC Dauchy / Tanel
AIX, ps would present the memory usage,
please refer the oracle script
$EMDROOT/sysman/admin/scripts/topProcs.pl
Best Regards.
@an oracle dba
Hi,
Ps wouldn’t probably work on AIX either.
The ps utilities have a shortcoming, they usually report the SGA as part of RSS for every process. That’s why tools like pmap are needed to peek inside the process address space for breaking the memory usage down by segment type.
Hello Tanel,
Still i think there is a bug for awk. I tried to run this
$ ./procmm.sh 1336
– procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )
– All numbers are shown in kilobytes
awk: syntax error near line 5
awk: illegal statement near line 5
@Taral
Hi Taral, which Solaris version are you running on? (uname -a)
@Tanel Poder
Hello Tanel,
I am using solaris 10 on sun virtualbox
$ uname -a
SunOS unknown 5.10 Generic_142901-04 i86pc i386 i86pc
Hi Tanel,
I’m having the same error Taral is having on Solaris 10 running on a M9000.
Do we need to use GNU awk?
Regards,
Hello Tanel,
Sorry for the delay, i check the options on procmm, but no option is available to get
the RSS and ANON info.
But i guess what i can understand (i am not an expert at all, just rrying to understand for the moment), the scripts “memdetails.sh” from IBM available in the following package :
ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/
Gives the following results : http://blog.krystalia.net/post/2008/01/27/aix-description-dtaille-de-l-utilisation-mmoire-avec-memdetailssh.html
Using svmon options, not sure you’ll have time to check out the script to grab the important information.
Thanks anyway for providing the script for linux and Solaris Users.
JC
@laurent DEMARET
Laurent you’re right, I had replaced awk with GNU awk in my test box for working around another problem in past. That’s why in my box everything worked. I don’t have time to test right now but if you replace awk with “nawk”, does the script still error out?
FYI:
> uname -a
SunOS proddb01 5.10 Generic_137137-09 sun4u sparc SUNW,SPARC-Enterprise
Replacing awk with nawk works.
I did run this script on Solaris 5.9(5.9 Generic_122300-13 sun4u sparc SUNW) with ‘nawk’ because the script was throwing “awk: syntax error near line 5
awk: illegal statement near line 5″ otherwise. For some reason I’m not getting ism_shmid(or dism for he same matter) segment in the script output. I did test with a process id for smon and I also tried for the whole instance , like -t `pgrep -f ora_.*DBTEST1`
@Taral I had the same problem. Try nawk instead of awk.
I’ve fixed the AWK problem in the script (by replacing awk with nawk), so it should work now…
till aix 6.1 , procmap command do not show anon segment and stack detail,
Maybe workaround way is utilizing “svmon -P” , but it reports in another style format.
@maclean
Thanks, do you have an example output of svmon -P? (believe it or not, but I don’t have an AIX box in my garage ;-)
By the way, I’ve updated the script and fixed the earlier problems with it (which caused AWK to fail)
Can anyone please explain the following output from “pmap” and “procmm.sh”, where the number for “Anon” from “pmap” is more likely “SWAP_RSVD” from “procmm.sh”?
# pmap -x 20211
20211: oraclelpiweb (LOCAL=NO)
Address Kbytes RSS Anon Locked Mode Mapped File
0000000100000000 103696 103392 – – r-x– oracle
0000000106642000 832 560 208 – rwx– oracle
…
FFFFFFFF7FFE8000 96 96 96 – rw— [ stack ]
—————- ———- ———- ———- ———-
total Kb 847512 837128 2736 716816
# ./procmm.sh 20211
– procmm.sh: Process Memory Matrix v1.02 by Tanel Poder ( http://tech.e2sn.com )
– All numbers are shown in kilobytes
PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD
—— ——————– ———— ———— ———— ———— ————
20211 stack 96 96 96 0 96
20211 anon 1048 960 816 0 1040
20211 lib 23064 14392 944 0 1512
—— ——————– ———— ———— ———— ———— ————
20211 TOTAL(kB) 24208 15448 1856 0 2648
@Max
Can you send me the full output of both commands?
As which user are you running this command? (is the oracle user or a member of dba group)?
procmm should report more output than the current (for example the shared mem segments etc).
Hi Tanel,
The original process has logged out. Here is another example when both commands ran immediately one after another:
>pmap -x 18918
18918: oraclelpiweb (LOCAL=NO)
Address Kbytes RSS Anon Locked Mode Mapped File
0000000100000000 103696 103624 – – r-x– oracle
0000000106642000 832 656 208 – rwx– oracle
0000000106712000 1960 912 888 – rwx– [ heap ]
0000000380000000 188416 188416 – 188416 rwxsR [ ism shmid=0x6a00002c ]
00000003C0000000 172032 172032 – 172032 rwxsR [ ism shmid=0x50000032 ]
0000000400000000 180224 180224 – 180224 rwxsR [ ism shmid=0x6c000038 ]
0000000440000000 176128 176128 – 176128 rwxsR [ ism shmid=0xf000042 ]
0000000480000000 16 16 – 16 rwxsR [ ism shmid=0x7b000043 ]
FFFFFFFF7BA60000 64 64 64 – rw— [ anon ]
FFFFFFFF7BAC0000 64 40 40 – rw— [ anon ]
FFFFFFFF7BAD0000 64 64 64 – rw— [ anon ]
FFFFFFFF7BAE0000 64 40 40 – rw— [ anon ]
FFFFFFFF7BB00000 16 16 – – r-x– liblgrp.so.1
FFFFFFFF7BC04000 8 8 8 – rwx– liblgrp.so.1
FFFFFFFF7BD00000 56 56 – – r-x– libmd.so.1
FFFFFFFF7BE0E000 8 8 8 – rwx– libmd.so.1
FFFFFFFF7BF00000 640 472 – – r-x– libm.so.2
FFFFFFFF7C09E000 40 32 24 – rwx– libm.so.2
FFFFFFFF7C100000 32 32 – – r-x– librt.so.1
FFFFFFFF7C208000 8 8 8 – rwx– librt.so.1
FFFFFFFF7C300000 32 32 – – r-x– libaio.so.1
FFFFFFFF7C408000 8 8 8 – rwx– libaio.so.1
FFFFFFFF7C500000 1264 896 – – r-x– libc.so.1
FFFFFFFF7C700000 64 32 32 – rwx– [ anon ]
FFFFFFFF7C73C000 72 72 72 – rwx– libc.so.1
FFFFFFFF7C74E000 8 – – – rwx– libc.so.1
FFFFFFFF7C800000 8 8 – – r-x– libdl.so.1
FFFFFFFF7C902000 8 8 8 – rwx– libdl.so.1
FFFFFFFF7CA00000 32 32 – – r-x– libgen.so.1
FFFFFFFF7CB08000 8 8 8 – rwx– libgen.so.1
FFFFFFFF7CC00000 56 56 – – r-x– libsocket.so.1
FFFFFFFF7CD00000 8 8 8 – rwx– [ anon ]
FFFFFFFF7CD0E000 16 16 16 – rwx– libsocket.so.1
FFFFFFFF7CE00000 688 448 – – r-x– libnsl.so.1
FFFFFFFF7CF00000 64 16 16 – rw— [ anon ]
FFFFFFFF7CF10000 64 64 64 – rw— [ anon ]
FFFFFFFF7CF20000 64 64 56 – rw— [ anon ]
FFFFFFFF7CF30000 64 64 64 – rw— [ anon ]
FFFFFFFF7CF40000 64 64 64 – rw— [ anon ]
FFFFFFFF7CF50000 64 64 64 – rw— [ anon ]
FFFFFFFF7CF60000 64 64 64 – rw— [ anon ]
FFFFFFFF7CF70000 64 64 64 – rw— [ anon ]
FFFFFFFF7CFAC000 64 64 64 – rwx– libnsl.so.1
FFFFFFFF7CFBC000 32 32 24 – rwx– libnsl.so.1
FFFFFFFF7D000000 8 8 – – r-x– libkstat.so.1
FFFFFFFF7D102000 8 8 8 – rwx– libkstat.so.1
FFFFFFFF7D200000 2184 408 – – r-x– libnnz10.so
FFFFFFFF7D500000 64 64 64 – rwx– [ anon ]
FFFFFFFF7D520000 240 216 200 – rwx– libnnz10.so
FFFFFFFF7D55C000 8 – – – rwx– libnnz10.so
FFFFFFFF7D600000 72 72 – – r-x– libdbcfg10.so
FFFFFFFF7D700000 24 16 16 – rwx– [ anon ]
FFFFFFFF7D710000 8 8 8 – rwx– libdbcfg10.so
FFFFFFFF7D770000 64 64 56 – rw— [ anon ]
FFFFFFFF7D780000 64 64 64 – rw— [ anon ]
FFFFFFFF7D790000 64 64 64 – rw— [ anon ]
FFFFFFFF7D7A0000 64 64 64 – rw— [ anon ]
FFFFFFFF7D7B0000 64 64 64 – rw— [ anon ]
FFFFFFFF7D7C0000 64 64 64 – rw— [ anon ]
FFFFFFFF7D7D0000 64 64 64 – rw— [ anon ]
FFFFFFFF7D7E0000 64 32 32 – rw— [ anon ]
FFFFFFFF7D800000 9272 8208 – – r-x– libjox10.so
FFFFFFFF7E200000 8 8 8 – rwx– [ anon ]
FFFFFFFF7E20C000 560 560 448 – rwx– libjox10.so
FFFFFFFF7E300000 1072 112 – – r-x– libclsra10.so
FFFFFFFF7E500000 8 8 8 – rwx– [ anon ]
FFFFFFFF7E50A000 56 40 40 – rwx– libclsra10.so
FFFFFFFF7E518000 8 – – – rwx– libclsra10.so
FFFFFFFF7E600000 1072 192 – – r-x– libocrutl10.so
FFFFFFFF7E800000 8 8 – – r-x– libc_psr.so.1
FFFFFFFF7E80A000 56 48 48 – rwx– libocrutl10.so
FFFFFFFF7E818000 8 – – – rwx– libocrutl10.so
FFFFFFFF7E900000 1376 192 – – r-x– libocrb10.so
FFFFFFFF7EB00000 8 8 8 – rwx– [ anon ]
FFFFFFFF7EB56000 64 56 56 – rwx– libocrb10.so
FFFFFFFF7EB66000 8 – – – rwx– libocrb10.so
FFFFFFFF7EC00000 1536 256 – – r-x– libocr10.so
FFFFFFFF7EE00000 8 8 8 – rwx– [ anon ]
FFFFFFFF7EE7E000 64 56 56 – rwx– libocr10.so
FFFFFFFF7EE8E000 8 – – – rwx– libocr10.so
FFFFFFFF7EF00000 8 8 – – r-x– libskgxn2.so
FFFFFFFF7F000000 8 8 8 – rwx– libskgxn2.so
FFFFFFFF7F100000 1752 792 – – r-x– libhasgen10.so
FFFFFFFF7F300000 8 8 8 – rwx– [ anon ]
FFFFFFFF7F3B4000 72 64 64 – rwx– libhasgen10.so
FFFFFFFF7F3C6000 16 8 8 – rwx– libhasgen10.so
FFFFFFFF7F400000 144 96 – – r-x– libskgxp10.so
FFFFFFFF7F500000 8 8 8 – rwx– [ anon ]
FFFFFFFF7F522000 16 16 16 – rwx– libskgxp10.so
FFFFFFFF7F600000 216 216 – – r-x– ld.so.1
FFFFFFFF7F700000 8 8 – – r-x– libsched.so.1
FFFFFFFF7F736000 16 16 16 – rwx– ld.so.1
FFFFFFFF7F73A000 8 8 8 – rwx– ld.so.1
FFFFFFFF7F7B8000 8 8 – – rwxs- [ anon ]
FFFFFFFF7FFDE000 136 136 136 – rw— [ stack ]
—————- ———- ———- ———- ———-
total Kb 848000 837472 3768 716816
>./procmm.sh 18918
– procmm.sh: Process Memory Matrix v1.02 by Tanel Poder ( http://tech.e2sn.com )
– All numbers are shown in kilobytes
PID SEGMENT_TYPE VIRTUAL RSS ANON LOCKED SWAP_RSVD
—— ——————– ———— ———— ———— ———— ————
18918 stack 136 136 136 0 136
18918 anon 1496 1328 1304 0 1488
18918 lib 23064 14000 1232 0 1512
—— ——————– ———— ———— ———— ———— ————
18918 TOTAL(kB) 24696 15464 2672 0 3136
Thanks,
Max
Also when man pmap, it says:
Anonymous Memory (Anon)
The number of pages, counted by using the system page
size, of anonymous memory associated with the specified
mapping. Anonymous memory shared with other address
spaces is not included, unless the -a option is speci-
fied.
Anonymous memory is reported for the process heap,
stack, for ‘copy on write’ pages with mappings mapped
with MAP_PRIVATE (see mmap(2)).
So “Anon” in pmap seems showing the number of pages, not in kb.