Check his blog entry out here:

http://karlarao.wordpress.com/2009/06/07/diagnosing-and-resolving-gc-block-lost/

Here’s a little situation on Linux where an Oracle shared memory segment was not released as someone was still using it.

$ ipcs -ma

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 393216     oracle    640        289406976  1          dest
0xbfb94e30 425985     oracle    640        289406976  18
0x3cf13430 557058     oracle    660        423624704  22

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xe2260ff0 1409024    oracle    640        154
0x9df96b74 1671169    oracle    660        154

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

The bold line should have disappeared after instance shutdown, but it didn’t. From “natcch” (number of attached processes) column I see there is still some process using the shared memory segment. Thus the segment was not released and even ipcrm command did not remove it (just like with normal files if someone has them open).

So, I needed to identify which process was still using the memory segment. If that had been a normal existing file, I’d could have used /sbin/fuser command to see which process still holds it open, but this only works for existing files with existing directory entries.

However for deleted files, sockets and shared memory segments, you can use lsof command (it’s normally installed by default on Linux, but for Unixes you need to separately download and install).

The SHM ID of that segment was 393216 as ipcs -ma showed, so I simply run lsof to show all open file descriptors and grep for that SHM ID:

$ lsof | egrep "393216|COMMAND"
COMMAND     PID      USER   FD      TYPE     DEVICE       SIZE       NODE NAME
python    18811    oracle  DEL       REG        0,8                393216 /SYSVbfb94e30

See how the NODE column corresponds to SHM ID in ipcs output.

So I kill the PID 18811 which is still attached to the SHM segment:

$ kill 18811

$ ipcs -ma

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0xbfb94e30 425985     oracle    640        289406976  18
0x3cf13430 557058     oracle    660        423624704  25

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xe2260ff0 1409024    oracle    640        154
0x9df96b74 1671169    oracle    660        154

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Now the shared memory segment is gone and its memory released.

Note that the lsof command is very useful for many other tasks as well. For example it allows you to list open sockets by network protocol, IP, port etc. For example you can determine to which client some server process is talking to, from OS level:

$ lsof -i:1521
COMMAND   PID   USER   FD   TYPE DEVICE SIZE NODE NAME
tnslsnr  6212 oracle   11u  IPv4  49486       TCP *:1521 (LISTEN)
tnslsnr  6212 oracle   13u  IPv4 276708       TCP linux03:1521->linux03:37277 (ESTABLISHED)
tnslsnr  6212 oracle   14u  IPv4 264894       TCP linux03:1521->linux03:41122 (ESTABLISHED)
oracle  22687 oracle   20u  IPv4 264893       TCP linux03:41122->linux03:1521 (ESTABLISHED)
oracle  25250 oracle   15u  IPv4 276707       TCP linux03:37277->linux03:1521 (ESTABLISHED)
oracle  25530 oracle   15u  IPv4 279910       TCP linux03:1521->192.168.247.1:nimsh (ESTABLISHED)

Unfortunately lsof is not installed by default in classic Unixes, but in some shops the sysadmins have chosen to install it. But even then, it may not work for regular users as lsof requires access to kernel memory structures through /dev/kmem or similar. If you can’t get access to lsof then there may be other tools available which can do some tricks lsof can do. For example on Solaris, there’s an useful command pfiles which can show open files of a process and since Solaris 9 ( I think ) it can also report the TCP connection endpoints of network sockets…

This year I promise to organize my blog a little better, to have some index of my articles, scripts etc :)

Here are the news…

I’ve moved my blog…

In December, when visiting Shanghai, I noticed that my blog was not accessible from there. Apparently the state firewall blocks all access to wordpress.com IPs.

So I have moved my blog to a virtual private server, off wordpress.com – my blog should be accessible from China now as well.

My Advanced Oracle Troubleshooting Seminars in year 2009

If you like the contents of my blog or conference presentations, you’ll sure like my 2-day seminar!

Here are the dates and links for more info:

3-4. February – NYOUG @ New York City – http://www.nyoug.org/etc/training/htm/NYOUG_Training_Session.htm

NB! As this is my first public seminar in US, you’ll be able to attend it for a very very good price! And the early bird registration (which gives you even better price) is open until end of today! More seminars in other US cities are planned for 2nd half of 2009.

2-3. April – Miracle @ Netherlands – http://www.miraclebv.nl/

Probably happens in Utrecht, I’ll keep you updated if it changes.

13-14. April – Oracle @ Singapore – http://www.oracle.com/education/apac/sg_tanel_poder.html

16-17. April – Oracle @ Sydney – http://www.oracle.com/education/apac/au_tanel_poder.html

20-21. April – Oracle @ Melbourne – http://www.oracle.com/education/apac/au_tanel_poder.html

23-24. April Oracle @ Auckland – http://www.oracle.com/education/apac/nz_tanel_poder.html

27-28. April – PiSec Ltd @ Edinburgh – http://www.pisec.org/index.php?option=com_content&view=article&id=6&Itemid=12

11-12. May – Oracle @ Spain – http://education.oracle.com/pls/web_prod-plq-dad/show_desc.redirect?dc=D70365_1060245&p_org_id=51&lang=E&source_call=

18-19. May – Miracle @ Denmark – http://www.miracleas.dk/index.php?option=com_content&view=article&id=100:advanced-oracle-troubleshooting&catid=19:info&Itemid=71

Note that the Advanced Oracle Troubleshooting seminar is meant mainly for experienced DBAs and performance engineers as I go very deep in Oracle internals and OS/hardware details.

See more details about my seminar here

Conferences in 2009

11-12. February – RMOUG Conference @ Denver, Colorado

I will be delivering two presentations about advanced Oracle troubleshooting, tuning and internals:

• Advanced Oracle Troubleshooting: No Magic is Needed, Systematic Approach Will Do
• Understanding Oracle Execution Plans: How SQL is Really Executed

8-12. March – Hotsos Symposium @ Dallas, Texas

I will be delivering two presentations:

• Understanding Oracle SQL Plan Execution: How It Really Works
• Zero Slides: The Scripts and Tools which Will Make Your Life Easier and Allow You to Troubleshoot Better

As the second title says, I don’t use any slides at that presentation, I will just demo some of the scripts and tools I use every day plus few case studies. It will be fun!

Also, I look forward attending the Training Day by Jonathan Lewis on Oracle troubleshooting, on last day of the Symposium.

I recently found a a pretty good article by Sun about Oracle Net8 tuning, SDU/TDU parameter sizes and some test results. Here it is:

http://www.sun.com/blueprints/1002/817-0370-10.pdf

Addition:

While at it, here’s also a another good document on TCP tuning in general. Recommended reading, especially if you’re dealing with tuning database (or application access) over WAN networks.

http://www.sun.com/blueprints/0304/817-5773.pdf

A quote is below:

We are seeing an insert statement reporting “SQL*Net break/reset to client” as over 1/3 of its time.
On the face of it this event suggests network issues but nothing else backs this up as the cause.
So I looked at the Java code in question and a trace of one of the sessions.
What is happening is that an attempt is made to insert a row, most of the time a duplicate error results, the code catches this exception and does an update.
I was wondering if its the duplicate error and the exception handling which results in this wait event showing up.

My answer to that was following:

Yes, a SQL*Net break/reset happens when an error/unhandled exception is raised during a call (which means that the call executed didn’t complete normally, thus the call state must be reset).

The approach of “insert -> if failed then update” is basically what MERGE does.

You need to change the application to use MERGE command. Alternatively you could use an “update -> if no-rows-updated then insert” approach, but MERGE makes much more sense nowadays.

So, these waits aren’t really indicating any network bottleneck, but rather just the fact that the client needs to be notified if it’s call has failed and Oracle has to clean up after the failed call. Apparently on some (network protocol) architectures or in some scenarios these operations have taken significant time (and may have required a system call), so a kernel developer has decided to wait instrument them.

In this blog entry I want to share a small test case which illustrates the point. First lets check how many times my session has waited for a SQL*Net break/reset wait event:

SQL> select event, total_waits from v$session_event where event like '%reset%' and sid = (select sid from v$mystat where rownum = 1);

no rows selected

As V$SESSION_EVENT (like also V$SYSTEM_EVENT) doesn’t display events for which nobody has waited yet, I see that my session’s wait count for that event is zero.

Now let’s create an error, but also let’s handle it in a PL/SQL exception handler:

SQL> begin
  2     execute immediate 'drop table non_existent';
  3  exception
  4     when others then null;
  5  end;
  6  /

PL/SQL procedure successfully completed.

SQL> select event, total_waits from v$session_event where event like '%reset%' and sid = (select sid from v$mystat where rownum = 1);

no rows selected

Still no breaks/resets as the error was handled and the OPI call completed properly.

Now let’s run the same PL/SQL block without exception handler:

SQL> begin
  2     execute immediate 'drop table non_existent';
  3  end;
  4  /
begin
*
ERROR at line 1:
ORA-00942: table or view does not exist
ORA-06512: at line 2

SQL> select event, total_waits from v$session_event where event like '%reset%' and sid = (select sid from v$mystat where rownum = 1);

EVENT                                                            TOTAL_WAITS
---------------------------------------------------------------- -----------
SQL*Net break/reset to client                                              2

You see, there are 2 break/reset waits registered.

Also normal SQL (not fired from PL/SQL but from top level instead) causes breaks/resets if it errors:

SQL> select * from non_existent;
select * from non_existent
              *
ERROR at line 1:
ORA-00942: table or view does not exist

SQL> select event, total_waits from v$session_event where event like '%reset%' and sid = (select sid from v$mystat where rownum = 1);

EVENT                                                            TOTAL_WAITS
---------------------------------------------------------------- -----------
SQL*Net break/reset to client                                              4

SQL>

Why 2 waits for a single error? I don’t know exact details, but the 10046 trace reveals that one of the waits is registered by a break operation and another one (apparently) not, thus it might be the call state reset operation just accounted separately:

=====================
PARSE ERROR #10:len=27 dep=0 uid=0 oct=3 lid=0 tim=1967444453 err=942
select * from non_existent
WAIT #10: nam='SQL*Net break/reset to client' ela= 2 driver id=1413697536 break?=1 p3=0 obj#=-1 tim=1967444525
WAIT #10: nam='SQL*Net break/reset to client' ela= 180 driver id=1413697536 break?=0 p3=0 obj#=-1 tim=1967444718
WAIT #10: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=1967444752

Anyway, as I said in my Oracle-L reply, these breaks are caused by bad application design which allows too many unhandled exceptions to be propagated all the way up to the client.

The solutions would be to reduce or eliminate the number of errors occurring, or at least put that code into PL/SQL blocks where errors would be handled and not propagated back to client every time.

The first session data unit (SDU) bufferful of return data is written to TCP socket buffer under “SQL*Net message to client” wait event.

If Oracle needs to write more data for a call than fits into the “first” SDU buffer, then further writes for that call are done under SQL*Net more data to client event.

So, whether and how much of the “SQL*Net more data to client” vs “SQL*Net message to client” waits you see depends on two things:

• Amount of data returned to client per call
• Oracle Net SDU size

A quick test with my Snapper performance tool illustrates this:

In one session run following:

SQL> set arraysize 1
SQL>
SQL> select rownum from dba_source;

    ROWNUM
----------
         1
         2
         3
         4
         5
         6
...many rows returned...

And I measure the waits in the other session:

SQL> @snapper out,gather=w 5 1 141

-- Session Snapper v1.06 by Tanel Poder ( http://www.tanelpoder.com )

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,      D/SEC,     HDELTA,     HD/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     141, 20080210 14:59:55,        5, WAIT, SQL*Net message to client               ,          6750,       1350,     6.75ms,     1.35ms
DATA,     141, 20080210 14:59:55,        5, WAIT, SQL*Net message from client             ,       4668258,     933652,      4.67s,   933.65ms
--  End of snap 1

No SQL*Net more data to client is evidenced above as due the arraysize = 1 setting Oracle fetches only 2 rows at a time. Therefore the returned resultset per fetch call always fits into the first SDU sized packet. Note that the reason why Oracle fetches 2 rows when arraysize is 1 is due an OCI optimization which tries to be clever and proactively detect end-of-resultset conditions when fetching a single row – that way there’s a chance to automatically cancel a cursor and release its pins without an explicit cursor cancel or close call. This should have positive effect reducing shared pool fragmentation in some conditions.

Anyway, now I set arraysize to 5000 and run the same SQL again:

SQL> set arraysize 5000
SQL>
SQL> select rownum from dba_source;

    ROWNUM
----------
         1
         2
         3
         4
         5
         6
...many rows returned...

And snapper output is below:

SQL> @snapper out,gather=w 5 1 141

-- Session Snapper v1.06 by Tanel Poder ( http://www.tanelpoder.com )

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,      D/SEC,     HDELTA,     HD/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     141, 20080210 15:00:35,        5, WAIT, SQL*Net message to client               ,            32,          6,       32us,      6.4us
DATA,     141, 20080210 15:00:35,        5, WAIT, SQL*Net more data to client             ,          2925,        585,     2.93ms,      585us
DATA,     141, 20080210 15:00:35,        5, WAIT, SQL*Net message from client             ,       4861859,     972372,      4.86s,   972.37ms
--  End of snap 1

Now we see SQL*Net more data to client waits as well as the 5000 rows returned for every fetch call just don’t fit into a single SDU buffer.

I’ll reiterate that both SQL*Net message to client and SQL*Net more data to client waits only record the time it took to write the return data from Oracle’s userland SDU buffer to OS kernel-land TCP socket buffer. Thus the wait times of only microseconds. Thanks to that, all of the time a TCP packet spent “flying” towards the client is actually accounted in SQL*Net message from client wait statistic. The problem here is though, that we don’t know how much of this time was spent on the wire and how much of it was application think time.

Therefore, unless you’re going to buy a tool which is able to interpret TCP ACK echo timestamps, you need to measure network latency using application side instrumentation.