Comments on: Systematic application troubleshooting in Unix

By: metin2 hileleri

metin2 hileleri — Mon, 01 Mar 2010 00:56:11 +0000

Thanks Hemant, i wrote a another quick blog entry about it!

By: 8 Useful Technical Posts … - Sinoptix

8 Useful Technical Posts … - Sinoptix — Sat, 02 Feb 2008 06:36:12 +0000

[...] my pockets, so to speak. First of all, I liked both the technical content of Tanel Poder’s blog on Systematic application troubleshooting in Unix and also the rants, despite the fact he tried not to spend too much time on the latter I hope he [...]

By: Virag Sharma

Virag Sharma — Sat, 19 Jan 2008 03:09:32 +0000

Thanks Tanel , your script sw.sql is very useful.

By: tanelp

tanelp — Tue, 15 Jan 2008 09:26:21 +0000

Hi Virag,

Note the context of my post, I deliberately mentioned “first round session troubleshooting” in italic in my post.

E.g. the first thing you check when you want to troubleshoot a specific session.

You can get there both ways, either start from v$session_wait or OS level stats.

The difference is that the v$session_wait will give you better, narrower focus (in other words it rules out more potential causes).

When you check for CPU usage of a process, then no matter what result you get (either the process is sleeping, is 50% on CPU or 100% on CPU), you still have to log on to database and continue your investigation there.

If you start from logging on to database and look into v$session_wait, then it can rule out many things which would require logging on to OS and using OS tools. For example (again) if you see the session waiting for TX enqueue for last 30minutes or for “log file switch (archiving needed”) then you probably do not care about the current CPU usage of the problem session.

Of course, checking “environmental factors” like current CPU runqueue lengths, disk IO queuing and service times, can very well be necessary in many cases – but it’s worth taking your time to check these only when a simple query against v$session_wait has ruled out many other “usual suspects”.

Note that after I wrote Snapper, I use sw.sql much less. Depending on the type of problem (and previous experience regarding that environment) I may start with Snapper instead as it gives a better overview of what’s going on when a session is actually doing something and not hung.

By: Virag Sharma

Virag Sharma — Sun, 13 Jan 2008 13:45:51 +0000

“Whenever you diagnose a single session hang or performance problem in Oracle, you should first look into V$SESSION_WAIT” ..

I think DBA still need to see OS level stats before peeping into V$SESSION_WAIT / v$session because it was noticed some time other processes consume most of CPU i.e. CPU usage 100% … and oracle processes not getting enough CPU resource …. Some time found sql execution plan changed due to stats collection / DB upgrade and session start consuming 70-80 % CPU …… in that case also OS command like top / ps help to immediately find out PID->SID-> SQL_ID -> execution plan and if needed set event on session and get all wait event details

As Old oracle Performance document suggest , if you are on production go left to right

OS-> Wait-> Lock/latch etc-> Design

Correct me , if am wrong :-)

By: tanelp

tanelp — Tue, 08 Jan 2008 15:19:32 +0000

Thanks Hemant,

I forgot to upload the most recent version. it’s done now + i wrote a another quick blog entry about it!

By: Hemant K Chitale

Hemant K Chitale — Tue, 08 Jan 2008 07:25:11 +0000

Neat ! Couldn’t find “P1TRANSL” in the code in sw.sql

By: tanelp

tanelp — Sun, 06 Jan 2008 05:44:47 +0000

Thanks Bas, yes 10g makes life easier thanks to v$session_wait contnts being in v$session.

I’m using this script in addition to Snapper for checking session wait info:

http://www.tanelpoder.com/files/scripts/sw.sql

My script works on 9i and 10g/11g. It also decodes the enqueue and latch names the session is waiting on, on 9i.

SQL> @sw 16


       SID STATE   EVENT                        SEQ# SECONDS_IN_WAIT         P1         P2         P3 P1TRANSL

---------- ------- ------------------- ---------- --------------- ---------- ---------- ---------- ----------------------

        16 WAITING enqueue                     26              18 1415053318     262156       5249 0x54580006: TX mode 6

I’ve written the script the waythat you can write the SID parameter as a subquery. This gives pretty good flexibility, especially when you want to report a group of sessions at a time:

SQL> @sw "select sid from v$session where status = 'ACTIVE'"


       SID STATE   EVENT                     SEQ# SECONDS_IN_WAIT         P1         P2         P3 P1TRANSL

---------- ------- ------------------- ---------- --------------- ---------- ---------- ---------- -------------------------

         1 WAITING pmon timer                 113             305        300          0          0

         2 WAITING rdbms ipc message          147             296        300          0          0

         3 WAITING rdbms ipc message         2029             159        300          0          0

         4 WAITING rdbms ipc message          261               0        300          0          0

         5 WAITING smon timer                 208             239        300          0          0

         6 WAITING rdbms ipc message            5             242     180000          0          0

         7 WAITING rdbms ipc message           74             278        500          0          0

        13 WORKING On CPU / runqueue           53               0 1413697536          1          0

        16 WAITING enqueue                     74             162 1415053318     262156       5249 0x54580006: TX mode 6

9 rows selected.

By: Bas

Bas — Sat, 05 Jan 2008 13:57:56 +0000

From 10.1 and up most columns of v$session_wait can also be found in v$session, it saves a join. My first query when troubleshooting a current performance problem is:

SELECT wait_class
, event
, COUNT (1) wait_count
FROM v$session
WHERE NOT username IS NULL
AND wait_class ‘Idle’
AND audsid SYS_CONTEXT (‘USERENV’, ‘SESSIONID’)
GROUP BY event
, wait_class
ORDER BY wait_class
, event
/