Yep the sawr$sess_event_delta should not be there anymore. This synonyms is a bug (and fixed in latest sesspack version). I’ll blog it soon.

I’m trying to install Sesspack 0.05 – I got it from the Hotsos Symposium 2008 download back in March.

It includes a synonym for sawr$sess_event_delta. But this view does not seem to be present in the install_sesspack_schema script in version 0.05 (it was there in 0.04). Do I need that view for 0.05?

br
peter

Yes, you’re right. Had the waits been for a previous cursor, we should have seen an EXEC# 25 or FETCH# 25 line after those waits, before PARSING IN.

My first hunch was that this may be due auditing (insert into AUD$ or FGA_LOG$), but audit inserts should have been executed through a separate, recursive cursor.

So what were those obj#’s seein in waits referring to? Were those normal tables/indexes?

Can you reproduce this issue at will? If yes and if you happen to be on Solaris 10, I can give you a DTrace script which dumps the process stack when it waits for buffer busy waits or enq TX: index contention.

This would give definitive proof on what caused those waits.

“WAIT lines appear in the trace data before the database call that motivated
them. This occurs because the Oracle kernel emits lines into the trace file as events
complete.”

Therefore, my interpretation is that EXECUTE database call begins for the INSERT (Cursor#25) and results ine WAITS on enq:TX and buffer busy waits. Only then is the data about cursor#25 output to the trace file.

Here is a more complete extract from the file(i’ve changed the names of the database objects):

WAIT #0: nam=’SQL*Net message to client’ ela= 2 driver id=1952673792 #bytes=1 p3=0 obj#=315994 tim=3273173063439
WAIT #0: nam=’SQL*Net message from client’ ela= 1566 driver id=1952673792 #bytes=1 p3=0 obj#=315994 tim=3273173065090
WAIT #0: nam=’SQL*Net message to client’ ela= 2 driver id=1952673792 #bytes=1 p3=0 obj#=315994 tim=3273173065421
WAIT #0: nam=’SQL*Net message from client’ ela= 833 driver id=1952673792 #bytes=1 p3=0 obj#=315994 tim=3273173066319
=====================
PARSING IN CURSOR #24 len=637 dep=1 uid=47 oct=3 lid=47 tim=3273173073507 hv=1331493982 ad=’57c2a718′
SELECT * FROM TABLE1 WHERE SOME_ID= :B1 FOR UPDATE
END OF STMT
EXEC #24:c=10000,e=4731,p=0,cr=31,cu=4,mis=0,r=0,dep=1,og=1,tim=3273173073489
FETCH #24:c=0,e=1553,p=0,cr=8,cu=0,mis=0,r=1,dep=1,og=1,tim=3273173077198
=====================
PARSING IN CURSOR #15 len=431 dep=1 uid=47 oct=3 lid=47 tim=3273173078097 hv=604810361 ad=’6b4deb48′
SELECT COUNT(*) FROM ( SELECT SOME_STUFF_FROM SOME_TABLES)
END OF STMT
EXEC #15:c=0,e=503,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=3273173078085
FETCH #15:c=0,e=1384,p=0,cr=19,cu=0,mis=0,r=1,dep=1,og=1,tim=3273173079686
WAIT #25: nam=’enq: TX – index contention’ ela= 6 name|mode=1415053316 usn<<16 | slot=2555913 sequence=228627 obj#=315994 tim=3273173081540
WAIT #25: nam=’enq: TX – index contention’ ela= 77446 name|mode=1415053316 usn<<16 | slot=2555913 sequence=228627 obj#=315994 tim=3273173159130
WAIT #25: nam=’buffer busy waits’ ela= 1109 file#=198 block#=742607 class#=1 obj#=315242 tim=3273173160444
WAIT #25: nam=’enq: TX – index contention’ ela= 8415 name|mode=1415053316 usn<<16 | slot=7602197 sequence=227627 obj#=315242 tim=3273173169242
WAIT #25: nam=’buffer busy waits’ ela= 3668 file#=198 block#=742607 class#=1 obj#=315242 tim=3273173173017
=====================
PARSING IN CURSOR #25 len=250 dep=1 uid=47 oct=2 lid=47 tim=3273173173513 hv=3689280581 ad=’57c2b178′
INSERT INTO TABLE_2 RETURNING A_COLUMN INTO :O0
END OF STMT
EXEC #25:c=0,e=93457,p=0,cr=11,cu=39,mis=0,r=1,dep=1,og=1,tim=3273173173484
FETCH #24:c=0,e=101,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=1,tim=3273173173976
FETCH #24:c=0,e=31,p=0,cr=1,cu=0,mis=0,r=0,dep=1,og=1,tim=3273173174119
=====================

So search upwards in your tracefile for PARSING IN CURSOR #25 and the match you hit first is the statement that caused those waits.

sorry to bother you !

But, I now have some raw trace file data related to this issue, that I’m unsure of.

First of all, 2 of the WAIT lines (emitted before an insert) report enq:TX index contention on a different table/index to that being inserted. That index is in a different tablespace and segment to the table being inserted into. Therefore, Im confused why this session has to wait on it.

EXEC #15:c=0,e=503,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=3273173078085
FETCH #15:c=0,e=1384,p=0,cr=19,cu=0,mis=0,r=1,dep=1,og=1,tim=3273173079686
WAIT #25: nam=’enq: TX – index contention’ ela= 6 name|mode=1415053316 usn<<16 | slot=2555913 sequence=228627 obj#=315994 tim=3273173081540
WAIT #25: nam=’enq: TX – index contention’ ela= 77446 name|mode=1415053316 usn<<16 | slot=2555913 sequence=228627 obj#=315994 tim=3273173159130
WAIT #25: nam=’buffer busy waits’ ela= 1109 file#=198 block#=742607 class#=1 obj#=315242 tim=3273173160444
WAIT #25: nam=’enq: TX – index contention’ ela= 8415 name|mode=1415053316 usn<<16 | slot=7602197 sequence=227627 obj#=315242 tim=3273173169242
WAIT #25: nam=’buffer busy waits’ ela= 3668 file#=198 block#=742607 class#=1 obj#=315242 tim=3273173173017
=====================
PARSING IN CURSOR #25 len=250 dep=1 uid=47 oct=2 lid=47 tim=3273173173513 hv=3689280581 ad=’57c2b178′
INSERT INTO SOME_TABLE

(I’m assuming that the obj# reported in the WAIT line refers to the object_id in dba_objects)

br
peter

The unaccounted time is likely one of the following:

1) Time spent waiting in CPU runqueue (Oracle thinks the process is on CPU and has stopped measuring wait time – but really the process is waiting in CPU runqueue, thus OS is not accounting any CPU time for the process either)

2) CPU Time not accounted due quantization error (as normally the CPU usage is measured once every tick and attributed to the process currently happening to be on the CPU)

3) Wait time not accounted a bug

Not quite the same topic but I thought you might have some ideas.

I am trying assemble a report about recent performance problems in our system.

I am looking at V$SQL to get a breakdown of wait time contributions to ELAPSED_TIME for some SQL statements using the

User_IO_Wait_Time,
Application_wait_time,
Concurrency_Wait_time,
PLSQL_Execution_Wait_time,
CPU_Time

columns. But there is always a percentage not accounted for by these columns. Could you suggest possible reasons for this unaccounted time?

I realise using V$ views is not ideal for this kind of task but its all I have (in the absence of properly scoped trace file data) now (the performance problem was fixed)

Cheers

–
LSC