http://blog.tanelpoder.com/seminar/expert-oracle-exadata-virtual-conference/

Also, our book will be out on Monday!

This is not really worth putting into my Oracle Exadata Performance series (which so far has only 1 article in it anyway) .. so this is a little stand-alone article …

Everybody knows that the Exadata smart scan can be used when scanning tables (and table partitions). You should also know that smart scan can be used with fast full scan on Oracle B-tree indexes (a fast full scan on an index segment is just like a full table scan, only on the index segment (and ignoring branch blocks)).

For some reason there’s a (little) myth circulating that smart scans aren’t used for scanning bitmap indexes.

So, here’s evidence, that smart scan can be used when scanning bitmap indexes:

SQL> select /*+ tanel3 */ count(*) from t1 where owner like '%XYZXYZ%';

...

Plan hash value: 39555139

-----------------------------------------------------------------------------------
| Id  | Operation                             | Name        | E-Rows | Cost (%CPU)|
-----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |             |        |   505 (100)|
|   1 |  SORT AGGREGATE                       |             |      1 |            |
|   2 |   BITMAP CONVERSION COUNT             |             |    400K|   505   (0)|
|*  3 |    BITMAP INDEX STORAGE FAST FULL SCAN| BI_T1_OWNER |        |            |
-----------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - storage(("OWNER" LIKE '%XYZXYZ%' AND "OWNER" IS NOT NULL))
       filter(("OWNER" LIKE '%XYZXYZ%' AND "OWNER" IS NOT NULL))

So, as you see the execution plan sure shows a FAST FULL SCAN on a BITMAP INDEX segment, which happens to be on Exadata STORAGE.

Also, you see a storage() predicate applied on the line 3 of the execution plan, which means that Oracle will attempt to use a smart scan predicate offload – but this can’t always be done!

So, you can’t really determine whether a smart scan happened during execution just by looking into the execution plan, you should really check some V$SESSION statistics too. That’s where my Snapper script becomes handy.

I started Snapper on my session just before running the above query. The “smart table scan” and “smart index scan” performance counters are updated right after Oracle has opened the segment header and determines, from the number of blocks in the segment, whether to call the smart scan codepath or not. In other words, the smart scan counters are inremented in the beginning of the segment scan.

The output is following (some irrelevant counters are stripped for brevity):

@snapper all 5 1 "301"
Sampling SID 301 with interval 5 seconds, taking 1 snapshots...
setting stats to all due to option = all

-- Session Snapper v3.52 by Tanel Poder @ E2SN ( http://tech.e2sn.com )

-------------------------------------------------------------------------------------------------------------------------------------
    SID, USERNAME  , TYPE, STATISTIC                                                 ,     HDELTA, HDELTA/SEC,    %TIME, GRAPH
-------------------------------------------------------------------------------------------------------------------------------------
    301, TANEL     , STAT, physical read total IO requests                           ,         13,        2.6,
    301, TANEL     , STAT, physical read total multi block requests                  ,          4,         .8,
    301, TANEL     , STAT, physical read requests optimized                          ,          1,         .2,
    301, TANEL     , STAT, physical read total bytes optimized                       ,      8.19k,      1.64k,
    301, TANEL     , STAT, physical read total bytes                                 ,      4.63M,     925.7k,
    301, TANEL     , STAT, cell physical IO interconnect bytes                       ,     10.02k,         2k,
    301, TANEL     , STAT, physical reads                                            ,        565,        113,
    301, TANEL     , STAT, physical reads cache                                      ,          1,         .2,
    301, TANEL     , STAT, physical reads direct                                     ,        564,      112.8,
    301, TANEL     , STAT, physical read IO requests                                 ,         13,        2.6,
    301, TANEL     , STAT, physical read bytes                                       ,      4.63M,     925.7k,
    301, TANEL     , STAT, db block changes                                          ,          1,         .2,
    301, TANEL     , STAT, cell physical IO bytes eligible for predicate offload     ,      4.62M,    924.06k,
    301, TANEL     , STAT, cell physical IO interconnect bytes returned by smart scan,      1.82k,      364.8,
    301, TANEL     , STAT, cell blocks processed by cache layer                      ,        564,      112.8,
    301, TANEL     , STAT, cell blocks processed by txn layer                        ,        564,      112.8,
    301, TANEL     , STAT, cell blocks processed by index layer                      ,        564,      112.8,
    301, TANEL     , STAT, cell blocks helped by minscn optimization                 ,        564,      112.8,
    301, TANEL     , STAT, cell index scans                                          ,          1,         .2,
    301, TANEL     , STAT, index fast full scans (full)                              ,          1,         .2,
    301, TANEL     , STAT, index fast full scans (direct read)                       ,          1,         .2,
    301, TANEL     , STAT, bytes sent via SQL*Net to client                          ,        334,       66.8,
    301, TANEL     , STAT, bytes received via SQL*Net from client                    ,        298,       59.6,
    301, TANEL     , STAT, SQL*Net roundtrips to/from client                         ,          2,         .4,
    301, TANEL     , STAT, cell flash cache read hits                                ,          1,         .2,
    301, TANEL     , TIME, hard parse elapsed time                                   ,     1.17ms,    233.8us,      .0%, |          |
    301, TANEL     , TIME, parse time elapsed                                        ,      1.5ms,    300.2us,      .0%, |          |
    301, TANEL     , TIME, DB CPU                                                    ,       11ms,      2.2ms,      .2%, |          |
    301, TANEL     , TIME, sql execute elapsed time                                  ,     82.2ms,    16.44ms,     1.6%, |@         |
    301, TANEL     , TIME, DB time                                                   ,    84.36ms,    16.87ms,     1.7%, |@         |
    301, TANEL     , WAIT, enq: KO - fast object checkpoint                          ,    16.18ms,     3.24ms,      .3%, |          |
    301, TANEL     , WAIT, gc cr grant 2-way                                         ,      223us,     44.6us,      .0%, |          |
    301, TANEL     , WAIT, gc current grant 2-way                                    ,      136us,     27.2us,      .0%, |          |
    301, TANEL     , WAIT, cell smart index scan                                     ,    56.04ms,    11.21ms,     1.1%, |@         |
    301, TANEL     , WAIT, SQL*Net message to client                                 ,        7us,      1.4us,      .0%, |          |
    301, TANEL     , WAIT, SQL*Net message from client                               ,      4.42s,   884.47ms,    88.4%, |@@@@@@@@@ |
    301, TANEL     , WAIT, cell single block physical read                           ,      541us,    108.2us,      .0%, |          |
    301, TANEL     , WAIT, events in waitclass Other                                 ,     2.22ms,    443.2us,      .0%, |          |
--  End of Stats snap 1, end=2011-03-13 19:36:31, seconds=5

As you see from the above “cell index scans” statistic – indeed one index segment was scanned using the cell smart scan method.

So, I would rather call this feature “smart segment scan” to reflect that smart scan can scan more than just tables…

I guess one of the reasons why few people have seen smart bitmap index scans in action is that (single-column) bitmap indexes tend to be small. Smaller than corresponding table segments and B-tree index segments. On partitioned tables they’re much more likely going to be under the “_small_table_threshold” calculation which is used for determining whether to do a direct path full segment scan or not (yes, the _small_table_threshold applies to fast full index scan and fast full bitmap index scan too, not just table scans). So, it’s likely that Oracle chooses to do a regular, buffered full bitmap segment scan and thus won’t even consider using smart scan (as smart scans require direct path reads).

By the way – the direct path read (or not) decision is done per segment – not per object (like a table or index). So if you have 10 partitions in a table (or index), half of them are large, half are smaller, then Oracle may end up using direct path reads (and smart scan) on 5 of them and buffered (dumb) scan on the other 5. If you run something like Snapper on the session, then you’d see the smart scan counters go up by 5 only. As written above, Oracle decides whether to do direct path reads (and smart scan) right after opening the header block of a segment (partition) and reading out how many blocks this partition’s segment has below HWM.

The above applied to serial direct path reads – the Parallel Execution slaves should always read using direct path mode, right? …. Wrong :)

Well, partially wrong… In 11.2.0.2, if the parallel_degree_policy = manual, then yes, PX slaves behave like usual and always force a direct path read (and try to use a smart scan). However, with parallel_degree_policy = AUTO, which is the future of PX auto-management, Oracle can decide to do a buffered parallel scan instead, again disabling the use of smart scan…

One more note – I didn’t say anything about whether you should or should not use (bitmap) indexes on Exadata, it’s an entirely different discussion. I just brought out that the smart scan is used for scanning table segments, B-tree index segments and bitmap index segments if conditions are right.

And in the end I have to say…. that even with this evidence you can’t be fully sure that a smart scan was used throughout the entire segment, but more about this in the book and perhaps in a later blog article. We have interesting times ahead ;-)

The syntax is the same, with Snapper you can now sample ASH-like data on Oracle 9.2 too. Instead of SQL_IDs it will display you SQL hash values:

SQL> @snapper ash,ash1=user+sql_id,ash2=sid+event 5 1 all
Sampling SID all with interval 5 seconds, taking 1 snapshots...

-- Session Snapper v3.52 by Tanel Poder @ E2SN ( http://tech.e2sn.com )

--------------------------------------------------
Active% | SQL_HASH_VAL | EVENT
--------------------------------------------------
    19% | 867131449    | db file scattered read
    19% | 1458866975   | db file scattered read
    13% | 1267657304   | db file scattered read
     6% | 884811952    | ON CPU
     6% | 581825634    | db file scattered read
     3% | 867131449    | ON CPU
     3% | 1267657304   | ON CPU
     3% | 1866659945   | ON CPU
     3% | 1671194465   | db file scattered read
     3% | 3021169464   | ON CPU

------------------------------------------------
Active% | USERNAME             | SQL_ID
------------------------------------------------
    23% | SYSTEM               | 867131449
    19% | SYSTEM               | 1458866975
    16% | SYSTEM               | 1267657304
     6% | SYSTEM               | 581825634
     6% | SYSTEM               | 884811952
     3% | SYSTEM               | 1558333473
     3% | SYSTEM               | 1671194465
     3% | SYSTEM               | 1866659945
     3% | SYSTEM               | 1927486197
     3% | SYSTEM               | 2700565926

--------------------------------------------
Active% |    SID | EVENT
--------------------------------------------
    77% |     18 | db file scattered read
    19% |     18 | ON CPU
     3% |     18 | db file sequential read

--  End of ASH snap 1, end=2011-01-10 03:02:58, seconds=5, samples_taken=31

PL/SQL procedure successfully completed.


Other than the 9i change, the rest of the snapper is pretty much the same as earlier, with some minor bugfixes and additions.

You can download it from here.

If you want to get the most out of snapper, read this article here (and make sure you look inside the script!)

Update: As the beginning says, this article was meant as something interesting about Oracle’s internals and CTRL+C / OCICancel() handling. There’s a more practical way for canceling session calls if you are running Oracle Enterprise Edition and are currently using resource manager:

You can set the consumer group for a session to CANCEL_SQL to cancel its current call:

DBMS_RESOURCE_MANAGER.SWITCH_CONSUMER_GROUP_FOR_SESS (
session_id IN NUMBER,
session_serial IN NUMBER,
consumer_group IN VARCHAR2);

Thanks to commenter “null” for this info. Note that I haven’t tested how/whether this feature works correctly so there’s homework for you ;-)

I recently received a question about how to cancel queries running in another Oracle session, so that the session would not be killed, but would remain alive.

Well, there’s no supported way I can tell you, but thanks to how Oracle handles out-of-band breaks on Unix platforms, you can cancel database calls using an OS tool – kill.

Before we go on, here’s how query cancellation (pressing CTRL+C in sqlplus for example) works in Oracle:

1. The user presses CTRL+C or clicks some button in App which runs OCICancel()
2. The client process sends an urgent TCP packet (which is a regular TCP packet with URG bit set) to the socket in the database server it is connected to
3. The server OS TCP stack receives the urgent TCP packet and sends URGENT signal to the process which owns the socket (the Oracle server process)
4. Unix interrupts whatever the current process was doing and calls the URGENT signal handler function (which Oracle has registered during process startup)
5. The urgent signal handler blindly assumes that the urgent signal has been received because user wants to cancel the query, stops the execution and returns back with an error: ORA-01013: user requested cancel of current operation

So, if we can’t make our application send the break packet, OCICancel() then we can just send the SIGURG signal to the Oracle process just like the OS TCP stack would do when it receives the packet with urgent bit set.

Here’s an example:

In one session I’m running a DBMS_STATS call:

SQL> exec dbms_stats.gather_database_stats;

I identify the SPID of that session’s process and send an URG signal to that process:

kill -URG 4476

And the call gets cancelled in the other session:

SQL> exec dbms_stats.gather_database_stats;
<pre>BEGIN dbms_stats.gather_database_stats; END;

*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
ORA-06512: at "SYS.DBMS_STATS", line 13336
ORA-06512: at "SYS.DBMS_STATS", line 13682
ORA-06512: at "SYS.DBMS_STATS", line 13826
ORA-06512: at "SYS.DBMS_STATS", line 13790
ORA-06512: at line 1

My session was not killed – I still can run queries in it:

SQL> select * from dual;

D
-
X

SQL>

This works only on Unix platforms. Also this does not work when your client application is Windows sqlplus! This is because Windows sqlplus does not set up the out-of-band break checking properly when connecting. Maybe this is because old Windows versions TCP stacks didn’t know anything about urgent TCP packets! :)

A word of warning – this stuff is not for your everyday production usage! While it works and we know how and why it works, it’s not a good idea to send “random” signals to Oracle processes at your will. So the better way is to make your application able to cancel its database calls when you want it, but well in real world its not always (or should I even say rarely) possible.

Another thing to consider is when you run Oracle with Direct NFS, there will be network connections to the NFS server used by your server process, in addition to the client-server communication. I haven’t tested what happens when you send URG packet to a process in the DNFS case.

So try this out at your own risk ;-)

If you want to know more about query cancelling and what the in-band and out-of-band break checking is then you can read one of my old blog posts about it:

• http://blog.tanelpoder.com/2008/02/05/oracle-hidden-costs-revealed-part-1/

The problem with V$PROCESS* views (and the V$SESSTAT) is that they will tell you what Oracle thinks it’s using, not the real amount of memory used. There will be discrepancies due how memory is actually allocated in OS, libraries out of Oracle’s control, the static memory areas inside Oracle binary and of course bugs.

I was working on one of such problems and decided to make my life easier by writing the script. It’s not so much about calculating the exact figures (they will never be 100% correct), but more about presenting the memory usage data in a better and more convenient fashion.

The script is called procmm and stands for Process Memory Matrix as it shows the memory usage in a matrix grid.

Here’s an example output to show what I’m talking about:

oracle@solaris02:~/research/memory$ ./procmm.sh -t `pgrep -f ora_.*SOL102`

-- procmm.sh: Process Memory Matrix v1.01 by Tanel Poder ( http://tech.e2sn.com )
-- All numbers are shown in kilobytes

Total PIDs 17, working: .................

PID            SEGMENT_TYPE      VIRTUAL          RSS         ANON       LOCKED    SWAP_RSVD
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                       lib       389844       388796        13180            0        17816
0                    oracle      1629064      1628908         3336            0        42012
0            ism_shmid=0x1d      6963336      6963336            0      6963336            0
0             hc_SOL102.dat           48           48            0            0            0
0                      anon        32936        15936        15452            0        32868
0                     stack         1660         1628         1592            0         1660
0                      heap        37004        18016        16844            0        37004
------ -------------------- ------------ ------------ ------------ ------------ ------------
0                 TOTAL(kB)      9053892      9016668        50404      6963336       131360

And here’s the full article and in there a link to the script:

• http://tech.e2sn.com/oracle/performance/unix-performance-tools/process-memory-matrix

Comments are welcome here, as I haven’t set up commenting on my other site yet…

You can follow me here:

http://twitter.com/tanelpoder

Or just check out the right hand twitter widget in my blog…

I in addition to useful (and less cool) stuff I showed some useful (and very cool) stuff there too. I demoed my PerfSheet tool and also something completely new what I’ve been working on for last few months. Alex has uploaded couple of videos in to youtube, if you’re interested what I’ve been up to in last few months, check them out here:

http://www.pythian.com/news/2590/sydney-oracle-meetup-2-visualizing-oracle-performance

I have been extremely busy over last couple of months, that’s why there haven’t been any blog entries (no, I haven’t ran out of good ideas ;-)
I just managed to find some time on a day-time 9-hour flight from Shanghai to Finland, thus here’s a blog entry about something what I had wanted to write about for long time… I’m talking about the PerfSheet tool I wrote a year ago and have been showing at few conferences already. It has been a great time saver for me over this year when working through performance data for troubleshooting or capacity planning tasks.

If you don’t want to read further through my comments, just download PerfSheet from http://www.tanelpoder.com/files/PerfSheet.zip and see how it works yourself!

For others, let me give some history first:

1. A client had some serious performance problems after an HBA driver upgrade on a weekend.
2. DBAs were blamed first and they started looking into Statspack reports (as that was the only performance history data they had).
3. Current Statspack reports didn’t show anything too extraordinary (as everybody knew this database was running quite non-optimally anyway).
4. By that time, both storage and Unix admins “had done the healthchecks and everything looked fine”
5. But something was definitely different today as the end user response times were much worse.
6. (This is a case where we didn’t care about general inefficiency of the application as this is what the application had been like all the time. Instead of systematically starting optimizing the database we just had to find what had gone wrong over the weekend and fix it. In other words, something had got F-ed up over the weekend and we needed to un-F it ;).
7. Thus the troubleshooters (including me) started comparing the “current” statspack reports with previous ones to see what were the main differences in wait and load profiles were.
8. But it wasn’t as easy as just taking last days snapshot from the same time as the load & wait profile of that system was fluctuating anyway as some heavily parallel batch jobs started and finished (at different times).
9. Based on the statspack numbers the DBAs suspected that there was something wrong with IO configuration (especially as there had been a HBA driver upgrade over the weekend)
10. Taking tens of 50-page statspack reports to a fingerpointing meeting with SAs, storage admins, high level directors etc didn’t work well…

Then I thought to visualize the numbers in Excel to get our point (and proof) through better.

Running Statspack reports manually and pasting numbers from there into Excel very boring, slow and error prone task. I actually started doing this but after 10 (painful) minutes I thought, heck, why am I doing all this manually?

This is how PerfSheet was born. I wrote the first version in couple of hours and it just ran an analytic query (downloaded the query from Tim Gorman’s site) against statspack repository. After fetching the SQL resultset to Excel, the sheet also automatically displayed the results in a graph (so we could refresh the graph after querying different stats out of statspack repository).

And armed with the wait profile graph above we were able to show hard evidence in a simple package (as opposed to tens of statspack reports) at the next meeting with storage and Unix teams. It’s very hard to argue with evidence like that, especially if managers are involved :) It’s much easier to discard a DBA with 500 pages of statspack reports blabbering about Oracle wait interface than someone who shows a visual overview of database health.

This led the Unix SAs to do another healthcheck and this time they found out that one of the two HBAs had been left deactivated after the driver upgrade. After the HBA was activated (online), the performance went back to normal.

So, sometimes, especially if multiple teams are involved in troubleshooting a sudden peformance problem (and there’s negative fingerpointing and protectionism going on), it’s hard to beat historical performance visualization. Instead of going through tens of pages of raw numbers you can just visualize that data and see if your eye catches a sudden change, trend or outlier.

And this leads me to version 2.0 (beta) of PerfSheet.

It allows you to do ad-hoc visualization of any SQL resultset. Excel runs your specified SQL statement, fetches data onto a worksheet and shows the results on a PivotChart. I have actually written only few lines of code to glue Excel’s SQL execution, data fetching and visualization modules together for convenience, Excel itself does most of the work.

When you open up the PerfSheet.xls, just fill in the connect details on the first page and click “Show me” and see what I mean (note that some queries do query v$active_session_history and DBA_HIST tables so make sure you have the proper licenses for querying these views).

Once the graph is displayed you can even drag and drop the data fields around into different axis to get a different breakdown of the statistics. Try it out and you’ll love it :)
Also, you can hover your mouse over some data element to get its description, as I’ve done here to see a SQLID matching an ASH sample on an ASH query:

As a next step, just navigate around at the Queries, Views and Help tabs and read the comments at column headers on how to define your own queries and visualization views. And if it doesn’t work, let me know.

Also, one more important thing. I used Excel for data fetching and visualization and I uset it on purpose. I haven’t seen a corporation yet where there is no Windows workstation and Excel available. Often I hear people saying that I should just use some Perl/Python/TCL/Java module or toolkit suth RRD etc as then you can put your graphs into web and “everybody has a web browser”.

I say that the mentioned technologies are for entirely different purpose. Yes, everybody does have a web browser, but In a corporation it’s not easy at all to get a (virtual) webserver for your purpose (and get all the software downloaded, configured and installed). When you have a serious production problem to troubleshoot, you don’t want to start installing and configuring webservers :)

The webserver approach is good for proactive, predetermined visualization, e.g. you know what you want to monitor and you make that data available in the intranet. But PerfSheet is perfect (thus the name :) for ad-hoc performance visualization of any SQL resultset on pretty much any corporate workstation.

So, Merry Christmas and happy visualization in the next year ;)

I will deliver an 1 hour version of my Advanced Oracle Troubleshooting presentation there, so if you’re in Austria and want to see what’s coming in my full length seminar I deliver on 3.-4. Nov also in Vienna, then you can see a preview on this Wednesday :)

There are multiple ways for getting stack traces for Oracle, but not all ways are equal. Some give you more contextual info, some less, but what I’m blogging about today is that some ways are less safe than others.

I was using pstack on Linux for diagnosing an IO related performance issue. I executed a create table as select statement and ran pstack in a loop for getting stack traces from the running process.

However in one of the test runs I got following error in my Oracle session:

SQL> create table t as select * from dba_source;
create table t as select * from dba_source
                                *
ERROR at line 1:
ORA-01115: IO error reading block from file 1 (block # 11161)
ORA-01110: data file 1: '/u01/oradata/LIN10G/system01.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Additional information: 3
Additional information: 11145
Additional information: 32768

I suspected that this issue was due Linux pstack, stopped the pstack script and ran my CTAS from the same Oracle session again:

SQL> create table t as select * from dba_source;

Table created.

The command now succeeded ok.

I tried to reproduce the failure again, but during the few minutes I spent testing, it didn’t occur again.

The failure happened likely due the fact that pstack on Linux is actually just a wrapper script around gdb. GDB in turn suspends the process under investigation and attaches to it through ptrace() syscall. And ptrace() syscall (and debuggers in general) have historically caused issues to host processes when communicating with kernel and other processes. For example they can block some signals or interrupts from being propagated back up to the “host” process.

Normally I have warned people about using debugger-based stack tracing due exactly those reasons, and now I managed to capture nice evidence.
I recommend to stay away with debugger from critical background processes on production systems, unless things have already collapsed anyway (that ought to be common sense anyway). So it’s good to know that Linux pstack is actually just a script with GDB backtrace command in it.

So, what are the safer and less safer stack tracing options:

• oradebug dump errorstack – unsafe for production (as dump errorstack actually alters the process under investigation from its original codepath)
• debugger based errorstack (gdb,dbx,mdb and Linux pstack) – can be unsafe for production due missed signals & interrupts if you get unlucky. Therefore you should stay away from at least the critical background processes with such tools
• pstack on Solaris (and procstack on AIX) – safe as they don’t use the ptrace() interface but just read the info required from /proc filesystem
• DTrace – safe by design

I haven’t checked how HP-UX pstack works, so can’t advise on that.