What the hell, shouldn’t the INDEX/TABLE access be the other way around?!

Also, how come it’s TABLE ACCESS FULL (and not by INDEX ROWID) in there?

This question is with a little gotcha, but can you come up with a query which produced such plan? ;-)

----------------------------------------------
| Id  | Operation          | Name   | E-Rows |
----------------------------------------------
|   0 | SELECT STATEMENT   |        |        |
|*  1 |  INDEX RANGE SCAN  | PK_EMP |      1 |
|*  2 |   TABLE ACCESS FULL| EMP    |      1 |
----------------------------------------------

I have uploaded the test case I used to my new website, to a section where I will upload some of my demo scripts which I show at my seminars (and people can download & test these themselves too):

http://tech.e2sn.com/oracle-seminar-demo-scripts

Basically what I do is this:

1. I run the query with bind variable values where only a handful of rows match the filter condition. Thus Oracle picks nested loop join (and indexed access path)
2. Then I run the same query with different bind values, where a lot of rows match the filter condition. Oracle reuses existing execution plan (with nested loops!!!). Oracle ends up looping through a lot of blocks again and again (because nested loop visits the “right” side of the join once for every row coming from the “left” side of the join).

Using nested loops over lots of rows is a sure way to kill your performance.

And an interesting thing with my script is that the problem still happens in Oracle 11.1 and 11.2 too!

Oracle 11g has Adaptive Cursor Sharing, right? This should take care of such a problem, right? Well no, adaptive bind variable peeking is a reactive technique – it only kicks in after the problem has happened!

So feel free to download the script, review it and test it out!

I have updated new seminar dates and cities in m new webpage:

• http://tech.e2sn.com/oracle-training-seminars

From April 2010 I offer total 3 different seminars

• Advanced Oracle Troubleshooting v2.0 (3 days)
• Advanced Oracle SQL Tuning (3 days)
• Partitioning and Parallel Execution for Performance (1 day)

I have rearranged the Advanced Oracle Troubleshooting class based on customer feedback, removed some content, added new content and I think this deserves a new version number, 2.0.

Also, I created an entirely new class Advanced Oracle SQL Tuning which should provide the same for SQL tuners that my Advanced Oracle Troubleshooting class has provided for database troubleshooters. This class will not start with CBO concepts and how SQL execution might work in theory, insead we will start from going very deep into understanding how Oracle really executes SQL execution plans and what is the data flow order and hierarchy in the execution plan tree.

From there we go on into learning how to read execution plans of any complexity and how to control SQL execution plans – how to make them do exactly what we want. And CBO topics will come in the end – by then the CBO fundamental concepts such as Cardinality, Density and IO/CPU Cost will make good sense and are not just some arbitrary names for some magic numbers coming from the optimizer ;-)

In addition, I separated Parallel Execution and Partitioning topics (which not everyone is using) into a separate 1-day seminar, Oracle Partitioning and Parallel Execution for Performance, which I usually deliver right after the 3-day SQL tuning seminar.

In coming days I also plan to upload some SQL tuning related content to tech.e2sn.com to show the quality of the upcoming seminar ;-)

So, feel free to check out the seminar dates and descriptions here:

• http://tech.e2sn.com/oracle-training-seminars

It’s time to publish the newsite now with an application demo rototype which gives some clue of what kind of features will there be in the secret project I’ve been working on for several months with my friend and business partner.

The website is located here:

http://tech.e2sn.com

E2SN does have a meaning, but I’ll leave it a secret for now ( you are free to guess ;-)

So, there’s not much technical content at the site yet, but there’s a cool online app which you should check if you deal with SQL tuning and execution plan analysis much.

It’s called PlanViz, Oracle Execution Plan Visualization app, you can check it out here:

http://tech.e2sn.com/apps/planviz

Oh, I’ve also created something called a “Living Book” into my website, where I will write about Oracle, performance, troubleshooting, etc. There is also a place where people can request what I should write about there!

And that’s all for today!

So, you better understand how NULLs interact with your SQL constructs if you call yourself a DBA or a database developer ;-)

Here’s another example about how misunderstanding NULLs may cause your application to return different results than what was intended.

I will create a little table with TWO rows in it:

SQL> create table t(a int);

Table created.

SQL> insert into t values(1);

1 row created.

SQL> insert into t values(null);

1 row created.

SQL> select avg(a) from t;

 AVG(A)
----------
         1

When I take an average of the 2 values in these rows I get average of 1.

Now lets update the NULL (no value) to 0 (an actual value of zero).

SQL> update t set a=0 where a is null;

1 row updated.

SQL> select avg(a) from t;

 AVG(A)
----------
        .5

As you see, as we now have an actual value in the other row (as opposed to “no value”), the AVG function takes that zero into account.

Hopefully this illustrates once more that NULL does not mean zero or any other value, it means NO value. If you do aggregation functions (count,avg) over NULLs then you must understand that Oracle treats NULLs as no value and doesn’t account these “no values”, thus your queries may behave differently than what your intuition might say (and yes its always good to read documentation about what exactly a given SQL construction/function does in the given database engine instead of relying on “common sense”).

I replied this, but thought to show an example test case of this problem too:

1) The optimizer statistics the EXPLAIN PLAN ends up using are different
from the statistics the other session ended up using

2) Explain plan does not use bind variable peeking thus will not optimize
for current bind variable values

3) Explain plan treats all bind variables as VARCHAR2, thus you ma have
implicit datatype conversion happening during the plan execution, (meaning
to_char,to_number functions are added around variables/columns) and this for
example may make optimizer to ignore some indexes if you get unlucky.

…Of course explain plan doesn’t really
execute the plan so the implicit datatype conversion you see is in the
explained plan only, but if you actually execute the statement (with correct
bind datatypes) then there’s no implicit datatype conversion. And that’s
where the difference comes from…

And here comes an example of condition number 3 above. Lets use a little bit of bad design out there and put numeric values into varchar2 columns:

SQL> create table t (id varchar2(10), name varchar2(100));

Table created.

SQL> insert into t select to_char(object_id), object_name from dba_objects;

51449 rows created.

Now we add a little index for lookup performance and gather stats:

SQL> create index i on t(id);

Index created.

SQL> exec dbms_stats.gather_table_stats(user,'T',cascade=>true);

PL/SQL procedure successfully completed.

Now lets define a bind variable of NUMBER type and set a value for it:

SQL> var x number
SQL>
SQL> exec :x:=99999

PL/SQL procedure successfully completed.

Now lets use “explain plan for” to estimate the execution plan:

SQL> explain plan for
 2  select sum(length(name)) from t where id >  :x;

Explained.

SQL> select * from table(dbms_xplan.display) ;

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------------
Plan hash value: 3694077449

-------------------------------------------------------------------------------------
| Id  | Operation                    | Name | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |     1 |    29 |    56   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE              |      |     1 |    29 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| T    |  2572 | 74588 |    56   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | I    |   463 |       |     3   (0)| 00:00:01 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 3 - access("ID">:X)

15 rows selected.

Explain plan command nicely reports that we’d be using an index range scan, which would be a good thing to do given my test data and search condition.

Now lets actually run the statement and see the REAL execution plan actually used for the execution. I’ll use dbms_xplan.display_CURSOR for this. If you don’t pass SQL_ID/child into that function it will just report the last SQL statement executed in your current session. But the key difference between the dbms_xplan.DISPLAY and DISPLAY_CURSOR is that the latter goes to library cache and fetches the actual SQL plan used from there. The explain plan command just reparses the statement and estimates a plan, ignoring any bind variable values and assuming that all bind variables are of type varchar2:

SQL> select sum(length(name)) from t where id >  :x;

SUM(LENGTH(NAME))
-----------------

SQL> select * from table(dbms_xplan.display_cursor);

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------
SQL_ID  7zm570j6kj597, child number 0
-------------------------------------
select sum(length(name)) from t where id >  :x

Plan hash value: 2966233522

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |    60 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    29 |            |          |
|*  2 |   TABLE ACCESS FULL| T    |  2572 | 74588 |    60   (5)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 2 - filter(TO_NUMBER("ID")>:X)

19 rows selected.

Whatta? We actually used a full table scan!

2nd part will follow soon :-)

Oracle is actually returning correct results, its just that sometimes the people asking the questions haven’t realized that COUNT(column) is something semantically different from COUNT(*).

COUNT(*) operation counts all rows fed to it by execution plan branch under it.

COUNT(column) operation on the other hand counts all non-null values in that column from rows fed to it by execution plan branch under it.

And here’s a little example:

SQL> select count(*) from v$session;

 COUNT(*)
----------
        23

SQL> select count(username) from v$session;

COUNT(USERNAME)
---------------
              1  <<-- only one non-null value in that column

SQL> select count(nvl(username,'blah')) from v$session; <<-- lets replace NULLs with non-nulls

COUNT(NVL(USERNAME,'BLAH'))
---------------------------
                         23

Here are few scripts which allow you to query SQL execution plans and their execution statistics out from V$SQL_PLAN and V$SQL_PLAN_STATISTICS yourself.

Why would you want to do this as there’s the DBMS_XPLAN.DISPLAY_CURSOR() function in Oracle 10g?

Well, my scripts work also on Oracle 9.2. Also they give you better detail (and flexibility) compared to DBMS_XPLAN.DISPLAY_CURSOR.

Of course in 9i there’s the DBMS_XPLAN.DISPLAY function which you can use in conjuction with EXPLAIN PLAN FOR command, but that approach has problems. EXPLAIN PLAN treats all bind variables as varchar2 datatype, possibly causing the reported execution plan to be different from reality. Also, if you have a long running report from last night and you run explain plan today morning, then if optimizer statistics have changed meanwhile, the explain plan command might come up with a different execution plan again, which is different from reality.

So that’s why in 10g we have the DBMS_XPLAN.DISPLAY_CURSOR which goes directly to required library cache child cursor and extracts (unparses) the execution plan from there. The function uses V$SQL_PLAN% views as its data source. And guess what – these views are there in version 9.2 already! And thats’ where my scripts come in:

Here’s an example. Let’s set statistics_level=all so we get rowsource level execution stats for the cursor (note that this parameter makes your query consume much more CPU so it should only be used at session level for troubleshooting a specific performance issue):

SQL> alter session set statistics_level = all;

Session altered.

SQL> select count(*) from all_users;

  COUNT(*)
----------
        36

I know the hash value of this query, so lets report its execution plan, directly from library cache. This is the REAL execution plan inside that child cursor, not some estimate like EXPLAIN PLAN command gives:

SQL> @xm 3416239794 %

Ch Pr   Op                                                         Objcect                        Optimizer Optim rows Optim bytes Optimizer
ld ed   ID Operation                                               Name                                Cost  from step   from step Mode
-- -- ---- ------------------------------------------------------- ------------------------------ --------- ---------- ----------- ----------
 0       0 SELECT STATEMENT                                                                              14                        ALL_ROWS
         1  SORT AGGREGATE                                                                                           1          15
   A     2   HASH JOIN                                                                                   14         36         540
   A     3    HASH JOIN                                                                                   9         36         432
         4     TABLE ACCESS FULL                                   TS$                                    5         13          39
    F    5     TABLE ACCESS FULL                                   USER$                                  3         36         324
         6    TABLE ACCESS FULL                                    TS$                                    5         13          39

7 rows selected.

Ch Op
ld ID     Predicate Information (identified by operation id):
-- ------ ----------------------------------------------------------------------------------------------------
 0     2  - access("U"."TEMPTS#"="TTS"."TS#")
       3  - access("U"."DATATS#"="DTS"."TS#")
       5  - filter("U"."TYPE#"=1)

XM means eXplain from Memory :)

As we had enabled rowsource level execution stats collection, we can use another script to report the response time and resource consumption at execution plan line level (if you’re interested how Oracle achieves this, check the last part of this blog post)

SQL> @xmsh 3416239794 %

SQL hash value:          3416239794    Cursor address:                  00000003A41E5290    |   Statement first parsed at: 2009-05-26/23:02:16  |  200 seconds ago

 Ch Pr   Op                                                         Object                           ms spent in     Estimated Real #rows  Rowsource Consistent    Current   Physical   Physical    Optimizer
 ld ed   ID Operation                                               Name                               operation   output rows   returned     starts       gets       gets      reads     writes         Cost
--- -- ---- ------------------------------------------------------- ------------------------------ ------------- ------------- ---------- ---------- ---------- ---------- ---------- ---------- ------------
  0       0 SELECT STATEMENT                                                                                                                                                                               14
          1  SORT AGGREGATE                                                                                 2.49             1          1          1         42          0          0          0
    A     2   HASH JOIN                                                                                     2.43            36         36          1         42          0          0          0           14
    A     3    HASH JOIN                                                                                    1.87            36         36          1         24          0          0          0            9
          4     TABLE ACCESS FULL                                   TS$                                      .12            13         13          1         18          0          0          0            5
     F    5     TABLE ACCESS FULL                                   USER$                                    .06            36         36          1          6          0          0          0            3
          6    TABLE ACCESS FULL                                    TS$                                      .05            13         13          1         18          0          0          0            5

 Ch Op
 ld ID     Predicate Information (identified by operation id):
--- ------ ----------------------------------------------------------------------------------------------------
  0     2  - access("U"."TEMPTS#"="TTS"."TS#")
        3  - access("U"."DATATS#"="DTS"."TS#")
        5  - filter("U"."TYPE#"=1)

XMSH means eXplain from Memory with Statistics by Hash value (well it did make sense to me when I wrote the script, you can rename it to something else if you like :)

You may need to make your browser window pretty wide to see the full output to see all the columns reported. I normally run sqlplus in Windows cmd.exe and make the screen buffer wide, so the output doesn’t wrap.

If you want to read what these output columns mean, read my presentation about how SQL plan execution works.

As I mentioned, the above scripts run also on Oracle 9.2 (that’s why I wrote them years ago).

Also there’s one more script, xms.sql which doesn’t take any parameters and shows you the execution plan of the latest SQL executed in current session. It uses v$session.prev_hash_value for determining this. However on Oracle 9.2 the prev_hash_value seems to be incorrectly reset and my script would display you wrong SQL statements execution plan. In 10g onwards Oracle has fixed this problem though. In 9i you need to identify the hash value of the statement and use xmsh.sql instead.

In 10g the xms works ok:

SQL> select count(*) from all_tables;

  COUNT(*)
----------
      1739

1 row selected.

SQL> @xms

SQL hash value:          3565202713    Cursor address:                  00000003A4311930    |   Statement first parsed at: 2009-05-26/23:05:59  |  2 seconds ago

 Ch Pr   Op                                          Object                             ms spent   Estimated Real #rows  Rowsource Consistent    Current   Physical   Physical  Optimizer
 ld ed   ID Operation                                Name                                 in op. output rows   returned     starts       gets       gets      reads     writes       Cost
--- -- ---- ---------------------------------------- ------------------------------ ------------ ----------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
  0       0 SELECT STATEMENT                                                                                                                                                          558
          1  SORT AGGREGATE                                                               418.63           1          1          1       4559          0          1          0
     F    2   FILTER                                                                      416.78                   1739          1       4559          0          1          0
    A     3    HASH JOIN RIGHT OUTER                                                      387.23        1750       1739          1       2568          0          0          0        558
          4     TABLE ACCESS FULL                    USER$                                   .12          74         74          1          6          0          0          0          3
    A     5     HASH JOIN OUTER                                                           377.55        1750       1739          1       2562          0          0          0        555
          6      NESTED LOOPS OUTER                                                       225.00        1750       1739          1       1913          0          0          0        410
    A     7       HASH JOIN                                                               218.02        1750       1739          1       1819          0          0          0        410
          8        TABLE ACCESS FULL                 USER$                                   .11          74         74          1          6          0          0          0          3
    A     9        HASH JOIN                                                              206.61        1750       1739          1       1813          0          0          0        406
    A    10         HASH JOIN RIGHT OUTER                                                  41.63        1750       1750          1       1164          0          0          0        260
         11          TABLE ACCESS FULL               SEG$                                   5.28        5249       5249          1        156          0          0          0         35
    A    12          HASH JOIN                                                             17.97        1750       1750          1       1008          0          0          0        224
         13           MERGE JOIN CARTESIAN                                                  6.62          13         13          1         18          0          0          0          6
    A    14            HASH JOIN                                                            6.40           1          1          1          0          0          0          0          1
     F   15             FIXED TABLE FULL             X$KSPPI                                 .95           1          1          1          0          0          0          0          0
         16             FIXED TABLE FULL             X$KSPPCV                               1.45        1440       1440          1          0          0          0          0          0
         17            BUFFER SORT                                                           .20          13         13          1         18          0          0          0          6
         18             TABLE ACCESS FULL            TS$                                     .09          13         13          1         18          0          0          0          5
     F   19           TABLE ACCESS FULL              TAB$                                   3.55        1750       1750          1        990          0          0          0        218
     F   20         TABLE ACCESS FULL                OBJ$                                  55.26       53718      53716          1        649          0          0          0        146
    A    21       INDEX UNIQUE SCAN                  I_OBJ1                                 3.36           1         92       1739         94          0          0          0          0
         22      TABLE ACCESS FULL                   OBJ$                                   1.26       53728      53727          1        649          0          0          0        144
         23    NESTED LOOPS                                                                25.93           1         69        995       1991          0          1          0          2
    A    24     INDEX RANGE SCAN                     I_OBJAUTH1                            22.25           1        149        995       1991          0          1          0          2
     F   25     FIXED TABLE FULL                     X$KZSRO                                 .91           1         69        149          0          0          0          0          0
     F   26      FIXED TABLE FULL                    X$KZSPR                                 .05          14          1          1          0          0          0          0          0

 Ch Op
 ld ID     Predicate Information (identified by operation id):
--- ------ ----------------------------------------------------------------------------------------------------
  0     2  - filter(("O"."OWNER#"=USERENV('SCHEMAID') OR  IS NOT NULL OR  IS NOT NULL))
        3  - access("CX"."OWNER#"="CU"."USER#")
        5  - access("T"."DATAOBJ#"="CX"."OBJ#")
        7  - access("O"."OWNER#"="U"."USER#")
        9  - access("O"."OBJ#"="T"."OBJ#")
       10  - access("T"."FILE#"="S"."FILE#" AND "T"."BLOCK#"="S"."BLOCK#" AND "T"."TS#"="S"."TS#")
       12  - access("T"."TS#"="TS"."TS#")
       14  - access("KSPPI"."INDX"="KSPPCV"."INDX")
       15  - filter("KSPPI"."KSPPINM"='_dml_monitoring_enabled')
       19  - filter(BITAND("T"."PROPERTY",1)=0)
       20  - filter(BITAND("O"."FLAGS",128)=0)
       21  - access("T"."BOBJ#"="CO"."OBJ#")
       24  - access("OA"."OBJ#"=:B1)
       25  - filter("GRANTEE#"="KZSROROL")
       26  - filter((((-"KZSPRPRV")=(-45) OR (-"KZSPRPRV")=(-47) OR (-"KZSPRPRV")=(-48) OR (-"KZSPRPRV")=(-49)
           OR (-"KZSPRPRV")=(-50)) AND "INST_ID"=USERENV('INSTANCE')))

I haven’t had a need to adjust the scripts to fix the problem what Jonathan Lewis has described earlier, but it should be easily doable (just need to make sure that the connect by recursive loop is done against a materialized resultset of v$sql_plan, not the view itself to avoid excessive latching).

The scripts are downloadable from here:

• xm.sql – explain from Memory
• xmsh.sql – eXplain from Memory lookup by Hash value
• xms.sql – eXplain from Memory (latest SQL in current session)

Update:

If you are on 10g and want to know how to convert SQL_ID’s to hash values for use with my XM scripts, read this:

http://blog.tanelpoder.com/2009/02/22/sql_id-is-just-a-fancy-representation-of-hash-value/

ORA-7445[XTYQBCB] OCCURS DURING EXECUTING SQL THAT USES 60000 BIND VARIABLES.

Wow! That’s about 100 times more bind variables in a single query than what I’ve seen in past. And I thought that query was bad!!! :)

I suspect this is a massive IN list passed to a query. Maybe it’s time to use a temporary table or a collection for passing in the IN values instead?

http://www.tanelpoder.com/files/TPT_public.zip

In the zip file there are over 300 Oracle sql scripts which I use for my everyday work. Also the demo scripts I show at Advanced Oracle Troubleshooting seminar are in there as well (in aot subdirectory).

To my seminar attendees in New York this week: I had a typo in the TPT script download link, the link above is the correct one!