• https://blogs.oracle.com/UPGRADE/entry/new_version_of_the_huge

It has lots of examples (from real customer upgrade cases) in it.

Thanks to Randolf Geist for telling me about this.

You can also download other interesting presentations from that page (check the right hand side download section), like the Oracle Database Behavior Changes 8.0 through 11.2 doc. Pretty cool!

The top google hits so far only return tips to disable the new editor completely, but I want to use the new one, just in a bigger window! There is a very easy workaround for that – and there’s no need to switch back to the old compose mode at all!

If you are using your mouse, then just:

1. Hold down SHIFT key when clicking the Compose button to get a stand-alone new window for composing a message.
2. Hold down CTRL (on Windows) or Command (on Mac OSX) key to get a full-screen new browser tab for composing a message.

Examples 1 & 2 below:

If you use Gmail keyboard shortcuts for productivity (they’re awesome!), then you can just:

1. Press “c” for the new (small) compose box
2. Press SHIFT-C (capital “C”) for the stand-alone new window for composing a message
3. Press D for a full screen new browser tab for composing a message

Note that you can view the Gmail shortcuts reference, if this feature is enabled, just by pressing the ? (question mark) key:

So, there’s no need to switch back to the old compose mode completely, just remember SHIFT, CTRL and the “D” keys :)

• 16 April 2013 – UKOUG Engineered Systems Summit
  • London, UK
  • I will talk about the common Exadata performance lessons learned in my “4 years of Exadata performance” talk
  • Andy Colvin and Martin Bach of Enkitec will also speak there (yes, Martin has joined Enkitec in Europe!!! :-)
• 30 April 2013 – NYOUG Training Day
  • Manhattan, NYC
  • This is the first ever public delivery of my new Getting the Most Out of Oracle’s Active Session History, Time Model and Wait events seminar!
• 2-3 May 2013 – Advanced Exadata Performance seminar – in-classroom!
  • First public delivery of this class – and we’ll be going very deep into Exadata internals and performance ;-)
  • Irving, TX (in Enkitec HQ)
• 13-16 May 2013 – Advanced Exadata Performance seminar – online
  • Same as previous, but online.
• 10-11 June 2013 – My new ASH seminar – online
  • Getting the Most Out of Oracle’s Active Session History, Time Model and Wait events
  • More details will appear in the training & seminar page soon
• 13-14 June 2013 – Oracle Data Warehouse Global Leaders forum (by invitation-only event run by Oracle)
  • Amsterdam, Netherlands
  • I will speak about my “4 years of Exadata performance” experience and probably learn from others’ experience too
• 5-6 Aug 2013 – Enkitec Extreme Exadata Expo (E4) conference
  • Irving, TX
  • I will be speaking there, haven’t set the exact topic yet, but it will include demos and hacking something I suspect ;-)
• 22-26 September 2013 – Oracle OpenWorld conference
  • San Francisco, CA
  • I will attend the ACE Directors briefing before the conference, likely speak, hang out at Enkitec booth, probably hack something at Enkitec’s Conference HQ for fun and definitely drink beer there. Maybe I’ll even attend the Wednesday’s party for a change!

As you see, my great plans to not travel much are not going to work out well :)

Actually it is better, this year I’ve managed to only travel twice so far (and one of the trips was for vacation!) and I haven’t had to do too many of the crazy around-the-world in 5 days trips I did when living in back Singapore… but looks like I’ll get to 2 million km nevertheless this year:

Well, see you in some corner of the world some day!

SQL> @hint merge

NAME                                                             VERSION                   VERSION_OUTLINE           INVERSE
---------------------------------------------------------------- ------------------------- ------------------------- ----------------------------------------------------------------
MERGE_CONST_ON                                                   8.0.0
MERGE_AJ                                                         8.1.0                     8.1.7
MERGE_SJ                                                         8.1.0                     8.1.7
MV_MERGE                                                         9.0.0
MERGE                                                            8.1.0                     10.1.0                    NO_MERGE
NO_MERGE                                                         8.0.0                     10.1.0                    MERGE
USE_MERGE_CARTESIAN                                              11.1.0.6                  11.1.0.6
USE_MERGE                                                        8.1.0                     8.1.7                     NO_USE_MERGE
NO_USE_MERGE                                                     10.1.0.3                  10.1.0.3                  USE_MERGE

But there’s more, (semi)undocumented views like V$SQL_FEATURE and V$SQL_FEATURE_HIERARCHY do give us more information about what these hints relate to. For example, if you have ever wondered why is there a MERGE hint and then also a USE_MERGE hint, you can check what do these hints control using my hinth.sql (Hint Hierarchy) script:

SQL> @hinth MERGE
Display Hint feature hierarchy for hints like MERGE

NAME                                                             PATH
---------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
MERGE                                                            ALL -> COMPILATION -> CBO -> CBQT -> CVM
MERGE                                                            ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> CVM
MERGE                                                            ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> CVM

So, the MERGE hints seem to affect the CBO’s query transformation code – (CBQT means Cost-Based Query Transformation and CVM means Complex View Merging, but more about that later).

SQL> @hinth USE_MERGE
Display Hint feature hierarchy for hints like USE_MERGE

NAME                                                             PATH
---------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
USE_MERGE                                                        ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_MERGE

And the USE_MERGE hint is about controlling the use of a join method – the sort-merge join.

Let’s list all hints having NL in them:

SQL> @hinth %NL%
Display Hint feature hierarchy for hints like %NL%

NAME                                                             PATH
---------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
INLINE_XMLTYPE_NT                                                ALL
NL_SJ                                                            ALL -> COMPILATION -> CBO
NL_AJ                                                            ALL -> COMPILATION -> CBO
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_USE_NL                                                        ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL                                                           ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL_WITH_INDEX                                                ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL_WITH_INDEX
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_CONNECT_BY_CB_WHR_ONLY                                        ALL -> COMPILATION -> TRANSFORMATION
CONNECT_BY_CB_WHR_ONLY                                           ALL -> COMPILATION -> TRANSFORMATION
INLINE                                                           ALL -> COMPILATION -> TRANSFORMATION
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_NLJ_BATCHING                                                  ALL -> EXECUTION
NLJ_BATCHING                                                     ALL -> EXECUTION
NO_NLJ_PREFETCH                                                  ALL -> EXECUTION
NLJ_PREFETCH                                                     ALL -> EXECUTION

Plenty of interesting stuff here – the new hint TABLE_LOOKUP_BY_NL that has showed up recently seems to have to do with star transformations for example (I just learned this myself from this output).

Interestingly the NLJ_BATCHING and NLJ_PREFETCH hints are considered as execution phase hints apparently (that was my term, I’m thinking about hints (also) affecting a decision in the execution phase, not just during optimization). For example, normally the NLJ prefetch feature can be dynamically turned on & off during the query execution, I guess with a hint this feature would be always enabled (I’m not sure about this here, just trying to reason why a hint is shown to be related to “execution” phase).

If optimizer feature terms like CBQT and CVM do not immediately ring a bell, you can use the V$SQL_FEATURE view (or my sqlfh.sql script) to list some more info about what these SQL feature name abbreviations mean and where in the hierarchy does this particular feature stand.

The script below doesn’t accept any parameters, prints out the entire SQL feature hierarchy (except the temporary bugfix features you can see from V$SYSTEM_FIX_CONTROL):

SQL> @sqlfh

SQL_FEATURE                                             DESCRIPTION
------------------------------------------------------- ----------------------------------------------------------------
ALL                                                     A Universal Feature
  COMPILATION                                           SQL COMPILATION
    CBO                                                 SQL Cost Based Optimization
      ACCESS_PATH                                       Query access path
        AND_EQUAL                                       Index and-equal access path
        BITMAP_TREE                                     Bitmap tree access path
        FULL                                            Full table scan
        INDEX                                           Index
        INDEX_ASC                                       Index (ascending)
        INDEX_COMBINE                                   Combine index for bitmap access
        INDEX_DESC                                      Use index (descending)
        INDEX_FFS                                       Index fast full scan
        INDEX_JOIN                                      Index join
        INDEX_RS_ASC                                    Index range scan
        INDEX_RS_DESC                                   Index range scan descending
        INDEX_SS                                        Index skip scan
        INDEX_SS_ASC                                    Index skip scan ascending
        INDEX_SS_DESC                                   Index skip scan descending
        SORT_ELIM                                       Sort Elimination Via Index
      CBQT                                              Cost Based Query Transformation
        CVM                                             Complex View Merging
        DIST_PLCMT                                      Distinct Placement
        JOINFAC                                         Join Factorization
        JPPD                                            Join Predicate Push Down
        PLACE_GROUP_BY                                  Group-By Placement
        PULL_PRED                                       pull predicates
        STAR_TRANS                                      Star Transformation
          TABLE_LOOKUP_BY_NL                            Table Lookup By Nested Loop
        TABLE_EXPANSION                                 Table Expansion
        UNNEST                                          unnest query block
      CURSOR_SHARING                                    Cursor sharing
      DML                                               DML
      JOIN_METHOD                                       Join methods
        USE_HASH                                        Hash join
        USE_MERGE                                       Sort-merge join
        USE_MERGE_CARTESIAN                             Merge join cartesian
        USE_NL                                          Nested-loop join
        USE_NL_WITH_INDEX                               Nested-loop index join
      JOIN_ORDER                                        Join order
      OPT_MODE                                          Optimizer mode
        ALL_ROWS                                        All rows (optimizer mode)
        CHOOSE                                          Choose (optimizer mode)
        FIRST_ROWS                                      First rows (optimizer mode)
      OR_EXPAND                                         OR expansion
      OUTLINE                                           Outlines
      PARTITION                                         Partition
      PQ                                                Parallel Query
        PARALLEL                                        Parallel table
        PQ_DISTRIBUTE                                   PQ Distribution method
        PQ_MAP                                          PQ slave mapper
        PX_JOIN_FILTER                                  Bloom filtering for joins
      STAR_TRANS                                        Star Transformation
        TABLE_LOOKUP_BY_NL                              Table Lookup By Nested Loop
      STATS                                             Optimizer statistics
        CARDINALITY                                     Cardinality computation
        COLUMN_STATS                                    Basic column statistics
        CPU_COSTING                                     CPU costing
        DBMS_STATS                                      Statistics gathered by DBMS_STATS
        DYNAMIC_SAMPLING                                Dynamic sampling
        DYNAMIC_SAMPLING_EST_CDN                        Estimate CDN using dynamic sampling
        GATHER_PLAN_STATISTICS                          Gather plan statistics
        INDEX_STATS                                     Basic index statistics
        OPT_ESTIMATE                                    Optimizer estimates
        TABLE_STATS                                     Basic table statistics
    QUERY_REWRITE                                       query rewrite with materialized views
    RBO                                                 SQL Rule Based Optimization
    SQL_CODE_GENERATOR                                  SQL Code Generator
    SQL_PLAN_MANAGEMENT                                 SQL Plan Management
    TRANSFORMATION                                      Query Transformation
      CBQT                                              Cost Based Query Transformation
        CVM                                             Complex View Merging
        DIST_PLCMT                                      Distinct Placement
        JOINFAC                                         Join Factorization
        JPPD                                            Join Predicate Push Down
        PLACE_GROUP_BY                                  Group-By Placement
        PULL_PRED                                       pull predicates
        STAR_TRANS                                      Star Transformation
          TABLE_LOOKUP_BY_NL                            Table Lookup By Nested Loop
        TABLE_EXPANSION                                 Table Expansion
        UNNEST                                          unnest query block
      HEURISTIC                                         Heuristic Query Transformation
        CNT                                             Count(col) to count(*)
        COALESCE_SQ                                     coalesce subqueries
        CSE                                             Common Sub-Expression Elimination
        CVM                                             Complex View Merging
        FILTER_PUSH_PRED                                Push filter predicates
        FULL_OUTER_JOIN_TO_OUTER                        Join Conversion
        JPPD                                            Join Predicate Push Down
        OBYE                                            Order-by Elimination
        OLD_PUSH_PRED                                   Old push predicate algorithm (pre-10.1.0.3)
        OUTER_JOIN_TO_ANTI                              Join Conversion
        OUTER_JOIN_TO_INNER                             Join Conversion
        PRED_MOVE_AROUND                                Predicate move around
        SET_TO_JOIN                                     Transform set operations to joins
        SVM                                             Simple View Merging
        TABLE_ELIM                                      Table Elimination
        UNNEST                                          unnest query block
        USE_CONCAT                                      Or-optimization
    XML_REWRITE                                         XML Rewrite
      CHECK_ACL_REWRITE                                 Check ACL Rewrite
      COST_XML_QUERY_REWRITE                            Cost Based XML Query Rewrite
      XMLINDEX_REWRITE                                  XMLIndex Rewrite
  EXECUTION                                             SQL EXECUTION

I highlighted the CVM and CBQT lines above…

Just for reference (and if you’re too lazy to run these scripts yourself), I’ve pasted the full output of the hint feature hierarchy script too (executed in my 11.2.0.3 DB):

SQL> @hinth %
Display Hint feature hierarchy for hints like %

NAME                                                             PATH
---------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
NO_XDB_FASTPATH_INSERT                                           ALL
XDB_FASTPATH_INSERT                                              ALL
NO_USE_HASH_GBY_FOR_PUSHDOWN                                     ALL
USE_HASH_GBY_FOR_PUSHDOWN                                        ALL
XMLINDEX_SEL_IDX_TBL                                             ALL
NO_DST_UPGRADE_INSERT_CONV                                       ALL
DST_UPGRADE_INSERT_CONV                                          ALL
NO_CONNECT_BY_ELIM_DUPS                                          ALL
CONNECT_BY_ELIM_DUPS                                             ALL
NO_MONITOR                                                       ALL
MONITOR                                                          ALL
NO_NATIVE_FULL_OUTER_JOIN                                        ALL
NATIVE_FULL_OUTER_JOIN                                           ALL
NO_CONNECT_BY_COMBINE_SW                                         ALL
CONNECT_BY_COMBINE_SW                                            ALL
OPT_PARAM                                                        ALL
OUTLINE_LEAF                                                     ALL
OUTLINE                                                          ALL
NO_CARTESIAN                                                     ALL
INCLUDE_VERSION                                                  ALL
RESTRICT_ALL_REF_CONS                                            ALL
NO_ACCESS                                                        ALL
HASH                                                             ALL
DRIVING_SITE                                                     ALL
CACHE_TEMP_TABLE                                                 ALL
QB_NAME                                                          ALL
NO_STATS_GSETS                                                   ALL
NO_USE_HASH_AGGREGATION                                          ALL
USE_HASH_AGGREGATION                                             ALL
NO_MODEL_PUSH_REF                                                ALL
MODEL_NO_ANALYSIS                                                ALL
SCN_ASCENDING                                                    ALL
TIV_GB                                                           ALL
PIV_GB                                                           ALL
TIV_SSF                                                          ALL
PIV_SSF                                                          ALL
NO_CONNECT_BY_FILTERING                                          ALL
CONNECT_BY_FILTERING                                             ALL
BYPASS_RECURSIVE_CHECK                                           ALL
SYS_RID_ORDER                                                    ALL
NO_BASETABLE_MULTIMV_REWRITE                                     ALL
NO_MULTIMV_REWRITE                                               ALL
REMOTE_MAPPED                                                    ALL
NO_GBY_PUSHDOWN                                                  ALL
GBY_PUSHDOWN                                                     ALL
IGNORE_OPTIM_EMBEDDED_HINTS                                      ALL
DB_VERSION                                                       ALL
OPTIMIZER_FEATURES_ENABLE                                        ALL
USE_WEAK_NAME_RESL                                               ALL
IGNORE_WHERE_CLAUSE                                              ALL
INLINE_XMLTYPE_NT                                                ALL
NESTED_TABLE_FAST_INSERT                                         ALL
NESTED_TABLE_SET_SETID                                           ALL
PRESERVE_OID                                                     ALL
NESTED_TABLE_GET_REFS                                            ALL
DEREF_NO_REWRITE                                                 ALL
NO_SQL_TUNE                                                      ALL
NO_MONITORING                                                    ALL
NO_OUTER_JOIN_TO_ANTI                                            ALL -> COMPILATION -> CBO
OUTER_JOIN_TO_ANTI                                               ALL -> COMPILATION -> CBO
NO_FULL_OUTER_JOIN_TO_OUTER                                      ALL -> COMPILATION -> CBO
FULL_OUTER_JOIN_TO_OUTER                                         ALL -> COMPILATION -> CBO
APPEND_VALUES                                                    ALL -> COMPILATION -> CBO
NUM_INDEX_KEYS                                                   ALL -> COMPILATION -> CBO
NO_DOMAIN_INDEX_FILTER                                           ALL -> COMPILATION -> CBO
DOMAIN_INDEX_FILTER                                              ALL -> COMPILATION -> CBO
NO_PARTIAL_COMMIT                                                ALL -> COMPILATION -> CBO
SKIP_UNQ_UNUSABLE_IDX                                            ALL -> COMPILATION -> CBO
X_DYN_PRUNE                                                      ALL -> COMPILATION -> CBO
ROWID                                                            ALL -> COMPILATION -> CBO
CLUSTER                                                          ALL -> COMPILATION -> CBO
NO_SWAP_JOIN_INPUTS                                              ALL -> COMPILATION -> CBO
SWAP_JOIN_INPUTS                                                 ALL -> COMPILATION -> CBO
INDEX_RRS                                                        ALL -> COMPILATION -> CBO
NO_SUBQUERY_PRUNING                                              ALL -> COMPILATION -> CBO
SUBQUERY_PRUNING                                                 ALL -> COMPILATION -> CBO
USE_SEMI                                                         ALL -> COMPILATION -> CBO
USE_ANTI                                                         ALL -> COMPILATION -> CBO
QUEUE_ROWP                                                       ALL -> COMPILATION -> CBO
QUEUE_CURR                                                       ALL -> COMPILATION -> CBO
CACHE_CB                                                         ALL -> COMPILATION -> CBO
NO_PARALLEL                                                      ALL -> COMPILATION -> CBO
CURSOR_SHARING_EXACT                                             ALL -> COMPILATION -> CBO
NO_BUFFER                                                        ALL -> COMPILATION -> CBO
BUFFER                                                           ALL -> COMPILATION -> CBO
NO_QKN_BUFF                                                      ALL -> COMPILATION -> CBO
BITMAP                                                           ALL -> COMPILATION -> CBO
RESTORE_AS_INTERVALS                                             ALL -> COMPILATION -> CBO
SAVE_AS_INTERVALS                                                ALL -> COMPILATION -> CBO
CUBE_GB                                                          ALL -> COMPILATION -> CBO
SYS_PARALLEL_TXN                                                 ALL -> COMPILATION -> CBO
OVERFLOW_NOMOVE                                                  ALL -> COMPILATION -> CBO
HWM_BROKERED                                                     ALL -> COMPILATION -> CBO
LOCAL_INDEXES                                                    ALL -> COMPILATION -> CBO
BYPASS_UJVC                                                      ALL -> COMPILATION -> CBO
NL_SJ                                                            ALL -> COMPILATION -> CBO
HASH_SJ                                                          ALL -> COMPILATION -> CBO
MERGE_SJ                                                         ALL -> COMPILATION -> CBO
NL_AJ                                                            ALL -> COMPILATION -> CBO
HASH_AJ                                                          ALL -> COMPILATION -> CBO
MERGE_AJ                                                         ALL -> COMPILATION -> CBO
SEMIJOIN_DRIVER                                                  ALL -> COMPILATION -> CBO
SKIP_EXT_OPTIMIZER                                               ALL -> COMPILATION -> CBO
DOMAIN_INDEX_NO_SORT                                             ALL -> COMPILATION -> CBO
DOMAIN_INDEX_SORT                                                ALL -> COMPILATION -> CBO
ORDERED_PREDICATES                                               ALL -> COMPILATION -> CBO
ORDERED                                                          ALL -> COMPILATION -> CBO
FBTSCAN                                                          ALL -> COMPILATION -> CBO
MERGE_CONST_ON                                                   ALL -> COMPILATION -> CBO
STREAMS                                                          ALL -> COMPILATION -> CBO
EXPR_CORR_CHECK                                                  ALL -> COMPILATION -> CBO
VECTOR_READ_TRACE                                                ALL -> COMPILATION -> CBO
VECTOR_READ                                                      ALL -> COMPILATION -> CBO
DML_UPDATE                                                       ALL -> COMPILATION -> CBO
SQLLDR                                                           ALL -> COMPILATION -> CBO
SYS_DL_CURSOR                                                    ALL -> COMPILATION -> CBO
NO_REF_CASCADE                                                   ALL -> COMPILATION -> CBO
REF_CASCADE_CURSOR                                               ALL -> COMPILATION -> CBO
NOAPPEND                                                         ALL -> COMPILATION -> CBO
APPEND                                                           ALL -> COMPILATION -> CBO
AND_EQUAL                                                        ALL -> COMPILATION -> CBO -> ACCESS_PATH -> AND_EQUAL
BITMAP_TREE                                                      ALL -> COMPILATION -> CBO -> ACCESS_PATH -> BITMAP_TREE
FULL                                                             ALL -> COMPILATION -> CBO -> ACCESS_PATH -> FULL
NO_USE_INVISIBLE_INDEXES                                         ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX
USE_INVISIBLE_INDEXES                                            ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX
NO_INDEX                                                         ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX
INDEX                                                            ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX
INDEX_ASC                                                        ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_ASC
INDEX_COMBINE                                                    ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_COMBINE
INDEX_DESC                                                       ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_DESC
NO_INDEX_FFS                                                     ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_FFS
INDEX_FFS                                                        ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_FFS
INDEX_JOIN                                                       ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_JOIN
INDEX_RS_ASC                                                     ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_RS_ASC
INDEX_RS_DESC                                                    ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_RS_DESC
NO_INDEX_SS                                                      ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_SS
INDEX_SS                                                         ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_SS
INDEX_SS_ASC                                                     ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_SS_ASC
INDEX_SS_DESC                                                    ALL -> COMPILATION -> CBO -> ACCESS_PATH -> INDEX_SS_DESC
NO_MERGE                                                         ALL -> COMPILATION -> CBO -> CBQT -> CVM
MERGE                                                            ALL -> COMPILATION -> CBO -> CBQT -> CVM
NO_PLACE_DISTINCT                                                ALL -> COMPILATION -> CBO -> CBQT -> DIST_PLCMT
PLACE_DISTINCT                                                   ALL -> COMPILATION -> CBO -> CBQT -> DIST_PLCMT
NO_FACTORIZE_JOIN                                                ALL -> COMPILATION -> CBO -> CBQT -> JOINFAC
FACTORIZE_JOIN                                                   ALL -> COMPILATION -> CBO -> CBQT -> JOINFAC
NO_PLACE_GROUP_BY                                                ALL -> COMPILATION -> CBO -> CBQT -> PLACE_GROUP_BY
PLACE_GROUP_BY                                                   ALL -> COMPILATION -> CBO -> CBQT -> PLACE_GROUP_BY
NO_PULL_PRED                                                     ALL -> COMPILATION -> CBO -> CBQT -> PULL_PRED
PULL_PRED                                                        ALL -> COMPILATION -> CBO -> CBQT -> PULL_PRED
NO_FACT                                                          ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS
FACT                                                             ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS
NO_STAR_TRANSFORMATION                                           ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS
STAR_TRANSFORMATION                                              ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS
STAR                                                             ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_EXPAND_TABLE                                                  ALL -> COMPILATION -> CBO -> CBQT -> TABLE_EXPANSION
EXPAND_TABLE                                                     ALL -> COMPILATION -> CBO -> CBQT -> TABLE_EXPANSION
NO_UNNEST                                                        ALL -> COMPILATION -> CBO -> CBQT -> UNNEST
UNNEST                                                           ALL -> COMPILATION -> CBO -> CBQT -> UNNEST
NO_BIND_AWARE                                                    ALL -> COMPILATION -> CBO -> CURSOR_SHARING
BIND_AWARE                                                       ALL -> COMPILATION -> CBO -> CURSOR_SHARING
RETRY_ON_ROW_CHANGE                                              ALL -> COMPILATION -> CBO -> DML
CHANGE_DUPKEY_ERROR_INDEX                                        ALL -> COMPILATION -> CBO -> DML
IGNORE_ROW_ON_DUPKEY_INDEX                                       ALL -> COMPILATION -> CBO -> DML
NO_USE_HASH                                                      ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_HASH
USE_HASH                                                         ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_HASH
NO_USE_MERGE                                                     ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_MERGE
USE_MERGE                                                        ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_MERGE
USE_MERGE_CARTESIAN                                              ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_MERGE_CARTESIAN
NO_USE_NL                                                        ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL                                                           ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL_WITH_INDEX                                                ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL_WITH_INDEX
LEADING                                                          ALL -> COMPILATION -> CBO -> JOIN_ORDER
ALL_ROWS                                                         ALL -> COMPILATION -> CBO -> OPT_MODE -> ALL_ROWS
CHOOSE                                                           ALL -> COMPILATION -> CBO -> OPT_MODE -> CHOOSE
FIRST_ROWS                                                       ALL -> COMPILATION -> CBO -> OPT_MODE -> FIRST_ROWS
OR_EXPAND                                                        ALL -> COMPILATION -> CBO -> OR_EXPAND
NO_PARALLEL_INDEX                                                ALL -> COMPILATION -> CBO -> PQ
PARALLEL_INDEX                                                   ALL -> COMPILATION -> CBO -> PQ
NO_STATEMENT_QUEUING                                             ALL -> COMPILATION -> CBO -> PQ -> PARALLEL
STATEMENT_QUEUING                                                ALL -> COMPILATION -> CBO -> PQ -> PARALLEL
SHARED                                                           ALL -> COMPILATION -> CBO -> PQ -> PARALLEL
NOPARALLEL                                                       ALL -> COMPILATION -> CBO -> PQ -> PARALLEL
PQ_DISTRIBUTE                                                    ALL -> COMPILATION -> CBO -> PQ -> PQ_DISTRIBUTE
PQ_NOMAP                                                         ALL -> COMPILATION -> CBO -> PQ -> PQ_MAP
PQ_MAP                                                           ALL -> COMPILATION -> CBO -> PQ -> PQ_MAP
NO_PX_JOIN_FILTER                                                ALL -> COMPILATION -> CBO -> PQ -> PX_JOIN_FILTER
PX_JOIN_FILTER                                                   ALL -> COMPILATION -> CBO -> PQ -> PX_JOIN_FILTER
NO_FACT                                                          ALL -> COMPILATION -> CBO -> STAR_TRANS
STAR                                                             ALL -> COMPILATION -> CBO -> STAR_TRANS
FACT                                                             ALL -> COMPILATION -> CBO -> STAR_TRANS
NO_STAR_TRANSFORMATION                                           ALL -> COMPILATION -> CBO -> STAR_TRANS
STAR_TRANSFORMATION                                              ALL -> COMPILATION -> CBO -> STAR_TRANS
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
CARDINALITY                                                      ALL -> COMPILATION -> CBO -> STATS
TABLE_STATS                                                      ALL -> COMPILATION -> CBO -> STATS
INDEX_STATS                                                      ALL -> COMPILATION -> CBO -> STATS
COLUMN_STATS                                                     ALL -> COMPILATION -> CBO -> STATS
NO_CPU_COSTING                                                   ALL -> COMPILATION -> CBO -> STATS -> CPU_COSTING
CPU_COSTING                                                      ALL -> COMPILATION -> CBO -> STATS -> CPU_COSTING
DBMS_STATS                                                       ALL -> COMPILATION -> CBO -> STATS -> DBMS_STATS
DYNAMIC_SAMPLING                                                 ALL -> COMPILATION -> CBO -> STATS -> DYNAMIC_SAMPLING
DYNAMIC_SAMPLING_EST_CDN                                         ALL -> COMPILATION -> CBO -> STATS -> DYNAMIC_SAMPLING_EST_CDN
GATHER_PLAN_STATISTICS                                           ALL -> COMPILATION -> CBO -> STATS -> GATHER_PLAN_STATISTICS
OPT_ESTIMATE                                                     ALL -> COMPILATION -> CBO -> STATS -> OPT_ESTIMATE
RBO_OUTLINE                                                      ALL -> COMPILATION -> RBO
RULE                                                             ALL -> COMPILATION -> RBO
NO_PRUNE_GSETS                                                   ALL -> COMPILATION -> TRANSFORMATION
MODEL_DONTVERIFY_UNIQUENESS                                      ALL -> COMPILATION -> TRANSFORMATION
MODEL_PUSH_REF                                                   ALL -> COMPILATION -> TRANSFORMATION
MODEL_COMPILE_SUBQUERY                                           ALL -> COMPILATION -> TRANSFORMATION
MODEL_DYNAMIC_SUBQUERY                                           ALL -> COMPILATION -> TRANSFORMATION
MODEL_MIN_ANALYSIS                                               ALL -> COMPILATION -> TRANSFORMATION
NO_EXPAND_GSET_TO_UNION                                          ALL -> COMPILATION -> TRANSFORMATION
EXPAND_GSET_TO_UNION                                             ALL -> COMPILATION -> TRANSFORMATION
MV_MERGE                                                         ALL -> COMPILATION -> TRANSFORMATION
NO_CONNECT_BY_COST_BASED                                         ALL -> COMPILATION -> TRANSFORMATION
CONNECT_BY_COST_BASED                                            ALL -> COMPILATION -> TRANSFORMATION
INLINE                                                           ALL -> COMPILATION -> TRANSFORMATION
MATERIALIZE                                                      ALL -> COMPILATION -> TRANSFORMATION
REWRITE_OR_ERROR                                                 ALL -> COMPILATION -> TRANSFORMATION
NO_REWRITE                                                       ALL -> COMPILATION -> TRANSFORMATION
REWRITE                                                          ALL -> COMPILATION -> TRANSFORMATION
NO_SEMIJOIN                                                      ALL -> COMPILATION -> TRANSFORMATION
SEMIJOIN                                                         ALL -> COMPILATION -> TRANSFORMATION
ANTIJOIN                                                         ALL -> COMPILATION -> TRANSFORMATION
NO_PUSH_SUBQ                                                     ALL -> COMPILATION -> TRANSFORMATION
PUSH_SUBQ                                                        ALL -> COMPILATION -> TRANSFORMATION
NO_QUERY_TRANSFORMATION                                          ALL -> COMPILATION -> TRANSFORMATION
OPAQUE_XCANONICAL                                                ALL -> COMPILATION -> TRANSFORMATION
OPAQUE_TRANSFORM                                                 ALL -> COMPILATION -> TRANSFORMATION
NO_CONNECT_BY_CB_WHR_ONLY                                        ALL -> COMPILATION -> TRANSFORMATION
CONNECT_BY_CB_WHR_ONLY                                           ALL -> COMPILATION -> TRANSFORMATION
NO_TRANSFORM_DISTINCT_AGG                                        ALL -> COMPILATION -> TRANSFORMATION
TRANSFORM_DISTINCT_AGG                                           ALL -> COMPILATION -> TRANSFORMATION
PRECOMPUTE_SUBQUERY                                              ALL -> COMPILATION -> TRANSFORMATION
LIKE_EXPAND                                                      ALL -> COMPILATION -> TRANSFORMATION
NO_ORDER_ROLLUPS                                                 ALL -> COMPILATION -> TRANSFORMATION
GBY_CONC_ROLLUP                                                  ALL -> COMPILATION -> TRANSFORMATION
USE_TTT_FOR_GSETS                                                ALL -> COMPILATION -> TRANSFORMATION
MERGE                                                            ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> CVM
NO_MERGE                                                         ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> CVM
NO_PLACE_DISTINCT                                                ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> DIST_PLCMT
PLACE_DISTINCT                                                   ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> DIST_PLCMT
FACTORIZE_JOIN                                                   ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> JOINFAC
NO_FACTORIZE_JOIN                                                ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> JOINFAC
PLACE_GROUP_BY                                                   ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> PLACE_GROUP_BY
NO_PLACE_GROUP_BY                                                ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> PLACE_GROUP_BY
PULL_PRED                                                        ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> PULL_PRED
NO_PULL_PRED                                                     ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> PULL_PRED
NO_FACT                                                          ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS
NO_STAR_TRANSFORMATION                                           ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS
STAR_TRANSFORMATION                                              ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS
STAR                                                             ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS
FACT                                                             ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS
TABLE_LOOKUP_BY_NL                                               ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_TABLE_LOOKUP_BY_NL                                            ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_EXPAND_TABLE                                                  ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> TABLE_EXPANSION
EXPAND_TABLE                                                     ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> TABLE_EXPANSION
NO_UNNEST                                                        ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> UNNEST
UNNEST                                                           ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> UNNEST
NO_COALESCE_SQ                                                   ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> COALESCE_SQ
COALESCE_SQ                                                      ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> COALESCE_SQ
MERGE                                                            ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> CVM
NO_MERGE                                                         ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> CVM
NO_PUSH_PRED                                                     ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> FILTER_PUSH_PRED
PUSH_PRED                                                        ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> FILTER_PUSH_PRED
ELIMINATE_OBY                                                    ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> OBYE
NO_ELIMINATE_OBY                                                 ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> OBYE
OLD_PUSH_PRED                                                    ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> OLD_PUSH_PRED
NO_OUTER_JOIN_TO_INNER                                           ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> OUTER_JOIN_TO_INNER
OUTER_JOIN_TO_INNER                                              ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> OUTER_JOIN_TO_INNER
SET_TO_JOIN                                                      ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> SET_TO_JOIN
NO_SET_TO_JOIN                                                   ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> SET_TO_JOIN
ELIMINATE_JOIN                                                   ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> TABLE_ELIM
NO_ELIMINATE_JOIN                                                ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> TABLE_ELIM
NO_UNNEST                                                        ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> UNNEST
UNNEST                                                           ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> UNNEST
NO_EXPAND                                                        ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> USE_CONCAT
USE_CONCAT                                                       ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> USE_CONCAT
XML_DML_RWT_STMT                                                 ALL -> COMPILATION -> XML_REWRITE
NO_XML_DML_REWRITE                                               ALL -> COMPILATION -> XML_REWRITE
NO_XML_QUERY_REWRITE                                             ALL -> COMPILATION -> XML_REWRITE
FORCE_XML_QUERY_REWRITE                                          ALL -> COMPILATION -> XML_REWRITE
CHECK_ACL_REWRITE                                                ALL -> COMPILATION -> XML_REWRITE -> CHECK_ACL_REWRITE
NO_CHECK_ACL_REWRITE                                             ALL -> COMPILATION -> XML_REWRITE -> CHECK_ACL_REWRITE
NO_COST_XML_QUERY_REWRITE                                        ALL -> COMPILATION -> XML_REWRITE -> COST_XML_QUERY_REWRITE
COST_XML_QUERY_REWRITE                                           ALL -> COMPILATION -> XML_REWRITE -> COST_XML_QUERY_REWRITE
NO_XMLINDEX_REWRITE_IN_SELECT                                    ALL -> COMPILATION -> XML_REWRITE -> XMLINDEX_REWRITE
XMLINDEX_REWRITE_IN_SELECT                                       ALL -> COMPILATION -> XML_REWRITE -> XMLINDEX_REWRITE
NO_XMLINDEX_REWRITE                                              ALL -> COMPILATION -> XML_REWRITE -> XMLINDEX_REWRITE
XMLINDEX_REWRITE                                                 ALL -> COMPILATION -> XML_REWRITE -> XMLINDEX_REWRITE
NO_SUBSTRB_PAD                                                   ALL -> EXECUTION
NO_NLJ_BATCHING                                                  ALL -> EXECUTION
NLJ_BATCHING                                                     ALL -> EXECUTION
NO_NLJ_PREFETCH                                                  ALL -> EXECUTION
NLJ_PREFETCH                                                     ALL -> EXECUTION
CACHE                                                            ALL -> EXECUTION
NO_RESULT_CACHE                                                  ALL -> EXECUTION
RESULT_CACHE                                                     ALL -> EXECUTION
TRACING                                                          ALL -> EXECUTION
NOCACHE                                                          ALL -> EXECUTION
NO_LOAD                                                          ALL -> EXECUTION

305 rows selected.

Lots of hints to remember and to try out some day… nah, I’ll just run on modern Oracle versions and gather the stats properly ;-)

SQL> @exadata/cellver
Show Exadata cell versions from V$CELL_CONFIG....

CELLNAME             CELLSRV_VERSION      FLASH_CACHE_MODE     CPU_COUNT  UPTIME               KERNEL_VERSION                 MAKE_MODEL
-------------------- -------------------- -------------------- ---------- -------------------- ------------------------------ --------------------------------------------------
192.168.12.10        11.2.3.2.1           WriteBack            24         8 days, 2:07         2.6.32-400.11.1.el5uek         Oracle Corporation SUN FIRE X4270 M2 SERVER SAS
192.168.12.11        11.2.3.2.1           WriteBack            24         8 days, 2:06         2.6.32-400.11.1.el5uek         Oracle Corporation SUN FIRE X4270 M2 SERVER SAS
192.168.12.8         11.2.3.2.1           WriteBack            24         8 days, 2:06         2.6.32-400.11.1.el5uek         Oracle Corporation SUN FIRE X4270 M2 SERVER SAS
192.168.12.9         11.2.3.2.1           WriteBack            24         8 days, 2:06         2.6.32-400.11.1.el5uek         Oracle Corporation SUN FIRE X4270 M2 SERVER SAS

The output is pretty self-explanatory. One thing to note is that the CPU_COUNT is not the number of CPU cores, but just the number of “virtual” CPU threads presented to the OS, in this case 24 threads over 12 cores.

The script itself is simple, just extracting some XML values from the “CELL” type config records:

SQL> l
  1  SELECT
  2      cellname cv_cellname
  3    , CAST(extract(xmltype(confval), '/cli-output/cell/releaseVersion/text()') AS VARCHAR2(20))  cv_cellVersion
  4    , CAST(extract(xmltype(confval), '/cli-output/cell/flashCacheMode/text()') AS VARCHAR2(20))  cv_flashcachemode
  5    , CAST(extract(xmltype(confval), '/cli-output/cell/cpuCount/text()')       AS VARCHAR2(10))  cpu_count
  6    , CAST(extract(xmltype(confval), '/cli-output/cell/upTime/text()')         AS VARCHAR2(20))  uptime
  7    , CAST(extract(xmltype(confval), '/cli-output/cell/kernelVersion/text()')  AS VARCHAR2(30))  kernel_version
  8    , CAST(extract(xmltype(confval), '/cli-output/cell/makeModel/text()')      AS VARCHAR2(50))  make_model
  9  FROM
 10      v$cell_config  -- gv$ isn't needed, all cells should be visible in all instances
 11  WHERE
 12      conftype = 'CELL'
 13  ORDER BY
 14*     cv_cellname

I will add some more scripts to the exadata directory over the coming days.

Enjoy! :)

I’m writing a similar article just because the word FORCE in the ALTER SESSION FORCE PARALLEL QUERY syntax. Force should mean that some behavior would always happen (when possible), right? Let’s test:

SQL> CREATE TABLE t AS SELECT * FROM dba_objects;

Table created.

SQL> CREATE INDEX i ON t(owner);

Index created.

SQL> @gts t
Gather Table Statistics for table t...

PL/SQL procedure successfully completed.

Now let’s “force” the parallel query in my session, run the query and check the execution plan:

SQL> ALTER SESSION FORCE PARALLEL QUERY PARALLEL 2;

Session altered.

SQL> SELECT SUM(object_id) FROM t WHERE owner LIKE 'S%';

SUM(OBJECT_ID)
--------------
     979900956

SQL> @x

---------------------------------------------------------------------------
| Id  | Operation                    | Name | E-Rows |E-Bytes| Cost (%CPU)|
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |      |        |       |   186 (100)|
|   1 |  SORT AGGREGATE              |      |      1 |    12 |            |
|   2 |   TABLE ACCESS BY INDEX ROWID| T    |   6741 | 80892 |   186   (0)|
|*  3 |    INDEX RANGE SCAN          | I    |   6741 |       |    18   (0)|
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - access("OWNER" LIKE 'S%')
       filter("OWNER" LIKE 'S%')

The output shows a regular, serial execution plan!

Hmm, let’s increase the “forced” parallelism from 2 to 3 and run exactly the same query again:

SQL> ALTER SESSION FORCE PARALLEL QUERY PARALLEL 3;

Session altered.

SQL> SELECT SUM(object_id) FROM t WHERE owner LIKE 'S%';

SUM(OBJECT_ID)
--------------
     979900956

SQL> @x

------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | E-Rows |E-Bytes| Cost (%CPU)|    TQ  |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |        |       |   128 (100)|        |      |            |
|   1 |  SORT AGGREGATE        |          |      1 |    12 |            |        |      |            |
|   2 |   PX COORDINATOR       |          |        |       |            |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |      1 |    12 |            |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |      1 |    12 |            |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |   6741 | 80892 |   128   (0)|  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| T        |   6741 | 80892 |   128   (0)|  Q1,00 | PCWP |            |
------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter("OWNER" LIKE 'S%')

Now the query will get a parallel plan!

The reason for this behavior is that the FORCE parallel query syntax doesn’t really force Oracle to use a parallel plan, but rather just reduces optimizer cost estimates for full table scans (the higher the parallelism, the lower the FTS costs – Jonathan already has details about this in his blog entry, so I won’t replicate this). But the optimizer is still free to choose some other, non-parallel execution plan if that has a lower cost than the best parallel one!

So what happened above is that with “forced” parallel degree 2, the parallel full table scan plan must have had a higher cost than the serial index range scan (186), but once I increased the parallelism “factor” to 3, then the final cost of the parallel full table scan plan ended up being lower (128) than the best serial plan found.

This is a good example showing that both the PARALLEL hints and the FORCE PARALLEL session settings really just adjust a narrow set of optimizer cost computation inputs and don’t really fix the resulting execution plan. If you really want to fix an execution plan, you need to tie optimizer “hands” in every aspect with a full set of hints just like the stored profiles do. That way, even if there is a lower cost plan available, the optimizer doesn’t know about it as you’ve prohibited it from doing any calculations other than your hints direct it to.

Note that when testing this, your mileage may vary, depending on how much data you have in your test table (or rather in the optimizer stats for that table) plus system stats.

The main performance impact of the old GATHER_PLAN_STATISTICS / statistics_level = ALL instrumentation came from the fact that expensive timing (gettimeofday()) system calls were used for getting A-Times of row sources.

That’s why the _rowsource_statistics_sampfreq was introduced, to not have to get the timing every switch between row sources, but only at every 128th time (by default). This caused some interesting measurement (or rather extrapolation) errors where the parent row source seemingly used less time than all of its children combined. (By the way, depending on your platform and recency of it, lightweight “fast trap” system calls may be used for getting the current timestamp, which reduces the overhead somewhat – Solaris since long time ago and RHEL 5.4+ for example).

Anyway, the row source timing info in SQL Monitoring comes from ASH sql_plan_line samples, so this instrumentation is always enabled anyway – SQL Monitoring will just query ASH for these records. The other data in V$SQL_MONITOR views doesn’t cause huge overhead – it depends on your query, but a heavy logical IO nested loop (my lotslios.sql) consumed around 0.6% more CPU with MONITOR hint compared to running with NO_MONITOR hint. I tested it on a VM and ran it only a couple of times, so this might just be statistical error or due to some other background activity. Not significant overhead in other words.

Now this was about long running queries, but you should not go and enable SQL Monitoring for all your short (OLTP) queries with a hint now as the SQL Monitoring data has to be kept and updated in a shared pool memory structure – and if all of your 1000 sessions suddenly start updating their monitoring data for every execution of a SQL, you will end up waiting for Real-time plan statistics latch contention (and some CPU overhead because so frequent searching and purging of previous SQL execution’s monitoring data).

The asqlmon (ASH-based SQL Monitoring like report) assumes that the corresponding execution plan is still present in library cache, it will take SQL_ID and child_number as a parameter. It does not show only a single execution as the SQL Monitoring report does, but  rather a plan-line level breakdown across all executions of that SQL/child captured in ASH data (you can change the WHERE clause to change the scope of the measurement):

SQL> @ash/asqlmon 6c45rgjx1myt6 0 

   SECONDS Activity Visual       Line ID Parent ASQLMON_OPERATION                                                                                    SESSION EVENT                                                            OBJ_ALIAS_QBC_NAME                                                                                 ASQLMON_PREDICATES                                                                                   PROJECTION
---------- -------- ------------ ------- ------ ---------------------------------------------------------------------------------------------------- ------- ---------------------------------------------------------------- -------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                  % |          |       0        SELECT STATEMENT
         5   10.4 % |#         |       1      0  SORT AGGREGATE                                                                                      ON CPU                                                                    [SEL$1]                                                                                                                                                                                                (#keys=0) COUNT(*)[22]
         3    6.3 % |#         |       2      1   COUNT STOPKEY                                                                                      ON CPU                                                                                                                                                                      [F:]ROWNUM<=1000000000
                  % |          |       3      2    MERGE JOIN CARTESIAN
                  % |          |       4      3     VIEW  [DBA_SOURCE]                                                                                                                                                        DBA_SOURCE@SEL$1 [SET$1]
                  % |          |       5      4      UNION-ALL                                                                                                                                                                 [SET$1]
                  % |          |       6      5       FILTER                                                                                                                                                                   [SEL$335DD26A]                                                                                    [F:](("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR
                                                                                                                                                                                                                                                                                                                                 (INTERNAL_FUNCTION("O"."TYPE#") AND ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND
                                                                                                                                                                                                                                                                                                                                 "U"."TYPE#"<>2) OR ("U"."TYPE#"=2 AND
                                                                                                                                                                                                                                                                                                                                 "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))

                  % |          |       7      6        HASH JOIN                                                                                                                                                                                                                                                                 [A:] "O"."OBJ#"="S"."OBJ#"                                                                           (#keys=1) "O"."OBJ#"[NUMBER,22], "U"."SPARE2"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |       8      7         HASH JOIN                                                                                                                                                                                                                                                                [A:] "O"."SPARE3"="U"."USER#"                                                                        (#keys=1) "U"."SPARE2"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "O"."OBJ#"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |       9      8          INDEX FULL SCAN [I_USER2]                                                                                                                                            U@SEL$2 [SEL$335DD26A]                                                                                                                                                                                  "U"."USER#"[NUMBER,22]
                  % |          |      10      8          HASH JOIN                                                                                                                                                                                                                                                               [A:] "O"."OWNER#"="U"."USER#"                                                                        (#keys=1) "U"."SPARE2"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "O"."OBJ#"[NUMBER,22], "O"."SPARE3"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |      11     10           INDEX FULL SCAN [I_USER2]                                                                                                                                           U@SEL$3 [SEL$335DD26A]                                                                                                                                                                                  "U"."USER#"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "U"."SPARE2"[NUMBER,22]
         1    2.1 % |          |      12     10           INDEX FAST FULL SCAN [I_OBJ2]                                                              ON CPU                                                                   O@SEL$3 [SEL$335DD26A]                                                                             [F:](("O"."TYPE#"=13 AND "O"."SUBNAME" IS NULL) OR INTERNAL_FUNCTION("O"."TYPE#"))                   "O"."OBJ#"[NUMBER,22], "O"."OWNER#"[NUMBER,22], "O"."TYPE#"[NUMBER,22], "O"."SPARE3"[NUMBER,22]
                  % |          |      13      7         INDEX FAST FULL SCAN [I_SOURCE1]                                                                                                                                      S@SEL$2 [SEL$335DD26A]                                                                                                                                                                                  "S"."OBJ#"[NUMBER,22]
                  % |          |      14      6        NESTED LOOPS                                                                                                                                                            [SEL$5]
                  % |          |      15     14         INDEX SKIP SCAN [I_USER2]                                                                                                                                             U2@SEL$5 [SEL$5]                                                                                   [A:] "U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))         "U2"."USER#"[NUMBER,22]
                                                                                                                                                                                                                                                                                                                                 [F:]("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))

                  % |          |      16     14         INDEX RANGE SCAN [I_OBJ4]                                                                                                                                             O2@SEL$5 [SEL$5]                                                                                   [A:] "O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#"
                  % |          |      17      5       NESTED LOOPS                                                                                                                                                             [SEL$68B588A0]
                  % |          |      18     17        NESTED LOOPS                                                                                                                                                                                                                                                                                                                                                                   "O"."SPARE3"[NUMBER,22]
                  % |          |      19     18         NESTED LOOPS                                                                                                                                                                                                                                                                                                                                                                  "O"."OWNER#"[NUMBER,22], "O"."TYPE#"[NUMBER,22], "O"."SPARE3"[NUMBER,22]
                  % |          |      20     19          FIXED TABLE FULL [X$JOXSCD]                                                                                                                                          S@SEL$6 [SEL$68B588A0]                                                                                                                                                                                  "S"."JOXFTOBN"[NUMBER,22]
                  % |          |      21     19          TABLE ACCESS BY INDEX ROWID [OBJ$]                                                                                                                                   O@SEL$7 [SEL$68B588A0]                                                                                                                                                                                  "O"."OWNER#"[NUMBER,22], "O"."TYPE#"[NUMBER,22], "O"."SPARE3"[NUMBER,22]
                  % |          |      22     21           INDEX RANGE SCAN [I_OBJ1]                                                                                                                                           O@SEL$7 [SEL$68B588A0]                                                                             [A:] "O"."OBJ#"="S"."JOXFTOBN" AND "O"."TYPE#"=28 [F:]"O"."TYPE#"=28                                 "O".ROWID[ROWID,10], "O"."OWNER#"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |      23     18         INDEX RANGE SCAN [I_USER2]                                                                                                                                            U@SEL$7 [SEL$68B588A0]                                                                             [A:] "O"."OWNER#"="U"."USER#" [F:](("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND "O"."TYPE#"<>10 AND "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR
                                                                                                                                                                                                                                                                                                                                 BITAND("U"."SPARE1",16)=0)

                  % |          |      24     17        INDEX RANGE SCAN [I_USER2]                                                                                                                                             U@SEL$6 [SEL$68B588A0]                                                                             [A:] "O"."SPARE3"="U"."USER#"
        39   81.3 % |########  |      25      3     BUFFER SORT                                                                                      ON CPU                                                                                                                                                                                                                                                                           (#keys=0)
                  % |          |      26     25      VIEW  [DBA_OBJECTS]                                                                                                                                                      DBA_OBJECTS@SEL$1 [SET$2]
                  % |          |      27     26       UNION-ALL                                                                                                                                                                [SET$2]
                  % |          |      28     27        FILTER                                                                                                                                                                  [SEL$18BE6699]                                                                                    [F:](("O"."TYPE#"<>4 AND "O"."TYPE#"<>5 AND "O"."TYPE#"<>7 AND "O"."TYPE#"<>8 AND "O"."TYPE#"<>9 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>11 AND "O"."TYPE#"<>12 AND "O"."TYPE#"<>13 AND "O"."TYPE#"<>14 AND "O"."TYPE#"<>22 AND
                                                                                                                                                                                                                                                                                                                                 "O"."TYPE#"<>87 AND "O"."TYPE#"<>88) OR BITAND("U"."SPARE1",16)=0 OR (INTERNAL_FUNCTION("O"."TYPE#")
                                                                                                                                                                                                                                                                                                                                 AND ((SYS_CONTEXT('userenv','current_edition_name')='ORA$BASE' AND "U"."TYPE#"<>2) OR ("U"."TYPE#"=2
                                                                                                                                                                                                                                                                                                                                 AND "U"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))) OR  IS NOT NULL)))

                  % |          |      29     28         HASH JOIN                                                                                                                                                                                                                                                                [A:] "O"."SPARE3"="U"."USER#"                                                                        (#keys=1) "U"."SPARE2"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "O"."OBJ#"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |      30     29          INDEX FULL SCAN [I_USER2]                                                                                                                                            U@SEL$10 [SEL$18BE6699]                                                                                                                                                                                 "U"."USER#"[NUMBER,22]
                  % |          |      31     29          HASH JOIN                                                                                                                                                                                                                                                               [A:] "O"."OWNER#"="U"."USER#"                                                                        (#keys=1) "U"."SPARE2"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "O"."OBJ#"[NUMBER,22], "O"."SPARE3"[NUMBER,22], "O"."TYPE#"[NUMBER,22]
                  % |          |      32     31           INDEX FULL SCAN [I_USER2]                                                                                                                                           U@SEL$12 [SEL$18BE6699]                                                                                                                                                                                 "U"."USER#"[NUMBER,22], "U"."TYPE#"[NUMBER,22], "U"."SPARE1"[NUMBER,22], "U"."SPARE2"[NUMBER,22]
                  % |          |      33     31           TABLE ACCESS FULL [OBJ$]                                                                                                                                            O@SEL$12 [SEL$18BE6699]                                                                            [F:]("O"."TYPE#"<>10 AND BITAND("O"."FLAGS",128)=0 AND "O"."NAME"<>'_NEXT_OBJECT' AND                "O"."OBJ#"[NUMBER,22], "O"."OWNER#"[NUMBER,22], "O"."TYPE#"[NUMBER,22], "O"."SPARE3"[NUMBER,22]
                                                                                                                                                                                                                                                                                                                                 "O"."NAME"<>'_default_auditing_options_' AND "O"."LINKNAME" IS NULL)

                  % |          |      34     28         NESTED LOOPS                                                                                                                                                           [SEL$14]
                  % |          |      35     34          INDEX SKIP SCAN [I_USER2]                                                                                                                                            U2@SEL$14 [SEL$14]                                                                                 [A:] "U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id'))         "U2"."USER#"[NUMBER,22]
                                                                                                                                                                                                                                                                                                                                 [F:]("U2"."TYPE#"=2 AND "U2"."SPARE2"=TO_NUMBER(SYS_CONTEXT('userenv','current_edition_id')))

                  % |          |      36     34          INDEX RANGE SCAN [I_OBJ4]                                                                                                                                            O2@SEL$14 [SEL$14]                                                                                 [A:] "O2"."DATAOBJ#"=:B1 AND "O2"."TYPE#"=88 AND "O2"."OWNER#"="U2"."USER#"
                  % |          |      37     27        NESTED LOOPS                                                                                                                                                            [SEL$15]
                  % |          |      38     37         INDEX FULL SCAN [I_LINK1]                                                                                                                                             L@SEL$15 [SEL$15]                                                                                                                                                                                       "L"."OWNER#"[NUMBER,22]
                  % |          |      39     37         INDEX RANGE SCAN [I_USER2]                                                                                                                                            U@SEL$15 [SEL$15]                                                                                  [A:] "L"."OWNER#"="U"."USER#"

40 rows selected.

The output is deliberately very wide – as I like to have relevant info to a plan line on the same text line visually. I navigate around such plans by visually identifying where most of the time was spent (Activity% and Visual columns), then double or triple-click that line in my terminal window to highlight the whole line (it becomes like a marker) and then scroll left-right to see more info about it, such as the filter conditions and projection etc.

Note that this script does not give you all the info that a SQL Monitoring report (or the old-fashioned SQL Plan Profiling would give), like A-rows and Starts, because ASH just doesn’t provide data at this scope. But again, this approach works with frequently executed OLTP-style queries and does not require you to enable any session settings or add hints for getting quick overview of on which execution plan line does your SQL spend the most of its time.

Update: I also have a RAC-aware / GV$-using gasqlmon.sql script, but I haven’t kept it up to date for a while, so it might return incorrect output at the moment (there are some notable challenges as the child# of interest may not be the same on all instances, so for simplicity I look into a single instance at a time – non-parallel SQL tuning rarely requires a whole cluster overview in one script).

I have split the recording of this session into 3 pieces and uploaded to enkitec.tv. The ExaSnapper v0.7 beta that I demoed is also available now at enkitec.com (registration needed). See the links below.

For quick reference, here’s the syntax of running ExaSnapper – there are two modes, one is the before/after capture (think Tom Kyte’s runstats, but for exadata metrics) and the other is more like a DBA-monitoring mode, where you can just measure a few seconds worth of a long-running query runtime and get the IO and efficiency figures from there.

Here’s an excerpt from the install script documentation section:

-- Usage:       Take a snapshot of a running session (use QC SID if PX):
--
--                a) Monitor a running query - "DBA mode" 
--
--                   SELECT * FROM TABLE(exasnap.display_sid(, [snap_seconds], [detail_level]));
--
--                   The SID argument can be just a number (SID in local instance) or a remote SID with
--                   @instance after it (like '123@4')
--
--                   SELECT * FROM TABLE(exasnap.display_sid(123));
--                   SELECT * FROM TABLE(exasnap.display_sid('123@4', p_detail=>'%');
--
--                b) Take Before & After snapshots of a query execution - "Developer Mode"
--
--                   1) SELECT exasnap.begin_snap(123) FROM dual;
--                        or
--                      EXEC :begin_snap_id := exasnap.begin_snap(123);
-- 
--                   2) Run your query, wait until it finishes (or CTRL+C)
--
--                   3) SELECT exasnap.end_snap(123) FROM dual;
--                        or
--                      EXEC :end_snap_id := exasnap.end_snap(123);
--
--                   4) SELECT * FROM TABLE(exasnap.display_snap(:begin_snap_id, :end_snap_id, '%'));
--

One slide illustrating the idea of ExaSnapper:

One  screenshot of ExaSnapper output:

Watch the videos to see more!

Videos:

• http://enkitec.tv/2013/02/22/tanel-poders-exadata-snapper-hacking-session-videos/

Slides:

• http://blog.tanelpoder.com/files/Tanel_Poder_Drilling_Deep_Into_Exadata_Performance.pdf

ExaSnapper install script:

• http://www.enkitec.com/products/exasnapper

Also, if you’re serious about learning more about Exadata, from The Exadata Experts, check out our training pages , I’m running my Advanced Exadata Performance seminar the first time in May :-)

Troubleshooting a “slow” process

Here’s one systematic troubleshooting example I reproduced in my laptop. A DBA was wondering why their find command had been running “much slower”, without returning any results for a while. Knowing the environment, we had a hunch, but I got asked about what would be the systematic approach for troubleshooting this – already ongoing – problem right now.

Luckily the system was running OEL6, so had a pretty new kernel. Actually the 2.6.39 UEK2.

So, let’s do some troubleshooting. First let’s see whether that find process is still alive:

[root@oel6 ~]# ps -ef | grep find
root     27288 27245  4 11:57 pts/0    00:00:01 find . -type f
root     27334 27315  0 11:57 pts/1    00:00:00 grep find

Yep it’s there – PID 27288 (I’ll use that pid throughout the troubleshooting example).

Let’s start from the basics and take a quick look what’s the bottleneck for this process – if it’s not blocked by anything (for example reading everything it needs from cache) it should be 100% on CPU. If it’s bottlenecked by some IO or contention issues, the CPU usage should be lower – or completely 0%.

[root@oel6 ~]# top -cbp 27288
top - 11:58:15 up 7 days,  3:38,  2 users,  load average: 1.21, 0.65, 0.47
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2026460k total,  1935780k used,    90680k free,    64416k buffers
Swap:  4128764k total,   251004k used,  3877760k free,   662280k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27288 root      20   0  109m 1160  844 D  0.0  0.1   0:01.11 find . -type f

Top tells me this process is either 0% on CPU or very close to zero percent (so it gets rounded to 0% in the output). There’s an important difference though, as a process that is completely stuck, not having a chance of getting onto CPU at all vs. a process which is getting out of its wait state every now and then (for example some polling operation times out every now and then and thee process chooses to go back to sleep). So, top on Linux is not a good enough tool to show that difference for sure – but at least we know that this process is not burning serious amounts of CPU time.

Let’s use something else then. Normally when a process seems to be stuck like that (0% CPU usually means that the process is stuck in some blocking system call – which causes the kernel to put the process to sleep) I run strace on that process to trace in which system call the process is currently stuck. Also if the process is actually not completely stuck, but returns from a system call and wakes up briefly every now and then, it would show up in strace (as the blocking system call would complete and be entered again a little later):

[root@oel6 ~]# strace -cp 27288
Process 27288 attached - interrupt to quit

^C
^Z
[1]+  Stopped                 strace -cp 27288

[root@oel6 ~]# kill -9 %%
[1]+  Stopped                 strace -cp 27288
[root@oel6 ~]# 
[1]+  Killed                  strace -cp 27288

Oops, the strace command itself got hung too! It didn’t print any output for a long time and didn’t respond to CTRL+C, so I had to put it into background with CTRL+Z and kill it from there. So much for easy diagnosis.

Let’s try pstack then (on Linux, pstack is just a shell script wrapper around the GDB debugger). While pstack does not see into kernel-land, it will still give us a clue about which system call was requested (as usually there’s a corresponding libc library call in the top of the displayed userland stack):

[root@oel6 ~]# pstack 27288

^C
^Z
[1]+  Stopped                 pstack 27288

[root@oel6 ~]# kill %%
[1]+  Stopped                 pstack 27288
[root@oel6 ~]# 
[1]+  Terminated              pstack 27288

Pstack also got stuck without returning anything!

So we still don’t know whether our process is 100% (hopelessly) stuck or just 99.99% stuck (the process keeps going back to sleep) – and where it’s stuck.

Ok, where else to look? There’s one more commonly available thing to check – the process status and WCHAN fields, which can be reported via good old ps (perhaps I should have ran this command earlier to make sure this process isn’t a zombie by now):

[root@oel6 ~]# ps -flp 27288
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 D root     27288 27245  0  80   0 - 28070 rpc_wa 11:57 pts/0    00:00:01 find . -type f

You should run ps multiple times in a row to make sure that the process is still in the same state (you don’t want to get misled by a single sample taken at an unlucky time), but I’m displaying it here only once for brevity.

The process is in state D (uninterruptible sleep) which is usually disk IO related (as the ps man pages also say). And the WCHAN (the function which caused this sleep/wait in this process) field is a bit truncated. I could use a ps option (read the man) to print it out a bit wider, but as this info comes from the proc filesystem anyway, let’s query it directly from the source (again, it would be a good idea to sample this multiple times as we are not sure yet whether our process is completely stuck or just mostly sleeping):

[root@oel6 ~]# cat /proc/27288/wchan
rpc_wait_bit_killable

Hmm… this process is waiting for some RPC call. RPC usually means that the process is talking to other processes (either in the local server or even remote servers). But we still don’t know why.

Is there any movement or is the process completely stuck?

Before we go on to the “meat” of this article, let’s figure out whether this process is completely stuck or not. The /proc/PID/status can tell us that on modern kernels. I have highlighted the interesting bits for our purpose:

[root@oel6 ~]# cat /proc/27288/status 
Name:	find
State:	D (disk sleep)
Tgid:	27288
Pid:	27288
PPid:	27245
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	256
Groups:	0 1 2 3 4 6 10 
VmPeak:	  112628 kB
VmSize:	  112280 kB
VmLck:	       0 kB
VmHWM:	    1508 kB
VmRSS:	    1160 kB
VmData:	     260 kB
VmStk:	     136 kB
VmExe:	     224 kB
VmLib:	    2468 kB
VmPTE:	      88 kB
VmSwap:	       0 kB
Threads:	1
SigQ:	4/15831
SigPnd:	0000000000040000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	0000000180000000
CapInh:	0000000000000000
CapPrm:	ffffffffffffffff
CapEff:	ffffffffffffffff
CapBnd:	ffffffffffffffff
Cpus_allowed:	ffffffff,ffffffff
Cpus_allowed_list:	0-63
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	9950
nonvoluntary_ctxt_switches:	17104

The process is in the state D – Disk Sleep (uninterruptible sleep). And look into the number of voluntary_ctxt_switches and nonvoluntary_ctxt_switches – this tells you how many times the process was taken off CPU (or put back). Then run the same command again a few seconds later and see if those numbers increase. In my case these numbers did not increase, thus I could conclude that the process was completely stuck (well, at least during the few seconds between these commands). So I can be more confident now that this process is completely stuck (and it’s not just flying under the radar by burning 0.04% of CPU all time).

By the way, there are two more locations where to get the context switch counts (and the 2nd one works on ancient kernels as well):

[root@oel6 ~]# cat /proc/27288/sched
find (27288, #threads: 1)
---------------------------------------------------------
se.exec_start                      :     617547410.689282
se.vruntime                        :       2471987.542895
se.sum_exec_runtime                :          1119.480311
se.statistics.wait_start           :             0.000000
se.statistics.sleep_start          :             0.000000
se.statistics.block_start          :     617547410.689282
se.statistics.sleep_max            :             0.089192
se.statistics.block_max            :         60082.951331
se.statistics.exec_max             :             1.110465
se.statistics.slice_max            :             0.334211
se.statistics.wait_max             :             0.812834
se.statistics.wait_sum             :           724.745506
se.statistics.wait_count           :                27211
se.statistics.iowait_sum           :             0.000000
se.statistics.iowait_count         :                    0
se.nr_migrations                   :                  312
se.statistics.nr_migrations_cold   :                    0
se.statistics.nr_failed_migrations_affine:                    0
se.statistics.nr_failed_migrations_running:                   96
se.statistics.nr_failed_migrations_hot:                 1794
se.statistics.nr_forced_migrations :                  150
se.statistics.nr_wakeups           :                18507
se.statistics.nr_wakeups_sync      :                    1
se.statistics.nr_wakeups_migrate   :                  155
se.statistics.nr_wakeups_local     :                18504
se.statistics.nr_wakeups_remote    :                    3
se.statistics.nr_wakeups_affine    :                  155
se.statistics.nr_wakeups_affine_attempts:                  158
se.statistics.nr_wakeups_passive   :                    0
se.statistics.nr_wakeups_idle      :                    0
avg_atom                           :             0.041379
avg_per_cpu                        :             3.588077
nr_switches                        :                27054
nr_voluntary_switches              :                 9950
nr_involuntary_switches            :                17104
se.load.weight                     :                 1024
policy                             :                    0
prio                               :                  120
clock-delta                        :                   72

Here you need to look into the nr_switches number (which equals nr_voluntary_switches + nr_involuntary_switches).

The total nr_switches is 27054 in the above output and this is also what the 3rd field in /proc/PID/schedstat shows:

[root@oel6 ~]# cat /proc/27288/schedstat 
1119480311 724745506 27054

And it doesn’t increase…

Peeking into kernel-land using /proc filesystem

So, it looks like our process is pretty stuck :) Strace and pstack were useless. They use ptrace() system call for attaching to the process and peeking into its memory, but as the process was hopelessly stuck, very likely in some system call, so I guess the ptrace() call got hung itself. (BTW, I once tried stracing the strace process itself when attaching to a target process and it crashed the target. You have been warned :)

So, how to see in which system call are we stuck – without strace or pstack? Luckily I was running on a modern kernel – say hi to /proc/PID/syscall!

[root@oel6 ~]# cat /proc/27288/syscall 
262 0xffffffffffffff9c 0x20cf6c8 0x7fff97c52710 0x100 0x100 0x676e776f645f616d 0x7fff97c52658 0x390e2da8ea

Ok WTF am I supposed to do with that?
Well, these numbers usually stand for something. If it’s “0xAVeryBigNumber” then it’s usually a memory address (and pmap-like tools could be used where they point), but if it’s a small number then perhaps it’s an index into some array – like open file descriptor array (which you can read from /proc/PID/fd) or in this case, as we are dealing with system calls – it’s the system call number where your process is currently in! So, this process is stuck in system call #262!

Note that the system call numbers may differ between OS type, OS version and platform, so you should read it from the proper .h file on your flavor of OS. Usually searching for “syscall*” from /usr/include is a good starting point. On my version and platform (64bit) of Linux the system calls are defined here in /usr/include/asm/unistd_64.h:

[root@oel6 ~]# grep 262 /usr/include/asm/unistd_64.h 
#define __NR_newfstatat				262

Here you go! The syscall 262 is something called newfstatat. Just run man to find out what it is. Here’s a hint regarding system call names – if man page doesn’t find a call, then try it without possible suffixes and prefixes (for example “man pread” instead of “man pread64″) – in this case, run man without the “new” prefix – man fstatat. Or just google.

Anyway, this system call “new-fstat-at” allows you to read file properties pretty much like the usual stat system call. And we are stuck in this file metadata reading operation. So we’ve gotten one step further. But we still don’t know why are we stuck there?

Well, Say Hello To My Little Friend /proc/PID/stack, which allows you to read kernel stack backtraces of a process by just cat’ing a proc file!!!

[root@oel6 ~]# cat /proc/27288/stack
[] rpc_wait_bit_killable+0x24/0x40 [sunrpc]
[] __rpc_execute+0xf5/0x1d0 [sunrpc]
[] rpc_execute+0x43/0x50 [sunrpc]
[] rpc_run_task+0x75/0x90 [sunrpc]
[] rpc_call_sync+0x42/0x70 [sunrpc]
[] nfs3_rpc_wrapper.clone.0+0x35/0x80 [nfs]
[] nfs3_proc_getattr+0x47/0x90 [nfs]
[] __nfs_revalidate_inode+0xcc/0x1f0 [nfs]
[] nfs_revalidate_inode+0x36/0x60 [nfs]
[] nfs_getattr+0x5f/0x110 [nfs]
[] vfs_getattr+0x4e/0x80
[] vfs_fstatat+0x70/0x90
[] sys_newfstatat+0x24/0x50
[] system_call_fastpath+0x16/0x1b
[] 0xffffffffffffffff

The top function is where in kernel code we are stuck right now – which is exactly what the WCHAN output showed us (note that actually there are some more functions on top of it, such as the kernel schedule() function which puts processes to sleep or on CPU as needed, but these are not shown here probably because they’re a result of this wait condition itself, not the cause).

Thanks to the full kernel-land stack of this task we can read it from bottom up to understand how did we end up calling this rpc_wait_bit_killable function in the top, which ended up calling the scheduler and put us into sleep mode.

The system_call_fastpath in the bottom is a generic kernel system call handler function which has called the kernel code for that newfstatat syscall we issued (sys_newfstatat). And then, continuing to go upwards through the “child” functions we end up seeing a bunch of NFS functions in the stack! This is a 100% undeniable proof that we are somewhere under NFS codepath. I’m not saying “in NFS codepath” as when you continue reading upwards past the NFS functions, you see that the topmost NFS function has in turn called some RPC function (rpc_call_sync) for talking to some other process – in this case probably the [kworker/N:N], [nfsiod], [lockd] or [rpciod] kernel IO threads. And we are never getting a reply from those threads for some reason (the usual suspect would be dropped network connection, packets or just network connectivity issues).

To see whether any of the helper threads are stuck on network-related code, you could sample their kernel stacks too, although the kworkers do much more than just NFS RPC communication. During a separate experiment (just copying large files over NFS), I have caught one of the kworkers waiting in networking code:

[root@oel6 proc]# for i in `pgrep worker` ; do ps -fp $i ; cat /proc/$i/stack ; done
UID        PID  PPID  C STIME TTY          TIME CMD
root        53     2  0 Feb14 ?        00:04:34 [kworker/1:1]

[] __cond_resched+0x2a/0x40
[] lock_sock_nested+0x35/0x70
[] tcp_sendmsg+0x29/0xbe0
[] inet_sendmsg+0x48/0xb0
[] sock_sendmsg+0xef/0x120
[] kernel_sendmsg+0x41/0x60
[] xs_send_kvec+0x8e/0xa0 [sunrpc]
[] xs_sendpages+0x173/0x220 [sunrpc]
[] xs_tcp_send_request+0x5d/0x160 [sunrpc]
[] xprt_transmit+0x83/0x2e0 [sunrpc]
[] call_transmit+0xa8/0x130 [sunrpc]
[] __rpc_execute+0x66/0x1d0 [sunrpc]
[] rpc_async_schedule+0x15/0x20 [sunrpc]
[] process_one_work+0x13e/0x460
[] worker_thread+0x17c/0x3b0
[] kthread+0x96/0xa0
[] kernel_thread_helper+0x4/0x10

It should be possible to enable kernel tracing for knowing which exact kernel thread communicates with other kernel threads, but I won’t go in there in this post – as it’s supposed to be a practical (and simple :) troubleshooting exercise!

Diagnosis and “fix”

Anyway, thanks to very easy kernel stack sampling in modern Linux kernels (I don’t know exact version when it was introduced) we managed to systematically figure out where the find command was stuck – in NFS code in the Linux kernel. And when you have NFS-related hangs, the most usual suspects are network issues. If you are wondering how did I reproduce this problem, I mounted a NFS volume from a VM, started the find command and then just suspended the VM. This produces similar symptoms as a network (configuration, firewall) issue where a connection is silently dropped without notifying the TCP endpoints or packets just don’t go through for some reason.

As the top function listed in the stack was one of the “killable” safe-to-kill functions (rpc_wait_bit_killable) then we can kill it with kill -9:

[root@oel6 ~]# ps -fp 27288
UID        PID  PPID  C STIME TTY          TIME CMD
root     27288 27245  0 11:57 pts/0    00:00:01 find . -type f
[root@oel6 ~]# kill -9 27288

[root@oel6 ~]# ls -l /proc/27288/stack
ls: cannot access /proc/27288/stack: No such file or directory

[root@oel6 ~]# ps -fp 27288
UID        PID  PPID  C STIME TTY          TIME CMD
[root@oel6 ~]#

The process is gone.

Poor-man’s kernel thread profiling

Note that as the /proc/PID/stack looks like just a plain text proc file, you can do poor-man’s stack profiling also on the kernel threads! This is an example of sampling the current system call and the kernel stack (if in system call) and aggregating it into a semi-hierarchical profile in a poor-man’s way:

[root@oel6 ~]# export LC_ALL=C ; for i in {1..100} ; do cat /proc/29797/syscall | awk '{ print $1 }' ; cat /proc/29797/stack | /home/oracle/os_explain -k ; usleep 100000 ; done | sort -r | uniq -c 
     69 running
      1 ffffff81534c83
      2 ffffff81534820
      6 247
     25 180

    100    0xffffffffffffffff 
      1     thread_group_cputime 
     27     sysenter_dispatch 
      3     ia32_sysret 
      1      task_sched_runtime 
     27      sys32_pread 
      1      compat_sys_io_submit 
      2      compat_sys_io_getevents 
     27       sys_pread64 
      2       sys_io_getevents 
      1       do_io_submit 
     27        vfs_read 
      2        read_events 
      1        io_submit_one 
     27         do_sync_read 
      1         aio_run_iocb 
     27          generic_file_aio_read 
      1          aio_rw_vect_retry 
     27           generic_file_read_iter 
      1           generic_file_aio_read 
     27            mapping_direct_IO 
      1            generic_file_read_iter 
     27             blkdev_direct_IO 
     27              __blockdev_direct_IO 
     27               do_blockdev_direct_IO 
     27                dio_post_submission 
     27                 dio_await_completion 
      6                  blk_flush_plug_list

This gives you a very rough sample of where in kernel this process spent its time – if at all. The system call numbers are separated, in bold, above – “running” means that this process was in userland (not in a system call) during the sample. So, 69% of the time during the sampling, this process was running userland code. 25% of time in syscall #180 (nfsservctl on my system) and 6% in syscall #247 (waitid).

There are two more “functions” seen in this output – but for some reason they haven’t been properly translated to a function name. Well, this address must stand for something, so let’s try our luck manually:

[root@oel6 ~]# cat /proc/kallsyms | grep -i ffffff81534c83
ffffffff81534c83 t ia32_sysret

So it looks that these samples are about a 32-bit architecture-compatible system call return function – but as this function isn’t a system call itself (just an internal helper function), perhaps that’s why the /proc/stack routine didn’t translate it. Perhaps these addresses show up because there’s no “read consistency” on the /proc views, when owning threads modify these memory structures and entries, the reading threads may sometimes read data “in-flux”.

Let’ check the other address too:

[root@oel6 ~]# cat /proc/kallsyms | grep -i ffffff81534820
[root@oel6 ~]#

Nothing? Well, the troubleshooting doesn’t have to stop yet – let’s see if there’s anything interesting around that address. I just removed a couple of trailing characters from the address:

[root@oel6 ~]# cat /proc/kallsyms | grep -i ffffff815348 
ffffffff8153480d t sysenter_do_call
ffffffff81534819 t sysenter_dispatch
ffffffff81534847 t sysexit_from_sys_call
ffffffff8153487a t sysenter_auditsys
ffffffff815348b9 t sysexit_audit

Looks like the sysenter_dispatch function starts just 1 byte before the original address displayed by /proc/PID/stack. So we had possibly already executed the one byte (possibly a NOP operation left in there for dynamic tracing probe traps). But nevertheless, looks like these stack samples were in the sysenter_dispatch function, which isn’t a system call itself, but a system call helper function.

More about stack profilers

Note that there are different types of stack samplers – the Linux Perf, Oprofile tools and Solaris DTrace’s profile provider do sample the instruction pointer (EIP on 32bit intel or RIP on x64) and stack pointer (ESP on 32bit and RSP on 64bit) registers of the currently running thread on CPU and walk the stack pointers backwards from there. Therefore, these tools only show threads which happen to be on CPU when sampling!!! This is of course perfect, when troubleshooting high CPU utilization issues, but not useful at all when troubleshooting hung processes or too long process sleeps/waits.

Tools like pstack on Linux,Solaris,HP-UX, procstack (on AIX), ORADEBUG SHORT_STACK and just reading the /proc/PID/stack pseudofile provide a good addition (not replacement) to the CPU profiling tools – as they access the probed process memory regardless of their CPU scheduling state and read the stack from there. If the process is sleeping, not on CPU, the starting point of the stack can be read from the stored context – which is saved to kernel memory by OS scheduler during the context switch.

Of course, the CPU event profiling tools can often do much more than just a pstack, OProfile, Perf and even DTrace’s CPC provider (in Solaris11) can set up and sample CPU internal performance counters to estimate things like the amount of CPU cycles stalled waiting for main memory, amount of L1/L2 cache misses, etc. Better read what Kevin Closson has to say on these topics: (Perf, Oprofile)

Enjoy :-)

You might also want to check how to enable the DBMS_OUTPUT display in SQL Developer post by “That” Jeff Smith :-)

Update: Niall Litchfield pointed out that the “set serverout on” command was ignored on SQL Developer because I was using the short syntax (serverout instead of serveroutput). Once I changed this, the SQL Developer doesn’t print the warning anymore and there’s no need to explicitly open up the separate DBMS_OUTPUT. I have uploaded Snapper v4.04 with this fix included. Thanks Niall!

Also, thanks go to:

• Hans-Peter Sloot – for building the first cut of RAC support into a previous Snapper version
• Ronald Rood – for adding the customizations for SQL Developer support into a previous Snapper version

Note that the manual before/after snapshots don’t work inside SQL Developer right now, will try to fix this later …