Seminar Files

December 1st, 2009

Here are links to relevant downloads for Advanced Oracle Troubleshooting seminar attendees (and also potential attendees ;)

Slides

Slides of my 1.5 hour “Advanced Oracle Troubleshooting: No magic is needed – systematic approach will do!” presentation. This session presents the approach and tools for advanced Oracle troubleshooting where classic methods don’t help. The slides are somewhat similar to the 1st session of the Advanced Oracle Troubleshooting seminar, so they should give you an idea what kind of content I have in the full seminar slides (which are available only to seminar attendees as a printout):

Seminar logs

Shell and sqlplus logs/transcript of Advanced Oracle Troubleshooting seminar. These will give you an idea of the level of detail we dive into there:

Link to Seminar Logs Page

Scripts & Tools

My TPT scripts (Tanel Poder’s Troubleshooting or Tanel’s Performance Tuning scripts, whichever you like better ;)

http://www.tanelpoder.com/files/TPT_public.zip

PerfSheet.xls tool which I use for performance data visualization:

http://www.tanelpoder.com/files/PerfSheet.zip

Happy troubleshooting ! :)

  • Share/Bookmark
  1. Trackbacks

  2. October 28th, 2009: Learning « OraStory
  1. Comments

  2. Julia Tokareva
    February 12th, 2009 at 10:42 | #1

    Advanced Troubleshooting Seminar is technically unique, which covers in details many aspects of the performance trouble-shooting and offers the methodological approach to deal with the real issues when the performance problems appear.
    The seminar material contains tremendous amounts of scripts that can be used on any environment to troubleshoot the most complicated performance problems
    Amazing seminar !!! Thank you Tanel !!!

  3. February 13th, 2009 at 01:08 | #2

    Thanks Julia! :)

    See you next time I’m in NYC!!!

  4. Venkata Giri
    February 13th, 2009 at 12:51 | #3

    Tanel,

    I attended Understanding Oracle execution plans seminar at Denver, CO. The way you dissected the Oracle execution plan to explain in detail to the audiences is very unique. The content is technically rich and really useful for DBA’s at all levels. I really appreciate your generosity for sharing all the scripts.

    Thanks
    Venkata Giri

  5. Alberto
    February 16th, 2009 at 04:27 | #4

    Tanel,
    Great job, I like your systematic approach for troubleshooting, thanks a lot for your script too, but when do you come in Italy for your seminar?

  6. Muru Balakrishnan
    February 16th, 2009 at 13:42 | #5

    Great presentation Tanel at RMOUG 2009! Keep up the good work!
    Thanks, Muru

  7. February 17th, 2009 at 11:22 | #6

    Tanel,

    I had the privilege of attending both of your sessions at RMOUG. Well done.

    Thank you for a great presentation on Advanced Oracle Troubleshooting; the tools you’ve shared are quite valuable.

    The session on SQL/Execution plans (not explain plans) was great.

    Cheers.

  8. February 17th, 2009 at 15:40 | #7

    Thanks for the great comments!

    I also enjoyed RMOUG, both speaking at and attending sessions.

  9. Wachyu Danyanto
    March 11th, 2009 at 01:55 | #8

    Thank you Tanel, amazing and great presentation that you share.

  10. March 14th, 2009 at 09:48 | #9

    I agree with your comments about explain plan, but dbms_xplan.display_cursor() requires access to the V$SESSION, V$SQL_PLAN and some other V$ tables. Unfortunately, if I don’t want to grant “select any dictionary” to public, I still have to use an old and battle tested EXPLAIN PLAN FOR.

  11. March 16th, 2009 at 03:29 | #10

    Hi Mladen,

    You don’t have to grant “select any dictionary” to PUBLIC. You can grant select on v$sql, v$sql_plan and v$sql_plan_statistics% views to only the users who require the privs. But you know that anyway.

    Basically the tradeoff is:

    1) do not grant these privs and risk getting incorrect execution plans with “explain plan” command

    2) grant these privs and get absolutely correct execution plan with dbms_xplan.display_cursor() or v$sql_plan

    3) do not grant these privs and use SQL trace for getting the plan, but this means you have to wait until the statement finishes (or cancel it and close the cursor). SQL trace also sets sqlstat_enabled to true, which enables rowsource level execution stats collection and makes the SQL run way slower… and not to mention that getting & parsing the tracefile from OS is less convenient than just running a query in sqlplus..

  12. suresh
    March 17th, 2009 at 14:23 | #11

    thanks dear , i learn lot by you…

  13. Kevin Seneviratne
    April 22nd, 2009 at 05:51 | #12

    Brilliant seminar in Melbourne. Absolutely love your approach to troubleshooting. Learnt lots of cool stuff but more importantly learnt a methodology. ‘As they say …Give a man a fish and he will eat for a day. Teach a man to fish and he will feed himself for years !!” Thanks for teaching me to fish Tanel.
    Kevin.

  14. April 22nd, 2009 at 07:10 | #13

    Hi Kevin,

    Thanks! I’m glad to hear you found it useful! :)

  15. April 22nd, 2009 at 16:35 | #14

    Hi Tanel,

    I enjoyed your seminar in NL very much. Are you going to post the AOT logs of that seminar as well?

    BTW I found a small error in your TPT_public.zip (snapper v1.10). Maybe that is to test if people are actually using it on Oracle v10?

    Regards,

    Patrick

  16. April 23rd, 2009 at 00:19 | #15

    What was the error?

    I’m still working on the NL seminar logs (I hoped to comment them but haven’t had time so far).

    I’ll upload them in raw tonight!

  17. Patrick
    April 24th, 2009 at 04:05 | #16

    It wasn’t a big error but you defined a variable called _IF_ORA_10_OR_HIGHER but then you use _IF_ORA10_OR_HIGHER in the rest of the script. That’s all.

    Take your time with the logs, I was just wondering if they were comming and now I know. :-)

  18. April 24th, 2009 at 05:59 | #17

    Ah yes, I remember that one. It should be fixed in snapper 2.0 (which is online now)

    Still working on the logs ;)

  19. ari rusanen
    May 6th, 2009 at 15:24 | #18

    Tanel!

    Thanks for great presentation at Miracle’s Mayday 2009 seminar.
    It is bitty, that I didn’t get opportunity to change some ideas after seminar..Mayby next time..

    Happy trails!
    Ari

  20. May 6th, 2009 at 15:33 | #19

    Hi Ari!

    I think I’ll be at Helsinki beer festival next friday with Tuomas, so next time is near! ;)

  21. Stano
    May 20th, 2009 at 04:50 | #20

    Hi Tanel,

    thanks for comming to Madrid and for the superb seminar. Also, thanks for the blog and the scripts!

    Stano.

  22. Santosh
    May 22nd, 2009 at 06:45 | #21

    Hey Tanel,

    I closely follow your blogs and I must say , your presentation has all together changed my views towards oracle internals.Are you planning to visit India any time ?

    Cheers,
    Santosh

  23. Santosh
    May 22nd, 2009 at 06:46 | #22

    Btw, I have a small query about v$osstat, would you mind if I post that here ?

    Thanks,
    Santosh

  24. May 22nd, 2009 at 07:45 | #23

    Hi Santosh,

    I planned an india visit some time ago, but my calendar for this year has been booked full already.

    Yep please ask the v$ossstat question here..

  25. Ivan Bajon
    May 24th, 2009 at 04:10 | #24

    Hi Tanel

    Thanks for the great seminar in Ballerup, Denmark. You’re certainly the master as your tag said but I was slightly disappointed that so much time was spent on basic stuff that there wasn’t much time for the juicy stuff. I’d like to suggest that you extend the seminar with at least one day and spend less time on stuff that the attendants are supposed to know already.

    Now, don’t get me wrong – I’ll be looking to join more Tanel Poder classes in the future. :o)

    I would also like to suggest that you put a new class together – containing your clever sqlplus tricks and hands-on training in how to create a perfsheet like your own.

    Best regards
    Ivan

  26. May 24th, 2009 at 05:14 | #25

    Hi Ivan,

    Thanks for the comments. Yes I have much more advanced slides in the material (in the end of each chapter, after “questions” slide) and at first seminars I showed stuff like how to interpret systemstate dumps and read SGA memory directly.

    However based on the feedback from people I started talking about more practical stuff for everyday troubleshooting and less about the extreme corner cases. Instead I’ve taken the deep dive when someone has a question about some specific case they’ve had (and that’s why I encourage questions during all the class).

    And yes – there should be more time for all that material (as I spend more than half of time doing demos). So for future 2-day classes I’ve removed the last 2 chapters and split chapter 1 and 3 to two parts.

    I’ve been thinking about a “Hard-Core Oracle Internals & Hacking Class” which would be some 4-5 days in length, which would go REALLY deep, so deep that average DBAs shouldn’t even care about it :) In addition to troubleshooting/problem diagnosis techniques I would also show how I do my research for finding out stuff about Oracle.
    I’d do it only once per year… so if anyone is interested, let me know and I start planning it ;)

  27. May 24th, 2009 at 05:18 | #26

    Regarding the hands-on sqlplus tricks, scripts and tools like perfsheet training – I think I’ll do something like this at Hotsos Symposium’s Training Day next year (yes I’ll be doing the training day next year – http://www.hotsos.com/sym10/sym_training.html)

    I’ll first show my new and improved scripts & tools (like Snapper v3.0, PerfSheet v3.0 and more) and then go through some new peformance diagnosis and troubleshooting case studies with them.

    If you plan to attend the Hotsos Symposium’s Training Day and you have further ideas what you’d want to hear there, let me know about it !

  28. Ivan Bajon
    May 24th, 2009 at 05:48 | #27

    I’ll send a reoccurring mail to my manager every day asking for go-ahead on both until he approves. :o)

    Cheers
    Ivan

  29. Santosh
    May 24th, 2009 at 23:02 | #28

    Hey Tanel,

    Here is my query regarding v$OSSTAT.
    I am working on a design where an application server needs current CPU consumption on database server.Because of business constraints, I can not setup passwordless ssh between 2 boxes.So I thought of getting the data from v$OSSTAT view. Being cumulative data in v$osstat for busy_time, idle_time,user_time metrics , I am finding the difference between these metrics over a period of time like 5 seconds and then % user cpu consumption =
    ( user_time/( busy_time+idle_time+iowait_time+nice_time).

    But sometimes I dont get the right data when compared with mpstat/TOP output.Do you think I am doing something wrong?

    Thanks,
    Santosh

  30. May 25th, 2009 at 05:32 | #29

    Which OS are you on?

    If you want just systemwide CPU utilization, then the formula should be something like util% = busy_time / ( busy_time + idle_time ). If you’re on linux there may be some more variables involved, this needs testing.

    I recommend you to send me some actual data from a case when you get “wrong” results (two samples of v$osstat 10 seconds apart) and also two samples of “vmstat -s” from the same interval.

  31. Joe Ramsey
    May 28th, 2009 at 03:14 | #30

    I’d be very interested in the 5 day class as well. Thanks for putting all of this great information out there Tanel!

  32. Santosh
    May 29th, 2009 at 06:52 | #31

    Hey Tanel,

    Sorry was busy with shared pool fragmentation problem in one of the production boxes , it was very interesting kind of problem :

    Its running on 10.2.0.2 on RHEL 3 ,hitting more often ora-04031 errors, when I checked the subpool configuration, came across weired thing :

    SQL> select KGHLUIDX,INST_ID,INDX,ADDR,KGHLUNFU from x$kghlu;

    KGHLUIDX INST_ID INDX ADDR KGHLUNFU
    ———- ———- ———- —————- ———-
    4 1 0 0000002A973A6ED8 0
    3 1 1 0000002A973A68A8 0
    2 1 2 0000002A973A7F08 435
    1 1 3 0000002A973A78D8 0

    All the failures were in subpool 2.After walking through trace files , concluded that oracle never switches the subpools even if there free large enough memory chunks in other subpools to satisfy the request.As we are running with low memory on the box , decided to reduce the number of subpools from 4 to 2 , going to do that very soon.

    Things which I didnt understand :

    1) How the distribution of sql stmts happens in these subpool , is it kind of hashing algorithm?
    2) If thats the case, then how can we find that in the database ?

    Thanks,
    Santosh

  33. June 4th, 2009 at 17:31 | #32

    Hi Santosh,

    I started answering your question with this article:

    http://blog.tanelpoder.com/2009/06/04/ora-04031-errors-and-monitoring-shared-pool-subpool-memory-utilization-with-sgastatxsql/

    I ended up writing much more than initially planned – and didn’t manage to answer the whole question there.. so I need to write a part 2 soon :)

  34. Santosh
    June 8th, 2009 at 02:57 | #33

    Hi Tanel,

    Thanks a lot for considering my questions valid enough to work on them.

    Eagerly waiting for your second post… :)

    Thanks a ton,
    Santosh

  35. Gary Olsen
    June 24th, 2009 at 17:11 | #34

    After attending the training course in Denver last week, I was on-call @ work for the rest of the week. During that time, I used my new troubleshooting skills twice.

    Thanks Tanel!

    Gary Olsen – Salt Lake City, UT

  36. June 24th, 2009 at 18:20 | #35

    Awesome! Good to hear that :)

    Just out of curiosity – what kind of problems did you troubleshoot?

  37. Gary Olsen
    July 8th, 2009 at 13:22 | #36

    One was related related to a problem that was showing up in the alert log every morning at around 5am. Rather than taking the ‘old’ approach of trying to look around the DB (early) in the morning, I just set an event to have the SQL dumped the next time the error condition occurred. I passed that along to the development database engineer so they could investigate further. I followed the systematic approach. (Was didn’t have to get up @ 5am in the morning!)

    Gary

  38. Chanson
    November 3rd, 2009 at 04:42 | #37

    I heard about you from a friend who had attended your seminar before. My friend said your seminar is very helpful for DBAs.
    And I found the scripts you privoded here is also very helpful.
    Thanks a lot.

  39. Zarier
    November 24th, 2009 at 03:58 | #38

    Hi Tanel,

    Just have a question, is there a way to identify a process that causes certain indexes to get corrupted, All I do know is that my client is running SAS jobs which calls stored proc’s, but not sure which process is causing the pains.

  40. November 24th, 2009 at 06:16 | #39

    So, how do you experience/see this corruption? ORA-600’s?

    Perhaps the jobs are not actually corrupting the index but they are just victims, the first ones who access a corrupt block.

    With corruptions, the sequence of things to do is roughtly this:

    1) check OS syslogs to find out whether any hardware issues are reported. You may also ask storage admins to check their sources about storage.

    2) run dbverify or RMAN test backup to detect the scale of corruptions in the database

    3) if this truly is an Oracle job which corrupts some blocks then its possible to find out more details using logminer, log dumps etc, however I would check point 1/2 first.

  41. January 25th, 2010 at 17:17 | #40

    HI , Veera Boda , having 6 + years exp as Oracle Apps DBA ( Oracle E – Business Suite and oracle Database ) . I did not work much on performance issues. I would like learn more about this topic .

    1.Can i know the training schedule ( I would like have online )

    2. Can you advise books /web sites which gives more fundamentals and performance tuning information .