“So, whether and how much of the “SQL*Net more data to client” vs “SQL*Net message to client” waits you see depends on two things:

Amount of data returned to client per call
Oracle Net SDU size “

I hit a network issue on May 26th. PD complained the slowness of their job at that time. I checked the running session using v$session_event and found the most time waiting for ‘SQL*Net more data to client’ and ‘virtual circuit status’(We use MTS). Using the stats$system_event(we use statspack with 15 minutes sampling), I found the average wait time(TIME_WAITED_MICRO/TOTAL_WAITS) of ‘SQL*Net more data to client’ increased from 200 to about 600. However, the average wait time of ‘SQL*Net message to client’ was always 4-5. Finally, I found the root cause is the network issue caused by failed switch.

My questions are,

1. Can we use average wait time of ‘SQL*Net more data to client’ to measure the network latency? (it’s not the first time we use this to detect the network issue).

2. Why the average wait time of ‘SQL*Net message to client’ didn’t change during the network issue since the ‘SQL*Net more data to client’ is just used to send more data over first SDU buffer as you stated above?

PS. the system event of ‘SQL*Net more data to client’
SNAP_TIME TIME_WAITED_MICRO TOTAL_WAITS TIME_WAITED_MICRO/TOTAL_WAITS
—————- —————– ———– —————————–
2010-05-25 01:00 1128567822 4496478 250.98929
…………
2010-05-25 23:15 1231169379 4703941 261.731467
2010-05-25 23:30 1282636291 4639954 276.432976
2010-05-25 23:45 1360151005 4496391 302.498383
2010-05-26 00:00 912223796 4210939 216.631919
2010-05-26 00:15 1507247039 5024973 299.951271
2010-05-26 00:30 1524815181 5234298 291.31226
2010-05-26 00:45 2691204366 5108648 526.793853
2010-05-26 01:00 3125441997 4723104 661.73474

the system event of ‘SQL*Net message to client’
SNAP_TIME TIME_WAITED_MICRO TOTAL_WAITS TIME_WAITED_MICRO/TOTAL_WAITS
—————- —————– ———– —————————–
2010-05-25 01:00 15541619 3162845 4.91380988
……….
2010-05-25 23:15 14801067 3046039 4.85911933
2010-05-25 23:30 14479453 2967351 4.8795889
2010-05-25 23:45 14589455 3004230 4.85630428
2010-05-26 00:00 14219425 2924318 4.86247563
2010-05-26 00:15 16295497 3308064 4.92599206
2010-05-26 00:30 18020082 3661416 4.92161557
2010-05-26 00:45 17216179 3507996 4.9076963
2010-05-26 01:00 16622061 3371335 4.93040917

Thanks for your help on this in advance.

Shenglin

I only got acceptable export speed when I changed the default buffer size to a high value:

exp user/password buffer=100000000

So, do you think there is some O/S kernel related parameter that I should to review?

Cheers

Thanks for the tips, I’ll check it out …

Cheers

If you think that everything is configured normally, there is no significant network latency nor application think time then you can look deeper – dump the network packets at client with tcpdump (if on linux) and analyze part of the transmission with Wireshark (www.wireshark.org). This will give you SDU/TDU sizes, resends, packet reassemblies and timestamps you need for finding where the time is going IF it’s a network transport issue.

1) time the result network packets are “flying” towards the client
2) client application think time – this is important as if you use some pipe for compression on the fly, the think time may become even longer
3) time the next request packet is “flying” towards the server

Thanks for the response. In fact, I have never faced this kind of problem before. I’m trying to understand why in the local server the export operation took so fast, and why the export operation from a remote machine took so long. 2 or 3 minutes is acceptable, but 1 hour? For 5 years, we have working in a Oracle 10g installed in another server, and this kind of problem have never happened before. The development team used to take exports (exp) and imports (imp) using default buffers size since then, without problems:

exp user/passwrod@service file=file.dmp …

The network analyst of the company made some tests using “scp” Linux commands in order to copy big files from that new Oracle 11g server to another server and vice versa without problems. The speed rate of copying a file was normal as expected (10MB/s) in a 100Mbs network.

The SDU/TDU SQLNet configuration is using the default values. The MTU of the LAN is fine too (1500 bytes).

So, I’m not sure how to find out either what could might causing this delay or contention. Maybe looking at Linux Kernel related network parameters? Maybe taking a test … like connecting a direct crossover cable connection between the server and a client machine?

Well, on next Monday I will try to take more tests …

Thanks.

See taht your session had made 26048 roundtrips during session startup. Now if every roundtrip takes let say 100ms then you’ll have ~2600 seconds of response time there. In fact in your case the time spent waiting for client (next fetch) was ~3000 seconds already. So it’s the network latency combined with high number of fetches/network roundtrips.

How did you perform the export? Make your export buffer size larger, this should reduce number of network roundtrips and response time.