Performance issues on Solaris:
The below mentioned key resources will play a huge role in system performance.
In the post we are going to see about disk performance issues and how to identify
IO bottle neck.
Generally a problem is identified when an application is not performing as we expected. In larger environments, servers are connected to SAN storage and if there is any issue with storage, it will impact application performance. The most efficiently constructed file system will not perform well if it is implemented on slow hardware. Here we are going to see how to identify the DISK I/O issues and along with some common problems and solutions.
Disk I/Os include the following components:
1. I/O bus access: If the bus is busy, the request is queued by the driver. The information is reported by “sar -d” command on avwait column and wait column on “iostat -x” command.
2. Bus transfer time: Arbitration time time to transfer the command (usually ~ 2 ms), data transfer time (in the case of a write).
3. Interrupt time: Time for the completion interrupt to be processed. This is very hard to measure, but high interrupt rates on the CPUs associated with this system board may be an indication of problems.
IOSTAT:
Disk performance is one of the complex issues and this blog neither attempts nor can solve all issues. Here we are going to see the logical approach of problem solving. There some tools like iostat, sar, vmstat very useful to troubleshoot the performance issues and these tools are commonly available in all Unix/Linux operating systems.
· If you run iostat command without any interval,it will give the below output but it’s very difficult to find the Disk I/O bottle neck.
bash-3.00# iostat -Cxn
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.5 2.3 13.9 13.6 0.0 0.0 0.0 7.3 0 1 c1
0.2 2.1 12.1 10.2 0.0 0.0 0.0 7.8 0 1 c1t0d0
0.0 0.0 0.0 0.3 0.0 0.0 0.0 3.4 0 0 c1t1d0
0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.9 0 0 c1t2d0
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.1 0 0 c1t3d0
0.1 0.1 0.7 2.5 0.0 0.0 0.1 13.1 0 0 c1t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 fd0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended disk statistics aggregated by controller id.
-x Report extended disk statistics.
-n Display names in descriptive format. For exam-
ple, cXtYdZ, rmt/N, server:/export/path.
From the above output, we need to look at the below columns to identify the I/O bottle necks.
1.asvc_t average service time of active transactions, in milliseconds. If the average service time is exceeding more than 25, then there is issue in the specific disk and need attention.
2.wsvc_t average service time in wait queue, in milliseconds. If the wait queue is crossed more than 10 then we need to check with SAN team regarding this.To more about the rest of the fields ,see man page of iostat.
· You can use the below command during the business hours to identify the real issue. If the average service time (asvc) value is more than 20 to 30 then there must be some issue with Disks I/O subsystem. The below command will give the Disk I/O statistics for every 30 seconds.
bash-3.00# iostat -xn 30
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 fd0
0.2 2.1 11.5 10.2 0.0 0.0 0.0 7.7 0 1 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.0 0.0 0.0 0.3 0.0 0.0 0.0 3.4 0 0 c1t1d0
0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.9 0 0 c1t2d0
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.1 0 0 c1t3d0
0.0 0.0 0.6 2.3 0.0 0.0 0.1 13.1 0 0 c1t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sfos:vold(pid1358)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 fd0
0.0 2.0 0.0 8.8 0.0 0.0 0.0 2.2 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0 0 c1t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t2d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t3d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sfos:vold(pid1358)
· The below command will be more use full than the above one. This will give statistics only if the disk is being used.
bash-3.00# iostat -xnmpz 5
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.2 2.1 11.4 10.2 0.0 0.0 0.0 7.6 0 1 c1t0d0
0.2 2.1 11.4 10.2 0.0 0.0 0.0 7.6 0 1 c1t0d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 c1t0d0s2
0.0 0.0 0.0 0.3 0.0 0.0 0.0 3.4 0 0 c1t1d0
0.0 0.0 0.0 0.3 0.0 0.0 0.0 4.9 0 0 c1t1d0s2
0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.9 0 0 c1t2d0
0.0 0.0 0.0 0.2 0.0 0.0 0.0 4.6 0 0 c1t2d0s2
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.1 0 0 c1t3d0
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.2 0 0 c1t3d0s2
0.0 0.0 0.6 2.3 0.0 0.0 0.1 13.1 0 0 c1t4d0
0.0 0.0 0.6 2.3 0.0 0.0 0.1 13.3 0 0 c1t4d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.1 0 0 c1t4d0s2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 c1t4d0s8
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
32.7 0.0 2048.7 0.0 0.0 1.7 0.0 53.3 0 50 c1t0d0
32.7 0.0 2048.7 0.0 0.0 1.7 0.0 53.3 0 50 c1t0d0s0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
42.2 0.0 2664.8 0.0 0.0 1.0 0.0 23.9 0 43 c1t0d0
42.2 0.0 2664.8 0.0 0.0 1.0 0.0 23.9 0 43 c1t0d0s0
Note:If you want to create real time disk I/O bottleneck on your test environment, you can use “find / > /dev/null 2>&1 &”.Make sure to kill the process after your test.
bash-3.00# iostat -xnmpz 5
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.2 2.1 11.4 10.2 0.0 0.0 0.0 7.6 0 1 c1t0d0
0.2 2.1 11.4 10.2 0.0 0.0 0.0 7.6 0 1 c1t0d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0 c1t0d0s2
0.0 0.0 0.0 0.3 0.0 0.0 0.0 3.4 0 0 c1t1d0
0.0 0.0 0.0 0.3 0.0 0.0 0.0 4.9 0 0 c1t1d0s2
0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.9 0 0 c1t2d0
0.0 0.0 0.0 0.2 0.0 0.0 0.0 4.6 0 0 c1t2d0s2
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.1 0 0 c1t3d0
0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.2 0 0 c1t3d0s2
0.0 0.0 0.6 2.3 0.0 0.0 0.1 13.1 0 0 c1t4d0
0.0 0.0 0.6 2.3 0.0 0.0 0.1 13.3 0 0 c1t4d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.1 0 0 c1t4d0s2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0 c1t4d0s8
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
32.7 0.0 2048.7 0.0 0.0 1.7 0.0 53.3 0 50 c1t0d0
32.7 0.0 2048.7 0.0 0.0 1.7 0.0 53.3 0 50 c1t0d0s0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
42.2 0.0 2664.8 0.0 0.0 1.0 0.0 23.9 0 43 c1t0d0
42.2 0.0 2664.8 0.0 0.0 1.0 0.0 23.9 0 43 c1t0d0s0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
41.3 0.0 2641.5 0.0 0.0 1.5 0.0 36.4 0 52 c1t0d0
41.3 0.0 2641.5 0.0 0.0 1.5 0.0 36.4 0 52 c1t0d0s0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
30.9 11.2 1881.3 56.1 0.0 1.0 0.0 24.1 0 41 c1t0d0
30.9 11.2 1881.3 56.1 0.0 1.0 0.0 24.1 0 41 c1t0d0s0
0.0 0.2 0.0 0.2 0.0 0.0 0.0 8.6 0 0 c1t1d0
0.0 0.2 0.0 0.2 0.0 0.0 0.0 8.6 0 0 c1t1d0s2
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
38.6 0.0 2219.3 0.0 0.0 1.4 0.0 35.2 0 51 c1t0d0
38.6 0.0 2219.3 0.0 0.0 1.4 0.0 35.2 0 51 c1t0d0s0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
30.9 0.0 1865.0 0.0 0.0 1.2 0.0 37.7 0 49 c1t0d0
30.9 0.0 1865.0 0.0 0.0 1.2 0.0 37.7 0 49 c1t0d0s0
Note: If iostat consistently reports %w > 7, the disk subsystem is too busy. To reduce the value, we have spread the filesystem on smaller LUNS. This will reduce the %w value.
SAR:
Apart from iostat, SAR also very helpful to identify the real disk I/O sub system issues. Disk I/O can be investigated to find out whether it is primarily random or sequential. If sar -d reports that
(blks/s) and (r+w/s) less than 16Kb , the I/O is predominantly random. If the ratio is more than 128Kb, it is predominantly sequential. This analysis may be useful when examining alternative disk configurations.bash-3.00# sar -d 5 5
SunOS sfos 5.10 Generic_142910-17 i86pc 07/15/2012
22:34:42 device %busy avque r+w/s blks/s avwait avserv
22:34:47 fd0 0 0.0 0 0 0.0 0.0
nfs1 0 0.0 0 0 0.0 0.0
sd0 0 0.0 0 0 0.0 0.0
sd0,a 0 0.0 0 0 0.0 0.0
sd0,c 0 0.0 0 0 0.0 0.0
sd0,i 0 0.0 0 0 0.0 0.0
sd0,q 0 0.0 0 0 0.0 0.0
sd0,r 0 0.0 0 0 0.0 0.0
sd1 0 0.0 0 0 0.0 0.0
sd2 0 0.0 0 0 0.0 0.0
sd2,c 0 0.0 0 0 0.0 0.0
sd2,h 0 0.0 0 0 0.0 0.0
sd2,i 0 0.0 0 0 0.0 0.0
sd2,q 0 0.0 0 0 0.0 0.0
sd3 0 0.0 0 0 0.0 0.0
sd3,c 0 0.0 0 0 0.0 0.0
sd3,h 0 0.0 0 0 0.0 0.0
sd3,i 0 0.0 0 0 0.0 0.0
sd3,q 0 0.0 0 0 0.0 0.0
sd4 0 0.0 0 0 0.0 0.0
sd4,c 0 0.0 0 0 0.0 0.0
sd4,h 0 0.0 0 0 0.0 0.0
sd4,i 0 0.0 0 0 0.0 0.0
sd4,q 0 0.0 0 0 0.0 0.0
sd5 0 0.0 0 0 0.0 0.0
sd5,a 0 0.0 0 0 0.0 0.0
sd5,c 0 0.0 0 0 0.0 0.0
sd5,h 0 0.0 0 0 0.0 0.0
sd5,i 0 0.0 0 0 0.0 0.0
sd5,q 0 0.0 0 0 0.0 0.0
Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.