Home / Performance issues / IO Bottleneck – Disk performance issue

IO Bottleneck – Disk performance issue

Performance issues on Solaris:

The below mentioned key resources will play a huge role in system performance.

In the post we are going to see about disk performance issues and  how to identify 
IO bottle neck.

Generally a problem is identified when an application is not performing as we expected. In larger environments, servers are connected to SAN storage and if there is any issue with storage, it will impact application performance. The most efficiently constructed file system will not perform well if it is implemented on slow hardware. Here we are going to see how to identify the DISK I/O issues and along with some common problems and solutions.
 
Disk I/Os include the following components:

1. I/O bus access: If the bus is busy, the request is queued by the driver. The information is reported by “sar -d” command on avwait column and wait column on “iostat -x” command.

2. Bus transfer time: Arbitration time time to transfer the command (usually ~ 2 ms), data transfer time (in the case of a write).

3. Interrupt time: Time for the completion interrupt to be processed. This is very hard to measure, but high interrupt rates on the CPUs associated with this system board may be an indication of problems.

 
IOSTAT:
Disk performance is one of the complex issues and this blog neither attempts nor can solve all issues. Here we are going to see the logical approach of problem solving. There some tools like iostat, sar, vmstat very useful to troubleshoot the performance issues and these tools are commonly available in all Unix/Linux operating systems.
 

·   If you run iostat command without any interval,it will give the below output but it’s very difficult to find the Disk I/O bottle neck.

bash-3.00# iostat -Cxn
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0
    0.5    2.3   13.9   13.6  0.0  0.0    0.0    7.3   0   1 c1
    0.2    2.1   12.1   10.2  0.0  0.0    0.0    7.8   0   1 c1t0d0
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    3.4   0   0 c1t1d0
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    0.9   0   0 c1t2d0
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.1   0   0 c1t3d0
    0.1    0.1    0.7    2.5  0.0  0.0    0.1   13.1   0   0 c1t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0

                 extended  disk  statistics  aggregated  by  controller id.

    -x          Report extended  disk  statistics.
    -n          Display names in descriptive format.  For  exam-
                 ple, cXtYdZ, rmt/N, server:/export/path.
 
From the above output, we need to look at the below columns to identify the I/O bottle necks.
1.asvc_t    average service time of  active  transactions,  in milliseconds. If the average service time is exceeding more than 25, then there is issue in the specific disk and need attention.
2.wsvc_t average  service  time  in  wait  queue,  in  milliseconds. If the wait queue is crossed more than 10 then we need to check with SAN team regarding this.To more about the rest of the fields ,see man page of iostat.
 
·    You can use the below command during the business hours to identify the real issue. If the average service time (asvc) value is more than 20 to 30 then there must be some issue with Disks I/O subsystem. The below command will give the Disk I/O statistics for every 30 seconds.
bash-3.00#  iostat -xn 30
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
    0.2    2.1   11.5   10.2  0.0  0.0    0.0    7.7   0   1 c1t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    3.4   0   0 c1t1d0
    0.0    0.0    0.0    0.2  0.0  0.0    0.0    0.9   0   0 c1t2d0
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.1   0   0 c1t3d0
    0.0    0.0    0.6    2.3  0.0  0.0    0.1   13.1   0   0 c1t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 sfos:vold(pid1358)
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
    0.0    2.0    0.0    8.8  0.0  0.0    0.0    2.2   0   0 c1t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.8   0   0 c1t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t2d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t3d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 sfos:vold(pid1358)
·        The below command will be more use full than the above one. This will give statistics only if the disk is being used.
bash-3.00# iostat -xnmpz 5
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.2    2.1   11.4   10.2  0.0  0.0    0.0    7.6   0   1 c1t0d0
    0.2    2.1   11.4   10.2  0.0  0.0    0.0    7.6   0   1 c1t0d0s0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.2   0   0 c1t0d0s2
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    3.4   0   0 c1t1d0
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    4.9   0   0 c1t1d0s2
    0.0    0.0    0.0    0.2  0.0  0.0    0.0    0.9   0   0 c1t2d0
    0.0    0.0    0.0    0.2  0.0  0.0    0.0    4.6   0   0 c1t2d0s2
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.1   0   0 c1t3d0
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.2   0   0 c1t3d0s2
    0.0    0.0    0.6    2.3  0.0  0.0    0.1   13.1   0   0 c1t4d0
    0.0    0.0    0.6    2.3  0.0  0.0    0.1   13.3   0   0 c1t4d0s0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   10.1   0   0 c1t4d0s2
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 c1t4d0s8
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   32.7    0.0 2048.7    0.0  0.0  1.7    0.0   53.3   0  50 c1t0d0
   32.7    0.0 2048.7    0.0  0.0  1.7    0.0   53.3   0  50 c1t0d0s0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   42.2    0.0 2664.8    0.0  0.0  1.0    0.0   23.9   0  43 c1t0d0
   42.2    0.0 2664.8    0.0  0.0  1.0    0.0   23.9   0  43 c1t0d0s0
Note:If you want to create real time disk I/O bottleneck on your test environment, you can use “find / > /dev/null 2>&1 &”.Make sure to kill the process after your test.
bash-3.00# iostat -xnmpz 5
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.2    2.1   11.4   10.2  0.0  0.0    0.0    7.6   0   1 c1t0d0
    0.2    2.1   11.4   10.2  0.0  0.0    0.0    7.6   0   1 c1t0d0s0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.2   0   0 c1t0d0s2
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    3.4   0   0 c1t1d0
    0.0    0.0    0.0    0.3  0.0  0.0    0.0    4.9   0   0 c1t1d0s2
    0.0    0.0    0.0    0.2  0.0  0.0    0.0    0.9   0   0 c1t2d0
    0.0    0.0    0.0    0.2  0.0  0.0    0.0    4.6   0   0 c1t2d0s2
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.1   0   0 c1t3d0
    0.2    0.0    0.2    0.3  0.0  0.0    0.0    0.2   0   0 c1t3d0s2
    0.0    0.0    0.6    2.3  0.0  0.0    0.1   13.1   0   0 c1t4d0
    0.0    0.0    0.6    2.3  0.0  0.0    0.1   13.3   0   0 c1t4d0s0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0   10.1   0   0 c1t4d0s2
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 c1t4d0s8
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   32.7    0.0 2048.7    0.0  0.0  1.7    0.0   53.3   0  50 c1t0d0
   32.7    0.0 2048.7    0.0  0.0  1.7    0.0   53.3   0  50 c1t0d0s0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   42.2    0.0 2664.8    0.0  0.0  1.0    0.0   23.9   0  43 c1t0d0
   42.2    0.0 2664.8    0.0  0.0  1.0    0.0   23.9   0  43 c1t0d0s0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   41.3    0.0 2641.5    0.0  0.0  1.5    0.0   36.4   0  52 c1t0d0
   41.3    0.0 2641.5    0.0  0.0  1.5    0.0   36.4   0  52 c1t0d0s0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   30.9   11.2 1881.3   56.1  0.0  1.0    0.0   24.1   0  41 c1t0d0
   30.9   11.2 1881.3   56.1  0.0  1.0    0.0   24.1   0  41 c1t0d0s0
    0.0    0.2    0.0    0.2  0.0  0.0    0.0    8.6   0   0 c1t1d0
    0.0    0.2    0.0    0.2  0.0  0.0    0.0    8.6   0   0 c1t1d0s2
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   38.6    0.0 2219.3    0.0  0.0  1.4    0.0   35.2   0  51 c1t0d0
   38.6    0.0 2219.3    0.0  0.0  1.4    0.0   35.2   0  51 c1t0d0s0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   30.9    0.0 1865.0    0.0  0.0  1.2    0.0   37.7   0  49 c1t0d0
   30.9    0.0 1865.0    0.0  0.0  1.2    0.0   37.7   0  49 c1t0d0s0

Note: If iostat consistently reports %w > 7, the disk subsystem is too busy. To reduce the value, we have spread the filesystem on smaller LUNS. This will reduce the %w value.
SAR:
Apart from iostat, SAR also very helpful to identify the real disk I/O sub system issues. Disk I/O can be investigated to find out whether it is primarily random or sequential. If sar -d reports that (blks/s) and (r+w/s) less than 16Kb , the I/O is predominantly random. If the ratio is  more than 128Kb, it is predominantly sequential. This analysis may be useful when examining alternative disk configurations.
bash-3.00# sar -d 5 5
SunOS sfos 5.10 Generic_142910-17 i86pc    07/15/2012
22:34:42   device        %busy   avque   r+w/s  blks/s  avwait  avserv
22:34:47   fd0               0     0.0       0       0     0.0     0.0
           nfs1              0     0.0       0       0     0.0     0.0
           sd0               0     0.0       0       0     0.0     0.0
           sd0,a             0     0.0       0       0     0.0     0.0
           sd0,c             0     0.0       0       0     0.0     0.0
           sd0,i             0     0.0       0       0     0.0     0.0
           sd0,q             0     0.0       0       0     0.0     0.0
           sd0,r             0     0.0       0       0     0.0     0.0
           sd1               0     0.0       0       0     0.0     0.0
           sd2               0     0.0       0       0     0.0     0.0
           sd2,c             0     0.0       0       0     0.0     0.0
           sd2,h             0     0.0       0       0     0.0     0.0
           sd2,i             0     0.0       0       0     0.0     0.0
           sd2,q             0     0.0       0       0     0.0     0.0
           sd3               0     0.0       0       0     0.0     0.0
           sd3,c             0     0.0       0       0     0.0     0.0
           sd3,h             0     0.0       0       0     0.0     0.0
           sd3,i             0     0.0       0       0     0.0     0.0
           sd3,q             0     0.0       0       0     0.0     0.0
           sd4               0     0.0       0       0     0.0     0.0
           sd4,c             0     0.0       0       0     0.0     0.0
           sd4,h             0     0.0       0       0     0.0     0.0
           sd4,i             0     0.0       0       0     0.0     0.0
           sd4,q             0     0.0       0       0     0.0     0.0
           sd5               0     0.0       0       0     0.0     0.0
           sd5,a             0     0.0       0       0     0.0     0.0
           sd5,c             0     0.0       0       0     0.0     0.0
           sd5,h             0     0.0       0       0     0.0     0.0
           sd5,i             0     0.0       0       0     0.0     0.0
           sd5,q             0     0.0       0       0     0.0     0.0
Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.
VMTURBO-CLOUD-CAPACITY

One comment

  1. Very good article which is very useful for the people who do not have idea on operating system and freshers.

Leave a Reply

Your email address will not be published. Required fields are marked *