Home / Performance issues / Performance issues-Solaris Part2-CPU

Performance issues-Solaris Part2-CPU

If its comes to performance issue in Unix operating systems, everybody will be behind on CPU.But not all the time CPU will cause for the performance problem. Because sometimes CPU may waiting for resources like Disk I/O, Swapping ,Network.As sysadmin you should able to pin point where the problem.vmstat is very useful command to determined where it the problem.

After introduction Solaris zones, sysadmin job is become bit difficult to isolate the problems from other local zones and global zone. prstat is very useful command to find local zones CPU usage from global zone. 

VMSTAT:
vmstat is very useful command to find the root cause for probing issues.last columns shows the CPU ideal status.The field which you need to look at in the vmstat are
r , b ,w,sr,cpu id

bash-3.00# vmstat 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 s2   in   sy   cs us sy id
 0 0 0 1061088 86408 36  99  0  0  1  0  6 -0  2 -0  0  328  458  186  0  1 98
 1 0 0 954432 93008  15  49  0  0  0  0  0  0  0  0  0  546  583  307  0  2 97
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  470  545  297  0  1 99
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  566  563  328  0  2 98
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  581  605  315  0  1 99
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  553  564  301  0  1 99
 2 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  550  574  315  0  4 95
 1 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  604  682  340  0  1 98
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  508  721  356  0  4 96
 1 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  519  575  321  0  2 97
The fields of vmstat’s display are
 procs 
  r     in run queue
  b     blocked for resources I/O, paging etc.
 w     swapped


     memory (in Kbytes)
swap –  amount  of  swap   space   currently   available
free   – size of the free list
     page ( in units per second).
re    page reclaims –  see  -S  option  for  how  this field is modified.
mf    minor faults –  see  -S  option  for  how    this field is modified.
pi    kilobytes paged in
po    kilobytes paged out
fr    kilobytes freed
de    anticipated short-term memory shortfall (Kbytes)
sr    pages scanned by clock algorithmdisk  ( operations per second )
           There are  slots for up to four disks, labeled with a single letter and number.
           The letter indicates  the  type  of  disk  (s = SCSI, i = IPI, etc) . The number is 
           the logical unit number.     faults
in    (non clock) device interrupts
sy    system calls
cs    CPU context switches cpu  –   breakdown of percentage usage of CPU  time.  On multiprocessors  this is an a

                verage across all processors.
us    user time
sy    system time
id    idle time


Problem symptoms:

1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.

 

Resolution :

Resolution to these kind of issues involves tuning of application procedures  to make efficient use of cpu  and as a last resort increasing the cpu power or adding more cpu to the system

Top:
Top is tool which is commonly available all the Unix/Linux.IN Solaris we need to install .To see top 10 process which are utilizing high CPU time,use below command.

bash-3.00# /usr/local/bin/top 10
last pid:5307;load avg:0.16,  1.25,  0.95; up 2+12:33:59 14:17:37                        
122 processes: 121 sleeping, 1 on cpu
CPU states: 99.0% idle,  0.5% user,  0.5% kernel,  0.0% iowait,  0.0% swap
Memory: 1024M phys mem, 91M free mem, 1124M total swap, 1054M free swap
   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 12321 root       1  59    0   67M 9688K sleep    3:07  0.17% mixer_applet2
  5306 root       1  52    0 3104K 1696K cpu/1    0:00  0.12% top
 12319 root       1  59    0   69M   11M sleep    1:19  0.07% gnome-netstatus
 12251 root       1  59    0   10M 6100K sleep    1:16  0.06% gconfd-2
 12226 root       1  59    0   25M 3596K sleep    1:15  0.06% vmtoolsd
 19353 noaccess  18  59    0  122M   97M sleep    2:16  0.04% java
  3961 noaccess  18  59    0  140M   87M sleep    6:17  0.03% java
  3869 root       1  59    0   73M   20M sleep    1:33  0.03% Xorg
  2015 root       1  59    0 7924K 3300K sleep    7:51  0.03% vmtoolsd
 12300 root       1  59    0   72M   12M sleep    0:39  0.03% gnome-panel


Prstat:
Prstat is in-built Solaris performance monitoring tool.Its very useful to find zone wise CPU utilization and also we can monitor amount of time threads wait for a processor using prstat -mL command in the LAT coloum.

bash-3.00# prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 18759 root     3856K 3144K cpu0    59    0   0:00:01 1.5% prstat/1
  2015 root     7572K 3744K sleep   59    0   0:05:25 0.2% vmtoolsd/1
 12226 root       25M 6696K sleep   59    0   0:00:28 0.2% vmtoolsd/1
 18497 root     6448K 3644K sleep   59    0   0:00:02 0.2% sshd/1
 12321 root       67M   10M sleep   59    0   0:00:52 0.2% mixer_applet2/1
     5 root        0K    0K sleep   99  -20   0:03:47 0.2% zpool-rpool/39
 12319 root       69M   13M sleep   59    0   0:00:23 0.1% gnome-netstatus/1
  3961 noaccess  140M   95M sleep   59    0   0:04:10 0.1% java/18
 12251 root       10M 7452K sleep   59    0   0:00:25 0.1% gconfd-2/1
  3869 root       73M   24M sleep   59    0   0:01:00 0.0% Xorg/1
 12300 root       72M   14M sleep   59    0   0:00:13 0.0% gnome-panel/1
 18508 root     2984K 1716K sleep   59    0   0:00:00 0.0% bash/1
   392 root     6156K 2776K sleep   59    0   0:00:07 0.0% nscd/33
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
    84 root      172M  207M    20%   0:14:20 2.8%
     1 noaccess  134M  101M   9.9%   0:04:10 0.1%
     7 daemon   7528K 6816K   0.7%   0:00:12 0.0%
Total: 92 processes, 356 lwps, load averages: 0.08, 0.05, 0.04

bash-3.00# prstat -Z
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  4901 root     3808K 3052K cpu1     0    0   0:00:00 1.3% prstat/1
  4879 root     6448K 3876K sleep   59    0   0:00:02 0.4% sshd/1
  2015 root     7908K 3796K sleep   59    0   0:07:45 0.2% vmtoolsd/1
     5 root        0K    0K sleep   99  -20   0:05:30 0.2% zpool-rpool/39
 12321 root       67M 9892K sleep   59    0   0:03:04 0.2% mixer_applet2/1
 19353 noaccess  122M  105M sleep   59    0   0:02:13 0.1% java/18
  3961 noaccess  140M   94M sleep   59    0   0:06:14 0.1% java/18
 12319 root       69M   12M sleep   59    0   0:01:15 0.1% gnome-netstatus/1
  4890 root     3000K 1852K sleep   54    0   0:00:00 0.1% bash/1
 12226 root       25M 4480K sleep   59    0   0:01:11 0.1% vmtoolsd/1
 12251 root       10M 6684K sleep   59    0   0:01:15 0.1% gconfd-2/1
  3869 root       73M   21M sleep   59    0   0:01:26 0.0% Xorg/1
 12300 root       72M   13M sleep   59    0   0:00:38 0.0% gnome-panel/1
   398 daemon   4260K 2212K sleep   59    0   0:00:02 0.0% kcfd/3
  1889 root     3888K 1172K sleep   59    0   0:00:01 0.0% sshd/1
  4878 root     4800K 2640K sleep   59    0   0:00:00 0.0% sshd/1
 18497 root     6448K 3140K sleep   59    0   0:00:12 0.0% sshd/1
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     0       95  318M  242M    24%   0:32:03 2.7% global
    10       30  157M  141M    14%   0:03:04 0.1% sol1

bash-3.00# prstat -mL
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
 27258 root     4.2  17 0.0 0.0 0.0 0.0  78 0.2  76   0 48K   0 prstat/1
  1907 root     0.4 0.6 0.0 0.1 0.0 0.0  99 0.3  51   2  2K   0 vmtoolsd/1
 21975 root     0.2 0.1 0.0 0.0 0.0 0.0 100 0.1  30   1 193   0 mixer_applet/1
 21291 root     0.2 0.0 0.0 0.1 0.1 0.0 100 0.2  51   0 212   0 vmtoolsd/1
 21971 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.1  10   0  61   0 gnome-netsta/1
 22001 root     0.1 0.1 0.0 0.0 0.0  98 2.2 0.1   6   0  94   0 firefox-bin/18
 21289 root     0.1 0.1 0.0 0.0 0.0 0.0 100 0.1  20   1 161   0 gconfd-2/1
  3803 noaccess 0.1 0.1 0.0 0.0 0.0 0.0  99 0.4  97   0  96   0 java/14
 24052 root     0.1 0.1 0.0 0.0 0.1 0.0  99 0.8  11   0  42   0 gnome-termin/1
  3803 noaccess 0.1 0.0 0.0 0.1 0.0 0.1 100 0.1   0   0  33   0 java/35
 13099 root     0.1 0.0 0.0 0.0 0.1 0.0 100 0.1  25   0 100  11 Xorg/1
 21583 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1  20   0 124   0 gnome-panel/1
     5 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/13
     5 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/39
 22001 root     0.0 0.0 0.0 0.3 0.3 0.0  99 0.0   3   0  14   0 firefox-bin/1
  3803 noaccess 0.0 0.0 0.0 0.0 0.0 100 0.0 0.1   5   0   5   0 java/7
  1597 daemon   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   4   0 nfsmapid/4
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0   0   0 zpool-rpool/12
    67 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   7   0 vxconfigd/1
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0   0   0 zpool-rpool/38
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/28
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/19
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/31
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/30
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/27
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/21
 24884 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   2   0 sshd/1

SAR:

The below command will CPU statistics for every 5 seconds for five times.We can add SAR in crontab to monitor CPU usage all the time.The load averages can be monitored extended time periods by looking at run queue lengths and the amount of time that the run queue is occupied using sar -q. High number of system calls per second per processor can  be monitored using sar -c.
bash-3.00# sar -u 5 5
SunOS sfos 5.10 Generic_142910-17 i86pc    
07/18/2012
14:05:40      usr    %sys    %wio   %idle
14:05:45       0       3       0      97
14:05:50       1       2       0      98
14:05:55       0       2       0      97
14:06:00       1       3       0      96
14:06:05       0       2       0      98
Average        0       2       0      97

bash-3.00# sar -q
SunOS sfos 5.10 Generic_142910-17 i86pc    04/29/2013
10:37:41 runq-sz %runocc swpq-sz %swpocc
10:40:01     1.3      24    39.0     100
10:45:00     1.3      19    39.0     100
10:50:01     1.0      18    39.0     100
Average      1.2      20    39.0     100

bash-3.00# sar -c
SunOS sfos 5.10 Generic_142910-17 i86pc    04/29/2013
10:37:41 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
10:40:01     933     115      60    0.16    0.15   11689    8178
10:45:00     886      68      32    0.78    0.77  113195    8021
10:50:01     744      45      21    0.40    0.40   58097    6056

Average      837      68      33    0.51    0.50   71549    7254
MPSTAT:

bash-3.00# mpstat
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   78   0    1   352  117  168    1    6    2    0   311    1   3   0  97
  1   71   0    1   127   29  158    2    6    3    0   328    1   2   0  97

Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.

VMTURBO-CLOUD-CAPACITY

2 comments

  1. Great Article… Very useful… It would be great if you can provide explanation of all the possible terms to look at and what they mean.

Leave a Reply

Your email address will not be published. Required fields are marked *