Home / Solaris 10 / Dtrace-To find high kernal usage

Dtrace-To find high kernal usage

Here i would like to share small piece of information about dtrace. I have used dtrace to find which process are making more system calls (i.e kernel usage). DTrace is very useful d to get the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.

There are few simple commands to find high kernel usage using dtrace.
Here we are just grepping all the system calls.

# dtrace -n 'syscall:::entry {@num[probefunc]=count();}'
dtrace: description 'syscall:::entry ' matched 237 probes   
 yield                          1096 
 portfs                         1400 
 getdents                       1440 
 p_online                       1536
accept                          8022
  modctl                        8430
  setcontext                    8658
  recv                          11367
  readlink                      14599
  lwp_park                      18311
unlink                         215507
  statvfs                      216256
  open                         257824
  read                         344775
  gtime                        351321
close                          353795
  getpid                       437632
  pollsys                      685370
  write                        739093

So the above highlighted process are making more system call through those “pollsys” and “write” functions.Using below command ,we are  finding what are process are calling “write” function more often.

# dtrace -n 'syscall::write:return {@num[execname]= count(); }'
dtrace: description 'syscall::write:return ' matched 1 probe  dtrace                                                            
  eauth_userpass                                  1
  rcapd                                           2
  nbrmms                                          3
  sbatchd                                         3                
expr                                              30
  init                                            60
  vemkd                                           60
  csthb.agt                                       154
  sas.e9bd41                                      188
  syslogd                                         202
  ls                                              212
  sas.e9bd12                                      292
  esd                                             510
  adclient                                        560
  java                                            880
  sas                                            1619
  jfd                                            30172
  spdslog                                        30616

Using below command . i am finding what are process are calling “pollsys” function more often.

# dtrace -n 'syscall::pollsys:return {@num[execname]= count(); }'
dtrace: description 'syscall::pollsys:return ' matched 1 probe  

  vxdclid                       1  
  esd                           2 
  nbrmms                        2
  vmd                           3
  cdcwatch                      6
  sbatchd                       6
  acsssi                        7
  vxpal                         7
  pem                           8
  syslogd                       8
  init                          9
  sendmail                     10
  mbschd                       11
  vemkd                        15
  lim                          22                  
  motifxsassm                  25
  mbatchd                      30
  egosc                        36     
  eauth_userpass               40 
 sshd                          46
  nrpe                        151
  jproxy                      294
  objspawn                    515
  top                         522
  sastcpd                     588
  sas.e9bd41                  811
  adclient                   1118
  sas.e9bd12                 1301
  cstd.agt                   1471
  sas                        2762
  java                      13360
  spdslog                   20554
  jfd                      171801

From the above output, process name,we can identify the pid’s which are making the more system calls.The below mentioned process are making more system calls.

bash-3.00# ps -ef |grep -i jfd
0041199 26430     1   0   May 25 ?          10:31 /comm/pm/3.0/sparc-sol7-32/etc/jfd0041248  
4491     1   0   May 25 ? 6:32 /comm/pm/3.0/sparc-sol7-32/etc/jfd0041199
11487     1   0   May 25 ?  835:06 /comm/ThirdParty/PlatformComputing/PM/7.1/sparc-sol10-64/etc/jfd0041248 10977     1   0   May 25 ?         437:53 /comm/ThirdParty/PlatformComputing/PM/7.1/sparc-sol10-64/etc/jfd    root 11652 23517   0 13:54:23 pts/25      0:00 grep -i jfd

Note:Dtrace command will not stop unless you press control+C.You have to terminate it after 2 to 3 minutes 

Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.

VMTURBO-CLOUD-CAPACITY