Here i would like to share small piece of information about dtrace. I have used dtrace to find which process are making more system calls (i.e kernel usage). DTrace is very useful d to get the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file.
There are few simple commands to find high kernel usage using dtrace.
Here we are just grepping all the system calls.
# dtrace -n 'syscall:::entry {@num[probefunc]=count();}'
dtrace: description 'syscall:::entry ' matched 237 probes
yield 1096
portfs 1400
getdents 1440
p_online 1536
accept 8022
modctl 8430
setcontext 8658
recv 11367
readlink 14599
lwp_park 18311
unlink 215507
statvfs 216256
open 257824
read 344775
gtime 351321
close 353795
getpid 437632
pollsys 685370
write 739093
So the above highlighted process are making more system call through those “pollsys” and “write” functions.Using below command ,we are finding what are process are calling “write” function more often.
# dtrace -n 'syscall::write:return {@num[execname]= count(); }'
dtrace: description 'syscall::write:return ' matched 1 probe dtrace
eauth_userpass 1
rcapd 2
nbrmms 3
sbatchd 3
expr 30
init 60
vemkd 60
csthb.agt 154
sas.e9bd41 188
syslogd 202
ls 212
sas.e9bd12 292
esd 510
adclient 560
java 880
sas 1619
jfd 30172
spdslog 30616
Using below command . i am finding what are process are calling “pollsys” function more often.
# dtrace -n 'syscall::pollsys:return {@num[execname]= count(); }'
dtrace: description 'syscall::pollsys:return ' matched 1 probe
vxdclid 1
esd 2
nbrmms 2
vmd 3
cdcwatch 6
sbatchd 6
acsssi 7
vxpal 7
pem 8
syslogd 8
init 9
sendmail 10
mbschd 11
vemkd 15
lim 22
motifxsassm 25
mbatchd 30
egosc 36
eauth_userpass 40
sshd 46
nrpe 151
jproxy 294
objspawn 515
top 522
sastcpd 588
sas.e9bd41 811
adclient 1118
sas.e9bd12 1301
cstd.agt 1471
sas 2762
java 13360
spdslog 20554
jfd 171801
From the above output, process name,we can identify the pid’s which are making the more system calls.The below mentioned process are making more system calls.
bash-3.00# ps -ef |grep -i jfd 0041199 26430 1 0 May 25 ? 10:31 /comm/pm/3.0/sparc-sol7-32/etc/jfd0041248 4491 1 0 May 25 ? 6:32 /comm/pm/3.0/sparc-sol7-32/etc/jfd0041199 11487 1 0 May 25 ? 835:06 /comm/ThirdParty/PlatformComputing/PM/7.1/sparc-sol10-64/etc/jfd0041248 10977 1 0 May 25 ? 437:53 /comm/ThirdParty/PlatformComputing/PM/7.1/sparc-sol10-64/etc/jfd root 11652 23517 0 13:54:23 pts/25 0:00 grep -i jfd
Note:Dtrace command will not stop unless you press control+C.You have to terminate it after 2 to 3 minutes
Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.