In this article, we will look at DTrace toolkit which is used to troubleshoot performance issues on Solaris operating systems. One of the co-authors of Solaris Performance and Tools [McDougall2007V2], Brendan Gregg, created this toolkit. It contains numerous d-scripts as well as Perl scripts that pre-process and post-process DTrace. DTrace toolkit is included in Oracle Solaris 11 and the package name is “dtrace-toolkit”. In Oracle Solaris 10, We need to download the toolkit separately from oracle website.
Installation :
1. Download DTrace Toolkit package from Oracle website. (only for Solaris 10).
- Oracle Solaris 10 – DTrace Toolkit Download
- Oracle Solaris 11 – Available as a package “dtrace-toolkit”
Unzip the downloaded file and untar it. Navigate to the Bin directroy.
# mkdir -p /usr/dtrace/DTT # cp DTraceToolkit.tar.gz /usr/dtrace/DTT # cd /usr/dtrace/DTT # gunzip DTraceToolkit.tar.gz # tar -xf DTraceToolkit.tar # cd DTraceToolkit-0.99/Bin #
2. In Oracle Solaris 11, if you don’t see the path “/usr/dtrace/DTT”, then you need to install the package. Server should have local or oracle repo configured.
# pkg install dtrace-toolkit
3. Once the packages are ready, navigate to the Bin directory to find the pre-built DTrace scripts.
# cd Bin # ls -lrt total 230 lrwxrwxrwx 1 100 other 19 Jun 12 09:20 pathopens.d -> ../Proc/pathopens.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 rb_funccalls.d -> ../Ruby/rb_funccalls.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 threaded.d -> ../Proc/threaded.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 pgpginbypid.d -> ../Mem/pgpginbypid.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 weblatency.d -> ../Apps/weblatency.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 writebytes.d -> ../Proc/writebytes.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_subcalls.d -> ../Perl/pl_subcalls.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 fspaging.d -> ../FS/fspaging.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 rb_calls.d -> ../Ruby/rb_calls.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 py_calltime.d -> ../Python/py_calltime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 hotspot.d -> ../Disk/hotspot.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 js_objgc.d -> ../JavaScript/js_objgc.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 j_events.d -> ../Java/j_events.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 connections -> ../Net/connections lrwxrwxrwx 1 100 other 21 Jun 12 09:20 php_calldist.d -> ../Php/php_calldist.d lrwxrwxrwx 1 100 other 27 Jun 12 09:20 js_flowtime.d -> ../JavaScript/js_flowtime.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 php_calltime.d -> ../Php/php_calltime.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 py_calldist.d -> ../Python/py_calldist.d lrwxrwxrwx 1 100 other 13 Jun 12 09:20 wpm.d -> ../Misc/wpm.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 seeksize.d -> ../Disk/seeksize.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 lockbyproc.d -> ../Locks/lockbyproc.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 tcl_stat.d -> ../Tcl/tcl_stat.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 statsnoop -> ../statsnoop lrwxrwxrwx 1 100 other 16 Jun 12 09:20 j_stat.d -> ../Java/j_stat.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 fsrw.d -> ../FS/fsrw.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 pfilestat -> ../Proc/pfilestat lrwxrwxrwx 1 100 other 15 Jun 12 09:20 vmstat.d -> ../Mem/vmstat.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 sh_calltime.d -> ../Shell/sh_calltime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 tcl_flow.d -> ../Tcl/tcl_flow.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 sh_lines.d -> ../Shell/sh_lines.d lrwxrwxrwx 1 100 other 14 Jun 12 09:20 woof.d -> ../Misc/woof.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 udpstat.d -> ../Net/udpstat.d lrwxrwxrwx 1 100 other 13 Jun 12 09:20 vopstat -> ../FS/vopstat lrwxrwxrwx 1 100 other 20 Jun 12 09:20 topsyscall -> ../System/topsyscall lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_calltime.d -> ../Perl/pl_calltime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 php_flow.d -> ../Php/php_flow.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 rb_objcpu.d -> ../Ruby/rb_objcpu.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 py_malloc.d -> ../Python/py_malloc.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 anonpgpid.d -> ../Mem/anonpgpid.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 zvmstat -> ../Zones/zvmstat lrwxrwxrwx 1 100 other 12 Jun 12 09:20 execsnoop -> ../execsnoop lrwxrwxrwx 1 100 other 17 Jun 12 09:20 sh_who.d -> ../Shell/sh_who.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 php_funccalls.d -> ../Php/php_funccalls.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 sh_cpudist.d -> ../Shell/sh_cpudist.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 sh_calldist.d -> ../Shell/sh_calldist.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 minfbypid.d -> ../Mem/minfbypid.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_calldist.d -> ../Perl/pl_calldist.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 rwbytype.d -> ../Proc/rwbytype.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 tcpsnoop.d -> ../Net/tcpsnoop.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 js_calls.d -> ../JavaScript/js_calls.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 rwbbypid.d -> ../Proc/rwbbypid.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 inttimes.d -> ../Cpu/inttimes.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 tcpstat.d -> ../Net/tcpstat.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcl_procflow.d -> ../Tcl/tcl_procflow.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 js_stat.d -> ../JavaScript/js_stat.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcl_flowtime.d -> ../Tcl/tcl_flowtime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 intoncpu.d -> ../Cpu/intoncpu.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 tcl_proccalls.d -> ../Tcl/tcl_proccalls.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 pl_flow.d -> ../Perl/pl_flow.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 cputimes -> ../Kernel/cputimes lrwxrwxrwx 1 100 other 15 Jun 12 09:20 guess.d -> ../Misc/guess.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 js_execs.d -> ../JavaScript/js_execs.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 php_flowinfo.d -> ../Php/php_flowinfo.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 cpuwalk.d -> ../Cpu/cpuwalk.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 sh_calls.d -> ../Shell/sh_calls.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 j_calls.d -> ../Java/j_calls.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 j_syscalls.d -> ../Java/j_syscalls.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 putnexts.d -> ../Kernel/putnexts.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 rb_flowinfo.d -> ../Ruby/rb_flowinfo.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 j_profile.d -> ../Java/j_profile.d lrwxrwxrwx 1 100 other 8 Jun 12 09:20 iotop -> ../iotop lrwxrwxrwx 1 100 other 23 Jun 12 09:20 sh_syscolors.d -> ../Shell/sh_syscolors.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 py_flowinfo.d -> ../Python/py_flowinfo.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 whatexec.d -> ../Kernel/whatexec.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 intbycpu.d -> ../Cpu/intbycpu.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 hotkernel -> ../hotkernel lrwxrwxrwx 1 100 other 19 Jun 12 09:20 tcpsnoop_snv -> ../Net/tcpsnoop_snv lrwxrwxrwx 1 100 other 23 Jun 12 09:20 kstat_types.d -> ../Kernel/kstat_types.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 rb_lines.d -> ../Ruby/rb_lines.d lrwxrwxrwx 1 100 other 14 Jun 12 09:20 kill.d -> ../Proc/kill.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 vmbypid.d -> ../Mem/vmbypid.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 vmstat-p.d -> ../Mem/vmstat-p.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 j_flowtime.d -> ../Java/j_flowtime.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 pl_cputime.d -> ../Perl/pl_cputime.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 creatbyproc.d -> ../Proc/creatbyproc.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 modcalls.d -> ../Kernel/modcalls.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 py_mallocstk.d -> ../Python/py_mallocstk.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 php_malloc.d -> ../Php/php_malloc.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 js_flow.d -> ../JavaScript/js_flow.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 readdist.d -> ../Proc/readdist.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 php_syscalls.d -> ../Php/php_syscalls.d lrwxrwxrwx 1 100 other 25 Jun 12 09:20 js_objnew.d -> ../JavaScript/js_objnew.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 py_syscalls.d -> ../Python/py_syscalls.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 j_thread.d -> ../Java/j_thread.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 py_who.d -> ../Python/py_who.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 j_calltime.d -> ../Java/j_calltime.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 pl_cpudist.d -> ../Perl/pl_cpudist.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 sh_flowtime.d -> ../Shell/sh_flowtime.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 sysbypid.d -> ../Proc/sysbypid.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 php_who.d -> ../Php/php_who.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_flowtime.d -> ../Perl/pl_flowtime.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 rb_objnew.d -> ../Ruby/rb_objnew.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 minfbyproc.d -> ../Mem/minfbyproc.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 sh_wasted.d -> ../Shell/sh_wasted.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 shellsnoop -> ../Apps/shellsnoop lrwxrwxrwx 1 100 other 17 Jun 12 09:20 newproc.d -> ../Proc/newproc.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 readbytes.d -> ../Proc/readbytes.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 rb_calltime.d -> ../Ruby/rb_calltime.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 iofile.d -> ../Disk/iofile.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 tcptop_snv -> ../Net/tcptop_snv lrwxrwxrwx 1 100 other 17 Jun 12 09:20 icmpstat.d -> ../Net/icmpstat.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 pidpersec.d -> ../Proc/pidpersec.d lrwxrwxrwx 1 100 other 13 Jun 12 09:20 tcptop -> ../Net/tcptop lrwxrwxrwx 1 100 other 19 Jun 12 09:20 j_cputime.d -> ../Java/j_cputime.d lrwxrwxrwx 1 100 other 25 Jun 12 09:20 syscallbysysc.d -> ../System/syscallbysysc.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 py_funccalls.d -> ../Python/py_funccalls.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 pl_who.d -> ../Perl/pl_who.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 rb_calldist.d -> ../Ruby/rb_calldist.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 j_calldist.d -> ../Java/j_calldist.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 cswstat.d -> ../Kernel/cswstat.d lrwxrwxrwx 1 100 other 13 Jun 12 09:20 rfsio.d -> ../FS/rfsio.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 sampleproc -> ../Proc/sampleproc lrwxrwxrwx 1 100 other 22 Jun 12 09:20 js_who.d -> ../JavaScript/js_who.d lrwxrwxrwx 1 100 other 24 Jun 12 09:20 py_syscolors.d -> ../Python/py_syscolors.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 j_cpudist.d -> ../Java/j_cpudist.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 rb_cputime.d -> ../Ruby/rb_cputime.d lrwxrwxrwx 1 100 other 10 Jun 12 09:20 rwsnoop -> ../rwsnoop lrwxrwxrwx 1 100 other 21 Jun 12 09:20 j_syscolors.d -> ../Java/j_syscolors.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 sh_syscalls.d -> ../Shell/sh_syscalls.d lrwxrwxrwx 1 100 other 15 Jun 12 09:20 runocc.d -> ../Cpu/runocc.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_syscalls.d -> ../Perl/pl_syscalls.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 iopending -> ../Disk/iopending lrwxrwxrwx 1 100 other 23 Jun 12 09:20 sh_pidcolors.d -> ../Shell/sh_pidcolors.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 sh_flowinfo.d -> ../Shell/sh_flowinfo.d lrwxrwxrwx 1 100 other 15 Jun 12 09:20 rfileio.d -> ../FS/rfileio.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcl_calltime.d -> ../Tcl/tcl_calltime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 rb_flow.d -> ../Ruby/rb_flow.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 pridist.d -> ../Kernel/pridist.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 rb_cpudist.d -> ../Ruby/rb_cpudist.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 pl_syscolors.d -> ../Perl/pl_syscolors.d lrwxrwxrwx 1 100 other 14 Jun 12 09:20 fddist -> ../Proc/fddist lrwxrwxrwx 1 100 other 17 Jun 12 09:20 rb_stat.d -> ../Ruby/rb_stat.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pl_flowinfo.d -> ../Perl/pl_flowinfo.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 php_flowtime.d -> ../Php/php_flowtime.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 tcl_who.d -> ../Tcl/tcl_who.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 j_objnew.d -> ../Java/j_objnew.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 xcallsbypid.d -> ../Cpu/xcallsbypid.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 py_flowtime.d -> ../Python/py_flowtime.d lrwxrwxrwx 1 100 other 10 Jun 12 09:20 dvmstat -> ../dvmstat lrwxrwxrwx 1 100 other 20 Jun 12 09:20 filebyproc.d -> ../Proc/filebyproc.d lrwxrwxrwx 1 100 other 15 Jun 12 09:20 crash.d -> ../Proc/crash.d lrwxrwxrwx 1 100 other 14 Jun 12 09:20 procsystime -> ../procsystime lrwxrwxrwx 1 100 other 19 Jun 12 09:20 httpdstat.d -> ../Apps/httpdstat.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 sh_cputime.d -> ../Shell/sh_cputime.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcl_calldist.d -> ../Tcl/tcl_calldist.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 j_flow.d -> ../Java/j_flow.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 pl_malloc.d -> ../Perl/pl_malloc.d lrwxrwxrwx 1 100 other 14 Jun 12 09:20 loads.d -> ../Cpu/loads.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 rwbypid.d -> ../Proc/rwbypid.d lrwxrwxrwx 1 100 other 9 Jun 12 09:20 dtruss -> ../dtruss lrwxrwxrwx 1 100 other 17 Jun 12 09:20 dapptrace -> ../Proc/dapptrace lrwxrwxrwx 1 100 other 10 Jun 12 09:20 iosnoop -> ../iosnoop lrwxrwxrwx 1 100 other 15 Jun 12 09:20 j_who.d -> ../Java/j_who.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 dexplorer -> ../dexplorer lrwxrwxrwx 1 100 other 17 Jun 12 09:20 sigdist.d -> ../Proc/sigdist.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 dappprof -> ../Proc/dappprof lrwxrwxrwx 1 100 other 23 Jun 12 09:20 j_methodcalls.d -> ../Java/j_methodcalls.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 py_cputime.d -> ../Python/py_cputime.d lrwxrwxrwx 1 100 other 23 Jun 12 09:20 syscallbyproc.d -> ../Proc/syscallbyproc.d lrwxrwxrwx 1 100 other 25 Jun 12 09:20 js_objcpu.d -> ../JavaScript/js_objcpu.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 bitesize.d -> ../Disk/bitesize.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 writedist.d -> ../Proc/writedist.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 py_flow.d -> ../Python/py_flow.d lrwxrwxrwx 1 100 other 27 Jun 12 09:20 js_flowinfo.d -> ../JavaScript/js_flowinfo.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 rb_syscalls.d -> ../Ruby/rb_syscalls.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 setuids.d -> ../User/setuids.d lrwxrwxrwx 1 100 other 10 Jun 12 09:20 errinfo -> ../errinfo lrwxrwxrwx 1 100 other 22 Jun 12 09:20 py_cpudist.d -> ../Python/py_cpudist.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 dnlcps.d -> ../Kernel/dnlcps.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 iopattern -> ../iopattern lrwxrwxrwx 1 100 other 22 Jun 12 09:20 rb_syscolors.d -> ../Ruby/rb_syscolors.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 shortlived.d -> ../Proc/shortlived.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 tcl_insflow.d -> ../Tcl/tcl_insflow.d lrwxrwxrwx 1 100 other 15 Jun 12 09:20 tcpsnoop -> ../Net/tcpsnoop lrwxrwxrwx 1 100 other 14 Jun 12 09:20 xvmstat -> ../Mem/xvmstat lrwxrwxrwx 1 100 other 17 Jun 12 09:20 swapinfo.d -> ../Mem/swapinfo.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 dnlcstat -> ../Kernel/dnlcstat lrwxrwxrwx 1 100 other 21 Jun 12 09:20 dnlcsnoop.d -> ../Kernel/dnlcsnoop.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 lastwords -> ../Proc/lastwords lrwxrwxrwx 1 100 other 19 Jun 12 09:20 stacksize.d -> ../Proc/stacksize.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 tcl_calls.d -> ../Tcl/tcl_calls.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 sh_flow.d -> ../Shell/sh_flow.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 tcpwdist.d -> ../Net/tcpwdist.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 tcl_syscolors.d -> ../Tcl/tcl_syscolors.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 pgpginbyproc.d -> ../Mem/pgpginbyproc.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 topsysproc -> ../Proc/topsysproc lrwxrwxrwx 1 100 other 19 Jun 12 09:20 j_package.d -> ../Java/j_package.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 sar-c.d -> ../System/sar-c.d lrwxrwxrwx 1 100 other 10 Jun 12 09:20 hotuser -> ../hotuser lrwxrwxrwx 1 100 other 22 Jun 12 09:20 py_profile.d -> ../Python/py_profile.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 rb_flowtime.d -> ../Ruby/rb_flowtime.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 php_cputime.d -> ../Php/php_cputime.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 mmapfiles.d -> ../Proc/mmapfiles.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 priclass.d -> ../Kernel/priclass.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 rb_who.d -> ../Ruby/rb_who.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 php_cpudist.d -> ../Php/php_cpudist.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 uname-a.d -> ../System/uname-a.d lrwxrwxrwx 1 100 other 27 Jun 12 09:20 js_calltime.d -> ../JavaScript/js_calltime.d lrwxrwxrwx 1 100 other 8 Jun 12 09:20 rwtop -> ../rwtop lrwxrwxrwx 1 100 other 19 Jun 12 09:20 rb_malloc.d -> ../Ruby/rb_malloc.d lrwxrwxrwx 1 100 other 27 Jun 12 09:20 js_calldist.d -> ../JavaScript/js_calldist.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 tcl_ins.d -> ../Tcl/tcl_ins.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcpsnoop_snv.d -> ../Net/tcpsnoop_snv.d lrwxrwxrwx 1 100 other 12 Jun 12 09:20 opensnoop -> ../opensnoop lrwxrwxrwx 1 100 other 21 Jun 12 09:20 tcl_syscalls.d -> ../Tcl/tcl_syscalls.d lrwxrwxrwx 1 100 other 16 Jun 12 09:20 diskhits -> ../Disk/diskhits lrwxrwxrwx 1 100 other 26 Jun 12 09:20 js_cputime.d -> ../JavaScript/js_cputime.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 syscallbypid.d -> ../Proc/syscallbypid.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 sh_stat.d -> ../Shell/sh_stat.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 j_classflow.d -> ../Java/j_classflow.d lrwxrwxrwx 1 100 other 22 Jun 12 09:20 php_syscolors.d -> ../Php/php_syscolors.d lrwxrwxrwx 1 100 other 19 Jun 12 09:20 nfswizard.d -> ../Apps/nfswizard.d lrwxrwxrwx 1 100 other 21 Jun 12 09:20 lockbydist.d -> ../Locks/lockbydist.d lrwxrwxrwx 1 100 other 26 Jun 12 09:20 js_cpudist.d -> ../JavaScript/js_cpudist.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 tcl_cputime.d -> ../Tcl/tcl_cputime.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 iofileb.d -> ../Disk/iofileb.d lrwxrwxrwx 1 100 other 18 Jun 12 09:20 cpudists -> ../Kernel/cpudists lrwxrwxrwx 1 100 other 17 Jun 12 09:20 cputypes.d -> ../Cpu/cputypes.d lrwxrwxrwx 1 100 other 20 Jun 12 09:20 tcl_cpudist.d -> ../Tcl/tcl_cpudist.d lrwxrwxrwx 1 100 other 17 Jun 12 09:20 dispqlen.d -> ../Cpu/dispqlen.d #
4. To know the top system calls, use the following script.
:# ./topsyscall 1 10 Tracing... Please wait. 2018 May 24 11:23:06, load average: 0.00, 0.00, 0.00 syscalls: 604 SYSCALL COUNT adjtime 1 fstat 1 mmap 1 schedctl 1 setcontext 1 sigsuspend 1 read 2 stat 3 sysconfig 3 write 3 sigaction 4 gtime 5 brk 6 pollsys 8 lwp_park 9 lwp_sigmask 13 p_online 256 ioctl 286 2018 May 24 11:23:07, load average: 0.00, 0.00, 0.00 syscalls: 381 SYSCALL COUNT adjtime 1 fstat 1 mmap 1 setcontext 1 sigsuspend 1 stat 1 close 2 lwp_park 2 open 2 pset 2 brk 4 read 5 pollsys 9 lwp_sigmask 11 gtime 12 write 26 ioctl 300
5. Here is another example to identify the CPU time.
# ./cputimes -at 8 5
2018 May 24 11:05:10,
THREADS TIME (ns)
nbdisco 356080
xntpd 416870
kslog 616290
dtrace 2879610
filmon 10186430
fsflush 31015160
KERNEL 54456080
IDLE 55791700420
2018 May 24 11:05:15,
THREADS TIME (ns)
xntpd 419660
fmd 466290
kslog 639300
dtrace 4409410
fsflush 5664960
filmon 6399760
KERNEL 51839790
IDLE 59893010430
6. To find wich LUN’s are heavily used in the system, use “iotop” script.
# ./iotop
Tracing... Please wait.
2018 Jun 12 08:04:27, load: 0.93, disk_r: 423853 KB, disk_w: 56700 KB
UID PID PPID CMD DEVICE MAJ MIN D BYTES
0 0 0 sched ssd162 242 1296 W 4096
0 0 0 sched ssd450 242 3600 W 4096
0 0 0 sched ssd452 242 3616 W 4096
0 0 0 sched ssd453 242 3624 W 4096
0 0 0 sched ssd454 242 3632 W 4096
0 0 0 sched ssd456 242 3648 W 4096
0 0 0 sched ssd460 242 3680 W 4096
0 0 0 sched ssd57 242 456 W 4096
0 0 0 sched ssd473 242 3784 R 6656
0 0 0 ssd51 242 408 7168
0 0 0 sched ssd399 242 3192 W 8192
There are almost 230 DTrace scripts available on the toolkit. You could give a try and it’s really helpful to identify the system bottlenecks. In one of my post, I have written manual DTrace query to identify the high kernel usage cause. But the pre-built scripts looks so easy compared to the manual queries.