Home / Solaris 10 / Solaris Processes Monitoring – prstat

Solaris Processes Monitoring – prstat

Solaris Processes Monitoring (PROC) 

In normal Unix operation ,you need to deal with process and this will be the  routine job in Unix administration .In Solaris we have number of tools to examine process details and those just reading the /proc directory.These tools are very useful in application troubleshooting and analyzing the system performance.This article is based on “Solaris™ Internals: Solaris 10 and OpenSolaris Kernel Architecture” book which is a very good one  for Solaris experts. I would strongly recommend this.

Here we have differentiated the process stuffs in five Categories.
1.Process status tools (Ex:prstat,ps)
2.Process controlling tools (Ex:pkill,prun)
3.Process inspection tools (Ex:pstack,pfiles)
4.Process Lock Activity examination tools (Ex:plockstat)
5.Process Tracing tools (Ex:Dtrace,apptrace,truss)

1.Process status tools:

The prstat statistics utility shows a top-level summary of the processes that are using system resources currently.The prstat utility summarizes this information for an every 5 seconds by default and reports the statistics for that period.

# prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 39 0 0:00:01 0.3% sshd/1
5 root 0K 0K sleep 99 -20 0:38:00 0.3% zpool-rpool/39
1921 root 6708K 4064K sleep 59 0 0:02:19 0.1% vmtoolsd/1
413 daemon 4260K 2384K sleep 59 0 0:00:02 0.1% kcfd/3
13170 root 3744K 2920K cpu0 19 0 0:00:00 0.1% prstat/1
3436 root 11M 7532K sleep 59 0 0:00:14 0.1% dtgreet/1
13161 root 3008K 1772K sleep 49 0 0:00:00 0.1% bash/1
3374 root 61M 15M sleep 59 0 0:00:18 0.1% Xorg/1
9 root 12M 10M sleep 59 0 0:00:02 0.1% svc.startd/13
1814 root 3888K 1336K sleep 59 0 0:00:00 0.0% sshd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:18 0.0% nfsmapid/4

The below table talks about default fields in prstat command output and its description.
table.tableizer-table { border: 1px solid #CCC; font-family: ; font-size: 12px; } .tableizer-table td { padding: 4px; margin: 3px; border: 1px solid #ccc; } .tableizer-table th { background-color: #104E8B; color: #FFF; font-weight: bold; }

PRSTAT FIELDS
Field Description
PID The unique process ID of the process
USERNAME Owner (user or uid) of the process
SIZE The total virtual memory size of mappings within the process, including all mapped files and devices.
RSS Resident set size. The amount of physical memory mapped into the process,including that shared with other processes.
STATE The state of the process
PRI The priority of the process. Larger numbers mean higher priority.
NICE Nice value used in priority computation.



The below table explains about various prstat options.

PRSTAT OPTIONS
Options Description
prstat This shows a top-level summary of the processes that are using system resources currently for every five mins
prstat -L This shows thread per line instead of one process per line.In last column you can see name of the process (executed file)
prstat -mL It shows microstates of process & sow per-thread ( -L)
prstat -s prstat ouput can be sorted using set of sub-options .sub options are cpu,pri,rss,size,time (Ex:prstat -s cpu)
cpu – sort by process cpu usage on the system
pri – sort the process by priority
rss – sort by resident set size (i.e physical memory)
size – sort by size of the process (i.e virtual memory)
time – sort by process execution time
prstat -t It provides complete users resource utlization
prstat -J It gives project wise resource utilization
prstat -Z It provides summary per local zone.

Here we will see one by one prstat output 
prstat -L

# prstat -L
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
13150 root 6448K 3744K sleep 59 0 0:00:01 0.2% sshd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:17 0.1% nfsmapid/3
1921 root 6708K 4064K sleep 59 0 0:02:21 0.1% vmtoolsd/1
13426 root 4032K 3220K cpu0 49 0 0:00:00 0.1% prstat/1
5 root 0K 0K sleep 0 -20 0:00:11 0.0% zpool-rpool/13
5 root 0K 0K sleep 0 -20 0:00:10 0.0% zpool-rpool/38
3374 root 61M 15M sleep 59 0 0:00:18 0.0% Xorg/1
3436 root 11M 7532K sleep 59 0 0:00:14 0.0% dtgreet/1
13161 root 3008K 1772K sleep 49 0 0:00:00 0.0% bash/1
Total: 70 processes, 337 lwps, load averages: 0.01, 0.02, 0.01

prstat -mL
This command is very useful to find out where is the actual issue on the system.The blow output shows the percentage of time spent in a given microstate over the last sampling period (default is 5 seconds), beginning with the USR column, up to and including the LAT column. The values in the columns USR through LAT should total 100, accounting for 100% of the threads time for the last sampling period. 

# prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
13660 root 11 88 0.0 0.0 0.0 0.0 0.0 0.3 0 227 52K 0 prstat/1
3437 root 5.0 3.3 0.0 0.0 0.0 0.0 89 3.1 5K 24 4K 0 /0
5 root 0.0 2.3 0.0 0.0 0.0 0.0 96 1.5 59 5 0 0 zpool-rpool/12
5 root 0.0 2.3 0.0 0.0 0.0 0.0 96 1.6 59 5 0 0 zpool-rpool/39
5 root 0.0 0.8 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/19
5 root 0.0 0.7 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/22
5 root 0.0 0.7 0.0 0.0 0.0 0.0 98 1.1 8 1 0 0 zpool-rpool/25

table.tableizer-table { border: 1px solid #CCC; font-family: ; font-size: 12px; } .tableizer-table td { padding: 4px; margin: 3px; border: 1px solid #ccc; } .tableizer-table th { background-color: #104E8B; color: #FFF; font-weight: bold; }

System Bottleneck determination using prstat -mL
Description Fields
Other users and application (Like top command usr field) USR
Kernel (Like top command sys fields) SYS
Monitor the amount of time threads wait for a processor (CPU bottleneck identification) LAT
Memory paging waits (Memory bottleneck identification) TFL and DTL
Disk and Network waits (I/O bottlenect identification ) SLP

prstat -s

# prstat -s cpu
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 39 0 0:00:01 0.5% sshd/1
5 root 0K 0K sleep 99 -20 0:38:09 0.2% zpool-rpool/39
1921 root 6724K 4076K sleep 59 0 0:02:29 0.2% vmtoolsd/1
1611 daemon 4728K 1364K sleep 59 0 0:00:22 0.0% nfsmapid/4
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
14506 root 3728K 2924K cpu0 59 0 0:00:00 0.0% prstat/1
# prstat -s pri
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
347 root 0K 0K sleep 99 -20 0:00:00 0.0% zpool-lzpool1/39
351 root 0K 0K sleep 99 -20 0:00:00 0.0% zpool-szpool/39
5 root 0K 0K sleep 99 -20 0:38:10 0.1% zpool-rpool/39
1793 daemon 2468K 1344K sleep 60 -20 0:00:00 0.0% nfsd/2
1620 daemon 2428K 1312K sleep 60 -20 0:00:00 0.0% lockd/2
1610 daemon 2448K 1288K sleep 60 -20 0:00:00 0.0% nfs4cbd/2
1772 root 3260K 2288K sleep 59 0 0:00:01 0.0% vold/6
# prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3374 root 61M 15M sleep 58 0 0:00:20 0.0% Xorg/1
1826 root 16M 12M sleep 59 0 0:00:02 0.0% fmd/19
11 root 12M 11M sleep 59 0 0:00:06 0.0% svc.configd/15
9 root 12M 10M sleep 59 0 0:00:02 0.0% svc.startd/13
67 root 12M 9536K sleep 59 0 0:00:02 0.0% vxconfigd/3
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
# prstat -s size
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
3374 root 61M 15M sleep 59 0 0:00:20 0.0% Xorg/1
1826 root 16M 12M sleep 59 0 0:00:02 0.0% fmd/19
9 root 12M 10M sleep 59 0 0:00:02 0.0% svc.startd/13
11 root 12M 11M sleep 59 0 0:00:06 0.0% svc.configd/15
67 root 12M 9536K sleep 59 0 0:00:02 0.0% vxconfigd/3
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
# prstat -s time
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1921 root 6724K 4080K run 59 0 0:02:29 0.2% vmtoolsd/1
3374 root 61M 15M sleep 59 0 0:00:20 0.0% Xorg/1
3436 root 11M 7532K sleep 59 0 0:00:16 0.0% dtgreet/1
45 root 2456K 1164K sleep 59 0 0:00:06 0.0% dhcpagent/1
3414 root 3968K 1532K sleep 59 0 0:00:02 0.0% veaintf/1
13150 root 6448K 3744K sleep 59 0 0:00:02 0.7% sshd/1

prstat -t

# prstat -t
NPROC USERNAME SWAP RSS MEMORY TIME CPU
62 root 94M 101M 14% 0:41:55 4.9%
8 daemon 8296K 7720K 1.0% 0:00:27 0.0%

prstat -J

# prstat -J
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13150 root 6448K 3744K sleep 59 0 0:00:02 0.2% sshd/1
1921 root 6724K 4080K sleep 59 0 0:02:30 0.2% vmtoolsd/1
13161 root 3012K 1776K sleep 39 0 0:00:00 0.1% bash/1
5 root 0K 0K sleep 99 -20 0:38:10 0.1% zpool-rpool/39
14651 root 3728K 2924K cpu1 49 0 0:00:00 0.1% prstat/1
PROJID NPROC SWAP RSS MEMORY TIME CPU PROJECT
1 4 3800K 7432K 1.0% 0:00:02 0.4% user.root
0 66 99M 103M 14% 0:42:18 0.3% system

prstat -Z

# prstat -Z
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5 root 0K 0K sleep 99 -20 0:38:15 8.5% zpool-rpool/39
14781 root 9388K 8152K run 59 0 0:00:03 5.4% svc.configd/11
1826 root 16M 12M sleep 59 0 0:00:08 5.4% fmd/19
14779 root 8440K 7372K run 59 0 0:00:02 2.6% svc.startd/54
349 root 6024K 3176K run 59 0 0:00:03 2.3% devfsadm/8
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 71 104M 109M 14% 0:42:40 25% global
1 6 13M 15M 2.0% 0:00:05 8.3% sol1

Hope you have learned many things about prstat command and Solaris process status monitoring 

Thank you for reading this article.Please leave a comment if you have any doubt .I will get back to you.

VMTURBO-CLOUD-CAPACITY

3 comments

  1. what is difference between swap ,RSS ,memory in prstat -a

  2. cpu utilisation in my server shows 100% but i see only 2-3 process that are using 8-9% of CPU. so why rest of 90% cpu is not idle.

Leave a Reply

Your email address will not be published. Required fields are marked *