Solaris 10 provides a bunch of observability tools to observe the processes, memory, processor, IO etc. Observing the system metrics data on solaris 10 can be done vi, SMC (Solaris Management Console) and command line stat utilities(iostat, vmstat etc).
Ideal way to observe a process/application is to observe it memory footprints, IO, processor needs etc. This process looks easy for small scenarios. However when we have a huge piles of data for analysis to find a root cause of problems, we end up looking up in verbose information. To make our life easier, we can stick with graphs generated with the system metrics data obtained earlier.
We will be using the outputs generated from the stats utilities. we shall extract the relevant information for our analysis by a shell script code. Later using OpenOffice/Staroffice/MS office spread sheet to draw necessary graphs.
Using Command line stat utilities have some benefits like mining for precise information, automation along
with any test procedure. vmstat, iostat, mpstat are available by default with the solaris 10. Nic stat is available under http://blogs.sun.com/timc/entry/nicstat_the_solaris_and_linux
with any test procedure. vmstat, iostat, mpstat are available by default with the solaris 10. Nic stat is available under http://blogs.sun.com/timc/entry/nicstat_the_solaris_and_linux
bash-3.00# iostat -xn 2 >iostat_xn.txt &
bash-3.00# mpstat 2 > mpstat.txt &
bash-3.00# mpstat -a 2 >mpstat_a.txt &
bash-3.00# vmstat 2 > vmstat.txt &
bash-3.00# nicstat 2 > nicstat.txt &
bash-3.00# ls
iostat_xn.txt mpstat.txt mpstat_a.txt nicstat.txt vmstat.txt
bash-3.00#
These stat commands output lot of information. However, we may be interesed in quite a few according
to the current context. To filter out the required information I have come up with a shell script name extract.sh'. This script understands the output formats from vmstat, iostat -xn, mpstat, nicstat etc.. and filters out the unwanted information.
VMSTAT
bash-3.00# more vmstat.txt
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m2 m2 s0 in sy cs us sy id
0 0 0 6904784 3290616 0 3 1 0 0 0 0 0 0 0 0 544 756 464 2 1 97
0 1 0 6279816 2622256 15 273 1660 12 12 0 0 0 0 0 225 1279 4322 1678 18 4 78
0 0 0 6279568 2618408 1 133 1955 0 0 0 0 0 0 0 261 1302 3847 1760 17 4 79
0 1 0 6279568 2614304 108 174 4376 0 0 0 0 0 0 0 199 1218 3740 1330 20 4 76
1 0 0 6279560 2606776 115 150 5324 0 0 0 0 0 0 0 113 1132 3868 1194 37 4 59
0 0 0 6279512 2597400 121 212 5824 0 0 0 0 0 0 0 96 1156 5323 1564 38 6 56
0 0 0 6279408 2590792 165 245 3679 0 0 0 0 0 0 0 141 1240 5056 1710 32 6 63
0 0 0 6267080 2569328 52 68 770 4 4 0 0 0 0 0 21 1063 4742 1227 51 3 46
1 1 0 6266968 2565368 140 228 3052 20 20 0 0 0 0 0 172 1234 6156 1875 35 6 59
0 1 0 6266824 2553392 178 246 2308 84 40 0 0 0 0 0 132 1266 7518 1984 48 5 47
4 0 0 6263736 2540344 185 286 2278 0 0 0 0 0 0 0 81 1557 10847 3006 58 7 35
2 0 0 6258344 2524432 166 172 2484 84 84 0 0 0 0 0 129 1560 11966 2182 76 8 17
4 0 0 6232992 2511904 139 381 1703 60 60 0 0 0 0 0 128 1251 4913 1303 63 5 32
bash-3.00#
bash-3.00# /deepak/extract.sh
Usage : extract <STAT NAME> <regular exp> <i/p file> <o/p file>
bash-3.00#
bash-3.00# /deepak/extract.sh vm " " vmstat.txt vmstat.out
extract exec begin
vm
extract exec done
bash-3.00# more vmstat.out
VM STAT begin
usr sys idle
2 1 97
18 4 78
17 4 79
20 4 76
37 4 59
38 6 56
32 6 63
51 3 46
35 6 59
48 5 47
58 7 35
76 8 17
63 5 32
VM STAT done
bash-3.00#
Launch Openoffice or Staroffice calc
Create a new Spreadsheet. Copy paste the values from .out file.
Select the all the values. Go to Insert menu -> Chart.
IOSTAT
bash-3.00# more iostat_xn.txt
extended device
statistics
statistics
r/s w/s kr/s kw/s wait actv
wsvc_t asvc_t %w %b device
wsvc_t asvc_t %w %b device
0.1 0.1 1.1 4.1 0.0 0.0
0.0 7.6 0 0 c0t0d0
0.0 7.6 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0
0.0 14.4 0 0 c0t1d0
0.0 14.4 0 0 c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0
0.0 0.2 0 0 c1t2d0
0.0 0.2 0 0 c1t2d0
.
.
.
.
bash-3.00# /deepak/extract.sh io_xn "c0t0d0" iostat_xn.txt iostat_xn.out
extract exec begin
io_xn
extract exec done
bash-3.00# more iostat_xn.out
IO STAT XN begin
IO device : c0t0d0
kr/s(c0t0d0) kw/s(c0t0d0)
asvc_t(c0t0d0) %b(c0t0d0) device(c0t0d0)
asvc_t(c0t0d0) %b(c0t0d0) device(c0t0d0)
1.1 4.1 7.6 0 c0t0d0
1398.3 514.3 3.5 75 c0t0d0
1739.4 0.0 3.1 81 c0t0d0
4176.6 1023.9 3.8 72 c0t0d0
5524.9 0.0 4.7 45 c0t0d0
5496.2 1072.3 5.1 39 c0t0d0
3774.6 335.7 3.2 41 c0t0d0
823.8 28.2 4.0 10 c0t0d0
2619.4 570.4 3.4 48 c0t0d0
2300.9 660.2 4.1 37 c0t0d0
1961.3 533.7 6.3 35 c0t0d0
2533.4 920.2 6.4 40 c0t0d0
1495.6 115.8 3.6 36 c0t0d0
done
bash-3.00#
MPSTAT
a. Individual processor reporting
bash-3.00# more mpstat.txt
CPU minf mjf xcal intr ithr csw icsw
migr smtx srw syscl usr sys wt idl
migr smtx srw syscl usr sys wt idl
0 1 0 3 428 147 328 3
9 2 0 429 2 2 0 96
9 2 0 429 2 2 0 96
1 2 0 2 116 54 136 2
10 2 0 327 2 1 0 98
10 2 0 327 2 1 0 98
CPU minf mjf xcal intr ithr csw icsw
migr smtx srw syscl usr sys wt idl
migr smtx srw syscl usr sys wt idl
0 226 51 210 773 418 874 77
59 8 1 2168 18 4 0 78
59 8 1 2168 18 4 0 78
1 224 138 363 517 296 841 92
60 10 0 2634 20 6 0 75
60 10 0 2634 20 6 0 75
.
.
.
.
.
bash-3.00# /deepak/extract.sh mp "0 1" mpstat.txt mpstat.out
extract exec begin
mp
extract exec done
bash-3.00# more mpstat.out
MP STAT XN begin
CPU : 0
minf(0) mjf(0) xcal(0) intr(0) csw(0)
icsw(0) migr(0) smtx(0) srw(0) syscl(0) usr(0) sys(0) idl(0)
icsw(0) migr(0) smtx(0) srw(0) syscl(0) usr(0) sys(0) idl(0)
1 0 3 428 147 328 9 2 0 429 2 2 96
226 51 210 773 418 874 59 8 1 2168 18 4
78
78
.
.
.
.
.
done
CPU : 1
minf(1) mjf(1) xcal(1) intr(1) csw(1)
icsw(1) migr(1) smtx(1) srw(1) syscl(1) usr(1) sys(1) idl(1)
icsw(1) migr(1) smtx(1) srw(1) syscl(1) usr(1) sys(1) idl(1)
2 0 2 116 54 136 10 2 0 327 2 1 98
224 138 363 517 296 841 60 10 0 2634 20
6 75
6 75
.
.
.
.
done
bash-3.00#
b. Processor set reporting
bash-3.00# /deepak/extract.sh mp_a "0" mpstat_a.txt mpstat_a.out
extract exec begin
mp_a
extract exec done
bash-3.00# more mpstat_a.out
MP STAT XN begin
SET : 0
minf(0) mjf(0) xcal(0) intr(0) csw(0)
icsw(0) migr(0) smtx(0) srw(0) syscl(0) usr(0) sys(0) idl(0) sze(0)
icsw(0) migr(0) smtx(0) srw(0) syscl(0) usr(0) sys(0) idl(0) sze(0)
3 0 5 544 201 464 19 4 0 756 2 1 97 2
358 190 560 1284 710 1686 118 18 1 4482
19 4 77 2
19 4 77 2
122 226 547 1307 734 1744 108 32 2 3654
17 4 79 2
17 4 79 2
.
.
.
.
done
bash-3.00#
NIC STAT
bash-3.00# more nicstat.txt
Time Int rKB/s wKB/s
rPk/s wPk/s rAvs wAvs %Util Sat
rPk/s wPk/s rAvs wAvs %Util Sat
12:40:30 lo0 0.00 0.00
0.03 0.03 0.00 0.00 0.00 0.00
0.03 0.03 0.00 0.00 0.00 0.00
12:40:30 bge0 4.15 0.10
3.87 1.49 1099.2 69.54 0.03 0.00
3.87 1.49 1099.2 69.54 0.03 0.00
Time Int rKB/s wKB/s
rPk/s wPk/s rAvs wAvs %Util Sat
rPk/s wPk/s rAvs wAvs %Util Sat
12:40:32 lo0 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
12:40:32 bge0 209.8 4.85
149.0 76.51 1441.3 64.90 1.76 0.00
149.0 76.51 1441.3 64.90 1.76 0.00
bash-3.00#
bash-3.00# /deepak/extract.sh nic "bge0" nicstat.txt nicstat.out
extract exec begin
nic
extract exec done
bash-3.00# more nicstat.out
NIC STAT begin
NIC : bge0
Int(bge0) rKB/s(bge0) wKB/s(bge0)
bge0 4.15 0.10
bge0 209.8 4.85
bge0 235.3 5.16
bge0 324.1 7.28
bge0 217.3 4.88
bge0 230.0 5.03
bge0 221.8 5.21
bge0 216.1 5.06
bge0 221.2 5.04
bge0 219.6 5.10
bge0 246.9 7.14
bge0 352.6 8.32
done
bash-3.00#
