Some UNIX Utilities for performance monitoring

vmstat command:
vmstat is a command which can be use to display system statistics. Syntax is given below:

vmstat <# second> < #number of times>

Example

vmstat 1 10

Above Command is to display the system statistics every second, 10 times.

Sample Output of the command:

vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 950888  57536 512868    0    0   559    45  479  671  9  2 73 15

Lets understand the column definition:
Procs Column would be having three subcolumns
r: count (processes waiting for run time)
b: count( Processes in uninterruptible sleep)
w: Count (Process which are swapped out and runnable) (on RHEL)

If any process in b and w column then DBA/System admin has to check the system.
Memory Column would be having following subcolumns:
swap: swap space currently available.(in kBs)
free: Amount of idle memory(kBs)
buff: Memory used as buffer (kBs)

SWAP information
si: memory swapped in from disk (kB/s)
so: memory swapped out to disk (kB/s)

I/O information:
bi: count (Number of blocks sent to a block device)(blocks/s)
bo: count (Number of blocks received from block device) (Blocks/s)

System Information:
in: Interrupts per second, including clock.
cs: CPU Switches

Some other percentage of cpu information:
us: user time
sy: system time
id: idle time

If id is zero then cpu is overburden. Also if si and so is high then there is lot of swapping happening in system.

To find more information on CPU sar command can be use. sar command can be use for getting the information of CPU Utilization.

Syntax: sar <#seconds> <# of times>

First row of sar command gives the information of System including version, OS information, machine name etc.
Real information sar command gives in five columns. First column (Time) is about timestamp/time i.e when information collected.
Second Column (%user) gives the information of load given by the users.
Third Column (%system) gives the information of load given by system processes including daemons.
Fourth Column (%IO) gives the information of load/processes waiting for I/O (ie hardware)
Fifth Column (%idle) gives the average idle load. In other words system is waiting for task to start.

If the idle percentage is less that equal to 20 for long time then system would be performing slow and system admin needs to look in the system. To reach at any conclusion user must need to watch the output of sar command for at least one day. If the system consistently having <=20% idle then system need to analyze the other processes and load. To find the information on I/O of System, user can use command iostat command.

syntax: iostat <# seconds> <# of times> [disk#]

iostat command gives the I/O information of all the disks.

First column is about Device (i.e hardisk)
Second column gives the information of block transfer per second, where transfer is an I/O request to the disk
Third column gives the blocks read/second (in kBs)
Fourth column gives the information of Blocks written per second.
Fifth Column gives Amount of Block read from device
Sixth Column gives information of total number of block written.

Sample Output is given below:

Linux 2.6.28-19-generic (vibhore-laptop)  10/31/2010  _i686_ (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.58    0.83    5.56   43.94    0.00   27.50

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             138.18      6067.27       446.48     809920      59600
sda1            137.70      6052.41       446.48     807936      59600
sda2              0.02         0.04         0.00          6          0
sda5              0.31        12.42         0.00       1658          0