<\/p>\n
System Admin Tools:First of all lets learn how to find out what is chocking or gobbling up the server resource. You can get the memory usage by issuing the command “free -m”<\/p>\n
[bash]<\/p>\n
root@spark [~]# free -m<\/p>\n
total used free shared buffers cached<\/p>\n
Mem: 4051 3052 999 0 249 1294<\/p>\n
-\/+ buffers\/cache: 1508 2543<\/p>\n
Swap: 4000 1 3999<\/p>\n
[\/bash]<\/p>\n
The top row ‘used’ (3052) value will almost always nearly match the top row mem value (4051). Since Linux likes to use any spare memory to cache disk blocks (1294).<\/p>\n
The key figure to look at is the buffers\/cache row used value (1508). This is how much space your applications are currently using. For best performance, this number should be less than your total (4051) memory.<\/p>\n
VMSTAT<\/strong><\/p>\n vmstat helps you to see, among other things, if your server is swapping. Take a look at the following run of vmstat doing a one second refresh for two iterations.<\/p>\n [bash]<\/p>\n root@spark [~]# vmstat 1 2<\/p>\n procs ———–memory———- —swap– —–io—- –system– —-cpu—-<\/p>\n r b swpd free buff cache si so bi bo in cs us sy id wa<\/p>\n 0 0 1172 1689332 333588 663092 0 0 19 113 1 2 3 1 95 1<\/p>\n 0 0 1172 1690320 332920 663100 0 0 352 256 355 681 5 3 91 2<\/p>\n [\/bash]<\/p>\n The first row shows your server averages. The si (swap in) and so (swap out) columns show if you have been swapping (i.e. needing to dip into ‘virtual’ memory) in order to run your server’s applications. The si\/so numbers should be 0 (or close to it). Numbers in the hundreds or thousands indicate your server is swapping heavily. This consumes a lot of CPU and other server resources and you would get a very significant benefit from adding more memory to your server.<\/p>\n Some other columns of interest: The r (runnable) b (blocked) and w (waiting) columns help see your server load. Waiting processes are swapped out. Blocked processes are typically waiting on I\/O. The runnable column is the number of processes trying to something. These numbers combine to form the ‘load’ value on your server. Typically you want the load value to be one or less per CPU in your server.<\/p>\n The bi (bytes in) and bo (bytes out) column show disk I\/O (including swapping memory to\/from disk) on your server. PS<\/strong><\/p>\n This command is used to know all the processes running in the server by System Admin Tools. It can be also used to find out process which is using most of the memory and cpu.<\/p>\n To find out top 3 memory consuming processes.<\/p>\n [bash]<\/p>\n ps -auxf | sort -nr -k 4 | head -3<\/p>\n [\/bash]<\/p>\n To find out top 3 cpu consuming processes<\/p>\n [bash]<\/p>\n ps -auxf | sort -nr -k 3 | head -3<\/p>\n [\/bash]<\/p>\n TOP<\/strong><\/p>\n Say the System Admin Tools is slow and you want to find out who is gobbling up all the CPU and\/or memory. To display the top processes, you use the command top.<\/p>\n Note that unlike other commands, top does not produce an output and sits still. It refreshes the screen to display new information. So, if you just issue top and leave the screen up, the most current information is always up. Top runs until you press “q” to quit top.<\/p>\n [bash]<\/p>\n $ top<\/p>\n 18:46:13 up 11 days, 21:50, 5 users, load average: 0.11, 0.19, 0.18<\/p>\n 151 processes: 147 sleeping, 4 running, 0 zombie, 0 stopped<\/p>\n CPU states: cpu user nice system irq softirq iowait idle<\/p>\n total 12.5% 0.0% 6.7% 0.0% 0.0% 5.3% 75.2%<\/p>\n Mem: 1026912k av, 999548k used, 27364k free, 0k shrd, 116104k buff<\/p>\n 758312k actv, 145904k in_d, 16192k in_c<\/p>\n Swap: 2041192k av, 122224k used, 1918968k free 590140k cached<\/p>\n PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n 451 spark 15 0 6044 4928 4216 S 0.1 0.4 0:20 0 tnslsnr<\/p>\n 8991 spark 15 0 1248 1248 896 R 0.1 0.1 0:00 0 top<\/p>\n 1 root 19 0 440 400 372 S 0.0 0.0 0:04 0 init<\/p>\n 2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd<\/p>\n 3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd<\/p>\n 4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd\/0<\/p>\n 7 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 bdflush<\/p>\n 5 root 15 0 0 0 0 SW 0.0 0.0 0:33 0 kswapd<\/p>\n 6 root 15 0 0 0 0 SW 0.0 0.0 0:14 0 kscand<\/p>\n 8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated<\/p>\n 9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd<\/p>\n … output snipped …<\/p>\n [\/bash]<\/p>\n Let’s examine the different types of information produced by System Admin Tools.<\/p>\n The first line:<\/p>\n [bash]<\/p>\n 18:46:13 up 11 days, 21:50, 5 users, load average: 0.11, 0.19, 0.18<\/p>\n [\/bash]<\/p>\n shows the current time (18:46:13), that system has been up for 11 days; that the System Admin Tools has been working for 21 hours 50 seconds. The load average of the system is shown (0.11, 0.19, 0.18) for the last 1, 5 and 15 minutes respectively. (By the way, you can also get this information by issuing the uptime command.)<\/p>\n If the load average is not required, press the letter “l” (lowercase L); it will turn it off. To turn it back on press l again. The second line: 151 processes: 147 sleeping, 4 running, 0 zombie, 0 stopped shows the number of processes, running, sleeping, etc. The third and fourth lines:<\/p>\n [bash]<\/p>\n CPU states: cpu user nice system irq softirq iowait idle<\/p>\n total 12.5% 0.0% 6.7% 0.0% 0.0% 5.3% 75.2%<\/p>\n [\/bash]<\/p>\n show the CPU utilization details. The above line shows that user processes consume 12.5% and system consumes 6.7%. The user processes include the Oracle processes. Press “t” to turn these three lines off and on. If there are more than one CPU, you will see one line per CPU.<\/p>\n The next two lines:<\/p>\n [bash]<\/p>\n Mem: 1026912k av, 1000688k used, 26224k free, 0k shrd, 113624k buff<\/p>\n 758668k actv, 146872k in_d, 14460k in_c Swap: 2041192k av, 122476k<\/p>\n used, 1918716k free 591776k cached<\/p>\n [\/bash]<\/p>\n show the memory available and utilized. Total memory is “1026912k av”, approximately 1GB, of which only 26224k or 26MB is free. The swap space is 2GB; but it’s almost not used. To turn it off and on, press “m”.<\/p>\n The rest of the display shows the processes in a tabular format. Here is the explanation of the columns:<\/p>\n Column Description<\/p>\n PID The process ID of the process<\/p>\n USER The user running the process<\/p>\n PRI The priority of the process<\/p>\n NI The nice value: The higher the value, the lower the priority of the task<\/p>\n SIZE Memory used by this process (code+data+stack)<\/p>\n RSS The physical memory used by this process<\/p>\n SHARE The shared memory used by this process<\/p>\n STAT<\/strong><\/p>\n The status of this process, shown in code. Some major status codes are:<\/p>\n R Running<\/p>\n S Sleeping<\/p>\n Z Zombie<\/p>\n T Stopped<\/p>\n You can also see second and third characters, which indicate:<\/p>\n W Swapped out process<\/p>\n N positive nice value<\/p>\n %CPU The percentage of CPU used by this process<\/p>\n %MEM The percentage of memory used by this process<\/p>\n TIME The total CPU time used by this process<\/p>\n CPU If this is a multi-processor System Admin Tools, this column indicates the ID of the CPU this process is running on.<\/p>\n COMMAND The command issued by this process<\/p>\n While the top is being displayed, you can press a few keys to format the display as you like. Pressing the uppercase M key sorts the output by memory usage. (Note that using lowercase m will turn the memory summary lines on or off at the top of the display.) This is very useful when you want to find out who is consuming the memory. Here is sample output:<\/p>\n [bash]<\/p>\n PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n 31903 spark 15 0 75760 72M 72508 S 0.0 7.2 0:01 0 ora_smon_PRODB2<\/p>\n 31909 spark 15 0 68944 66M 64572 S 0.0 6.6 0:03 0 ora_mmon_PRODB2<\/p>\n 31897 spark 15 0 53788 49M 48652 S 0.0 4.9 0:00 0 ora_dbw0_PRODB2<\/p>\n [\/bash]<\/p>\n Now that you learned how to interpret the output, let’s see how to use command line parameters.<\/p>\n The most useful is -d, which indicates the delay between the screen refreshes. To refresh every second, use top -d 1.<\/p>\n The other useful option is -p. If you want to monitor only a few processes, not all, you can specify only those after the -p option. To monitor processes 13609, 13608 and 13554, issue: top -p 13609 -p 13608 -p 13554 SKILL & SNICE<\/strong><\/p>\n From the previous discussion you learned how to identify a CPU consuming resource. What if you find that a process is consuming a lot of CPU and memory, but you don’t want to kill it? Consider the top output below:<\/p>\n [bash]<\/p>\n $ top -c -p 16514<\/p>\n 23:00:44 up 12 days, 2:04, 4 users, load average: 0.47, 0.35, 0.31<\/p>\n 1 processes: 1 sleeping, 0 running, 0 zombie, 0 stopped<\/p>\n CPU states: cpu user nice system irq softirq iowait idle<\/p>\n total 0.0% 0.6% 8.7% 2.2% 0.0% 88.3% 0.0%<\/p>\n Mem: 1026912k av, 1010476k used, 16436k free, 0k shrd, 52128k buff<\/p>\n 766724k actv, 143128k in_d, 14264k in_c<\/p>\n Swap: 2041192k av, 83160k used, 1958032k free 799432k cached<\/p>\n PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n 16514 spark 19 4 28796 26M 20252 D N 7.0 2.5 0:03 0 sparkPRODB2…<\/p>\n [\/bash]<\/p>\n Now that you confirmed the process 16514 is consuming a lot of memory, you can “freeze” it but not kill it using the skill command.<\/p>\n [bash]<\/p>\n $ skill -STOP 1 1 processes: 0 sleeping, 0 running, 0 zombie, 1 stopped<\/p>\n CPU states: cpu user nice system irq softirq iowait idle<\/p>\n total 2.3% 0.0% 0.3% 0.0% 0.0% 2.3% 94.8%<\/p>\n Mem: 1026912k av, 1008756k used, 18156k free, 0k shrd, 3976k buff<\/p>\n 770024k actv, 143496k in_d, 12876k in_c<\/p>\n Swap: 2041192k av, 83152k used, 1958040k free 851200k cached<\/p>\n PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n 16514 spark 19 4 28796 26M 20252 T N 0.0 2.5 0:04 0 sparkPRODB2…<\/p>\n [\/bash]<\/p>\n The CPU is now 94% idle from 0%. The process is effectively frozen. After some time, you may want to revive the process from coma:<\/p>\n [bash]<\/p>\n $ skill -CONT 16514<\/p>\n [\/bash]<\/p>\n This approach is immensely useful for temporarily freezing processes to make room for more important processes to complete.<\/p>\n The command is very versatile. If you want to stop all processes of the user “spark”, only one command does it all:<\/p>\n [bash]<\/p>\n $ skill -STOP spark><\/p>\n [\/bash]<\/p>\n You can use a user, a PID, a command or terminal id as argument. The following stops all rman commands.<\/p>\n [bash]<\/p>\n $ skill -STOP rman<\/p>\n [\/bash]<\/p>\n As you can see, skill decides that argument you entered a process ID, userid, or command and acts appropriately. This may cause an issue in some cases, where you may have a user and a command in the same name. The best example is the “spark” process, which is typically run by the user “spark”. So, when you want to stop the process called “spark” and you issue:<\/p>\n [bash]<\/p>\n $ skill -STOP spark<\/p>\n [\/bash]<\/p>\n all the processes of user “spark” stop, including the session you may be on. To be completely unambiguous you can optionally give a new parameter to specify the type of the parameter. To stop a command called spark, you can give:<\/p>\n [bash]<\/p>\n $ skill -STOP -c spark<\/p>\n [\/bash]<\/p>\n The command snice is similar. Instead of stopping a process it makes its priority a lower one. First, check the top output:<\/p>\n [bash]<\/p>\n PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n 3 root 15 0 0 0 0 RW 0.0 0.0 0:00 0 kapmd<\/p>\n 13680 spark 15 0 11336 10M 8820 T 0.0 1.0 0:00 0 spark<\/p>\n 13683 spark 15 0 9972 9608 7788 T 0.0 0.9 0:00 0 spark<\/p>\n 13686 spark 15 0 9860 9496 7676 T 0.0 0.9 0:00 0 spark<\/p>\n 13689 spark 15 0 10004 9640 7820 T 0.0 0.9 0:00 0 spark<\/p>\n 13695 spark 15 0 9984 9620 7800 T 0.0 0.9 0:00 0 spark<\/p>\n 13698 spark 15 0 10064 9700 7884 T 0.0 0.9 0:00 0 spark<\/p>\n 13701 spark 15 0 22204 21M 16940 T 0.0 2.1 0:00 0 spark<\/p>\n [\/bash]<\/p>\n Now, drop the priority of the processes of “spark” by four points. Note that the higher the number, the lower the priority.<\/p>\n [bash]<\/p>\n $ snice +4 -u spark 16894 spark 20 4 38904 32M 26248 D N 5.5 3.2 0:01 0 spark<\/p>\n [\/bash]<\/p>\n Note how the NI column (for nice values) is now 4 and the priority is now set to 20, instead of 15. This is quite useful in reducing priorities.<\/p>\n Now lets get over to trace the process and manipulate it System Admin Tools.<\/p>\n lsof<\/strong><\/p>\n The command lsof shows a list of processes attached to open files or network ports. List processes attached to a given file: lsof filenmame<\/p>\n [bash]<\/p>\n List all open files on system: (Long list)<\/p>\n List all files opened by user:<\/p>\n [\/bash]<\/p>\n The commands netstat -punta and socklist will list open network connections. Use the command lsof -i TCP:port-number to see the processes attached to the port.<\/p>\n [bash]<\/p>\n root@spark [~]# lsof -i TCP:25 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME exim 11350 mailnull 4u IPv4 1226820346 TCP 201.201.203.43:smtp->esp222.neoplus.adsl.tpnet.pl:2466 (ESTABLISHED) exim 11350 mailnull 5u IPv4 1226820346 TCP 201.201.203.43:smtp->esp222.neoplus.adsl.tpnet.pl:2466 (ESTABLISHED) exim 11811 mailnull 4u IPv4 1226830746 TCP 201.201.203.43:smtp->ANantes-151-1-39-245.w83-195.abo.wanadoo.fr:627 31 (ESTABLISHED) exim 11811 mailnull 5u IPv4 1226830746 TCP 201.201.203.43:smtp->ANantes-151-1-39-245.w83-195.abo.wanadoo.fr:627 31 (ESTABLISHED) exim 14135 mailnull 4u IPv4 1226855434 TCP 201.201.203.43:smtp->125.115.215.112:3653 (ESTABLISHED) exim 14135 mailnull 5u IPv4 1226855434 TCP 201.201.203.43:smtp->125.115.215.112:3653 (ESTABLISHED)<\/p>\n [\/bash]<\/p>\n
The us (user), sy (system) and id (idle) show the amount of CPU your server is using. The higher the idle value, the better.<\/p>\n
This will show results in the same format as the top command, but only those specific processes.<\/p>\n
After this, check the top output:
23:01:11 up 12 days, 2:05, 4 users, load average: 1.20, 0.54, 0.38<\/p>\n
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND<\/p>\n
lsof<\/p>\n