I would like to monitor the free memory on four machines each running Windows Server 2003 R2 SP2 64-bit. Each box has 31.7gb of RAM. I would like to periodically run a command line tool so I can collect the output and later make charts with it.
I ran some tests and collected vmstat output periodically, using Cygwin. I'm seeing numbers like this:
0 0 1235228 4194303 0 0 0 0 0 0 4652 3089 1 5 94 0
0 0 1235228 4194303 0 0 0 0 0 0 4718 7591 5 4 91 0
0 0 1235228 4194303 0 0 0 0 0 0 5027 7816 5 4 92 0
0 0 1235228 4194303 0 0 0 0 0 0 4886 7099 3 3 93 0
On a completely different machine I see these numbers:
0 0 10344 4194303 0 0 32 0 1 0 3113 6492 32 10 58 0
0 0 10340 4194303 0 0 0 0 0 0 3908 6180 38 11 51 0
0 0 10340 4194303 0 0 0 0 0 0 2094 4501 23 5 72 0
0 0 10340 4194303 0 0 0 0 0 0 2435 3792 32 5 63 0
I intended to use the 4th column, "free", to determine the amount of free memory. However, this column is 4194303 on all four boxes! This doesn't seem to correlate to any number in task manager. I guess this column isn't useful? Is 4194303 some magic number?
I was unable to find a command line tool to monitor per process CPU usage. The Cygwin tools don't seem to do this. I ended up writing a tool I call cpumem (Google Code). My tool also outputs memory per process. Maybe this could be useful to determine the total used memory, which I could use to determine free memory? Here is some example output from my tool:
Name PID CPU PageFaultCount PeakWorkingSetSize WorkingSetSize QuotaPeakPagedPoolUsage QuotaPagedPoolUsage QuotaPeakNonPagedPoolUsage QuotaNonPagedPoolUsage PagefileUsage PeakPagefileUsage
Idle 0 362
System 4 2 17019 4857856 258048 0 0 80 80 45056 45056
smss 408 0 1371 1257472 684032 27704 7696 1520 1120 229376 2527232
csrss 456 0 265739 4042752 4005888 206112 205264 10608 10160 1781760 1789952
winlogon 480 0 4223496 15458304 15282176 150992 126576 130272 129120 10924032 11526144
services 528 6 8328 5210112 5144576 66024 63400 14416 12256 2703360 2928640
lsass 540 0 14312 12607488 12558336 105200 100400 30976 25344 11763712 12058624
svchost 716 0 1774 4636672 4608000 57704 53832 8512 5264 1634304 1806336
svchost 788 0 8397 5877760 5804032 68488 66992 35536 33984 2318336 2457600
svchost 852 0 7341 7426048 7352320 113496 100176 21330 16342 6791168 6991872
svchost 888 0 1321 5218304 5197824 57680 57648 10256 8544 2134016 2469888
svchost 904 0 319648 37355520 28160000 227912 206056 52384 38384 20017152 35856384
spoolsv 1036 0 82386 10686464 8921088 149696 106656 12528 10224 5988352 7823360
msdtc 1064 0 1814 7053312 6979584 63264 62656 12752 11424 2785280 2916352
svchost 1260 0 25164 5328896 3710976 77576 49096 5072 4080 1306624 3690496
svchost 1412 0 723 2912256 2908160 35200 35008 4432 3984 1187840 1191936
VMwareService 2008 0 1797 6971392 6897664 119624 118888 11120 6944 3108864 3231744
svchost 432 0 7845 7294976 7151616 98552 96304 12512 11232 3588096 24743936
dllhost 2096 0 2966 10526720 10362880 75568 74464 12128 9984 4001792 4362240
wmiprvse 2736 0 3955 12021760 9736192 73360 70112 8320 6960 4620288 7536640
logon.scr 1112 0 630 2564096 2543616 76648 76080 2960 2960 1187840 1187840
httpd 5744 0 59245 8650752 8650752 83728 66840 128494 100894 4268032 4460544
cmd 6336 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
cmd 6840 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
rotatelogs 8212 0 857 3366912 3366912 66080 51840 4464 4464 1617920 1617920
rotatelogs 3296 0 846 3321856 3321856 66080 51840 4400 4384 1581056 1581056
httpd 7536 0 42975 11481088 11460608 82280 70664 34976 30624 8179712 8241152
cmd 5948 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 2968 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
cmd 2532 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 8584 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
wrapper 5800 0 998 4038656 4005888 39992 39960 7846 6960 1929216 1970176
java 7708 0 1873243 161222656 134750208 125392 121616 515910 122640 130805760 157409280
csrss 1520 0 3747 5390336 4505600 108024 100128 9040 6960 2060288 2097152
winlogon 2804 0 4568 11296768 4702208 165832 163496 14944 13552 4583424 4943872
rdpclip 1736 0 1331 5201920 5185536 124832 114664 5520 5040 2105344 2183168
explorer 2072 0 83569 24793088 23375872 209560 194280 20704 18320 11202560 12865536
jusched 5716 0 1769 6991872 6991872 130320 124592 9264 8640 2850816 2850816
ApacheMonitor 3648 0 977 3858432 3825664 109320 95720 4832 4240 1638400 1679360
VMwareTray 3740 0 1394 5521408 5496832 117352 104184 6672 6080 2478080 2551808
VMwareUser 2820 0 1759 7024640 6995968 119552 107184 7472 7120 4067328 4108288
SciTE 4692 0 3158 9125888 8888320 112560 98560 5280 4880 6103040 6381568
mmc 2812 0 6443 23494656 23470080 205416 203064 14640 13520 12255232 16740352
eclipse 8956 0 205139 184983552 178651136 225112 217320 42560 40720 168902656 174735360
firefox 7672 0 19757 58945536 58462208 174896 171416 33568 27328 47968256 49152000
wrapper 3132 0 985 3973120 3944448 39712 39680 8438 6640 1900544 1941504
java 7456 13 1996365 -1154555904 -1194131456 1061632 865376 2342660 1668320 -1 -1
cmd 8700 0 613 2527232 2527232 33680 33672 3600 3040 2150400 2150400
java 7140 0 9411 26324992 23179264 54408 54408 12928 11120 581910528 586002432
java 228 0 22616 69480448 60051456 54344 54344 10464 10336 593469440 604434432
cmd 5664 0 606 2494464 2494464 38040 38040 7072 6512 2064384 2064384
cpumem 6040 13 3393 4390912 4222976 37280 37200 5360 4960 2260992 2433024
_Total 0 366
Idle 0 366
System 4 1 17019 4857856 258048 0 0 80 80 45056 45056
smss 408 0 1371 1257472 684032 27704 7696 1520 1120 229376 2527232
csrss 456 0 265739 4042752 4005888 206112 205264 10608 10160 1781760 1789952
winlogon 480 0 4223496 15458304 15282176 150992 126576 130272 129120 10924032 11526144
services 528 4 8328 5210112 5144576 66024 63400 14416 12256 2703360 2928640
lsass 540 0 14312 12607488 12558336 105200 100400 30976 25344 11763712 12058624
svchost 716 0 1774 4636672 4608000 57704 53832 8512 5264 1634304 1806336
svchost 788 0 8397 5877760 5804032 68488 66992 35536 33984 2318336 2457600
svchost 852 0 7341 7426048 7352320 113496 100176 21330 16342 6791168 6991872
svchost 888 0 1321 5218304 5197824 57680 57648 10256 8544 2134016 2469888
svchost 904 0 319648 37355520 28160000 227912 206056 52384 38384 20017152 35856384
spoolsv 1036 0 82386 10686464 8921088 149696 106656 12528 10224 5988352 7823360
msdtc 1064 0 1814 7053312 6979584 63264 62656 12752 11424 2785280 2916352
svchost 1260 0 25164 5328896 3710976 77576 49096 5072 4080 1306624 3690496
svchost 1412 0 723 2912256 2908160 35200 35008 4432 3984 1187840 1191936
VMwareService 2008 0 1797 6971392 6897664 119624 118888 11120 6944 3108864 3231744
svchost 432 0 7845 7294976 7151616 98552 96304 12512 11232 3588096 24743936
dllhost 2096 0 2966 10526720 10362880 75568 74464 12128 9984 4001792 4362240
wmiprvse 2736 0 3955 12021760 9736192 73360 70112 8320 6960 4620288 7536640
logon.scr 1112 0 630 2564096 2543616 76648 76080 2960 2960 1187840 1187840
httpd 5744 0 59245 8650752 8650752 83728 66840 128494 100894 4268032 4460544
cmd 6336 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
cmd 6840 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
rotatelogs 8212 0 857 3366912 3366912 66080 51840 4464 4464 1617920 1617920
rotatelogs 3296 0 846 3321856 3321856 66080 51840 4400 4384 1581056 1581056
httpd 7536 0 42975 11481088 11460608 82280 70664 34976 30624 8179712 8241152
cmd 5948 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 2968 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
cmd 2532 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 8584 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
wrapper 5800 0 998 4038656 4005888 39992 39960 7846 6960 1929216 1970176
java 7708 0 1873243 161222656 134750208 125392 121616 515910 122640 130805760 157409280
csrss 1520 0 3747 5390336 4505600 108024 100128 9040 6960 2060288 2097152
winlogon 2804 0 4568 11296768 4702208 165832 163496 14944 13552 4583424 4943872
rdpclip 1736 0 1331 5201920 5185536 124832 114664 5520 5040 2105344 2183168
explorer 2072 0 83569 24793088 23375872 209560 194280 20704 18320 11202560 12865536
jusched 5716 0 1769 6991872 6991872 130320 124592 9264 8640 2850816 2850816
ApacheMonitor 3648 0 977 3858432 3825664 109320 95720 4832 4240 1638400 1679360
VMwareTray 3740 0 1394 5521408 5496832 117352 104184 6672 6080 2478080 2551808
VMwareUser 2820 0 1759 7024640 6995968 119552 107184 7472 7120 4067328 4108288
SciTE 4692 0 3158 9125888 8888320 112560 98560 5280 4880 6103040 6381568
mmc 2812 0 6443 23494656 23470080 205416 203064 14640 13520 12255232 16740352
eclipse 8956 0 205139 184983552 178651136 225112 217320 42560 40720 168902656 174735360
firefox 7672 0 19757 58945536 58462208 174896 171416 33568 27328 47968256 49152000
wrapper 3132 0 985 3973120 3944448 39712 39680 8438 6640 1900544 1941504
java 7456 12 1996365 -1154555904 -1194131456 1061632 865376 2342660 1668320 -1 -1
cmd 8700 0 613 2527232 2527232 33680 33672 3600 3040 2150400 2150400
java 7140 0 9411 26324992 23179264 54408 54408 12928 11120 581910528 586002432
java 228 0 22616 69480448 60051456 54344 54344 10464 10336 593469440 604434432
cmd 5664 0 606 2494464 2494464 38040 38040 7072 6512 2064384 2064384
cpumem 6040 14 5567 4395008 4227072 37280 37200 5360 4960 2260992 2433024
_Total 0 369
Idle 0 365
System 4 0 17019 4857856 258048 0 0 80 80 45056 45056
smss 408 0 1371 1257472 684032 27704 7696 1520 1120 229376 2527232
csrss 456 0 265739 4042752 4005888 206112 205264 10608 10160 1781760 1789952
winlogon 480 0 4223496 15458304 15282176 150992 126576 130272 129120 10924032 11526144
services 528 1 8328 5210112 5144576 66024 63400 14416 12256 2703360 2928640
lsass 540 0 14312 12607488 12558336 105200 100400 30976 25344 11763712 12058624
svchost 716 0 1774 4636672 4608000 57704 53832 8512 5264 1634304 1806336
svchost 788 0 8397 5877760 5804032 68488 66992 35536 33984 2318336 2457600
svchost 852 0 7341 7426048 7352320 113496 100176 21330 16342 6791168 6991872
svchost 888 0 1321 5218304 5197824 57680 57648 10256 8544 2134016 2469888
svchost 904 0 319648 37355520 28160000 227912 206056 52384 38384 20017152 35856384
spoolsv 1036 0 82386 10686464 8921088 149696 106656 12528 10224 5988352 7823360
msdtc 1064 0 1814 7053312 6979584 63264 62656 12752 11424 2785280 2916352
svchost 1260 0 25164 5328896 3710976 77576 49096 5072 4080 1306624 3690496
svchost 1412 0 723 2912256 2908160 35200 35008 4432 3984 1187840 1191936
VMwareService 2008 0 1797 6971392 6897664 119624 118888 11120 6944 3108864 3231744
svchost 432 0 7845 7294976 7151616 98552 96304 12512 11232 3588096 24743936
dllhost 2096 0 2966 10526720 10362880 75568 74464 12128 9984 4001792 4362240
wmiprvse 2736 0 3955 12021760 9736192 73360 70112 8320 6960 4620288 7536640
logon.scr 1112 0 630 2564096 2543616 76648 76080 2960 2960 1187840 1187840
httpd 5744 0 59245 8650752 8650752 83728 66840 128494 100894 4268032 4460544
cmd 6336 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
cmd 6840 0 754 2797568 2797568 33584 33000 3760 3600 2203648 2236416
rotatelogs 8212 0 857 3366912 3366912 66080 51840 4464 4464 1617920 1617920
rotatelogs 3296 0 846 3321856 3321856 66080 51840 4400 4384 1581056 1581056
httpd 7536 0 42975 11481088 11460608 82280 70664 34976 30624 8179712 8241152
cmd 5948 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 2968 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
cmd 2532 0 753 2793472 2793472 33584 33000 3760 3600 2203648 2236416
rotatelogs 8584 0 863 3391488 3391488 66080 51840 4464 4464 1650688 1650688
wrapper 5800 0 998 4038656 4005888 39992 39960 7846 6960 1929216 1970176
java 7708 0 1873243 161222656 134750208 125392 121616 515910 122640 130805760 157409280
csrss 1520 0 3747 5390336 4505600 108024 100128 9040 6960 2060288 2097152
winlogon 2804 0 4568 11296768 4702208 165832 163496 14944 13552 4583424 4943872
rdpclip 1736 0 1331 5201920 5185536 124832 114664 5520 5040 2105344 2183168
explorer 2072 0 83569 24793088 23375872 209560 194280 20704 18320 11202560 12865536
jusched 5716 0 1769 6991872 6991872 130320 124592 9264 8640 2850816 2850816
ApacheMonitor 3648 0 977 3858432 3825664 109320 95720 4832 4240 1638400 1679360
VMwareTray 3740 0 1394 5521408 5496832 117352 104184 6672 6080 2478080 2551808
VMwareUser 2820 0 1759 7024640 6995968 119552 107184 7472 7120 4067328 4108288
SciTE 4692 0 3158 9125888 8888320 112560 98560 5280 4880 6103040 6381568
mmc 2812 0 6443 23494656 23470080 205416 203064 14640 13520 12255232 16740352
eclipse 8956 0 205139 184983552 178651136 225112 217320 42560 40720 168902656 174735360
firefox 7672 0 19757 58945536 58462208 174896 171416 33568 27328 47968256 49152000
wrapper 3132 0 985 3973120 3944448 39712 39680 8438 6640 1900544 1941504
java 7456 15 1996365 -1154555904 -1194131456 1061632 866240 2342660 1670336 -1 -1
cmd 8700 0 613 2527232 2527232 33680 33672 3600 3040 2150400 2150400
java 7140 0 9411 26324992 23179264 54408 54408 12928 11120 581910528 586002432
java 228 0 22617 69480448 60055552 54344 54344 10464 10336 593469440 604434432
cmd 5664 0 606 2494464 2494464 38040 38040 7072 6512 2064384 2064384
cpumem 6040 11 7794 4616192 4448256 47488 47408 6320 5920 2334720 2506752
_Total 0 370
That is one run of the tool. The machine is a quad core, so I guess the output quadrupled so that each core's CPU usage could be seen for each process. The memory values look the same though. Which of these columns should I sum to determine the total used memory? Is this a reasonable way to do it?
Is there a better way to determine free or used memory from the command line on Windows?
