Suddenly Ubuntu crashes every night with oom-killer in the logs

0

Suddenly started crashing right after I added about a million files from my server to one of the folders in my truecrypt partition.

The strange part is that it does it each night right around 1am...checked my crons and nothing is running at that time.

Here is some info from my kern.log file from last night:

Dec 20 01:01:58 Lubuntu kernel: [142653.102582] lightdm-gtk-gre invoked oom-killer: gfp_mask=0x44d0, order=2, oom_adj=0, oom_score_adj=0
Dec 20 01:01:58 Lubuntu kernel: [142653.102588] lightdm-gtk-gre cpuset=/ mems_allowed=0
Dec 20 01:01:58 Lubuntu kernel: [142653.102591] Pid: 1786, comm: lightdm-gtk-gre Tainted: G           O 3.5.0-17-generic #28-Ubuntu
Dec 20 01:01:58 Lubuntu kernel: [142653.102593] Call Trace:
Dec 20 01:01:58 Lubuntu kernel: [142653.102602]  [<c15c01c4>] dump_header.isra.10+0x86/0x1b4
Dec 20 01:01:58 Lubuntu kernel: [142653.102607]  [<c1104a1a>] oom_kill_process+0x23a/0x270
Dec 20 01:01:58 Lubuntu kernel: [142653.102611]  [<c1104b3e>] ? select_bad_process.constprop.15+0xee/0x170
Dec 20 01:01:58 Lubuntu kernel: [142653.102614]  [<c1104f53>] out_of_memory+0x163/0x1c0
Dec 20 01:01:58 Lubuntu kernel: [142653.102617]  [<c1108abf>] __alloc_pages_nodemask+0x68f/0x750
Dec 20 01:01:58 Lubuntu kernel: [142653.102621]  [<c1108bfc>] __get_free_pages+0x1c/0x40
Dec 20 01:01:58 Lubuntu kernel: [142653.102625]  [<c113dd82>] kmalloc_order_trace+0x22/0x90
Dec 20 01:01:58 Lubuntu kernel: [142653.102628]  [<c114008a>] __kmalloc_track_caller+0x14a/0x160
Dec 20 01:01:58 Lubuntu kernel: [142653.102631]  [<c113df88>] ? kmem_cache_alloc+0xe8/0x120
Dec 20 01:01:58 Lubuntu kernel: [142653.102636]  [<c14caf2f>] ? sock_alloc_send_pskb+0x17f/0x2b0
Dec 20 01:01:58 Lubuntu kernel: [142653.102639]  [<c14cfb65>] __alloc_skb+0x55/0x200
Dec 20 01:01:58 Lubuntu kernel: [142653.102642]  [<c14caf2f>] sock_alloc_send_pskb+0x17f/0x2b0
Dec 20 01:01:58 Lubuntu kernel: [142653.102645]  [<c14cb07d>] sock_alloc_send_skb+0x1d/0x30
Dec 20 01:01:58 Lubuntu kernel: [142653.102649]  [<c1556cb3>] unix_stream_sendmsg+0x2f3/0x3e0
Dec 20 01:01:58 Lubuntu kernel: [142653.102652]  [<c14c74bf>] sock_aio_write+0xff/0x120
Dec 20 01:01:58 Lubuntu kernel: [142653.102655]  [<c14c73c0>] ? sock_register+0x90/0x90
Dec 20 01:01:58 Lubuntu kernel: [142653.102659]  [<c114f42c>] do_sync_readv_writev+0xac/0xe0
Dec 20 01:01:58 Lubuntu kernel: [142653.102664]  [<c125d3f4>] ? security_file_permission+0x24/0xb0
Dec 20 01:01:58 Lubuntu kernel: [142653.102667]  [<c114eba3>] ? rw_verify_area+0x63/0x110
Dec 20 01:01:58 Lubuntu kernel: [142653.102670]  [<c114f6bf>] do_readv_writev+0x9f/0x190
Dec 20 01:01:58 Lubuntu kernel: [142653.102672]  [<c14c73c0>] ? sock_register+0x90/0x90
Dec 20 01:01:58 Lubuntu kernel: [142653.102677]  [<c10735a2>] ? check_preempt_curr+0x72/0x90
Dec 20 01:01:58 Lubuntu kernel: [142653.102680]  [<c10735e8>] ? ttwu_do_wakeup+0x28/0x110
Dec 20 01:01:58 Lubuntu kernel: [142653.102683]  [<c114f7e7>] vfs_writev+0x37/0x60
Dec 20 01:01:58 Lubuntu kernel: [142653.102685]  [<c114f94d>] sys_writev+0x3d/0xa0
Dec 20 01:01:58 Lubuntu kernel: [142653.102689]  [<c15cff5f>] sysenter_do_call+0x12/0x28
Dec 20 01:01:58 Lubuntu kernel: [142653.102692]  [<c15c0000>] ? filter_check_discard.isra.15.part.16+0xb/0x25
Dec 20 01:01:58 Lubuntu kernel: [142653.102694] Mem-Info:
Dec 20 01:01:58 Lubuntu kernel: [142653.102696] DMA per-cpu:
Dec 20 01:01:58 Lubuntu kernel: [142653.102698] CPU    0: hi:    0, btch:   1 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102699] CPU    1: hi:    0, btch:   1 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102701] CPU    2: hi:    0, btch:   1 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102702] CPU    3: hi:    0, btch:   1 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102704] Normal per-cpu:
Dec 20 01:01:58 Lubuntu kernel: [142653.102705] CPU    0: hi:  186, btch:  31 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102707] CPU    1: hi:  186, btch:  31 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102709] CPU    2: hi:  186, btch:  31 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102710] CPU    3: hi:  186, btch:  31 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102711] HighMem per-cpu:
Dec 20 01:01:58 Lubuntu kernel: [142653.102713] CPU    0: hi:  186, btch:  31 usd:  17
Dec 20 01:01:58 Lubuntu kernel: [142653.102715] CPU    1: hi:  186, btch:  31 usd:  19
Dec 20 01:01:58 Lubuntu kernel: [142653.102717] CPU    2: hi:  186, btch:  31 usd:   0
Dec 20 01:01:58 Lubuntu kernel: [142653.102718] CPU    3: hi:  186, btch:  31 usd:  19
Dec 20 01:01:58 Lubuntu kernel: [142653.102722] active_anon:109650 inactive_anon:110327 isolated_anon:0
Dec 20 01:01:58 Lubuntu kernel: [142653.102722]  active_file:317770 inactive_file:89531 isolated_file:0
Dec 20 01:01:58 Lubuntu kernel: [142653.102722]  unevictable:7798 dirty:0 writeback:0 unstable:0
Dec 20 01:01:58 Lubuntu kernel: [142653.102722]  free:177821 slab_reclaimable:182861 slab_unreclaimable:6900
Dec 20 01:01:58 Lubuntu kernel: [142653.102722]  mapped:565603 shmem:5873 pagetables:3184 bounce:56
Dec 20 01:01:58 Lubuntu kernel: [142653.102729] DMA free:4232kB min:784kB low:980kB high:1176kB active_anon:0kB inactive_anon:0kB active_file:432kB inactive_file:412kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:10548kB slab_unreclaimable:80kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:76kB writeback_tmp:0kB pages_scanned:3741 all_unreclaimable? yes
Dec 20 01:01:58 Lubuntu kernel: [142653.102731] lowmem_reserve[]: 0 869 6072 6072
Dec 20 01:01:58 Lubuntu kernel: [142653.102739] Normal free:44208kB min:44216kB low:55268kB high:66324kB active_anon:0kB inactive_anon:0kB active_file:4152kB inactive_file:3972kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:890008kB mlocked:0kB dirty:0kB writeback:0kB mapped:4880kB shmem:0kB slab_reclaimable:720896kB slab_unreclaimable:27520kB kernel_stack:3416kB pagetables:756kB unstable:0kB bounce:148kB writeback_tmp:0kB pages_scanned:25082 all_unreclaimable? yes
Dec 20 01:01:58 Lubuntu kernel: [142653.102741] lowmem_reserve[]: 0 0 41626 41626
Dec 20 01:01:58 Lubuntu kernel: [142653.102748] HighMem free:662844kB min:512kB low:66692kB high:132872kB active_anon:438600kB inactive_anon:441308kB active_file:1266496kB inactive_file:353740kB unevictable:31192kB isolated(anon):0kB isolated(file):0kB present:5328224kB mlocked:31192kB dirty:0kB writeback:0kB mapped:2257532kB shmem:23492kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:11980kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 20 01:01:58 Lubuntu kernel: [142653.102750] lowmem_reserve[]: 0 0 0 0
Dec 20 01:01:58 Lubuntu kernel: [142653.102754] DMA: 41*4kB 29*8kB 77*16kB 51*32kB 6*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4284kB
Dec 20 01:01:58 Lubuntu kernel: [142653.102763] Normal: 1764*4kB 1524*8kB 1029*16kB 79*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 44256kB
Dec 20 01:01:58 Lubuntu kernel: [142653.102773] HighMem: 4662*4kB 1936*8kB 842*16kB 5637*32kB 3319*64kB 916*128kB 149*256kB 21*512kB 1*1024kB 1*2048kB 13*4096kB = 662872kB
Dec 20 01:01:58 Lubuntu kernel: [142653.102787] 415373 total pagecache pages
Dec 20 01:01:58 Lubuntu kernel: [142653.102789] 722 pages in swap cache
Dec 20 01:01:58 Lubuntu kernel: [142653.102790] Swap cache stats: add 5238, delete 4516, find 3096/3220
Dec 20 01:01:58 Lubuntu kernel: [142653.102792] Free swap  = 1937052kB
Dec 20 01:01:58 Lubuntu kernel: [142653.102793] Total swap = 1953120kB
Dec 20 01:01:58 Lubuntu kernel: [142653.121617] 1703920 pages RAM
Dec 20 01:01:58 Lubuntu kernel: [142653.121622] 1475586 pages HighMem
Dec 20 01:01:58 Lubuntu kernel: [142653.121623] 693511 pages reserved
Dec 20 01:01:58 Lubuntu kernel: [142653.121624] 367083 pages shared
Dec 20 01:01:58 Lubuntu kernel: [142653.121625] 555463 pages non-shared
Dec 20 01:01:58 Lubuntu kernel: [142653.121627] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Dec 20 01:01:58 Lubuntu kernel: [142653.121638] [  403]     0   403      705      148   3       0             0 upstart-udev-br
Dec 20 01:01:58 Lubuntu kernel: [142653.121642] [  436]     0   436      843      412   0     -17         -1000 udevd
Dec 20 01:01:58 Lubuntu kernel: [142653.121645] [  573]     0   573      842      323   2     -17         -1000 udevd
Dec 20 01:01:58 Lubuntu kernel: [142653.121648] [  574]     0   574      844      321   2     -17         -1000 udevd
Dec 20 01:01:58 Lubuntu kernel: [142653.121651] [  774]     0   774      704      149   1       0             0 upstart-socket-
Dec 20 01:01:58 Lubuntu kernel: [142653.121654] [  866]     0   866     1176      240   2       0             0 vsftpd
Dec 20 01:01:58 Lubuntu kernel: [142653.121657] [  975]     0   975     5740      996   2       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121660] [  978]     0   978     1673      602   3     -17         -1000 sshd
Dec 20 01:01:58 Lubuntu kernel: [142653.121663] [  980]   102   980      979      434   2       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121666] [  989]     0   989     3325      439   1       0             0 nmbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121669] [ 1001]     0  1001     5864      334   3       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121672] [ 1052]     0  1052     1854      639   1       0             0 modem-manager
Dec 20 01:01:58 Lubuntu kernel: [142653.121675] [ 1057]     0  1057     1195      337   2       0             0 bluetoothd
Dec 20 01:01:58 Lubuntu kernel: [142653.121678] [ 1067]     0  1067     6299     1327   2       0             0 NetworkManager
Dec 20 01:01:58 Lubuntu kernel: [142653.121682] [ 1081]   101  1081     7768      329   0       0             0 rsyslogd
Dec 20 01:01:58 Lubuntu kernel: [142653.121685] [ 1092]   108  1092      895      425   3       0             0 avahi-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121688] [ 1097]   108  1097      862      106   0       0             0 avahi-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121690] [ 1124]     0  1124     4229      847   3       0             0 cupsd
Dec 20 01:01:58 Lubuntu kernel: [142653.121694] [ 1146]     0  1146     6363      995   2       0             0 polkitd
Dec 20 01:01:58 Lubuntu kernel: [142653.121696] [ 1208]     0  1208     1158      211   1       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121699] [ 1214]     0  1214     1158      208   1       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121702] [ 1220]     0  1220     1158      211   1       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121705] [ 1221]     0  1221     1158      210   1       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121708] [ 1224]     0  1224     1158      211   1       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121711] [ 1241]     0  1241      544      163   1       0             0 acpid
Dec 20 01:01:58 Lubuntu kernel: [142653.121714] [ 1242]     0  1242     8529     7790   3       0             0 lightdm
Dec 20 01:01:58 Lubuntu kernel: [142653.121717] [ 1252]     0  1252      955      160   3       0             0 irqbalance
Dec 20 01:01:58 Lubuntu kernel: [142653.121720] [ 1261]     0  1261      655      228   3       0             0 cron
Dec 20 01:01:58 Lubuntu kernel: [142653.121723] [ 1262]     0  1262      619       31   1       0             0 atd
Dec 20 01:01:58 Lubuntu kernel: [142653.121725] [ 1265]   107  1265     6483     1119   1       0             0 whoopsie
Dec 20 01:01:58 Lubuntu kernel: [142653.121729] [ 1277]   116  1277    80506    40846   1       0             0 mysqld
Dec 20 01:01:58 Lubuntu kernel: [142653.121731] [ 1327]   103  1327     8876     1018   3       0             0 colord
Dec 20 01:01:58 Lubuntu kernel: [142653.121735] [ 1392]     0  1392    25298     3293   1       0             0 Xorg
Dec 20 01:01:58 Lubuntu kernel: [142653.121738] [ 1608]   117  1608     1938      252   0       0             0 exim4
Dec 20 01:01:58 Lubuntu kernel: [142653.121740] [ 1670]     0  1670     5870     1056   1       0             0 lightdm
Dec 20 01:01:58 Lubuntu kernel: [142653.121743] [ 1673]     0  1673     6183      782   0       0             0 accounts-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121746] [ 1693]     0  1693     8603      889   0       0             0 console-kit-dae
Dec 20 01:01:58 Lubuntu kernel: [142653.121749] [ 1780]   105  1780      558      137   3       0             0 lightdm-greeter
Dec 20 01:01:58 Lubuntu kernel: [142653.121752] [ 1785]   105  1785      845      254   2       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121755] [ 1786]   105  1786    27032     3122   2       0             0 lightdm-gtk-gre
Dec 20 01:01:58 Lubuntu kernel: [142653.121758] [ 1788]   105  1788    11091      737   1       0             0 at-spi-bus-laun
Dec 20 01:01:58 Lubuntu kernel: [142653.121761] [ 1792]   105  1792      812      357   1       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121764] [ 1795]   105  1795     4272      749   3       0             0 at-spi2-registr
Dec 20 01:01:58 Lubuntu kernel: [142653.121767] [ 1804]   105  1804     6687      656   2       0             0 gvfsd
Dec 20 01:01:58 Lubuntu kernel: [142653.121770] [ 1808]   105  1808    10691      748   2       0             0 gvfsd-fuse
Dec 20 01:01:58 Lubuntu kernel: [142653.121773] [ 1816]     0  1816     7163      926   0       0             0 upowerd
Dec 20 01:01:58 Lubuntu kernel: [142653.121775] [ 1976]     0  1976      640      237   1       0             0 hddtemp
Dec 20 01:01:58 Lubuntu kernel: [142653.121779] [ 2188]     0  2188    10280     1878   2       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121782] [ 2226]    33  2226    10340     1319   0       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121784] [ 2227]    33  2227    10340     1335   1       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121787] [ 2228]    33  2228    10296     1301   3       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121790] [ 2229]    33  2229    10296     1301   1       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121793] [ 2230]    33  2230    10340     1323   2       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121796] [ 2275]     0  2275     1158      209   2       0             0 getty
Dec 20 01:01:58 Lubuntu kernel: [142653.121799] [ 2562]   106  2562     1439      505   3       0             0 ntpd
Dec 20 01:01:58 Lubuntu kernel: [142653.121802] [ 2567]     0  2567     3533      781   1       0             0 lightdm
Dec 20 01:01:58 Lubuntu kernel: [142653.121804] [ 2572]    33  2572    10340     1329   0       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121807] [ 2573]    33  2573    10340     1320   0       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121810] [ 2574]    33  2574    10340     1319   0       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121813] [ 2832]   113  2832     1675      730   1       0             0 nxssh
Dec 20 01:01:58 Lubuntu kernel: [142653.121816] [ 2833]     0  2833     4257     1087   0       0             0 sshd
Dec 20 01:01:58 Lubuntu kernel: [142653.121819] [ 2934]  1000  2934     4257      457   1       0             0 sshd
Dec 20 01:01:58 Lubuntu kernel: [142653.121822] [ 2935]  1000  2935     3734     3226   1       0             0 nxnode
Dec 20 01:01:58 Lubuntu kernel: [142653.121825] [ 2959]  1000  2959    36002    26402   2       0             0 nxagent
Dec 20 01:01:58 Lubuntu kernel: [142653.121828] [ 2961]   113  2961     5200     2747   3       0             0 nxserver
Dec 20 01:01:58 Lubuntu kernel: [142653.121831] [ 2964]  1000  2964     3767     2017   1       0             0 nxnode
Dec 20 01:01:58 Lubuntu kernel: [142653.121833] [ 2974]  1000  2974    12069     1414   1       0             0 lxsession
Dec 20 01:01:58 Lubuntu kernel: [142653.121836] [ 2977]  1000  2977     4482     1937   0       0             0 openbox
Dec 20 01:01:58 Lubuntu kernel: [142653.121839] [ 2982]  1000  2982    48577     3282   0       0             0 lxpanel
Dec 20 01:01:58 Lubuntu kernel: [142653.121842] [ 2983]  1000  2983     1875      592   0       0             0 xscreensaver
Dec 20 01:01:58 Lubuntu kernel: [142653.121845] [ 2985]  1000  2985    31394     3145   2       0             0 pcmanfm
Dec 20 01:01:58 Lubuntu kernel: [142653.121848] [ 2986]  1000  2986     7310     1356   2       0             0 polkit-gnome-au
Dec 20 01:01:58 Lubuntu kernel: [142653.121850] [ 2990]  1000  2990      961      139   2       0             0 dbus-launch
Dec 20 01:01:58 Lubuntu kernel: [142653.121854] [ 2991]  1000  2991      938      308   2       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121856] [ 2993]  1000  2993    11092      736   2       0             0 at-spi-bus-laun
Dec 20 01:01:58 Lubuntu kernel: [142653.121859] [ 2994]  1000  2994     7349     1041   0       0             0 xfce4-power-man
Dec 20 01:01:58 Lubuntu kernel: [142653.121862] [ 3001]  1000  3001    12608     2707   1       0             0 update-notifier
Dec 20 01:01:58 Lubuntu kernel: [142653.121865] [ 3005]  1000  3005      812      357   1       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121868] [ 3011]  1000  3011    50324     3472   2       0             0 nm-applet
Dec 20 01:01:58 Lubuntu kernel: [142653.121870] [ 3016]  1000  3016     4273      763   0       0             0 at-spi2-registr
Dec 20 01:01:58 Lubuntu kernel: [142653.121873] [ 3019]  1000  3019     1627      597   2       0             0 xfconfd
Dec 20 01:01:58 Lubuntu kernel: [142653.121876] [ 3031]  1000  3031     1487      456   3       0             0 menu-cached
Dec 20 01:01:58 Lubuntu kernel: [142653.121880] [ 3042]  1000  3042     2204      579   0       0             0 gconfd-2
Dec 20 01:01:58 Lubuntu kernel: [142653.121882] [ 3046]  1000  3046    22131      893   3       0             0 pulseaudio
Dec 20 01:01:58 Lubuntu kernel: [142653.121886] [ 3048]   115  3048     5334      315   1       0             0 rtkit-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121888] [ 3069]  1000  3069    11507     1082   2       0             0 indicator-appli
Dec 20 01:01:58 Lubuntu kernel: [142653.121891] [ 3274]  1000  3274     8400      613   0       0             0 dconf-service
Dec 20 01:01:58 Lubuntu kernel: [142653.121894] [ 3305]  1000  3305     4970     1243   3       0             0 VBoxXPCOMIPCD
Dec 20 01:01:58 Lubuntu kernel: [142653.121897] [ 3310]  1000  3310    10332     2611   1       0             0 VBoxSVC
Dec 20 01:01:58 Lubuntu kernel: [142653.121900] [ 3356]  1000  3356    34122     4071   3       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121903] [ 3357]  1000  3357     8258      520   1       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121905] [ 3373]     0  3373     6216      715   3       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121908] [ 3377]     0  3377     6216      332   3       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121911] [ 3379]     0  3379    40907      465   3       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121915] [ 3419]     0  3419     6216      334   3       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121918] [ 3421]     0  3421    39029      496   2       0             0 truecrypt
Dec 20 01:01:58 Lubuntu kernel: [142653.121922] [ 3444]  1000  3444   331341   284523   3       0             0 VirtualBox
Dec 20 01:01:58 Lubuntu kernel: [142653.121924] [ 3991]  1000  3991   345903   282194   1       0             0 VirtualBox
Dec 20 01:01:58 Lubuntu kernel: [142653.121927] [ 4395]     0  4395    17967     1184   2       0             0 udisksd
Dec 20 01:01:58 Lubuntu kernel: [142653.121930] [ 4872]    33  4872    10296     1302   2       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121933] [ 5576]     0  5576     5964     1019   2       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121936] [ 5646]     0  5646     4272     1113   0       0             0 sshd
Dec 20 01:01:58 Lubuntu kernel: [142653.121939] [ 5826]   113  5826     4309      491   2       0             0 sshd
Dec 20 01:01:58 Lubuntu kernel: [142653.121942] [ 5827]   113  5827     5169     4672   2       0             0 nxserver
Dec 20 01:01:58 Lubuntu kernel: [142653.121944] [ 6057]   113  6057     1675      567   0       0             0 nxssh
Dec 20 01:01:58 Lubuntu kernel: [142653.121947] [ 6139]    33  6139    10340     1299   0       0             0 apache2
Dec 20 01:01:58 Lubuntu kernel: [142653.121950] [ 6162]  1000  6162     1303      299   0       0             0 su-to-root
Dec 20 01:01:58 Lubuntu kernel: [142653.121953] [ 6168]  1000  6168    24381     2066   0       0             0 gksu
Dec 20 01:01:58 Lubuntu kernel: [142653.121956] [ 6175]     0  6175      961       85   1       0             0 dbus-launch
Dec 20 01:01:58 Lubuntu kernel: [142653.121958] [ 6176]     0  6176      842      169   2       0             0 dbus-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121961] [ 6185]     0  6185    14470     1015   0       0             0 kdeinit4
Dec 20 01:01:58 Lubuntu kernel: [142653.121964] [ 6186]     0  6186    16613     1439   0       0             0 klauncher
Dec 20 01:01:58 Lubuntu kernel: [142653.121967] [ 6189]     0  6189    18512     1834   0       0             0 kded4
Dec 20 01:01:58 Lubuntu kernel: [142653.121970] [ 6197]     0  6197     6351      837   3       0             0 udisks-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121973] [ 6198]     0  6198     1655      108   2       0             0 udisks-daemon
Dec 20 01:01:58 Lubuntu kernel: [142653.121976] [ 6213]     0  6213    22899     1784   2       0             0 kactivitymanage
Dec 20 01:01:58 Lubuntu kernel: [142653.121979] [ 6262]  1000  6262    26641     2961   0       0             0 lxterminal
Dec 20 01:01:58 Lubuntu kernel: [142653.121981] [ 6263]  1000  6263      601      182   1       0             0 gnome-pty-helpe
Dec 20 01:01:58 Lubuntu kernel: [142653.121984] [ 6264]  1000  6264     1558      664   2       0             0 bash
Dec 20 01:01:58 Lubuntu kernel: [142653.121987] [ 7259]     0  7259     5957      972   2       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121990] [24575]     0 24575     5954     1194   3       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121993] [24843]     0 24843     5991     1142   2       0             0 smbd
Dec 20 01:01:58 Lubuntu kernel: [142653.121996] [25240]     0 25240    18206     2749   0       0             0 kuiserver
Dec 20 01:01:58 Lubuntu kernel: [142653.122000] [27635]     0 27635     2684      597   3       0             0 cron
Dec 20 01:01:58 Lubuntu kernel: [142653.122003] [27636]     0 27636      558      137   3       0             0 sh
Dec 20 01:01:58 Lubuntu kernel: [142653.122006] [27637]     0 27637      558      136   0       0             0 backintime
Dec 20 01:01:58 Lubuntu kernel: [142653.122009] [27638]     0 27638   114984   113351   1       0             0 python
Dec 20 01:01:58 Lubuntu kernel: [142653.122014] [ 3194]     0  3194      558      137   0       0             0 sh
Dec 20 01:01:58 Lubuntu kernel: [142653.122017] [ 3195]     0  3195     1244      324   0       0             0 chmod
Dec 20 01:01:58 Lubuntu kernel: [142653.122019] Out of memory: Kill process 3444 (VirtualBox) score 139 or sacrifice child
Dec 20 01:01:58 Lubuntu kernel: [142653.122025] Killed process 3444 (VirtualBox) total-vm:1325364kB, anon-rss:19736kB, file-rss:1118356kB

It just goes on and on like this and the server is totally frozen. It runs Lubuntu 12.10 32 bit

user277244

Posted 2013-12-21T18:10:38.607

Reputation: 251

Since oom means “out of memory”, do you have any logs or ideas on what consumes all the memory? How much memory (physical and swap) does the system have? – David Foerster – 2013-12-21T19:40:07.297

It has 6gb of physical and 2gb of ram. Other than the kern.log file, where do I check for out of memory logs?

Also, seems to me that in the kern.log file it says total swap and free swap are about the same, so it looks like it's not even filling up the swap. – user277244 – 2013-12-21T19:55:26.063

It all seems to have started when I added the million or so files. Could that somehow increase ram usage? I use backintime nightly to create backups, rsync to copy the original to another computer, and crashplan to backup everything online. So I disabled rsync and crashplan and it still crashed at the same time. I run two windows 2000 virtual machines with 1gb of ram each but have run them for a few years without issues on any systems. Nothing else runs on the server. Could it be that I'm running a 32 bit kernal? – user277244 – 2013-12-21T19:58:40.873

A large amount of files to rsync could consume lots of memory, but you disabled that. If it's always around the same time you could run a job that periodically logs the top memory consumers every few seconds. – David Foerster – 2013-12-21T20:06:55.350

What exactly could I run to log the top memory consumers? Do you have any examples? – user277244 – 2013-12-22T00:35:17.870

Answers

0

The OOM log shows which processes it saw running when it encountered the low-memory condition, then which ones it attempted to kill. See the table beginning with the following heading:

 [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name

total_vm is the column showing the total mapped memory (RAM + Swap) for the process - I note the big processes are VirtualBox (x2) and python (pid == 27638).

I'd assume you knew about VirtualBox, but it would be worth finding out what that python process was. Could be a script gone awry? Something that monitors added files? Some sort of media scanner that looks for new files (eg a media server such as plex, rygel or similar?), or your backup manager?

The simplest way to track memory usage (but its strictly online/interactive) would be using top and sorting by memory (hit M - that's capital M, - ie shift-M). The first memory row will also tell you how much physical RAM the OS thinks you have "Mem:" which is useful if you think you're missing some of your RAM - due to 32-bit issues.

You could cobble some sort of monitoring script, although off the top of my head I don't know a nice way to do it. As a start I've got:

top -n 1 -b | sed '1,/^$/d' | sort -r -n -k10

This dumps the top output, trims off the header up to the blank line (from 1 to "^$"), then sorts the output by the 10th column (for me its memory percentage), in reverse numeric order. Run that regularly dumping to a file might help.

Greg

Posted 2013-12-21T18:10:38.607

Reputation: 865

Python is probably from backintime since the pid is right after backintime and I believe it uses python. Does that number mean it's using 100 and some mb of ram? That's not that much. The virtualbox looks like 300 or so mb ram but I give 1gb of ram to each virtualbox instance. What's most alarming to me is that the system crashes with oom-killer errors but the swap does not appear to be used. Why would oom-killer start doing its thing if I have swap available? – user277244 – 2013-12-23T23:04:03.903

The units are pages - 4kB each so virtualbox is using ~1.3GB each – Greg – 2013-12-24T23:58:00.443

Not all allocations can be satisfied by moving pages to swap - eg larger allocation than free swap, or the allocations in the other process may be of a type that can't be paged. Here's another question that suggests tweaking the allocator to fail the alloc rather than trying to honour it by invoking the oom handler to kill someone else.http://askubuntu.com/questions/161521/why-does-my-server-freeze-everyday-at-the-same-time

– Greg – 2013-12-25T00:17:17.437

Asked: 2013-12-21T18:10:38.607

Viewed: 1 296 times

Active: 2015-10-17T07:08:21.547