my server crashed tonight :(

[Hal9000]

Hello!
Something wicked happened to my server tonight, it suddenly locked up after 60 days uptime. I was running 2.4.32-grsec.

Jan 26 01:35:42 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:35:49 localhost kernel: VM: killing process ircd
Jan 26 01:38:00 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:38:02 localhost kernel: VM: killing process python
Jan 26 01:39:06 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:39:22 localhost kernel: VM: killing process imapd
Jan 26 01:40:42 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:40:43 localhost kernel: VM: killing process cron
Jan 26 01:41:14 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:41:15 localhost kernel: VM: killing process mrtg
Jan 26 01:41:43 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:41:44 localhost kernel: VM: killing process MusiBOT.botchk
Jan 26 01:42:10 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:42:11 localhost kernel: VM: killing process couriertls
Jan 26 01:42:55 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:42:55 localhost kernel: VM: killing process python
Jan 26 01:44:01 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:44:02 localhost kernel: VM: killing process apache2
Jan 26 01:44:13 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:44:14 localhost kernel: VM: killing process mysqld
Jan 26 01:45:41 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:47:04 localhost kernel: VM: killing process apache2
Jan 26 01:47:05 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:47:07 localhost kernel: VM: killing process apache2
Jan 26 01:47:08 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:47:08 localhost kernel: VM: killing process apache2
Jan 26 01:47:50 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:47:50 localhost kernel: VM: killing process apache2
Jan 26 01:47:53 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:47:53 localhost kernel: VM: killing process named
Jan 26 01:49:08 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:49:10 localhost kernel: VM: killing process apache2
Jan 26 01:50:41 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:50:42 localhost kernel: VM: killing process apache2
Jan 26 01:50:45 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:50:45 localhost kernel: VM: killing process apache2
Jan 26 01:52:49 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:52:49 localhost kernel: VM: killing process SeenBOT.botchk
Jan 26 01:52:50 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:52:50 localhost kernel: VM: killing process apache2
Jan 26 01:53:40 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:53:40 localhost kernel: VM: killing process apache2
Jan 26 01:53:42 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:53:42 localhost kernel: VM: killing process python
Jan 26 01:55:55 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:55:56 localhost kernel: VM: killing process couriertls
Jan 26 01:56:11 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:56:11 localhost kernel: VM: killing process monit
Jan 26 01:59:21 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 01:59:22 localhost kernel: VM: killing process svscan
Jan 26 02:00:46 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:00:54 localhost kernel: VM: killing process couriertls
Jan 26 02:01:01 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:02:07 localhost kernel: VM: killing process suexec2
Jan 26 02:02:37 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:02:38 localhost kernel: VM: killing process apache2
Jan 26 02:02:38 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:02:51 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:02:51 localhost kernel: VM: killing process tcp-env
Jan 26 02:02:59 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:02:59 localhost kernel: VM: killing process relaylock
Jan 26 02:04:18 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:04:56 localhost kernel: VM: killing process apache2
Jan 26 02:04:58 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:04:59 localhost kernel: VM: killing process apache2
Jan 26 02:04:59 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:03 localhost kernel: VM: killing process services.chk
Jan 26 02:05:06 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:06 localhost kernel: VM: killing process eggdrop
Jan 26 02:05:07 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:23 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:23 localhost kernel: VM: killing process python
Jan 26 02:05:48 localhost kernel: VM: killing process named
Jan 26 02:05:51 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:51 localhost kernel: VM: killing process apache2
Jan 26 02:05:51 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:05:51 localhost kernel: VM: killing process ircd
Jan 26 02:06:46 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:06:46 localhost kernel: VM: killing process couriertls
Jan 26 02:07:02 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:07:03 localhost kernel: VM: killing process apache2
Jan 26 02:07:49 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:07:49 localhost kernel: VM: killing process couriertcpd
Jan 26 02:07:57 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:07:57 localhost kernel: VM: killing process inetd
Jan 26 02:07:59 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:07:59 localhost kernel: VM: killing process couriertcpd
Jan 26 02:08:04 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:08:04 localhost kernel: VM: killing process apache2
Jan 26 02:08:14 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:08:14 localhost kernel: VM: killing process apache2
Jan 26 02:09:42 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:09:42 localhost kernel: VM: killing process apache2
Jan 26 02:09:42 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:09:42 localhost kernel: VM: killing process apache2
Jan 26 02:10:06 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:10:07 localhost kernel: VM: killing process apache2
Jan 26 02:10:27 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:10:28 localhost kernel: VM: killing process python
Jan 26 02:12:37 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:12:42 localhost kernel: VM: killing process seenbot.conf
Jan 26 02:12:53 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:12:54 localhost kernel: VM: killing process apache2
Jan 26 02:15:28 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:15:29 localhost kernel: VM: killing process monit
Jan 26 02:15:39 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:15:49 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:15:49 localhost kernel: VM: killing process apache2
Jan 26 02:15:56 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:16:03 localhost kernel: VM: killing process apache2
Jan 26 02:16:08 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:16:10 localhost kernel: VM: killing process apache2
Jan 26 02:16:12 localhost kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
Jan 26 02:16:16 localhost kernel: VM: killing process python

I then reset the machine and now everything seems back to normal...
Any ideas what might be causing this?
greets
hal

[spender]

Without more information, the only thing I can suggest is to upgrade to 2.1.8.
Did you notice anything before the crash occurred? Was memory usage unusually high?

-Brad

[Hal9000]

Hello!
Sorry, i don't know quite what information would be useful...
I was using grsec 2.1.7 of course, as the kernel was compiled back in november.
Here is a graph from cacti that shows the memory usage. At 1:30 am the box locked up, at about 2:30 i reset the machine.
It seems something went wrong with the memory, but i can't tell what and why... http://www.musichat.net/porcodio.png
greets
hal

[overdose]

pkoi vous parler en anglais alors que vous etes tout les deux francais ...

onaqr qr pbaaneqf

[Hal9000]

parce-que je ne suis pas un francais fanatique... pour être precis, je ne suis pas fraincais du tout... et spender je crois quand-même.
et btw, ou est ton problème... l'anglais est la langue universelle du internet, si je veux parler en francais je vais aux moulin rouge, pas dans le forum de grsecurity.