Major Problem With Grsecurity 1.9.6 And Kernel 2.4.19

[ChrisO]

I'm having A LOT of trouble with kernel version 2.4.19, Grsecurity 1.9.6, and Apache 1.3.26 with Mod_Perl, Mod_SSL, and PHP installed into it (all the latest versions).

Everytime I boot up, when Redhat 7.3 goes to "setup the swap space" it hangs for a good minute or so. And after it boots up, Apache runs, but none of the pages that it serves comes up (you just get a "Connection Refused"). Nothing turns up in the logs (either Apache's logs or Syslog's) and Apache correctly opens up the ports that it should (and I can connect to them). Upon looking into it further, I've discovered that Apache is causing insanely high load on my box. Here's the proof:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
apache 3625 19.4 3.0 24148 15664 ? R 01:11 2:09 /usr/local/www/bin/httpd -DSSL
apache 3047 19.4 3.0 24148 15664 ? R 01:11 2:09 /usr/local/www/bin/httpd -DSSL
apache 3604 19.4 3.0 24148 15664 ? R 01:11 2:09 /usr/local/www/bin/httpd -DSSL
apache 15797 19.4 3.0 24148 15664 ? R 01:11 2:09 /usr/local/www/bin/httpd -DSSL
apache 1234 19.4 3.0 24148 15664 ? R 01:11 2:09 /usr/local/www/bin/httpd -DSSL

and

1:25am up 16 min, 2 users, load average: 5.10, 4.83, 3.08

Furthermore, when the httpd processes are all killed off, the load is reduced completely.

I have narrowed it down to Grsecurity 1.9.6 (with kernel 2.4.19), but not a specific Grsecurity option. Kernel 2.4.18 with Grsecurity 1.9.5 works fine and kernel 2.4.19 without Grsecurity options enabled (the patch is applied, however) works like a charm.

I'm really pulling out my hair on this one. I wrote this forum in the effort that maybe it'll raise awareness of a bug or if someone knows of a specific Grsecurity option that might be causing this issue. Feel free to ask for further information. I'd REALLY like to get this resolved.

[PaX Team]

Upon looking into it further, I've discovered that Apache is causing insanely high load on my box. Here's the proof:

I have narrowed it down to Grsecurity 1.9.6 (with kernel 2.4.19), but not a specific Grsecurity option. Kernel 2.4.18 with Grsecurity 1.9.5 works fine and kernel 2.4.19 without Grsecurity options enabled (the patch is applied, however) works like a charm.

if it's a PaX problem, you can help to nail it down by first trying to disable
PaX in your grsec config (if it wasn't enabled, then ignore the rest . if
PaX causes the problem, then i need to know what options you enabled
in the kernel config and whether you enabled/disabled anything on apache
itself (using chpax). also, it would help if you could submit the outputs of
'cat /proc/<apache_pids>/maps' and maybe attach to the tasks with gdb
and look at where they're stuck.

[ChrisO]

It does have to do with PaX. As soon as I disabled PaX and rebooted into kernel 2.4.19 with Grsecurity 1.9.6, it worked fine.

I did not enable/disable anything on Apache using chpax.

My kernel configuration file can be found at: http://chris.daran.org/kernel.conf

The "cat /proc/<pid>/maps" result can be found at: http://chris.daran.org/maps

And from GDB, here's the only stuff of interest that I get:

[New Thread 1024 (LWP 9282)]
Error while reading shared library symbols:
Can't attach LWP 9282: No such process
*snip*
Loaded symbols for /lib/libnss_nisplus.so.2
0x080d41f0 in ap_update_child_status ()

And furthermore, when I try to get out of GDB, it won't let me becuase it gives me a similiar error message to "Can't attach LWP 9282: No such process".

[PaX Team]

It does have to do with PaX. As soon as I disabled PaX and rebooted into kernel 2.4.19 with Grsecurity 1.9.6, it worked fine.

I did not enable/disable anything on Apache using chpax.

could you try to enable PaX in the kernel config and disable features on the apache binary (/usr/local/www/bin/httpd) just to see which one is causing the problem? also, one more combination to try would be to enable PaX and RANDEXEC (in the kernel config, no need to enable the latter on apache itself).

The "cat /proc/<pid>/maps" result can be found at: http://chris.daran.org/maps

hmm, this is interesting:

08255000-08c8f000 rwxp 00000000 00:00 0

it is the bss+sbrk() managed heap, and it should not be executable at all. can you verify that your mm/mmap.c:do_brk() actually has the patch that removes VM_EXEC/VM_MAYEXEC? also you may want to run apache thru strace to see better what's going on.

[ChrisO]

Hopefully, I'll have a reply to all of this in an hour or so. It's just really quite a pain to keep recompiling the kernel.

[ChrisO]

Ok, I enabled PaX in the kernel configuration. It, of course, still didn't work. However, when I used chpax to disable the enforcing of PAGE_EXEC, Apache worked. When I re-enabled it, Apache didn't work.

The PaX and RANDEXEC combination did not work. Apache was still messed up.

Also, what should I be looking for mm/mmap.c:do_brk()?

I ran Apache through strace prior to your request, but I didn't see anything abnormal. Anything in particular you would like me to look for?

[PaX Team]

Ok, I enabled PaX in the kernel configuration. It, of course, still didn't work. However, when I used chpax to disable the enforcing of PAGE_EXEC, Apache worked. When I re-enabled it, Apache didn't work.

so this is a non-exec page problem then and i'm quite puzzled at it too. which apache version are you using and in what setup (eg. multithreaded, prefork, etc)? also, are you using any special module that you didn't have before?

Also, what should I be looking for mm/mmap.c:do_brk()?

ok, don't worry about this one, i've just figured out that grsec had this patch applied to the wrong function, will be fixed soon.

I ran Apache through strace prior to your request, but I didn't see anything abnormal. Anything in particular you would like me to look for?

yes, syscalls to open/old_mmap/mprotect/munmap/ipc because these would show what happens to the address space of apache. don't forget to trace children as well as the problem may originate in them. and thanks for your help, i'll try to keep kernel compilations to a minimum .

[ChrisO]

I'm running Apache version 1.3.26. I'm pretty sure that I'm using multithreaded. This is an utterly stupid question, but how can I make sure?

No, I am not using any special modules. I assume you're referring to kernel modules. If not, the box's setup hasn't changed at all since when the kernel was still working on the latest version of Grsecurity with kernel 2.4.18 just prior to 2.4.19's and 1.9.6's release.

The only thing that I saw of interest in the strace was this (and I got a lot of it):

9417 time(NULL) = 1028760580
9417 wait4(-1, 0xb99e3f7c, WNOHANG, NULL) = -1 ECHILD (No child processes)
9417 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
9417 time(NULL) = 1028760581
9417 wait4(-1, 0xb99e3f7c, WNOHANG, NULL) = -1 ECHILD (No child processes)
9417 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)

You can view the entire result at: http://chris.daran.org/strace

But, I warn you, it's quite hefty.

[ChrisO]

Nevermind, maybe that thing of interest really isn't of interest, as the working Apache does the exact same thing. To be quite honest, I don't know what to look for in the strace result.

[PaX Team]

I'm running Apache version 1.3.26. I'm pretty sure that I'm using multithreaded. This is an utterly stupid question, but how can I make sure?

the configuration file should contain information about what execution model you're using, in particular i'd be interested in the ThreadsPerChild setting. based on the maps output you gave me before it seems that it came from the main control process, not from one of the actual servers (where the problem is likely to be). could you post one for them as well please?

No, I am not using any special modules. I assume you're referring to kernel modules. If not, the box's setup hasn't changed at all since when the kernel was still working on the latest version of Grsecurity with kernel 2.4.18 just prior to 2.4.19's and 1.9.6's release.

actually i was thinking of apache modules, but if you haven't added/changed anything since last time, this question is irrelevant then.

in any case, i'm in thinking mode for now (and bughunting too as there are other issues as well), if i come up with something, i'll let you know.