segfault with 2.6.30.6-200909121646 (works with 2.6.27.7)

[sfaerber]

Hi,

i've discovered a strange problem with 2.6.30.6 and grsecurity-2.1.14-2.6.30.6-200909121646.
A custom PHP-Application (in CGI-Mode, using ZendOptimizer) is suddenly segfaulting after upgrading.
If i switch back to my previous kernel (2.6.27.7 and grsecurity-2.1.12-2.6.27.7-200811291903) the problem disappears.
I've also discovered that disabling the ZendOptimizer in the PHP Application solves the problem.

I see errors like the following in "dmesg":
example.php[6683]: segfault at b63ddec8 ip b63cc8e8 sp bf02a630 error 7 in ld-2.3.4.so[b63c7000+15000]
example.php[6731]: segfault at a3ec7ec8 ip a3eb68e8 sp b2228e70 error 7 in ld-2.3.4.so[a3eb1000+15000]

I also did an strace to get an idea where the segfault occurs:

6945 mmap2(NULL, 285444, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xa46a1000
6945 mmap2(0xa46e2000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x41) = 0xa46e2000
6945 mprotect(0xa4d7c000, 3788, PROT_READ|PROT_WRITE) = -1 EACCES (Permission denied)
6945 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
5750 <... read resumed> "", 4096) = 0
5750 --- SIGCHLD (Child exited) @ 0 (0) ---


Any idea how to debug this further or what might cause this problem?
I'll try to get as much information as needed, just point me in the right direction.

Regards,

Sebastian

[PaX Team]

6945 mmap2(NULL, 285444, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xa46a1000
6945 mmap2(0xa46e2000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x41) = 0xa46e2000
6945 mprotect(0xa4d7c000, 3788, PROT_READ|PROT_WRITE) = -1 EACCES (Permission denied)

you should find out what is mapped at 0xa4d7c000 (turn off randomization to make it easier for yourself) to see why this mprotect failed. to me it looks like an attempt to apply text relocations, so i'd look for them in libraries loaded into your apache process (most likely Zend related). if you send me a full strace i can also figure it out if you want. as a sidenote, is your system an amd64 kernel with 32 bit userland?

[sfaerber]

I turned off address randomization (CONFIG_PAX_ASLR is not set) but couldn't find out what's at that address ...
You can download the full strace here: http://v6-net.de/grsec/strace-2.6.30.6
I had to do some obfuscation, paths/ip addresses only

I also compared this strace with one i did using 2.6.27.7 and noticed that the same mprotect fails there too. But using 2.6.27.7 there is no segfault, PHP
just closes fd 4 and continues to open the next file ... strange.

No 64bit involved, the kernel is 32bit, PAE and BIG_SMP (config: http://v6-net.de/grsec/2.6.30.6.config).
Any thoughts?

[PaX Team]

I also compared this strace with one i did using 2.6.27.7 and noticed that the same mprotect fails there too. But using 2.6.27.7 there is no segfault, PHP
just closes fd 4 and continues to open the next file ... strange.

strange indeed. the history of that mapping (as far as it can be determined from the strace) is:

Code: Select all

6910  mprotect(0xb8014000, 3788, PROT_READ|PROT_WRITE) = 0
6910  mprotect(0xb8014000, 3788, PROT_READ) = 0
6910  mprotect(0xb8014000, 3788, PROT_READ|PROT_WRITE) = -1 EACCES (Permission denied)

so for some reason it was mprotected multiple times over its lifetime and the last one failed, probably because PaX removed VM_MAYWRITE on the mapping. that should only occur for RELRO segments but then they should not be mprotected more than once. it would be important to learn what is mapped at that address (at that point it can only be the vdso or the stack, neither of which should be mprotected this way though). so could you run this php script from gdb and when it crashes and stops in gdb, can you take its /proc/pid/maps file and send it to me? i also noticed that after loading the Zend optimizer library, ld.so tried to make the stack executable, that probably means that this library lacks a GNU_STACK header or has an RWE one, i'd run execstack -c on it as you'll need to tweak your MPROTECT settings anyway if this library really wants an executable stack (or worse, generates code at runtime).

[sfaerber]

I didn't get a chance to run the PHP Script through gdb yet but i tried your suggesting regard execstack.
I cleared the bit via "execstack -c ZendOptimizer.so" and now everything is working, no more segfaults ...
before that i queried the setting via "-q" and got "?" back, so apparently no value was set before.

[PaX Team]

I cleared the bit via "execstack -c ZendOptimizer.so" and now everything is working, no more segfaults ...

ok, i think i know what happened here: glibc/ld.so has a variable that stores the stack access rights that the geniuses put into the RELRO segment. which would be all fine were it not for dlopen'd libraries that will force an executable stack and the need to update this already read-only variable. so a further stroke of genius figured that mprotecting the RELRO segment temporarily is the best way to security (remember what the whole idea behind RELRO was). so since PaX actually enforces RELRO this attempt obviously failed.

[closet geek]

Hi,

We have also run into this exact problem. execstack -c works but is not really a solution for us in the long term. What option should we disable in the .config to prevent this check from taking place?

Thanks.

[spender]

A long term solution would be to bug the software distributor (this is for-pay software we're talking about correct?) to mark their binaries properly so that they're not reducing the security of every system that uses their software (then you wouldn't have to mark them after-the-fact yourself). PAX_MPROTECT is the option performing the enforcement, but neither I nor the PaX team recommend disabling this as a long-term solution to the issue.

Meanwhile, people using SELinux are running their webservers with silently-degraded security thanks to PT_GNU_STACK, and no one seems to have noticed.

-Brad