dovecot vs 3.3.0-grsec

[Dwokfur]

I've recently tried hardened-sources-3.3.0 (grsecurity-2.9-3.3.0-201203251922) and dovecot stopped working properly. All other deamons seem to tolerate eachother with 3.3.0-grsec, except for dovecot.

Here are the error messages I see in mail.log:
Apr 4 21:55:55 replaced dovecot: imap: Error: dovecot/imap: error while loading shared libraries: libpthread.so.0: failed to map segment from shared object: Cannot allocate memory
Apr 4 21:55:55 replaced dovecot: master: Error: service(imap): command startup failed, throttling for 2 secs
Apr 4 21:55:55 replaced dovecot: imap: Fatal: master: service(imap): child 6275 returned error 127
Apr 4 21:55:55 replaced dovecot: imap-login: Error: read(imap) failed: Connection reset by peer
Apr 4 21:55:55 replaced dovecot: imap-login: Internal login failure (pid=6272 id=1) (internal failure, 1 succesful auths): user=<replaced>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Apr 4 21:56:13 replaced dovecot: master: Error: service(imap-login): command startup failed, throttling for 2 secs
Apr 4 21:56:13 replaced dovecot: imap-login: Fatal: master: service(imap-login): child 6309 killed with signal 9

restarting the daemon
Apr 4 21:59:43 replaced dovecot: master: Warning: Killed with signal 15 (by pid=6390 uid=0 code=kill)
Apr 4 21:59:53 replaced dovecot: master: Dovecot v2.1.3 starting up (core dumps disabled)
daemon restarted

Apr 4 22:00:43 replaced dovecot: master: Error: service(imap-login): command startup failed, throttling for 2 secs
Apr 4 22:00:43 replaced dovecot: imap-login: Fatal: master: service(imap-login): child 6450 killed with signal 9
Apr 4 22:05:12 replaced dovecot: imap-login: Login: user=<replaced>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=6484, secured
Apr 4 22:05:12 replaced dovecot: imap(replaced): Disconnected: Logged out in=44 out=721
Apr 4 22:05:13 replaced dovecot: imap-login: Error: dovecot/imap-login: error while loading shared libraries: libcrypto.so.1.0.0: failed to map segment from shared object: Cannot allocate memory
Apr 4 22:05:13 replaced dovecot: master: Error: service(imap-login): command startup failed, throttling for 2 secs
Apr 4 22:05:13 replaced dovecot: imap-login: Fatal: master: service(imap-login): child 6486 returned error 127
Apr 4 22:05:15 replaced dovecot: imap-login: Login: user=<replaced>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=6488, secured
Apr 4 22:05:17 replaced dovecot: imap(replaced): Disconnected: Logged out in=43541 out=178193

I only see some RLIMIT_AS lines in grsec.log, no other relevant messages:
Apr 4 22:00:43 replaced kernel: grsec: From 10.97.100.79: (root:U:/usr/libexec/dovecot/imap-login) denied resource overstep by requesting 63205376 for RLIMIT_AS against limit 16777216 for /usr/libexec/dovecot/imap-login[imap-login:6450] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/dovecot[dovecot:6409] uid/euid:0/0 gid/egid:0/0
Apr 4 22:05:13 replaced kernel: grsec: (root:U:/usr/libexec/dovecot/imap-login) denied resource overstep by requesting 17612800 for RLIMIT_AS against limit 16777216 for /usr/libexec/dovecot/imap-login[imap-login:6486] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/dovecot[dovecot:6409] uid/euid:0/0 gid/egid:0/0

The symptom is that I cannot log on to squirrelmail. I could get in eventually, but most of the time it fails. The symptoms are present with or without activated RBAC.

There were no RLIMIT_AS grsec messages or failed shared library loads using hardened-sources-3.2.9 (grsecurity-2.9-3.2.9-201203022148) or hardened-sources-3.2.9-r1 (grsecurity-2.9-3.2.9-201203062051).

Please give me some advice.

[PaX Team]

Apr 4 21:55:55 replaced dovecot: imap: Error: dovecot/imap: error while loading shared libraries: libpthread.so.0: failed to map segment from shared object: Cannot allocate memory
Apr 4 22:05:13 replaced dovecot: imap-login: Error: dovecot/imap-login: error while loading shared libraries: libcrypto.so.1.0.0: failed to map segment from shared object: Cannot allocate memory

would it be possible to strace the daemon and see what the failing mmap request looked like? (even better would be to also capture the maps file at the time of the failure)

[Dwokfur]

Apr 4 21:55:55 replaced dovecot: imap: Error: dovecot/imap: error while loading shared libraries: libpthread.so.0: failed to map segment from shared object: Cannot allocate memory
Apr 4 22:05:13 replaced dovecot: imap-login: Error: dovecot/imap-login: error while loading shared libraries: libcrypto.so.1.0.0: failed to map segment from shared object: Cannot allocate memory

would it be possible to strace the daemon and see what the failing mmap request looked like? (even better would be to also capture the maps file at the time of the failure)

Is it necessary to strace dovecot with 3.3.0-grsec, or is it enough to do it with 3.2.11-grsec?
About capturing the maps file: how would I do that?

Thx:
Dw.

[PaX Team]

Is it necessary to strace dovecot with 3.3.0-grsec, or is it enough to do it with 3.2.11-grsec?
About capturing the maps file: how would I do that?

any kernel is fine where the problem occurs but in the meantime spender, our resident genious, had an insight that would explain the situation. unfortunately the fix is a bit ugly, so i don't yet know what i'll do about it. however the lesson for userland is that if it wants to track its own resources, it had better take into account what the kernel does to those resources as well.

[spender]

This should be fixed in the latest patches. Let me know if you still experience problems.

-Brad

[Dwokfur]

This should be fixed in the latest patches. Let me know if you still experience problems.

-Brad

I was about to test PaxTeam's advice on gathering strace of dovecot, but I booted a new kernel in order to do so.
I cannot reproduce the symptom using grsecurity-2.9-3.3.1-201204062021. I guess it got fixed in the mean time.

Big thanks:
Dw.

[rainbow]

Two days ago I have installed the 3.2.39 kernel with grsecurity-2.9.1-3.2.39-201303012254.patch into a kvm guest but I run up against the same issuse wrote by Dwokfur.
The configuration is this http://bpaste.net/show/81282/
Patch regression?
thanks

[spender]

Hi,

Yes, the PaX Team recently added random mapped "gaps" at the top and bottom of the userland address space, but their sizes weren't removed when calculating RLIMIT_AS. I created a patch against 3.8 which may or may not apply to your kernel. If it does, could you test it and let me know if it resolves the issue?

http://grsecurity.net/~spender/rlimit_as.diff

-Brad

[rainbow]

the patch apply but the /init (of initrd) failed to start.
Disabling grsecurity everything works great.

[spender]

Can you give more detail about the failure?

-Brad

[rainbow]

In the last message I forgot to say: thank you very much for the patch
The error is the following:
[ 0.695321] Freeing unused kernel memory: 632k freed
[ 0.696050] Failed to execute /init
[ 0.696426] Kernel panic - not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[ 0.697200] Pid: 1, comm: init Not tainted 3.2.39-grsec+ #1
[ 0.697642] Call Trace:
[ 0.698001] [<0046e68f>] ? panic+0x6f/0x14f
[ 0.698394] [<0046cd8c>] ? init_post+0xa1/0xa1
[ 0.698775] [<00805351>] ? 0x805350
[ 0.699135] [<008050f3>] ? 0x8050f2
[ 0.699502] [<00477306>] ? kernel_thread_helper+0x6/0xd

[zImage]

It looks like I'm hitting the same thing. I can reproduce it with either dovecot or php.
With 2.6.32.49-grsec & 3.2.40 without grsec php works with quite low RLIMIT_AS (ulimit -v):

Code: Select all

# uname -a ; for i in $(seq 40000 10000 200000); do  ulimit -v $i ; echo $i; php -i |grep -m1 "PHP Ver" ;done
Linux spare 2.6.32.49-grsec #1 SMP Wed Nov 30 07:38:53 EST 2011 x86_64 GNU/Linux
40000
/usr/local/php53/bin/php: error while loading shared libraries: libxml2.so.2: failed to map segment from shared object: Cannot allocate memory
50000
PHP Version => 5.3.17


And here's with 3.2.40 patched with grsecurity-2.9.1-3.2.40-201303142234.patch:

Code: Select all

# uname -a ; for i in $(seq 60000 10000 200000); do  ulimit -v $i ; echo $i; php -i |grep -m1 "PHP Ver" ;done
Linux spare 3.2.40-grsec #1 SMP Thu Mar 14 08:18:13 EDT 2013 x86_64 GNU/Linux
60000
Killed
...
190000
Killed
200000
PHP Version => 5.3.17


Here's 3.2 patched with grsec, but disabled in menuconfig:

Code: Select all

# uname -a ; for i in $(seq 40000 10000 200000); do  ulimit -v $i ; echo $i; php -i |grep -m1 "PHP Ver" ;done
Linux spare 3.2.40-grsec #5 SMP Fri Mar 15 17:22:17 EDT 2013 x86_64 GNU/Linux
40000
/usr/bin/php: error while loading shared libraries: libxml2.so.2: failed to map segment from shared object: Cannot allocate memory
50000
PHP Version => 5.3.17


I tried adding http://grsecurity.net/~spender/rlimit_as.diff, but boot failed similarly to the one shown by rainbow in previous post:

Code: Select all

[ 4.296421] Kernel panic - not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[ 4.297204] Pid: 1, comm: init Not tainted 3.2.40-grsec #4
[ 4.297642] Call Trace:
[ 4.298007] [<ffffffff817f72ae>] panic+0xa4/0x1ad
[ 4.298394] [<ffffffff81000237>] init_post+0x87/0xc0
[ 4.298773] [<ffffffff81c49743>] 0xffffffff81c49742
[ 4.299131] [<ffffffff81037082>] ? schedule_tail+0x22/0xa0
[ 4.299502] [<ffffffff817fcb84>] kernel_thread_helper+0x4/0x10
[ 4.299802] [<ffffffff81c4961d>] ? ffffffff81c4961c
[ 4.300501] [<ffffffff817fcb80>] ? gs_change+0xb/0xb


Similar to https://bugs.gentoo.org/show_bug.cgi?id=459268 isn't it?

Let me know if I could provide any helpful info...

[spender]

Are you able to try 3.8? It has a different fix for this implemented.

-Brad

[crusader]

Hello spender,

I'm a colleague of zImage.

We experience the same problem with kernel 3.8.3 and grsecurity-2.9.1-3.8.3-201303142235.patch.

Code: Select all

#  uname -a ; for i in $(seq 40000 10000 200000); do  ulimit -v $i ; echo $i; php -i |grep -m1 "PHP Ver" ;done
3.8.3-grsec #1 SMP Mon Mar 18 13:57:04 EDT 2013 x86_64 GNU/Linux
40000
Killed
50000
Killed
60000
Killed
70000
80000
90000
Killed
100000
PHP Version => 5.3.17
110000
/usr/bin/php: line 6:  8593 Killed                  /usr/local/php53/bin/php -c "$INIDIR" "$@"
120000
Killed
130000
Killed
140000
PHP Version => 5.3.17
150000
/usr/bin/php: line 6:  8614 Killed                  /usr/local/php53/bin/php -c "$INIDIR" "$@"
Killed
160000
/usr/bin/php: line 6:  8622 Killed                  /usr/local/php53/bin/php -c "$INIDIR" "$@"
170000
/usr/bin/php: line 6:  8630 Killed                  /usr/local/php53/bin/php -c "$INIDIR" "$@"
180000
PHP Version => 5.3.17
190000
PHP Version => 5.3.17
200000
PHP Version => 5.3.17


[spender]

Can you try the 3.8.3 patch released last night? The PaX team is unable to reproduce the bug with that patch using the same commands you gave on a variety of userland/kernel 32/64bit combinations.

-Brad