grsecurity forums

Posted: **Fri Jan 18, 2013 8:56 pm**

Hi, I'm having kernel panics on grsec-2.9.1-3.2.36-201301032034 (and few previous 3.2 versions). It seems to always (6 out of 6 times) be triggered by vsftpd FTP daemon. I've managed to grab output from serial console:

Code: Select all: grsec: From 93.141.53.168: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:27314] uid/eu id:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0 grsec: From 195.190.136.139: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:29783] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0 grsec: From 195.190.136.139: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:31607] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0 BUG: unable to handle kernel NULL pointer dereference at 0000000000000030 IP: [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b PGD 5c1545000 Thread overran stack, or stack corrupted Oops: 0000 [#1] SMP CPU 1 Modules linked in: ip6t_REJECT ip6t_LOG ip6table_filter ip6_tables ipt_LOG ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables fuse xfs exportfs ipv6 nls_iso8859_1 nls_cp437 ext2 ide_generic ide_core isofs vfat fat r8169 mii softdog Pid: 19815, comm: vsftpd Not tainted 3.2.36-grsec-2.9.1-3.2.36-201301032034 #6 MSI MS-7522/MSI X58 Pro-E (MS-7522) RIP: 0010:[<ffffffff810deef4>] [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b RSP: 0018:ffff8805bf4f3ca8 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff8806159d8e10 RCX: 0000000000000000 RDX: ffff8806159d8e10 RSI: ffff880617830b18 RDI: ffff8805c2e6d398 RBP: ffff8805bf4f3cc8 R08: 0000000000000000 R09: 0000000000004842 R10: 0000000000004842 R11: 0000000000000200 R12: ffff880617830b18 R13: ffff8806159d8e50 R14: ffff8806159d8e10 R15: ffff8805c2e6d398 FS: 0000000000000000(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000030 CR3: 000000000143b000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process vsftpd (pid: 19815, threadinfo ffff8805f9aba1f8, task ffff8805f9ab9dc0) Stack: 0000000000000000 ffff8805c2e6d398 ffff880617830b00 ffff8805c2e6d398 ffff8805bf4f3cf8 ffffffff810df0e1 ffff8805c2e6d0b8 ffff8805bf4f3d60 0000000000000000 0000000000000000 ffff8805bf4f3d48 ffffffff810d8d49 Call Trace: [<ffffffff810df0e1>] unlink_file_vma+0x3d/0x54 [<ffffffff810d8d49>] free_pgtables+0x3e/0xce [<ffffffff810e0206>] exit_mmap+0xca/0x114 [<ffffffff81038a8d>] mmput+0x61/0x108 [<ffffffff8103e754>] exit_mm+0x175/0x188 [<ffffffff8103e9c5>] do_exit+0x25e/0x756 [<ffffffff8103f197>] do_group_exit+0x67/0x9a [<ffffffff8103f1dd>] sys_exit_group+0x13/0x13 [<ffffffff81428d10>] system_call_fastpath+0x18/0x1d Code: 5d 48 0f ba 2c 24 3f c3 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 48 83 ec 10 48 89 7d e8 48 8b 7d e8 f6 47 31 08 74 1a 49 8b 04 24 <48> 8b 40 30 f0 ff 80 24 01 00 00 71 09 f0 ff 88 24 01 00 00 cd RIP [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b RSP <ffff8805bf4f3ca8> CR2: 0000000000000030 ---[ end trace 1325c063a4ec72cb ]--- Kernel panic - not syncing: Fatal exception

The vsftpd is from debian stable - version 2.3.2-3+squeeze2

The same system works normally on grsec-2.9.1-2.6.32.60-201212271948 (for long time - days).

It also seems to work normally on vanilla 3.2.36 without grsec (although it has only been tested for an hour to minimize risk to production machine, it didn't crash, while 3.2.36-grsec-2.9.1-3.2.36-201301032034 crashed 6 times in 5-20 minutes since boot each time).
It also survives on 3.2.36-grsec-2.9.1-3.2.36-201301032034 for an hour at least if I shutdown vsftpd.

Setting up the test machine with same debian/kernel/grsec/vsftpd and logging in and out of FTP and transfering files does not seem to trigger the bug. So either the test machine is immune, or only specific condition triggers the bug and I'm not managing to get it.

what more info can I provide or try to help hunt this down? thanks!

Posted: **Fri Jan 18, 2013 9:27 pm**

did you try the latest patch?

Posted: **Tue Jan 22, 2013 5:33 pm**

I tried 3.2.37-grsec-2.9.1-3.2.37-201301181518 now (it seems latest at the moment, I just doublechecked

, unfortunately that one didn't even finish the booting sequence:

Loading kernel module ext2.
Loading kernel module ext3.
Loading kernel module nls_cp437.
Loading kernel module nls_iso8859_1.
Loading kernel module zlib_inflate.
Loading kernel module isofs.
Loading kernel module ipv6.
NET: Registered protocol family 10
Loading kernel module xfs.
BUG: unable to handle kernel paging request at ffffffffa016ca68
IP: [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
PGD 1445067 PUD 144b063 PMD 6165e0063 PTE 616077161
Thread overran stack, or stack corrupted
Oops: 0003 [#1] SMP
CPU 1
Modules linked in: xfs(+) exportfs ipv6 nls_iso8859_1 nls_cp437 ext2 ide_generic ide_core isofs vfat fat r8169 mii softdog

Pid: 1025, comm: modprobe Not tainted 3.2.37-grsec-2.9.1-3.2.37-201301181518 #7 MSI MS-7522/MSI X58 Pro-E (MS-7522)
RIP: 0010:[<ffffffff810a42f7>] [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
RSP: 0018:ffff88061669dce8 EFLAGS: 00010246
RAX: ffffffffa016ca60 RBX: ffffffffa00ec028 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000031e RDI: 0000000000000131
RBP: ffff88061669dd08 R08: ffffffffa00fe1d0 R09: ffffffffa00fe100
R10: ffff88063fffbe00 R11: ffffffffa00ec028 R12: ffffffff81627810
R13: ffffffffa00ec038 R14: ffffffffa00fdd40 R15: 00000000fffffffd
FS: 000002ee5cbdc700(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa016ca68 CR3: 000000000143a000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 1025, threadinfo ffff880617927b38, task ffff880617927700)
Stack:
ffffffffa00f7590 ffffffffa00ec000 ffffffffa00f6d18 ffffffffa00f7590
ffff88061669dd28 ffffffff810a86ae ffffffffa00fe030 ffffffffa00ec000
ffff88061669dd88 ffffffff810a93bb ffffffffa00fe100 ffffffffa00fe1d0
Call Trace:
[<ffffffff810a86ae>] trace_event_raw_init+0x12/0x24
[<ffffffff810a93bb>] __trace_add_event_call+0x51/0x38d
[<ffffffff810a99ea>] trace_module_notify+0x114/0x1cd
[<ffffffff81061d86>] notifier_call_chain+0x46/0x77
[<ffffffff81061f79>] __blocking_notifier_call_chain+0x46/0x69
[<ffffffff81061fc7>] blocking_notifier_call_chain+0x2b/0x33
[<ffffffff81074dc3>] sys_init_module+0x69/0x1dd
[<ffffffff81428510>] system_call_fastpath+0x18/0x1d
[<ffffffff81427c73>] ? retint_swapgs+0xc/0xd
Code: e9 c8 00 00 00 e8 c6 fe ff ff 48 85 c0 75 ef 48 8b 43 28 48 83 38 00 75 07 48 c7 00 02 2f 0a 81 48 8b 43 28 48 83 78 08 00 75 08 <48> c7 40 08 02 2f 0a 81 48 8b 43 28 48 83 78 10 00 75 08 48 c7
RIP [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
RSP <ffff88061669dce8>
CR2: ffffffffa016ca68
---[ end trace ee7f9c1cb2e16c09 ]---
Kernel panic - not syncing: grsec: halting the system due to suspicious kernel crash caused by root

Any fix for that?

Posted: **Tue Jan 22, 2013 9:29 pm**

it was fixed in PaX, spender should update grsec soon.

Posted: **Wed Jan 23, 2013 8:15 am**

It's fixed in grsecurity as well, as of last night.

-Brad

Posted: **Thu Jan 24, 2013 2:42 pm**

Thanks!

I've installed 3.2.37-grsec-2.9.1-3.2.37-201301230047, and it not only boots, but vsftpd has survived for more than 24 hours.
So I'd consider this solved!

only new thing I've noticed compared to 2.6.32 series is that now vsftpd asks for CAP_SYS_ADMIN for some reason (but seems to function without it)

Code: Select all: grsec: From 120.43.28.176: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:6061] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0

I wasn't seeing that in 2.6.32.60-grsec-2.9.1-2.6.32.60-201212271948

Posted: **Thu Jan 24, 2013 3:37 pm**

It's probably from it trying to use some namespace support that wasn't present in 2.6.32. You could verify with an strace.

-Brad

Posted: **Thu Jan 24, 2013 7:24 pm**

I could, if I knew what to look for

i've tried stracing the vsftpd and grepping EPERM gives me only those:

Code: Select all: clone(child_stack=0, flags=0x40000000|SIGCHLD) = -1 EPERM (Operation not permitted)

and when it fails it falls back to

Code: Select all: clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2f314a949d0) = 11255

0x40000000 seems to be CLONE_NEWNET, and vsftpd-2.3.2 does mention

Where available, use CLONE_NEWNET to isolate the untrusted processes so that
they can't do arbitrary connect() and instead have to ask the privileged
process for sockets. Moderate code disturbance - hope for no breakage :-/

What I'm not sure however, is whether giving all-powerfull-and-problematic CAP_SYS_ADMIN to vsftpd (which does seems to know how to handle capabilities to some point at least) is going to do more harm or good (security-wise). If someone at least somewhat familiar with the vsftpd code would care to comment, that would be quite interesting.

grsecurity forums

kernel panic in vsftpd on grsec-2.9.1-3.2.36-201301032034

kernel panic in vsftpd on grsec-2.9.1-3.2.36-201301032034

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203

Re: kernel panic in vsftpd on grsec-2.9.1-3.2.36-20130103203