kernel panic in vsftpd on grsec-2.9.1-3.2.36-201301032034

[mnalis]

Hi, I'm having kernel panics on grsec-2.9.1-3.2.36-201301032034 (and few previous 3.2 versions). It seems to always (6 out of 6 times) be triggered by vsftpd FTP daemon. I've managed to grab output from serial console:

Code: Select all

grsec: From 93.141.53.168: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:27314] uid/eu
id:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0
grsec: From 195.190.136.139: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:29783] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0
grsec: From 195.190.136.139: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:31607] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b
PGD 5c1545000
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP
CPU 1
Modules linked in: ip6t_REJECT ip6t_LOG ip6table_filter ip6_tables ipt_LOG ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables fuse xfs exportfs ipv6 nls_iso8859_1 nls_cp437 ext2 ide_generic ide_core isofs vfat fat r8169 mii softdog

Pid: 19815, comm: vsftpd Not tainted 3.2.36-grsec-2.9.1-3.2.36-201301032034 #6 MSI MS-7522/MSI X58 Pro-E (MS-7522)
RIP: 0010:[<ffffffff810deef4>]  [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b
RSP: 0018:ffff8805bf4f3ca8  EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff8806159d8e10 RCX: 0000000000000000
RDX: ffff8806159d8e10 RSI: ffff880617830b18 RDI: ffff8805c2e6d398
RBP: ffff8805bf4f3cc8 R08: 0000000000000000 R09: 0000000000004842
R10: 0000000000004842 R11: 0000000000000200 R12: ffff880617830b18
R13: ffff8806159d8e50 R14: ffff8806159d8e10 R15: ffff8805c2e6d398
FS:  0000000000000000(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000030 CR3: 000000000143b000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vsftpd (pid: 19815, threadinfo ffff8805f9aba1f8, task ffff8805f9ab9dc0)
Stack:
 0000000000000000 ffff8805c2e6d398 ffff880617830b00 ffff8805c2e6d398
 ffff8805bf4f3cf8 ffffffff810df0e1 ffff8805c2e6d0b8 ffff8805bf4f3d60
 0000000000000000 0000000000000000 ffff8805bf4f3d48 ffffffff810d8d49
Call Trace: 
 [<ffffffff810df0e1>] unlink_file_vma+0x3d/0x54
 [<ffffffff810d8d49>] free_pgtables+0x3e/0xce
 [<ffffffff810e0206>] exit_mmap+0xca/0x114
 [<ffffffff81038a8d>] mmput+0x61/0x108
 [<ffffffff8103e754>] exit_mm+0x175/0x188
 [<ffffffff8103e9c5>] do_exit+0x25e/0x756
 [<ffffffff8103f197>] do_group_exit+0x67/0x9a
 [<ffffffff8103f1dd>] sys_exit_group+0x13/0x13
 [<ffffffff81428d10>] system_call_fastpath+0x18/0x1d
Code: 5d 48 0f ba 2c 24 3f c3 55 48 89 e5 41 54 49 89 f4 53 48 89 d3 48 83 ec 10 48 89 7d e8 48 8b 7d e8 f6 47 31 08 74 1a 49 8b 04 24 <48> 8b 40 30 f0 ff 80 24 01 00 00 71 09 f0 ff 88 24 01 00 00 cd
RIP  [<ffffffff810deef4>] __remove_shared_vm_struct.isra.25+0x23/0x7b
 RSP <ffff8805bf4f3ca8>
CR2: 0000000000000030
---[ end trace 1325c063a4ec72cb ]---
Kernel panic - not syncing: Fatal exception


The vsftpd is from debian stable - version 2.3.2-3+squeeze2

The same system works normally on grsec-2.9.1-2.6.32.60-201212271948 (for long time - days).

It also seems to work normally on vanilla 3.2.36 without grsec (although it has only been tested for an hour to minimize risk to production machine, it didn't crash, while 3.2.36-grsec-2.9.1-3.2.36-201301032034 crashed 6 times in 5-20 minutes since boot each time).
It also survives on 3.2.36-grsec-2.9.1-3.2.36-201301032034 for an hour at least if I shutdown vsftpd.

Setting up the test machine with same debian/kernel/grsec/vsftpd and logging in and out of FTP and transfering files does not seem to trigger the bug. So either the test machine is immune, or only specific condition triggers the bug and I'm not managing to get it.

what more info can I provide or try to help hunt this down? thanks!

[PaX Team]

did you try the latest patch?

[mnalis]

I tried 3.2.37-grsec-2.9.1-3.2.37-201301181518 now (it seems latest at the moment, I just doublechecked , unfortunately that one didn't even finish the booting sequence:

Loading kernel module ext2.
Loading kernel module ext3.
Loading kernel module nls_cp437.
Loading kernel module nls_iso8859_1.
Loading kernel module zlib_inflate.
Loading kernel module isofs.
Loading kernel module ipv6.
NET: Registered protocol family 10
Loading kernel module xfs.
BUG: unable to handle kernel paging request at ffffffffa016ca68
IP: [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
PGD 1445067 PUD 144b063 PMD 6165e0063 PTE 616077161
Thread overran stack, or stack corrupted
Oops: 0003 [#1] SMP
CPU 1
Modules linked in: xfs(+) exportfs ipv6 nls_iso8859_1 nls_cp437 ext2 ide_generic ide_core isofs vfat fat r8169 mii softdog

Pid: 1025, comm: modprobe Not tainted 3.2.37-grsec-2.9.1-3.2.37-201301181518 #7 MSI MS-7522/MSI X58 Pro-E (MS-7522)
RIP: 0010:[<ffffffff810a42f7>] [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
RSP: 0018:ffff88061669dce8 EFLAGS: 00010246
RAX: ffffffffa016ca60 RBX: ffffffffa00ec028 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000031e RDI: 0000000000000131
RBP: ffff88061669dd08 R08: ffffffffa00fe1d0 R09: ffffffffa00fe100
R10: ffff88063fffbe00 R11: ffffffffa00ec028 R12: ffffffff81627810
R13: ffffffffa00ec038 R14: ffffffffa00fdd40 R15: 00000000fffffffd
FS: 000002ee5cbdc700(0000) GS:ffff88063fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffa016ca68 CR3: 000000000143a000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process modprobe (pid: 1025, threadinfo ffff880617927b38, task ffff880617927700)
Stack:
ffffffffa00f7590 ffffffffa00ec000 ffffffffa00f6d18 ffffffffa00f7590
ffff88061669dd28 ffffffff810a86ae ffffffffa00fe030 ffffffffa00ec000
ffff88061669dd88 ffffffff810a93bb ffffffffa00fe100 ffffffffa00fe1d0
Call Trace:
[<ffffffff810a86ae>] trace_event_raw_init+0x12/0x24
[<ffffffff810a93bb>] __trace_add_event_call+0x51/0x38d
[<ffffffff810a99ea>] trace_module_notify+0x114/0x1cd
[<ffffffff81061d86>] notifier_call_chain+0x46/0x77
[<ffffffff81061f79>] __blocking_notifier_call_chain+0x46/0x69
[<ffffffff81061fc7>] blocking_notifier_call_chain+0x2b/0x33
[<ffffffff81074dc3>] sys_init_module+0x69/0x1dd
[<ffffffff81428510>] system_call_fastpath+0x18/0x1d
[<ffffffff81427c73>] ? retint_swapgs+0xc/0xd
Code: e9 c8 00 00 00 e8 c6 fe ff ff 48 85 c0 75 ef 48 8b 43 28 48 83 38 00 75 07 48 c7 00 02 2f 0a 81 48 8b 43 28 48 83 78 08 00 75 08 <48> c7 40 08 02 2f 0a 81 48 8b 43 28 48 83 78 10 00 75 08 48 c7
RIP [<ffffffff810a42f7>] register_ftrace_event+0x127/0x1e6
RSP <ffff88061669dce8>
CR2: ffffffffa016ca68
---[ end trace ee7f9c1cb2e16c09 ]---
Kernel panic - not syncing: grsec: halting the system due to suspicious kernel crash caused by root

Any fix for that?

[PaX Team]

it was fixed in PaX, spender should update grsec soon.

[spender]

It's fixed in grsecurity as well, as of last night.

-Brad

[mnalis]

Thanks!

I've installed 3.2.37-grsec-2.9.1-3.2.37-201301230047, and it not only boots, but vsftpd has survived for more than 24 hours.
So I'd consider this solved!

only new thing I've noticed compared to 2.6.32 series is that now vsftpd asks for CAP_SYS_ADMIN for some reason (but seems to function without it)

Code: Select all

grsec: From 120.43.28.176: (default:D:/usr/sbin/vsftpd) use of CAP_SYS_ADMIN denied for /usr/sbin/vsftpd[vsftpd:6061] uid/euid:0/0 gid/egid:0/0, parent /usr/sbin/vsftpd[vsftpd:1865] uid/euid:0/0 gid/egid:0/0


I wasn't seeing that in 2.6.32.60-grsec-2.9.1-2.6.32.60-201212271948

[spender]

It's probably from it trying to use some namespace support that wasn't present in 2.6.32. You could verify with an strace.

-Brad

[mnalis]

I could, if I knew what to look for

i've tried stracing the vsftpd and grepping EPERM gives me only those:

Code: Select all

clone(child_stack=0, flags=0x40000000|SIGCHLD) = -1 EPERM (Operation not permitted)


and when it fails it falls back to

Code: Select all

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2f314a949d0) = 11255

0x40000000 seems to be CLONE_NEWNET, and vsftpd-2.3.2 does mention

Where available, use CLONE_NEWNET to isolate the untrusted processes so that
they can't do arbitrary connect() and instead have to ask the privileged
process for sockets. Moderate code disturbance - hope for no breakage :-/

What I'm not sure however, is whether giving all-powerfull-and-problematic CAP_SYS_ADMIN to vsftpd (which does seems to know how to handle capabilities to some point at least) is going to do more harm or good (security-wise). If someone at least somewhat familiar with the vsftpd code would care to comment, that would be quite interesting.