Page 1 of 1

Kernel Panic on udp networking

PostPosted: Mon Aug 25, 2014 10:33 am
by coredumb
Hello,

I'm running dnsmasq as my DNS servers on EL6 with grsecurity and I'm being hit quite often by kernel panics, both on 3.2.55 and 3.14.13.
No RBAC used.
Here's the best trace i could get.
Image
Hope it's sufficient.

-C.

Re: Kernel Panic on udp networking

PostPosted: Mon Aug 25, 2014 12:23 pm
by PaX Team
can you resolve the RIP address to a symbol in vmlinux? something like addr2line -e vmlinux -fip ffffffff815b6667 should be enough.

Re: Kernel Panic on udp networking

PostPosted: Tue Aug 26, 2014 3:22 am
by coredumb
With debug enabled, i get only this:

udp_sendmsg
/root/rpmbuild/BUILD/kernel-3.14.13.el6/linux-3.14.13-100.el6.x86_64/net/ipv4/udp.c:869

Re: Kernel Panic on udp networking

PostPosted: Tue Sep 02, 2014 12:55 am
by coredumb
Any pointers? I've not yet tried vanilla, but i've seen this issue on last 3.2.55 grsec as well.

Re: Kernel Panic on udp networking

PostPosted: Wed Sep 17, 2014 5:13 am
by coredumb
I've finally been able to capture the full panic
Image

Hope this helps

Re: Kernel Panic on udp networking

PostPosted: Wed Sep 17, 2014 8:38 am
by spender
Hi,

This is a classic use-after-free condition caught by PAX_MEMORY_SANITIZE. Notice that R12 is set to a poisoned fefefe value, causing a GPF on dereference. The faulting instruction is:
cmp r13, [r12-28h]
I'm quite sure we don't cause this problem -- if you were to disable PAX_MEMORY_SANITIZE, the "problem" would disappear. It's an upstream bug, possibly related to the following:
http://sourceforge.net/p/e1000/mailman/ ... /31312382/

I don't see that the issues raised in those posts have been resolved.

If you can send your vmlinux file we can investigate it further.

Thanks,
-Brad

Re: Kernel Panic on udp networking

PostPosted: Wed Sep 17, 2014 9:18 am
by coredumb
Hi Brad,

Thanks for your reply, here's the vmlinux of this version recompiled with enough debug flags. Last time on IRC pipacs seemed to say that the vmlinux wasn't so correct but could find his way in the disasm code.

Hope this helps,
Olivier

Re: Kernel Panic on udp networking

PostPosted: Wed Oct 08, 2014 1:57 am
by coredumb
Image
Seems i have the exact same issue with vmxnet3 driver

Re: Kernel Panic on udp networking

PostPosted: Sun Dec 28, 2014 7:16 pm
by PaX Team
do you still have this problem? if i'm not mistaken you had SANITIZE enabled where we fixed a slab/rcu problem a while ago and i think this is what you experienced as well.

Re: Kernel Panic on udp networking

PostPosted: Thu Feb 19, 2015 9:01 am
by coredumb
Oh my sorry i actually did patch my kernel for this specific case and forgot about this thread :(
I've a fresh 3.14.33 build i can stress test right now, i'll let you know.

By patch i mean:
Code: Select all
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index cdc3258..984427d 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -2220,7 +2220,7 @@ struct proto udp_prot = {
.sysctl_wmem = &sysctl_udp_wmem_min,
.sysctl_rmem = &sysctl_udp_rmem_min,
.obj_size = sizeof(struct udp_sock),
- .slab_flags = SLAB_DESTROY_BY_RCU,
+ .slab_flags = SLAB_DESTROY_BY_RCU | SLAB_NO_SANITIZE,
.h.udp_table = &udp_table,
#ifdef CONFIG_COMPAT
.compat_setsockopt = compat_udp_setsockopt,

Re: Kernel Panic on udp networking

PostPosted: Thu Feb 19, 2015 9:06 am
by spender
You shouldn't need that patch anymore as we exempt slabs with SLAB_DESTROY_BY_RCU from sanitization.

-Brad

Re: Kernel Panic on udp networking

PostPosted: Thu Feb 19, 2015 10:07 am
by coredumb
Yes i'm validating that right now with a fresh 3.14.33 build :)

Re: Kernel Panic on udp networking

PostPosted: Mon Feb 23, 2015 2:32 am
by coredumb
All good after 4 days of stress testing.