Page 1 of 1

'halt due to suspicious kernel crash' during poweroff

PostPosted: Sat Nov 29, 2014 7:33 pm
by curiousity
Related to VirtualBox modules? Kernel version is 3.17.4 gentoo hardened, with linux ZFS (current git) and virtualbox guest additions 4.3.20 added.

Image

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 7:51 am
by PaX Team
this looks like a use after free bug caught by SANITIZE (do you have it enabled?), probably in vboxguest, but i can tell more if you reproduce this with frame pointers enabled.

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 8:13 am
by curiousity
PaX Team wrote:this looks like a use after free bug caught by SANITIZE (do you have it enabled?), probably in vboxguest, but i can tell more if you reproduce this with frame pointers enabled.


Erp, sorry about the lack of frame pointers.

Seeing it took a good few hours before you guys timezone rolled around I disabled some options as a workaround ... I'll revert to a previous snapshot, alter the frame pointer status, give it a go at reproducing and post the output now.

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 9:37 am
by curiousity
I did have PAX_MEMORY_SANITIZE enabled but I think the culprit is actually CONFIG_PAX_KERNEXEC. I am currently verifying this by recompiling without it... will post an update in a minute.

Here's the panic image with frame pointers .. almost the same as I see it, with the addition of a VBoxGuest_RTHeapSimpleSize entry in the call trace.

Image

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 9:51 am
by curiousity
curiousity wrote:I did have PAX_MEMORY_SANITIZE enabled but I think the culprit is actually CONFIG_PAX_KERNEXEC. I am currently verifying this by recompiling without it... will post an update in a minute.


No, that wasn't it. I will try disabling PAX_MEMORY_SANITIZE and re-enabling CONFIG_PAX_KERNEXEC, recompile all the modules again and see what happens.

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 9:54 am
by curiousity
curiousity wrote:
curiousity wrote:I did have PAX_MEMORY_SANITIZE enabled but I think the culprit is actually CONFIG_PAX_KERNEXEC. I am currently verifying this by recompiling without it... will post an update in a minute.


No, that wasn't it. I will try disabling PAX_MEMORY_SANITIZE and re-enabling CONFIG_PAX_KERNEXEC, recompile all the modules again and see what happens.


Actually I will leave PAX_MEMORY_SANITIZE on, but try disabling PAX_MEMORY_STACKLEAK. Also still re-enabling CONFIG_PAX_KERNEXEC.

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 10:16 am
by curiousity
curiousity wrote:Actually I will leave PAX_MEMORY_SANITIZE on, but try disabling PAX_MEMORY_STACKLEAK. Also still re-enabling CONFIG_PAX_KERNEXEC.


Still fails. Time to try PAX_MEMORY_SANITIZE off, PAX_MEMORY_STACKLEAK on, CONFIG_PAX_KERNEXEC on...

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 7:23 pm
by curiousity
curiousity wrote:Still fails. Time to try PAX_MEMORY_SANITIZE off, PAX_MEMORY_STACKLEAK on, CONFIG_PAX_KERNEXEC on...


That fixed it again.

So yes, it's dying repeatably on multiple kernel builds if PAX_MEMORY_SANITIZE is enabled, while unloading the vboxguest module.

Is this something I can whitelist around and/or whinge about to the virtualbox people upstream? I'd prefer not to switch the whole option off just to get clean shutdowns happening. :D

Re: 'halt due to suspicious kernel crash' during poweroff

PostPosted: Sun Nov 30, 2014 7:29 pm
by PaX Team
i think the problem is in vboxguest/VBoxGuest-linux.c:vboxguestLinuxTermInputDevice, the call to input_free_device is probably not necessary but you should ask kernel and/or vbox developers to be sure.