native_machine_emergency_restart panic since 2.3.37

[gaima]

Hi,

I'm using Gentoo hardened-sources in my Xen based VMs (Citrix XenServer, and XenSource with hardened-sources)
Hardened 3.2.36 works perfectly. Vanilla 3.2.37 (and 3.2.36) also works perfectly.
3.2.37 with grsecurity-2.9.1-3.2.37-201301311810 and 3.2.38 don't work so well, and I can't figure out why (although I'm no developer, and can barely read C let alone comprehend it).

In operation everything seems perfectly normal.
Reboots and halts not so much.

Halts will not "power off", in that the VM never actually stops. If connected to the console during a halt "[253231.583212] System halted." is printed but no actual halt occurs.

Rebooting is even more interesting. The following happens.

Code: Select all

�182106.764440� Restarting system.
�182109.945327� int3: 0000 �#1� SMP
�182109.945335� CPU 0
�182109.945338� Modules linked in:
�182109.945342�
�182109.945346� Pid: 10849, comm: reboot Not tainted 3.2.37-hardened-r2 #2 
�182109.945352� RIP: e030:�<ffffffff8101f39e>�  �<ffffffff8101f39e>� native_machine_emergency_restart+0x176/0x1cc
�182109.945367� RSP: e02b:ffff880079ad7cd8  EFLAGS: 00000246
�182109.945371� RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000056bce4f6
�182109.945377� RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81803d3c
�182109.945382� RBP: 0000000000000cf9 R08: 0000000000000040 R09: 0720072007200720
�182109.945388� R10: 0720072007200720 R11: 0720072007200720 R12: ffffffff814bdaf0
�182109.945393� R13: 0000000000000ffe R14: 0000000000000061 R15: 0000000000000000
�182109.945401� FS:  000077978e80e700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
�182109.945407� CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
�182109.945412� CR2: 000077978e335030 CR3: 0000000079b2e000 CR4: 0000000000002660
�182109.945417� DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
�182109.945423� DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
�182109.945428� Process reboot (pid: 10849, threadinfo ffff88007a8924a0, task ffff88007a892080)
�182109.945433� Stack:
�182109.945436�  ffff88007a892080 01ffffff00000001 0000000028121969 0000000000000000
�182109.945444�  0000000000000000 0000000028121969 0000000001234567 0000000000000000
�182109.945452�  0000000028121969 0000000001234567 0000000000000fcc 0000053e8c004298
�182109.945460� Call Trace:
�182109.945465�  �<ffffffff8101f420>� ? native_machine_restart+0x2c/0x2c
�182109.945471�  �<ffffffff8101f426>� ? machine_restart+0x6/0x6
�182109.945478�  �<ffffffff8104b13d>� ? kernel_restart+0x46/0x46
�182109.945483�  �<ffffffff8104b273>� ? sys_reboot+0x12d/0x17f
�182109.945491�  �<ffffffff81237bad>� ? gr_acl_is_capable+0x29/0x1f6
�182109.945498�  �<ffffffff81005730>� ? pte_mfn_to_pfn+0x54/0xc4
�182109.945503�  �<ffffffff81004069>� ? __raw_callee_save_xen_pmd_val+0x11/0x1e
�182109.945510�  �<ffffffff8123012d>� ? gr_is_capable+0x15/0x20
�182109.945518�  �<ffffffff81043e48>� ? ns_capable+0x4c/0x62
�182109.945524�  �<ffffffff8106be6f>� ? kernel_kexec+0x56/0x5b
�182109.945529�  �<ffffffff8104b28d>� ? sys_reboot+0x147/0x17f
�182109.945535�  �<ffffffff810294ff>� ? do_page_fault+0x312/0x37a
�182109.945540�  �<ffffffff81003e83>� ? xen_mc_flush+0xe3/0x108
�182109.945548�  �<ffffffff8143d8cd>� ? system_call_fastpath+0x18/0x1d
�182109.945556�  �<ffffffff8101177c>� ? pax_randomize_kstack+0x4c/0x60
�182109.945562�  �<ffffffff8143d8f5>� ? sysret_check+0x1e/0x5a
�182109.945566� Code: c7 05 13 9a 6c 00 61 00 00 00 b3 01 e9 5b ff ff ff c7 05 02 9a 6c 00 74 00 00 00 e9 4c ff ff ff 4c 89 e7 e8 b7 47 fe ff 66 90 cc <eb> 45 e8 eb a7 25 00 eb 3e c6 05 e2 c0 7e 00 01 80 3d db c0 7e
�182109.945615� RIP  �<ffffffff8101f39e>� native_machine_emergency_restart+0x176/0x1cc
�182109.945622�  RSP <ffff880079ad7cd8>
�182109.945631� ---� end trace 125bc25cd52863f9 �---
�182109.945637� Kernel panic - not syncing: Fatal exception


You'll to excuse the unusual characters in place of square brackets, I don't know why that happens.

What's most interesting to me is "native_machine_emergency_restart+0x176/0x1cc".
0x176/0x1cc is the same everytime, even if I recompile the kernel with a different version of GCC, and subtly different options.

The version of GCC I've tried are

Code: Select all

gcc version 4.5.4 (Gentoo Hardened 4.5.4 p1.1, pie-0.4.7)
gcc version 4.6.3 (Gentoo Hardened 4.6.3 p1.11, pie-0.5.2)


And my .config has the following in

Code: Select all

CONFIG_PAX_USERCOPY_SLABS=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CONFIG_AUTO=y
# CONFIG_GRKERNSEC_CONFIG_CUSTOM is not set
CONFIG_GRKERNSEC_CONFIG_SERVER=y
# CONFIG_GRKERNSEC_CONFIG_DESKTOP is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_NONE is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_GUEST is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_HOST=y
CONFIG_GRKERNSEC_CONFIG_VIRT_EPT=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_SOFT is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_XEN=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_VMWARE is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_KVM is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_VIRTUALBOX is not set
CONFIG_GRKERNSEC_CONFIG_PRIORITY_PERF=y
# CONFIG_GRKERNSEC_CONFIG_PRIORITY_SECURITY is not set
CONFIG_GRKERNSEC_PROC_GID=1001
CONFIG_GRKERNSEC_TPE_UNTRUSTED_GID=100
CONFIG_GRKERNSEC_SYMLINKOWN_GID=100
CONFIG_PAX=y
# CONFIG_PAX_SOFTMODE is not set
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_XATTR_PAX_FLAGS=y
# CONFIG_PAX_NO_ACL_FLAGS is not set
CONFIG_PAX_HAVE_ACL_FLAGS=y
# CONFIG_PAX_HOOK_ACL_FLAGS is not set
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
# CONFIG_PAX_EMUTRAMP is not set
CONFIG_PAX_MPROTECT=y
# CONFIG_PAX_MPROTECT_COMPAT is not set
# CONFIG_PAX_ELFRELOCS is not set
CONFIG_PAX_KERNEXEC_PLUGIN_METHOD=""
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDKSTACK=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y
# CONFIG_PAX_MEMORY_SANITIZE is not set
# CONFIG_PAX_MEMORY_STACKLEAK is not set
CONFIG_PAX_REFCOUNT=y
CONFIG_PAX_USERCOPY=y
CONFIG_PAX_SIZE_OVERFLOW=y
CONFIG_PAX_LATENT_ENTROPY=y
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_RAND_THREADSTACK=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODHARDEN=y
CONFIG_GRKERNSEC_HIDESYM=y
CONFIG_GRKERNSEC_KERN_LOCKOUT=y
# CONFIG_GRKERNSEC_NO_RBAC is not set
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
# CONFIG_GRKERNSEC_PROC_USER is not set
CONFIG_GRKERNSEC_PROC_USERGROUP=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_SYMLINKOWN=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_SYSFS_RESTRICT=y
# CONFIG_GRKERNSEC_ROFS is not set
CONFIG_GRKERNSEC_DEVICE_SIDECHANNEL=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_MOUNT=y
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_CHMOD=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_CAPS=y
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
# CONFIG_GRKERNSEC_EXECLOG is not set
CONFIG_GRKERNSEC_RESLOG=y
# CONFIG_GRKERNSEC_CHROOT_EXECLOG is not set
# CONFIG_GRKERNSEC_AUDIT_PTRACE is not set
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
# CONFIG_GRKERNSEC_AUDIT_MOUNT is not set
CONFIG_GRKERNSEC_SIGNAL=y
# CONFIG_GRKERNSEC_FORKFAIL is not set
CONFIG_GRKERNSEC_TIME=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_RWXMAP_LOG=y
# CONFIG_GRKERNSEC_AUDIT_TEXTREL is not set
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_HARDEN_PTRACE=y
CONFIG_GRKERNSEC_PTRACE_READEXEC=y
CONFIG_GRKERNSEC_SETXID=y
CONFIG_GRKERNSEC_TPE=y
CONFIG_GRKERNSEC_TPE_ALL=y
# CONFIG_GRKERNSEC_TPE_INVERT is not set
CONFIG_GRKERNSEC_TPE_GID=100
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_BLACKHOLE=y
CONFIG_GRKERNSEC_NO_SIMULT_CONNECT=y
# CONFIG_GRKERNSEC_SOCKET is not set
CONFIG_GRKERNSEC_SYSCTL=y
# CONFIG_GRKERNSEC_SYSCTL_DISTRO is not set
CONFIG_GRKERNSEC_SYSCTL_ON=y
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=6


I have tried changing CONFIG_VIRT_GUEST for CONFIG_VIRT_HOST.
There was no change in behaviour or output.


At this point I'm stumped.
Any help would be greatly appreciated.

Thanks

[PaX Team]

can you post the PARAVIRT/XEN related options from your .config?

[gaima]

can you post the PARAVIRT/XEN related options from your .config?

Sure, here it is.

Code: Select all

CONFIG_PARAVIRT_GUEST=y
# CONFIG_PARAVIRT_TIME_ACCOUNTING is not set
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_VIRT_TO_BUS=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
# CONFIG_NETXEN_NIC is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# Virtio drivers
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
# CONFIG_VIRTIO_MMIO is not set
# Xen driver support
CONFIG_XEN_BALLOON=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PCIDEV_BACKEND=m
CONFIG_VIRT_DRIVERS=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_NONE is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_GUEST is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_HOST=y
CONFIG_GRKERNSEC_CONFIG_VIRT_EPT=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_SOFT is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_XEN=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_VMWARE is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_KVM is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_VIRTUALBOX is not set
CONFIG_VIRTUALIZATION=y


That's every entry with "xen" or "virt" in.

3.2.36/7/8 all have exactly the same XEN/VIRT config.


Also, I hadn't noticed before, but there are tiny differences between the GRSec/PAX configs between 36 and 37/38.

Code: Select all

# diff -u 36 37-r2
--- 36  2013-02-13 15:44:15.000000000 +0000
+++ 37-r2       2013-02-13 15:39:31.000000000 +0000
@@ -15,7 +15,7 @@
 # CONFIG_GRKERNSEC_CONFIG_VIRT_VIRTUALBOX is not set
 CONFIG_GRKERNSEC_CONFIG_PRIORITY_PERF=y
 # CONFIG_GRKERNSEC_CONFIG_PRIORITY_SECURITY is not set
-CONFIG_GRKERNSEC_PROC_GID=10
+CONFIG_GRKERNSEC_PROC_GID=1001
 CONFIG_GRKERNSEC_TPE_UNTRUSTED_GID=100
 CONFIG_GRKERNSEC_SYMLINKOWN_GID=100
 CONFIG_PAX=y
@@ -44,6 +44,7 @@
 CONFIG_PAX_LATENT_ENTROPY=y
 CONFIG_GRKERNSEC_KMEM=y
 CONFIG_GRKERNSEC_IO=y
+CONFIG_GRKERNSEC_RAND_THREADSTACK=y
 CONFIG_GRKERNSEC_PROC_MEMMAP=y
 CONFIG_GRKERNSEC_BRUTE=y
 CONFIG_GRKERNSEC_MODHARDEN=y
@@ -62,6 +63,7 @@
 CONFIG_GRKERNSEC_FIFO=y
 CONFIG_GRKERNSEC_SYSFS_RESTRICT=y
 # CONFIG_GRKERNSEC_ROFS is not set
+CONFIG_GRKERNSEC_DEVICE_SIDECHANNEL=y
 CONFIG_GRKERNSEC_CHROOT=y
 CONFIG_GRKERNSEC_CHROOT_MOUNT=y
 CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
@@ -94,11 +96,12 @@
 CONFIG_GRKERNSEC_PTRACE_READEXEC=y
 CONFIG_GRKERNSEC_SETXID=y
 CONFIG_GRKERNSEC_TPE=y
-# CONFIG_GRKERNSEC_TPE_ALL is not set
+CONFIG_GRKERNSEC_TPE_ALL=y
 # CONFIG_GRKERNSEC_TPE_INVERT is not set
 CONFIG_GRKERNSEC_TPE_GID=100
 CONFIG_GRKERNSEC_RANDNET=y
 CONFIG_GRKERNSEC_BLACKHOLE=y
+CONFIG_GRKERNSEC_NO_SIMULT_CONNECT=y
 # CONFIG_GRKERNSEC_SOCKET is not set
 CONFIG_GRKERNSEC_SYSCTL=y
 # CONFIG_GRKERNSEC_SYSCTL_DISTRO is not set


Code: Select all

# diff -u 37-r2 38-r1
--- 37-r2       2013-02-13 15:39:31.000000000 +0000
+++ 38-r1       2013-02-13 15:39:39.000000000 +0000
@@ -40,6 +40,7 @@
 # CONFIG_PAX_MEMORY_STACKLEAK is not set
 CONFIG_PAX_REFCOUNT=y
 CONFIG_PAX_USERCOPY=y
+# CONFIG_PAX_USERCOPY_DEBUG is not set
 CONFIG_PAX_SIZE_OVERFLOW=y
 CONFIG_PAX_LATENT_ENTROPY=y
 CONFIG_GRKERNSEC_KMEM=y


I'm going to try turning off TPE in 37/38 and report back

[gaima]

I'm going to try turning off TPE in 37/38 and report back

Disabling CONFIG_GRKERNSEC_TPE_ALL has unfortunately made no change

Code: Select all

�   97.826294� Restarting system.
�  101.007058� int3: 0000 �#1� SMP
�  101.007065� CPU 0
�  101.007067� Modules linked in: nfs lockd fscache auth_rpcgss sunrpc usbcore usb_common
�  101.007080�
�  101.007084� Pid: 2327, comm: reboot Not tainted 3.2.38-hardened-r1 #3 
�  101.007090� RIP: e030:�<ffffffff8101f3a2>�  �<ffffffff8101f3a2>� native_machine_emergency_restart+0x176/0x1cc
�  101.007104� RSP: e02b:ffff880079fa3d98  EFLAGS: 00000246
�  101.007109� RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00000000f2a24928
�  101.007114� RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81805d3c
�  101.007119� RBP: 0000000000000cf9 R08: 0000000000000040 R09: 0720072007200720
�  101.007124� R10: 0720072007200720 R11: 0720072007200720 R12: ffffffff814beaf0
�  101.007129� R13: 0000000000000ffe R14: 0000000000000061 R15: 0000000000000000
�  101.007137� FS:  000071133473b700(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
�  101.007143� CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
�  101.007147� CR2: 0000711334262030 CR3: 00000000799e5000 CR4: 0000000000002660
�  101.007152� DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
�  101.007158� DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
�  101.007163� Process reboot (pid: 2327, threadinfo ffff88007afc14a0, task ffff88007afc1080)
�  101.007169� Stack:
�  101.007172�  ffff88007afc1080 01ffffff00000001 0000000028121969 0000000000000000
�  101.007179�  0000000000000000 0000000028121969 0000000001234567 0000000000000000
�  101.007188�  0000000028121969 0000000001234567 0000000000000fcc 000005e71d419298
�  101.007196� Call Trace:
�  101.007201�  �<ffffffff8101f424>� ? native_machine_restart+0x2c/0x2c
�  101.007207�  �<ffffffff8101f42a>� ? machine_restart+0x6/0x6
�  101.007214�  �<ffffffff8104b149>� ? kernel_restart+0x46/0x46
�  101.007220�  �<ffffffff8104b27f>� ? sys_reboot+0x12d/0x17f
�  101.007225�  �<ffffffff810032d1>� ? xen_load_sp0+0x77/0x7e
�  101.007234�  �<ffffffff8143d5b5>� ? page_fault+0x25/0x30
�  101.007241�  �<ffffffff8101177c>� ? pax_randomize_kstack+0x4c/0x60
�  101.007247�  �<ffffffff8143d328>� ? retint_swapgs+0xc/0xd
�  101.007253�  �<ffffffff81003e83>� ? xen_mc_flush+0xe3/0x108
�  101.007258�  �<ffffffff810032d1>� ? xen_load_sp0+0x77/0x7e
�  101.007264�  �<ffffffff8143da4d>� ? system_call_fastpath+0x18/0x1d
�  101.007269� Code: c7 05 0f ba 6c 00 61 00 00 00 b3 01 e9 5b ff ff ff c7 05 fe b9 6c 00 74 00 00 00 e9 4c ff ff ff 4c 89 e7 e8 b3 47 fe ff 66 90 cc <eb> 45 e8 9b a8 25 00 eb 3e c6 05 de e0 7e 00 01 80 3d d7 e0 7e
�  101.007321� RIP  �<ffffffff8101f3a2>� native_machine_emergency_restart+0x176/0x1cc
�  101.007329�  RSP <ffff880079fa3d98>
�  101.007335� ---� end trace 4bfd42b262369a6f �---
�  101.007340� Kernel panic - not syncing: Fatal exception


[PaX Team]

can you send me arch/x86/xen/enlighten.o please?

[PaX Team]

i think i figured it out, can you send me arch/x86/kernel/reboot.o (of a non-working kernel)?

[gaima]

I saw the change released last night (964588cb0830ac56f245b09f5a926cc63c1bdacf), and recompiled with it this morning.
No more shutdown hangs, or reboot panics!

Thanks very much guys.