Kernel panic with 2.6.24 under Xen, take two

[sfaerber]

As of the patches uploaded last night, 32bit xen domU should be compatible with grsecurity/PaX. Please let us know if you continue to have any issues.

-Brad

Great News!
I just tried grsecurity-2.1.14-2.6.32.12-201005012055.patch with a 32bit Xen domU but it still doesn't boot. I complains about Warnings in
arch/x86/xen/multicalls.c:182
I've uploaded the complete console output to
http://pastebin.com/6bUKYfG2
and my .config to http://pastebin.com/iJcXdjw6

Note: I compiled this kernel on a 64bit Debian Lenny System, setting ARCH=i386.
I'm running the Xen Hypervisor that comes with RHEL 5.5

Would be great if you could take a look, i can supply additional information if necessary.
Thanks!
- Sebastian

[spender]

Could you disable HIDESYM, enable KALLSYMS, and boot with loglevel=8 on the kernel commandline? The resulting output will help us debug the problem.

Thanks,
-Brad

[sfaerber]

Here's the console output:

http://pastebin.com/0K2XS0yR

I also noticed these messages from the Hypervisor (via xm dmesg):

Code: Select all

(XEN) traps.c:1878:d5 Domain attempted WRMSR 00000000c0000080 from 00000000:00000d00 to 00000000:00000500.
(XEN) mm.c:1530:d5 Bad L2 flags 80
(XEN) mm.c:1530:d5 Bad L2 flags 80
(XEN) mm.c:1530:d5 Bad L2 flags 80
(XEN) mm.c:1530:d5 Bad L2 flags 80
(XEN) mm.c:1530:d5 Bad L2 flags 80


Thanks!

-Sebastian

[PaX Team]

1. can you look up c15d88e0 in System.map? (may not be exact address, my guess is that it's a page table so a more likely address you'll find is c15d8000 or so).
2. what is your xen and xen dom0 kernel version?

[cormander]

In my build system, the "latest" (2.6.33.3) xenU 32bit build worked, but there was a few stack traces in the console output. See below:

http://build.cormander.com/job/linux-2. ... 13/console

The latest "stable" (2.6.32.12) kernel doesn't boot, it loops a stack trace and never finishes. You can see its console here:

http://build.cormander.com/job/linux-2. ... onsoleText

If you need the vmlinux and system.map files, they are available in the workspace of the projects.

[sfaerber]

1. can you look up c15d88e0 in System.map? (may not be exact address, my guess is that it's a page table so a more likely address you'll find is c15d8000 or so).
2. what is your xen and xen dom0 kernel version?

Code: Select all

# grep -10 c15d8000 System.map
c15d70f0 d xfrm_state_hashmax
c15d70f4 d secpath_cachep
c15d70f8 d xs_tcp_fin_timeout
c15d70fc d rpc_task_slabp
c15d7100 d rpc_buffer_slabp
c15d7104 d rpc_task_mempool
c15d7108 d rpc_buffer_mempool
c15d710c d rpc_mount
c15d7110 d rpc_inode_cachep
c15d7114 D _edata
c15d8000 T __init_begin
c15d8000 D __per_cpu_load
c15d8000 A per_cpu_load
c15de000 T _sinittext
c15de000 A init_begin
c15de000 T startup_xen
c15de042 T i386_start_kernel
c15de0a9 T reserve_ebda_region
c15de120 t nosmp
c15de136 t set_reset_devices
c15de14a t debug_kernel
c15de15b t quiet_kernel
c15de16c t init_setup


My dom0 is running 2.6.18-194.el5.xen (RHEL 5.5) and the Xen Hypervisor is at Version 3.1.2-194.el5 (RHEL 5.5 too).

-Sebastian

[PaX Team]

In my build system, the "latest" (2.6.33.3) xenU 32bit build worked, but there was a few stack traces in the console output. See below:

http://build.cormander.com/job/linux-2. ... 13/console

The latest "stable" (2.6.32.12) kernel doesn't boot, it loops a stack trace and never finishes. You can see its console here:

http://build.cormander.com/job/linux-2. ... onsoleText

If you need the vmlinux and system.map files, they are available in the workspace of the projects.

can you enable KALLSYMS for future builds please? also set loglevel=8 on the kernel command line so we see more messages. what is your xen and dom0 kernel version? and if it's not 3.4.2/2.6.32 based, can you try that setup too? it's what i have here and none of these multicall errors show up for me...

[PaX Team]

Code: Select all

c15d8000 D __per_cpu_load
c15d8000 A per_cpu_load
c15de000 T _sinittext

so it's a per-cpu variable, weird. can you grep 8e0 or whatever is around there from System.map (will be somewhere at the beginning of that file)?

My dom0 is running 2.6.18-194.el5.xen (RHEL 5.5) and the Xen Hypervisor is at Version 3.1.2-194.el5 (RHEL 5.5 too).

i don't know how much work it'd be, but could you try a newer xen and dom0 kernel similar to what i have (see my previous post)? it'd help me determine if it's some older xen bug or something still in PaX.

[sfaerber]

Code: Select all

c15d8000 D __per_cpu_load
c15d8000 A per_cpu_load
c15de000 T _sinittext

so it's a per-cpu variable, weird. can you grep 8e0 or whatever is around there from System.map (will be somewhere at the beginning of that file)?

My dom0 is running 2.6.18-194.el5.xen (RHEL 5.5) and the Xen Hypervisor is at Version 3.1.2-194.el5 (RHEL 5.5 too).

i don't know how much work it'd be, but could you try a newer xen and dom0 kernel similar to what i have (see my previous post)? it'd help me determine if it's some older xen bug or something still in PaX.

You can get my System.map here: http://sd1.abcde.biz/grsec/System.map
The first occurance of 8e0 is "i8237A_suspend":

Code: Select all

c100f460 t ati_force_enable_hpet
c100f660 t nvidia_force_enable_hpet
c100f720 T force_hpet_resume
c100f8e0 t i8237A_suspend
c100f8f0 t i8237A_resume
c100f9d0 T arch_unregister_cpu
c100f9f0 t text_poke_early
c100fa50 T text_poke
c100fb20 t add_nops
c100fbb0 T apply_paravirt


But i'm not sure that's the symbol you're looking for?
I can't test a newer Version of Xen right now, i'll look into that tomorrow.
But 2.6.32.11 32bit vanilla (no grsec, no pax) works fine here.

-Sebastian

[PaX Team]

I can't test a newer Version of Xen right now, i'll look into that tomorrow.
But 2.6.32.11 32bit vanilla (no grsec, no pax) works fine here.

in the meantime i could debug this myself and it seems that CONFIG_NUMA isn't exactly supported by xen/domU, i'm surprised that a vanilla kernel works with it at all. in any case, turn it off for PaX and it should all work fine.

[Shadow]

For what it's worth building a dom0 off the latest xen-unstable and using the 2.6.32.21 trunk kernel with grsec added, works perfectly fine. Have several boxes in production running with it without issue. All 64bit, no pax.