xen + pax + 2.6.27

[bplant]

It was "Overlapping early reservations". Here is the backtrace in case you want it. Note: this is a UP kernel running with 64MB RAM.

thanks but i'd need to see the full message to know what previous region was reserved over the same range (as you can guess there should be nothing else in the kernel's code/data area, so there's definitely something wrong here).

Code: Select all

#0  make_lowmem_page_readwrite (vaddr=0xffffffff80331000) at arch/x86/xen/mmu.c:228
#1  0xffffffff803e913f in xen_smp_prepare_boot_cpu () at include/asm/desc.h:176
#2  0xffffffff803e79c7 in start_kernel () at include/asm/smp.h:76
#3  0xffffffff803e7279 in x86_64_start_reservations (real_mode_data=0xffffffff80331000 "") at arch/x86/kernel/head64.c:144
#4  0xffffffff803e8cf0 in xen_start_kernel () at arch/x86/xen/enlighten.c:1764

i think i know what xen's trying to do here, i'll try to fix it in the next test patch. if you want to experiment, you could remove the make_lowmem_page_readwrite() call in xen_smp_prepare_boot_cpu().

You were right. Removing the call to make_lowmem_page_readwrite() resulted in the kernel getting to the point where it tries to mount a filesystem (which failed because I didn't specify one!)

[PaX Team]

So you want me to set a breakpoint at reserve_early and __reserve_early and record all the parameters passed? If not, what "full message" are you referring to?

the "Overlapping early reservations" message doesn't end there, the kernel also prints out the conflicting regions and their names, at least if i can trust the printk format string . based on what you posted next, i don't see where the conflict comes from, so it'd really be useful to see the full message.

[bplant]

So you want me to set a breakpoint at reserve_early and __reserve_early and record all the parameters passed? If not, what "full message" are you referring to?

the "Overlapping early reservations" message doesn't end there, the kernel also prints out the conflicting regions and their names, at least if i can trust the printk format string . based on what you posted next, i don't see where the conflict comes from, so it'd really be useful to see the full message.

Ok, I get it. Problem is, the console hasn't started working yet so nothing is printed out. I inserted a breakpoint in panic() and printed out the local variable buf after it is filled on kernel/panic.c:80.

Code: Select all

Overlapping early reservations 200000-5fffff TEXT DATA BSS to 467000-46dfff XEN PAGETABLES\n

Is this sufficient?

I also tried it with an initrd since the kernel was also crashing in __reserve_early here too. Looks like the "TEXT DATA BSS" reservation is problematic.

Code: Select all

Overlapping early reservations 44f000-c4ffff RAMDISK to 200000-5fffff TEXT DATA BSS

[bplant]

[quote="bplant"]

Code: Select all

Overlapping early reservations 200000-5fffff TEXT DATA BSS to 467000-46dfff XEN PAGETABLES\n

For what it's worth; I removed the ". = ALIGN(2*1024*1024);" from line 234 (just above "_end = .;") of vmlinux_64.lds.S since I noticed it wasn't in the vanilla linker script. The VM then booted with less than 880MB of memory and initrd worked too.

Are there implications of removing the "ALIGN(2*1024=1024)"? Performance? Making PaX not work properly?

Thought I would take the opportunity to run "paxtest blackhat".

Code: Select all

Executable anonymous mapping             : Vulnerable
Executable bss                           : Vulnerable
Executable data                          : Vulnerable
Executable heap                          : Vulnerable
Executable stack                         : Vulnerable
Executable anonymous mapping (mprotect)  : Vulnerable
Executable bss (mprotect)                : Vulnerable
Executable data (mprotect)               : Vulnerable
Executable heap (mprotect)               : Vulnerable
Executable stack (mprotect)              : Vulnerable
Executable shared library bss (mprotect) : Vulnerable
Executable shared library data (mprotect): Vulnerable
Writable text segments                   : Vulnerable
Anonymous mapping randomisation test     : 33 bits (guessed)
Heap randomisation test (ET_EXEC)        : 40 bits (guessed)
Heap randomisation test (ET_DYN)         : 40 bits (guessed)
Main executable randomisation (ET_EXEC)  : 33 bits (guessed)
Main executable randomisation (ET_DYN)   : 33 bits (guessed)
Shared library randomisation test        : 33 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 40 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 40 bits (guessed)
Return to function (strcpy)              : Killed
Return to function (memcpy)              : Killed
Return to function (strcpy, RANDEXEC)    : Killed
Return to function (memcpy, RANDEXEC)    : Killed
Executable shared library bss            : Vulnerable
Executable shared library data           : Killed

It looks like the randomisation stuff is working, but I'm wondering of the NX bit isn't getting set correctly or at all? All the "executable anonymous mappings" are killed in the host system, so the NX bit is enabled.

[PaX Team]

Are there implications of removing the "ALIGN(2*1024=1024)"? Performance? Making PaX not work properly?

ok, that's indeed a bug, the alignment should be inside a section (i wonder how on earth relocatable kernel support works on amd64 at all, it's full of absolute symbols...), i'll fix it. this alignment is useful in that the last large page mapping the kernel data/bss doesn't get split up into 4k mappings.

It looks like the randomisation stuff is working, but I'm wondering of the NX bit isn't getting set correctly or at all? All the "executable anonymous mappings" are killed in the host system, so the NX bit is enabled.

does xen give access to the NX bit in the guest? what does the guest's /proc/cpuinfo report? what's the paxtest result under a vanilla guest kernel?

[cormander]

does xen give access to the NX bit in the guest? what does the guest's /proc/cpuinfo report? what's the paxtest result under a vanilla guest kernel?

I believe so, as /proc/cpuinfo does show the "nx" bit inside a paravirtulized guest, when the host has it.

[bplant]

It looks like the randomisation stuff is working, but I'm wondering of the NX bit isn't getting set correctly or at all? All the "executable anonymous mappings" are killed in the host system, so the NX bit is enabled.

does xen give access to the NX bit in the guest? what does the guest's /proc/cpuinfo report? what's the paxtest result under a vanilla guest kernel?

Xen does support the NX bit in the guest because I patched a 2.6.18 xen kernel with PaX/grsec some time ago and paxtest reported everything working. What I don't know is that if paravirt_ops supports the NX bit.

nx is listed in /proc/cpuinfo

Code: Select all

processor   : 0
vendor_id   : GenuineIntel
cpu family   : 6
model      : 15
model name   : Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz
stepping   : 7
cpu MHz      : 1596.042
cache size   : 4096 KB
physical id   : 0
siblings   : 1
core id      : 0
cpu cores   : 1
apicid      : 0
initial apicid   : 0
fpu      : yes
fpu_exception   : yes
cpuid level   : 10
wp      : yes
flags      : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc up rep_good nopl pni ssse3 cx16 lahf_lm
bogomips   : 3192.08
clflush size   : 64
cache_alignment   : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

paxtest blackhat on a vanilla guest kernel

Code: Select all

Executable anonymous mapping             : Vulnerable
Executable bss                           : Vulnerable
Executable data                          : Vulnerable
Executable heap                          : Vulnerable
Executable stack                         : Vulnerable
Executable anonymous mapping (mprotect)  : Vulnerable
Executable bss (mprotect)                : Vulnerable
Executable data (mprotect)               : Vulnerable
Executable heap (mprotect)               : Vulnerable
Executable stack (mprotect)              : Vulnerable
Executable shared library bss (mprotect) : Vulnerable
Executable shared library data (mprotect): Vulnerable
Writable text segments                   : Vulnerable
Anonymous mapping randomisation test     : 28 bits (guessed)
Heap randomisation test (ET_EXEC)        : 28 bits (guessed)
Heap randomisation test (ET_DYN)         : 28 bits (guessed)
Main executable randomisation (ET_EXEC)  : 28 bits (guessed)
Main executable randomisation (ET_DYN)   : 28 bits (guessed)
Shared library randomisation test        : 28 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 28 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 28 bits (guessed)
Return to function (strcpy)              : Killed
Return to function (memcpy)              : Killed
Return to function (strcpy, RANDEXEC)    : Killed
Return to function (memcpy, RANDEXEC)    : Killed
Executable shared library bss            : Vulnerable
Executable shared library data           : Killed

[PaX Team]

Xen does support the NX bit in the guest because I patched a 2.6.18 xen kernel with PaX/grsec some time ago and paxtest reported everything working. What I don't know is that if paravirt_ops supports the NX bit.

well it seems that even a vanilla kernel doesn't get the NX bit (does any older vanilla guest kernel get it under this xen host?), so the problem is not PaX related probably. you should ask the upstream xen guys as well because if it's really their or vanilla's problem, it's a security related bug.

[bplant]

Xen does support the NX bit in the guest because I patched a 2.6.18 xen kernel with PaX/grsec some time ago and paxtest reported everything working. What I don't know is that if paravirt_ops supports the NX bit.

well it seems that even a vanilla kernel doesn't get the NX bit (does any older vanilla guest kernel get it under this xen host?), so the problem is not PaX related probably. you should ask the upstream xen guys as well because if it's really their or vanilla's problem, it's a security related bug.

Unfortunately 2.6.27 is the first kernel with 64bit xen support. I could try an earlier 32bit kernel but my understanding was that the nx bit is a 64bit only thing. I'll try and ask on the xen-devel mailing list.

[PaX Team]

Unfortunately 2.6.27 is the first kernel with 64bit xen support. I could try an earlier 32bit kernel but my understanding was that the nx bit is a 64bit only thing. I'll try and ask on the xen-devel mailing list.

on capable CPUs the nx bit is available in 32 bit mode as well, but you need to enable PAE (or HIGHMEM64G) for kernel support.

[bplant]

Code: Select all

Executable anonymous mapping             : Vulnerable
Executable bss                           : Vulnerable
Executable data                          : Vulnerable
Executable heap                          : Vulnerable
Executable stack                         : Vulnerable
Executable anonymous mapping (mprotect)  : Vulnerable
Executable bss (mprotect)                : Vulnerable
Executable data (mprotect)               : Vulnerable
Executable heap (mprotect)               : Vulnerable
Executable stack (mprotect)              : Vulnerable
Executable shared library bss (mprotect) : Vulnerable
Executable shared library data (mprotect): Vulnerable
Writable text segments                   : Vulnerable
Anonymous mapping randomisation test     : 16 bits (guessed)
Heap randomisation test (ET_EXEC)        : 14 bits (guessed)
Heap randomisation test (ET_DYN)         : 13 bits (guessed)
Main executable randomisation (ET_EXEC)  : No randomisation
Main executable randomisation (ET_DYN)   : 16 bits (guessed)
Shared library randomisation test        : 16 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 19 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 19 bits (guessed)
Return to function (strcpy)              : paxtest: bad luck, try different compiler options.
Return to function (memcpy)              : Vulnerable
Return to function (strcpy, RANDEXEC)    : paxtest: bad luck, try different compiler options.
Return to function (memcpy, RANDEXEC)    : Vulnerable
Executable shared library bss            : Killed
Executable shared library data           : Killed

It would seem that the NX bit isn't working in a 32 bit kernel either. Before I post to the xen-devel mailing list; is there a kernel option I need to include to activate the NX bit support or is it "always on"? HIGHMEM64 and PAE are enabled and the nx flag is listed in /proc/cpuinfo.

[PaX Team]

It would seem that the NX bit isn't working in a 32 bit kernel either.

does this happen with a vanilla guest as well?

Before I post to the xen-devel mailing list; is there a kernel option I need to include to activate the NX bit support or is it "always on"? HIGHMEM64 and PAE are enabled and the nx flag is listed in /proc/cpuinfo.

NX doesn't have any extra config option, the kernel detects and makes use of it when in PAE mode automatically. on vanilla kernels there's a noexec kernel cmdline option, but under PaX/PAGEEXEC i disable it.

[bplant]

It would seem that the NX bit isn't working in a 32 bit kernel either.

does this happen with a vanilla guest as well?

It was a vanilla kernel. A 32-bit PaX kernel doesn't seem to boot. I'm guessing this may be the relocation stuff talked about in another thread.

Before I post to the xen-devel mailing list; is there a kernel option I need to include to activate the NX bit support or is it "always on"? HIGHMEM64 and PAE are enabled and the nx flag is listed in /proc/cpuinfo.

NX doesn't have any extra config option, the kernel detects and makes use of it when in PAE mode automatically. on vanilla kernels there's a noexec kernel cmdline option, but under PaX/PAGEEXEC i disable it.

I guess it should have worked then if it was going to. I have posted to the xen-devel mailing list asking if the NX bit is meant to work under paravirt_ops.

[cormander]

I upgraded my kernel to pax-linux-2.6.27.4-test13.patch and it boots under xen 3.3!

One thing I did notice was in the menuconfig is on the two PAX submenus:

Non-executable pages
Address Space Layout Randomization

They don't have anything in them .... I went to remove the paravirtualization piece and they still weren't there. Any ideas?

I greped the resulting .config for PAX and all I see is:

Code: Select all

CONFIG_PAX=y
CONFIG_PAX_SOFTMODE=y
# CONFIG_PAX_EI_PAX is not set
# CONFIG_PAX_PT_PAX_FLAGS is not set
CONFIG_PAX_NO_ACL_FLAGS=y
# CONFIG_PAX_HAVE_ACL_FLAGS is not set
# CONFIG_PAX_HOOK_ACL_FLAGS is not set
CONFIG_PAX_MEMORY_SANITIZE=y
# CONFIG_PAX_REFCOUNT is not set


[PaX Team]

One thing I did notice was in the menuconfig is on the two PAX submenus:

Non-executable pages
Address Space Layout Randomization

They don't have anything in them .... I went to remove the paravirtualization piece and they still weren't there. Any ideas?

besides checking COMPAT_VDSO for it disables the NOEXEC options (KERNEXEC also won't work with PARAVIRT), you must also select a PaX flag control mechanism.