Xen Paravirtualization + grsecurity working!

[john_anderson_ii]

Greetings!

Well, I've been working on this for a while, but I think I finally have grsecurity working with Xen 3.0.2 paravirtualization. I tried this before, but it was nearly impossible because Xen treated the linux kernel like a subsystem of itself. Patching a vanilla kernel with Xen completely replaced several parts of the kernel. To circumvent this we decided to try out Xen's HVM full virtualization. That didn't work out so well. Not only did we have to buy a really expensive server to get Intel VT-x enabled chips, but the performance is horrible. I know the guys over at xensource are working hard on replacing the qemu emulation system with a better and faster system, but there is no way I can deploy the current implementation on production systems. That's when I started thinking.... Xen is trying to merge itself with the mainstream kernel. So in the current release, Xen doesn't become it's own arch and replace large parts of the kernel. Instead it tries to appear more like a driver or a service. As a matter of fact, I think it's offically a subarch of i386 and x86_64 now.

Then I thought If I'm going to be running linux on Xen, why the heck would I ever want to fully virtualize something that should be paravirtualized. All I'm getting out of the deal is a big performance hit.

Anyway, I got cracking on fixing the .rej's that are produced when applying grsecurity-2.1.9-2.6.16.19-200606041421.patch to a Xen patched kernel tree. After the smoke cleared, I fired up Dom0 and lo and behold grsec worked there! Then I fired up a paravirtualized DomU and again grsec worked there too!

I haven't tested everything, just x86_64 Dom0 and DomU's. I'm afraid to try the i386 arch because Xen replaces arch/{i386,x86_64}/mm/fault.c with fault-xen.c. The x86_64 version was almost identical to the kernel.org version, so finding out where the code in the patch should go was easy. The i386 version wasn't so easy, and I'm not sure if I got it right.

I didn't have to change any code at all, I just had to find the proper place to put the code when the patch failed. I'm a lousy programmer, so I'm glad I didn't have to guess. If any real developers out there use Xen, please give this a try and let me know if I screwed anything up.

How to use the patch.

First, get the Xen Source. Then unpack it, chdir into the top level tree and do a make. This will download the vanilla 2.6.16 kernel from kernel.org, patch the hell out of it and build it. Once that's done, cd into linux-2.6.12-xen and make mrproper (I suppose make clean would work as just well), then apply this patch. Then cd back into the Xen source directory and run rm -fR dist/install/* && make linux-2.6-xen-config CONFIGMODE={menuconfig,xconfig} and configure your kenrel and grsec. Next run make linux-2.6-xen-build && make linux-2.6.-xen-install. Then cd dist && ./install.sh.

I tried to create a patch that could just be copied into the xen-3.0.2/patches/linux-2.6.16/ directory and auto-applied but I'm not having any luck there.


I'm hosting this Xen patch from my box at home, so it's availability is really up to Cox Communications and the electric company. If anyone has a place more accessible, please stash it and link to it. If not, I'll try to get it up somewhere on CCBill's network.

[bplant]

Hi John,

Firstly, I would like to say a big thank you for your efforts. I believe many people will benefit from what you have achieved here. I know that I have been wanting to run Xen and grsecurity concurrently for a while now, but I have not been able to find any solution.... until now that is

I have tried to compile Xen + grsec for i386, but unfortunately the process does not get very far. I get the following error:

CHK include/linux/version.h
CC arch/i386/kernel/asm-offsets.s
arch/i386/kernel/asm-offsets.c: In function `foo':
arch/i386/kernel/asm-offsets.c:74: error: `PTRS_PER_PTE' undeclared (first use in this function)
arch/i386/kernel/asm-offsets.c:74: error: (Each undeclared identifier is reported only once
arch/i386/kernel/asm-offsets.c:74: error: for each function it appears in.)
make[1]: *** [arch/i386/kernel/asm-offsets.s] Error 1
make: *** [prepare0] Error 2

Unfortunately my skill set does not extend to kernel programming (although I would like to change that one day!) and I am not sure how to go about fixing this issue. If anyone has had any success compiling the kernel for i386 I would be very interested to hear how this issue was circumvented.

I do have new dual core Intel CPU's, so I believe I could try a 64 bit kernel like yourself, however after reading some posts by the pax team, I would prefer to use i386. The 900 series Intel CPU does not appear to have the nx feature and PaX on x86_64 doesn't seem to be fully matured yet.
http://forums.grsecurity.net/viewtopic.php?t=1432 (second post)
http://forums.grsecurity.net/viewtopic.php?t=1426 (last post on first page)

Cheers,

Brad

[john_anderson_ii]

Try to #include <asm/pgtable.h> in arch/i386/kernel/asm-offsets.c. I wonder if that's the way it's setup in 2.6.16.19. I used the grsec patch for that later version to apply grsec to the 2.6.16.13-xen kernel.

[bplant]

Hi John,

Including asm/pgtable.h worked a treat. I have now come across this error:

$ make
CHK include/linux/version.h
CC arch/i386/kernel/asm-offsets.s
GEN include/asm-i386/asm-offsets.h
HOSTCC scripts/conmakehash
CC init/main.o
CHK include/linux/compile.h
UPD include/linux/compile.h
CC init/version.o
CC init/do_mounts.o
CC init/do_mounts_rd.o
CC init/do_mounts_initrd.o
LD init/mounts.o
CC init/initramfs.o
CC init/calibrate.o
LD init/built-in.o
HOSTCC usr/gen_init_cpio
CHK usr/initramfs_list
UPD usr/initramfs_list
CPIO usr/initramfs_data.cpio
GZIP usr/initramfs_data.cpio.gz
AS usr/initramfs_data.o
LD usr/built-in.o
CC arch/i386/kernel/process-xen.o
CC arch/i386/kernel/semaphore.o
CC arch/i386/kernel/signal.o
arch/i386/kernel/signal.c: In function `do_signal':
arch/i386/kernel/signal.c:603: warning: implicit declaration of function `user_mode_novm'
AS arch/i386/kernel/entry-xen.o
CC arch/i386/kernel/traps-xen.o
arch/i386/kernel/traps-xen.c: In function `math_state_restore':
arch/i386/kernel/traps-xen.c:986: error: invalid `asm': operand number out of range
arch/i386/kernel/traps-xen.c:986: error: invalid `asm': operand number out of range
{standard input}: Assembler messages:
{standard input}:2097: Error: suffix or operands invalid for `frstor'
{standard input}:2109: Error: suffix or operands invalid for `fxrstor'
make[1]: *** [arch/i386/kernel/traps-xen.o] Error 1
make: *** [arch/i386/kernel] Error 2

I am using a 2.6.16.18-xen kernel (provided by Gentoo). The grsec patch you provided appeared to apply ok this this kernel with just a handful of offsets. Once we've got the i386 kernel to compile and work I might look at updating the patch to apply without the offsets

Thanks for your help John.

Cheers,

Brad

[john_anderson_ii]

Ok. It looks grsecurity-2.1.9-2.6.16.19-200606041421.patch makes significant changes to arch/i386/kernel/traps.c. Unfortunately Xen doesn't use this file, it uses arch/i386/kernel/traps-xen.c. So I'll have to fix my patch to modify traps-xen.c instead of traps.c.

Actually it looks like I'm going to be spending quite a bit more time getting Xen and grsecurity to work together. At first I just hacked it in, fixed the x86_64 problems and said , "good enough for me." (I only use x86_64 stuff at work and home.)

But Ill clean everything up and probably split it into two patches. The first patch to be applied by the xen make process after all other patches are applied but before the xen-sparse tree is applied. Then the other half will be applied so we can deal with the files that Xen replaces with it's own.

That's mainly because Xen symlinks some of it's *-xen.c into the place it would go in the kernel tree, and patches the Makefiles to use the symlinks just as it has been done with the traps files above.

[bplant]

It sure does sound like the x86_64 case is easier than the i386 one. I tried to apply parts of the grsec patch to the *-xen.{c,S} files, but I didn't get past patching head-xen.S. There seem to be major differences between head.S and head-xen.S.

But like I said earlier, I have never programmed in the kernel, so I really don't know how to go about trying to resolve these issues when it isn't clear where stuff should go. Hopefully you will have more luck!

Btw, have you run paxtest on your patched kernels to verify that the PaX components of the grsec patch are working? Would you mind posting the output from paxtest?

Cheers,

Brad

[john_anderson_ii]

I'll check out the pax test suite when I get i386 to compile and boot. I tested RBAC, process and chroot restrictions on x86_64 Xen and they worked out alright. Knowing what I know now I wouldn't be surprised at all to find that PAX isn't working properly and thinking that was PAX is working when it isn't is probably a worse scenario than no PAX at all.

I've made a Xen version of the grsecurity-2.1.9 patch that applies cleanly after the Xen build process applies the content of /patches/2.6.16 to the kernel source tree. I'm going through the linux-2.6-xen-sparse tree right now and applying code from the grsec patch to (*hopefully*) the right places.

[john_anderson_ii]

I've still been working at this and here's where I'm at so far. I can finally get i386 & X86_64 kernels to compile cleanly.

Here's the issues I'm having:

i386 will not boot. The Xen microkernel fails to recognize the i386 kernel as a Dom0 image. I'm pretty sure I know why. head-xen.S has to be modified to get the kernel to compile and I'm thinking I messing up those modifications. head-xen.S is about 1/4 the size of a vanilla kernel's head.S and it doesn't define cpu_gdt_descr. If I don't modify head-xen.S to define cpu_gdt_descr cpu/common-xen.c doesn't compile with the grsec modifications. All of the Xen code uses "struct Xgt_desc_struct *gdt_descr = &per_cpu(cpu_gdt_descr, vcpu);" as the gdt_descr initializer all over their code. I moved all the code to use "gdt_descr = &cpu_gdt_descr[cpu];" , but that doesn't do much good if I can't get cpu_gdt_descr defined as a .globl in head-xen.S


CONFIG_PAX_RANDKSTACK is not working under any architecture. Xen does not use any sort of tss struct and when CONFIG_XEN is selected then CONFIG_X86_NO_TSS is also selected which means INIT_TSS is not defined. In other words, I had to wrap the insides of asmlinkage void pax_randomize_kstack(void) with ifdefs since that function didn't have access to the tss_struct initializer. I'm working with xen-devel to try to figure out what I can do. I think, since this is linux virtualized, they use a thread_struct that serves the same purpose as a tss_struct. I'm trying to learn how to get a handle on them.

There's probably more stuff that isn't working, but it's a start.

If anyone would like to tinker with it, get the xen 3.0 testing (3.0.2-3) source from here. Then get this patch and this patch. Untar the xen sources and apply grsecurity-2.1.9-xen-3.0-testing.patch to the linux-2.6-xen-sparse directory. Then copy the z_grsecurity-2.1.9-xen-3.0.2t.patch to patches/2.6.16.13/ directory and run make from the root of the Xen source directory.

The good news that it seems the grsecurity (non PAX) features of X86_64 Xen is working fine.

UPDATE: With the help of Keir Fraser from xen-devel we were able to *hopefully* fix the pax_randomize_kstack function to not use TSS but still be effective if CONFIG_X86_NO_TSS is defined. I've updated the patch linked to above.

[bplant]

Hi John,

The issues you describe with cpu_gdt_descr in head-xen.S sound all too familiar. Unfortunately I was not able to sort it out when I was trying to patch the i386 kernel.

I have used your latest patches and compiled a dom0 system for x86_64. From what I can see, grsecurity is working. I haven't done a lot of testing though.

PaX also seems to be working. The output of paxtest is listed below:

Mode: blackhat
Linux xen2 2.6.16.18-xen-grsec #2 SMP Wed Jun 28 12:44:40 EST 2006 x86_64 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux

Executable anonymous mapping : Killed
Executable bss : Killed
Executable data : Killed
Executable heap : Killed
Executable stack : Killed
Executable anonymous mapping (mprotect) : Killed
Executable bss (mprotect) : Killed
Executable data (mprotect) : Killed
Executable heap (mprotect) : Killed
Executable stack (mprotect) : Killed
Executable shared library bss (mprotect) : Killed
Executable shared library data (mprotect): Killed
Writable text segments : Killed
Anonymous mapping randomisation test : 33 bits (guessed)
Heap randomisation test (ET_EXEC) : 13 bits (guessed)
Heap randomisation test (ET_DYN) : 40 bits (guessed)
Main executable randomisation (ET_EXEC) : No randomisation
Main executable randomisation (ET_DYN) : No randomisation
Shared library randomisation test : 33 bits (guessed)
Stack randomisation test (SEGMEXEC) : No randomisation
Stack randomisation test (PAGEEXEC) : 40 bits (guessed)
Return to function (strcpy) : paxtest: bad luck, try different compiler options.
Return to function (memcpy) : Killed
Return to function (strcpy, RANDEXEC) : paxtest: bad luck, try different compiler options.
Return to function (memcpy, RANDEXEC) : Killed
Executable shared library bss : Killed
Executable shared library data : Killed

I'm not sure about the "Main executable randomisation" tests. Should they have some randomisation? Unfortunately I don't have a non-xen x86_64 machine to compare against.

Btw, I am still yet to try a domU kernel. I will report on that later.

Again, thanks for all your work John. Things are looking pretty good.

Cheers,

Brad

[john_anderson_ii]

Those are the same results I saw. The main executable randomization I think is referring to RANDSTACK, comparing the results of "paxtest blackhat" of my x86_64 Dom0 to a non-xen X86_64 box with the same PAX configuration the results are identical, so I'm not entirely sure RANDSTACK is supposed to work on x86_64, I'm still looking into it though. Another thing that worries me is I wasn't able to compile paxtest in 64-bit mode. I have a multilib system so I compiled it in 32 bit mode and ran it from there. I wish I could get it to compile in native 64.

When I try to compile in 64 bit mode, I get this error

Code: Select all

crt1S.S: Assembler messages:
crt1S.S:5: Error: suffix or operands invalid for `pop'
crt1S.S:10: Error: suffix or operands invalid for `pop'
crt1S.S:11: Internal error, aborting at ../../binutils-2.16.1/gas/config/tc-i386.c line 3652 in output_imm
Please report this bug.
make[1]: *** [crt1S.o] Error 1
make[1]: Leaving directory `/usr/src/gx/paxtest/paxtest-0.9.7-pre4'
make: *** [generic] Error 2


I'm pretty sure this error is way over my head.

I'm going to appeal to the xen-dev mailing list today and see if any ASM expert on their side can help me fix the i386 head-xen.S and entry-xen.S problem. entry-xen.S is almost identical to the vanilla kernel's entry.S, it just has some Xen DomU stuff appended to the end of it. head-xen.S is completely different though, since I don't know ASM, I have no idea where the PAX code goes, or how to properly define and populate cpu_gdt_descr and cpu_gdt_table.

[bplant]

I am using Gentoo, so all the compiling is done for me, but it would appear that paxtest has been compiled for native x86_64:

Code: Select all

$ file /usr/lib64/paxtest/anonmap
/usr/lib64/paxtest/anonmap: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped


After looking at the ebuild for 0.9.6 of paxtest: http://www.gentoo.org/cgi-bin/viewcvs.cgi/app-admin/paxtest/paxtest-0.9.6.ebuild?rev=1.12&view=auto it looks as though a different Makefile is used: http://www.gentoo.org/cgi-bin/viewcvs.cgi/app-admin/paxtest/files/Makefile-portable?rev=1.1&view=auto Perhaps you could try compiling with that instead.

Good luck with head-xen.S

Cheers,

Brad

[PaX Team]

I'm not sure about the "Main executable randomisation" tests. Should they have some randomisation? Unfortunately I don't have a non-xen x86_64 machine to compare against.

RANDEXEC has been removed from PaX/2.6 (others will follow too), so the ET_EXEC randomization test should show 0, however the ET_DYN one should show the mmap randomization value.

[PaX Team]

Another thing that worries me is I wasn't able to compile paxtest in 64-bit mode. I have a multilib system so I compiled it in 32 bit mode and ran it from there. I wish I could get it to compile in native 64.

try -test5 from my test dir along with Makefile.psm, that should work on amd64 as well.

I'm going to appeal to the xen-dev mailing list today and see if any ASM expert on their side can help me fix the i386 head-xen.S and entry-xen.S problem. entry-xen.S is almost identical to the vanilla kernel's entry.S, it just has some Xen DomU stuff appended to the end of it. head-xen.S is completely different though, since I don't know ASM, I have no idea where the PAX code goes, or how to properly define and populate cpu_gdt_descr and cpu_gdt_table.

it's not going to be easy, i can tell you that in advance . there're several changes on i386 in PaX that are not strictly security related but make some features a lot easier or even possible to implement. here's some list that should guide anyone looking into porting them to Xen (i won't do it myself until Xen hits mainline):

1. early PAE boot: normal linux doesn't boot into PAE mode when configured for PAE, instead in early boot it runs in 2-level paging mode and enables PAE mode (3-level page tables) later. this presents problems on its own even for normal linux (look at the EFI boot code for some horror), and it would complicate KERNEXEC a lot, so i rewrote this code to engage PAE as soon as paging itself is turned on - there is no mode switching later. furthermore, the kernel's own top level page directory page(s) are no longer allocated dynamically, they're defined in entry.S and placed into a special section at link time (again, this is needed to simplify KERNEXEC).

2. per-CPU GDT: sometime during 2.6 development some misguided soul decided that any NR_CPUS array is best turned into a dynamically allocatable per-CPU variable (for CPU hotplug and whatnot). this is a very stupid idea for the GDT (look at the the contortions that arch/i386/kernel/cpu/common.c:cpu_init() goes through to allocate the GDT page) and is directly against the goals of KERNEXEC - so i ended up reverting that change. it affects more than entry.S by the way, look for cpu_gdt_table and related symbols in PaX to find the relevant chunks.

3. userland ptr deref/KERNEXEC and GDT patching: there're a few features that required changing the default linux GDT descriptors, i don't know whether all of them can be expressed under Xen though.

the easy one is the unconditional setting of the Accessed bit in all descriptors, that allows to make the GDT to be read-only at all and i think Xen can only benefit from this.

the harder one will be the KERNEXEC ring-0 code segment, i don't know what assumptions Xen makes about the guest kernel's segmentation needs, it may or may not fly.

the hardest part will the new userland ptr dereference protection as it relies on an otherwise rarely used and hard to emulate (under Xen/VMware) feature: expand down data segments. i'm almost sure that the Xen folks will say it's not possible to port w/o an unreasonable performance impact (just try it out under VMware to see what i mean), so you may as well just put the original __USER_DS descriptor back (like PaX does at startup_32).

finally, as KERNEXEC implements proper read-only kernel data as well even for otherwise writable pieces, there's code that allows turning off this write-protection temporarily, that code needs Xen support as it writes to cr0. the C code is probably covered by Xen's redefining the cr0 access macros but the assembly in entry.S needs patching.

the above may make you think that turning off KERNEXEC will alleviate the porting issues, but that's not the case as not everything is under #ifdef (some of the above changes just can't coexist with the vanilla kernel's behaviour without an #ifdef mess).

[bplant]

bplant wrote:
I'm not sure about the "Main executable randomisation" tests. Should they have some randomisation? Unfortunately I don't have a non-xen x86_64 machine to compare against.

RANDEXEC has been removed from PaX/2.6 (others will follow too), so the ET_EXEC randomization test should show 0, however the ET_DYN one should show the mmap randomization value.

Any ideas why the ET_DYN test isn't showing the mmap randomisation value? Here's my PaX config:

Code: Select all

#
# PaX
#
CONFIG_PAX=y

#
# PaX Control
#
# CONFIG_PAX_SOFTMODE is not set
CONFIG_PAX_EI_PAX=y
CONFIG_PAX_PT_PAX_FLAGS=y
# CONFIG_PAX_NO_ACL_FLAGS is not set
CONFIG_PAX_HAVE_ACL_FLAGS=y
# CONFIG_PAX_HOOK_ACL_FLAGS is not set

#
# Non-executable pages
#
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_MPROTECT=y
CONFIG_PAX_NOELFRELOCS=y

#
# Address Space Layout Randomization
#
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y


Cheers,

Brad

[PaX Team]

Any ideas why the ET_DYN test isn't showing the mmap randomisation value?

maybe the test itself was miscompiled (as the kernel part is really trivial and hard to get wrong, look at the code using delta_exec). first make sure you're using -test5 and Makefile.psm then verify that getmain2 is ET_DYN and when run manually, it prints random numbers, not all 0s.