grsecurity forums

by **bplant** » Tue Jul 04, 2006 10:17 am

bplant wrote:
Any ideas why the ET_DYN test isn't showing the mmap randomisation value?

maybe the test itself was miscompiled (as the kernel part is really trivial and hard to get wrong, look at the code using delta_exec). first make sure you're using -test5 and Makefile.psm then verify that getmain2 is ET_DYN and when run manually, it prints random numbers, not all 0s.

I downloaded paxtest-0.9.7-pre5.tar.gz and compiled it by running "make -f Makefile.psm". This produced the following output:

Code: Select all: Mode: blackhat Linux vancouver 2.6.16.18-xen-grsec #5 SMP Wed Jun 28 17:10:35 EST 2006 x86_64 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux Executable anonymous mapping : Killed Executable bss : Killed Executable data : Killed Executable heap : Killed Executable stack : Killed Executable anonymous mapping (mprotect) : Killed Executable bss (mprotect) : Killed Executable data (mprotect) : Killed Executable heap (mprotect) : Killed Executable stack (mprotect) : Killed Executable shared library bss (mprotect) : Killed Executable shared library data (mprotect): Killed Writable text segments : Killed Anonymous mapping randomisation test : 33 bits (guessed) Heap randomisation test (ET_EXEC) : 40 bits (guessed) Heap randomisation test (ET_DYN) : 40 bits (guessed) Main executable randomisation (ET_EXEC) : 32 bits (guessed) Main executable randomisation (ET_DYN) : 32 bits (guessed) Shared library randomisation test : 33 bits (guessed) Stack randomisation test (SEGMEXEC) : No randomisation Stack randomisation test (PAGEEXEC) : 40 bits (guessed) Return to function (strcpy) : Killed Return to function (memcpy) : Killed Return to function (strcpy, RANDEXEC) : Killed Return to function (memcpy, RANDEXEC) : Killed Executable shared library bss : Killed Executable shared library data : Killed

For interest sake, I recompiled using "make -f Makefile.Adamantix" and reran paxtest. Most of the results were the same, except for these:

Code: Select all: Stack randomisation test (SEGMEXEC) : 40 bits (guessed) Return to function (strcpy, RANDEXEC) : paxtest: return address contains a NULL byte.

I couldn't compile the paxtest using the Gentoo makefile, even though I run Gentoo

Is there a reason why the results are different when the different Makefiles are used? If one method can circumvent the randomisation, does this mean that the machine is more vulnerable? Or are they different because paxtest simulates exploits, not applications?

Cheers,

Brad

by **PaX Team** » Tue Jul 04, 2006 12:00 pm

bplant wrote:I couldn't compile the paxtest using the Gentoo makefile, even though I run Gentoo

Is there a reason why the results are different when the different Makefiles are used? If one method can circumvent the randomisation, does this mean that the machine is more vulnerable? Or are they different because paxtest simulates exploits, not applications?

paxtest was originally written for i386, everything else is not really supported. amd64 happens to work mostly by chance (e.g., the opcode for 'retn' is the same as on i386) and some things were not 64 bit clean. also there're different ways of compiling PIEs and apparently the makefiles are not always picking up the right one. so before jumping to conclusions about the test results it's always good to verify what you actually measured ;-)

.

by **john_anderson_ii** » Tue Jul 04, 2006 3:19 pm

Alright. I got paxtest to compile 64-bit, and I ran three different tests. One on a non-xen kernel, one on a Xen Dom0 and one on a Xen DomU. Here's the results:

linux-2.6.14.6 GRSecurity 2.1.8

Code: Select all: Mode: blackhat Linux apollo-r.beer.usmc 2.6.14.6-grsec #4 SMP Sun Apr 9 23:31:15 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux Executable anonymous mapping : Killed Executable bss : Killed Executable data : Killed Executable heap : Killed Executable stack : Killed Executable anonymous mapping (mprotect) : Killed Executable bss (mprotect) : Killed Executable data (mprotect) : Killed Executable heap (mprotect) : Killed Executable stack (mprotect) : Killed Executable shared library bss (mprotect) : Killed Executable shared library data (mprotect): Killed Writable text segments : Killed Anonymous mapping randomisation test : 33 bits (guessed) Heap randomisation test (ET_EXEC) : 13 bits (guessed) Heap randomisation test (ET_DYN) : 40 bits (guessed) Main executable randomisation (ET_EXEC) : 33 bits (guessed) Main executable randomisation (ET_DYN) : 33 bits (guessed) Shared library randomisation test : 33 bits (guessed) Stack randomisation test (SEGMEXEC) : 40 bits (guessed) Stack randomisation test (PAGEEXEC) : 40 bits (guessed) Return to function (strcpy) : paxtest: return address contains a NULL byte. Return to function (memcpy) : Vulnerable Return to function (strcpy, RANDEXEC) : paxtest: return address contains a NULL byte. Return to function (memcpy, RANDEXEC) : Vulnerable Executable shared library bss : Killed Executable shared library data : Killed

linux-2.6.16.13 GRSecurity 2.1.9 Xen Dom0

Code: Select all: Mode: blackhat Linux apollo-r.beer.usmc 2.6.16.13-xen-grsec #1 SMP Tue Jun 27 19:20:04 MST 2006 x86_64 x86_64 x86_64 GNU/Linux Executable anonymous mapping : Killed Executable bss : Killed Executable data : Killed Executable heap : Killed Executable stack : Killed Executable anonymous mapping (mprotect) : Killed Executable bss (mprotect) : Killed Executable data (mprotect) : Killed Executable heap (mprotect) : Killed Executable stack (mprotect) : Killed Executable shared library bss (mprotect) : Killed Executable shared library data (mprotect): Killed Writable text segments : Killed Anonymous mapping randomisation test : 33 bits (guessed) Heap randomisation test (ET_EXEC) : 13 bits (guessed) Heap randomisation test (ET_DYN) : 40 bits (guessed) Main executable randomisation (ET_EXEC) : 33 bits (guessed) Main executable randomisation (ET_DYN) : 33 bits (guessed) Shared library randomisation test : 33 bits (guessed) Stack randomisation test (SEGMEXEC) : 40 bits (guessed) Stack randomisation test (PAGEEXEC) : 40 bits (guessed) Return to function (strcpy) : paxtest: return address contains a NULL byte. Return to function (memcpy) : Vulnerable Return to function (strcpy, RANDEXEC) : paxtest: return address contains a NULL byte. Return to function (memcpy, RANDEXEC) : Vulnerable Executable shared library bss : Killed Executable shared library data : Killed

linux-2.6.16.13 GRSecurity-2.1.9 Xen DomU

Code: Select all: Mode: blackhat Linux test.beer.usmc 2.6.16-xen-grsec #4 SMP Fri Jun 16 22:17:59 MST 2006 x86_64 x86_64 x86_64 GNU/Linux Executable anonymous mapping : Killed Executable bss : Killed Executable data : Killed Executable heap : Killed Executable stack : Killed Executable anonymous mapping (mprotect) : Killed Executable bss (mprotect) : Killed Executable data (mprotect) : Killed Executable heap (mprotect) : Killed Executable stack (mprotect) : Vulnerable Executable shared library bss (mprotect) : Killed Executable shared library data (mprotect): Killed Writable text segments : Killed Anonymous mapping randomisation test : 33 bits (guessed) Heap randomisation test (ET_EXEC) : 13 bits (guessed) Heap randomisation test (ET_DYN) : 40 bits (guessed) Main executable randomisation (ET_EXEC) : 33 bits (guessed) Main executable randomisation (ET_DYN) : 33 bits (guessed) Shared library randomisation test : 33 bits (guessed) Stack randomisation test (SEGMEXEC) : 40 bits (guessed) Stack randomisation test (PAGEEXEC) : 40 bits (guessed) Return to function (strcpy) : paxtest: return address contains a NULL byte. Return to function (memcpy) : Vulnerable Return to function (strcpy, RANDEXEC) : paxtest: return address contains a NULL byte. Return to function (memcpy, RANDEXEC) : Vulnerable Executable shared library bss : Killed Executable shared library data : Killed

Everything is the same except for the "Executable stack (mprotect)" under the DomU. I don't know why it's not working maybe it can't, but I'll look into it.

by **john_anderson_ii** » Tue Jul 04, 2006 3:48 pm

PaX Team wrote:
john_anderson_ii wrote:I'm going to appeal to the xen-dev mailing list today and see if any ASM expert on their side can help me fix the i386 head-xen.S and entry-xen.S problem. entry-xen.S is almost identical to the vanilla kernel's entry.S, it just has some Xen DomU stuff appended to the end of it. head-xen.S is completely different though, since I don't know ASM, I have no idea where the PAX code goes, or how to properly define and populate cpu_gdt_descr and cpu_gdt_table.
it's not going to be easy, i can tell you that in advance . there're several changes on i386 in PaX that are not strictly security related but make some features a lot easier or even possible to implement. here's some list that should guide anyone looking into porting them to Xen (i won't do it myself until Xen hits mainline):

I don't blame you. With the amount of development that goes on daily at Xen it would be almost impossible to keep up. They will be mainline soon enough I guess and that will give you guys a more stable release cycle to work with.

Pax Team wrote:1. early PAE boot: normal linux doesn't boot into PAE mode when configured for PAE, instead in early boot it runs in 2-level paging mode and enables PAE mode (3-level page tables) later. this presents problems on its own even for normal linux (look at the EFI boot code for some horror), and it would complicate KERNEXEC a lot, so i rewrote this code to engage PAE as soon as paging itself is turned on - there is no mode switching later. furthermore, the kernel's own top level page directory page(s) are no longer allocated dynamically, they're defined in entry.S and placed into a special section at link time (again, this is needed to simplify KERNEXEC)..........the above may make you think that turning off KERNEXEC will alleviate the porting issues, but that's not the case as not everything is under #ifdef (some of the above changes just can't coexist with the vanilla kernel's behaviour without an #ifdef mess).

Thanks for the info. I've tried to the best of my ability to fix head-xen.s to work with Pax and I can't solve the problem. I'll forward your post over to the Xen developers, but they haven't shown any interest in the past.

I'm willing to say that realistically this is as far as we can go until Xen goes mainstream and it's codebase becomes less invasive to the vanilla Kernel. I'm really happy we got this far and have x86_64 (mostly) working with Xen Dom0 and DomU's. I wish I had a better understanding of what the code in head-xen.S was actually doing, it doesn't look like it even bothers with PAE at all.

At work (http://www.ccbill.com/www.cwie.net) we use only x86_64, the performance gains we get on our databases is worth it. Since we are a large credit card processor Visa, MasterCard and our auditors are all over our full deployment of GRSec/PAX. In that we respect we are always above and beyond our competitors. The bean counters wanted to reduce hardware costs and free-up rack space etc. Moving to Xen and dropping GRSecurity was what they wanted to do, and considering the amount of money that will save us anually, they made a strong business case. Now it seems I won't have to make that decision.

I've been testing a Dell 1850 EM64T Dual-Core for about two weeks. Hammering databases and webservers and it all seems to be stable, (even using OCFS2 between DomU's etc.). It looks like all of GRSecurity is working. I haven't tried learning mode yet. I really should get our software testing/QA team up to speed on GRSec/PAX and let them run through it with a fine tooth comb.

by **bplant** » Tue Jul 04, 2006 6:32 pm

Everything is the same except for the "Executable stack (mprotect)" under the DomU. I don't know why it's not working maybe it can't, but I'll look into it.

I failed to mention that I was running paxtest on a domU. I downloaded paxtest-0.9.7-pre5 and compiled it on the dom0 system using Makefile.psm. I got the same results as when using Makefile.psm in the domU system.

Did you use the same PaX configuration on both dom0 and domU machines?

Cheers,

Brad

by **john_anderson_ii** » Wed Jul 05, 2006 5:47 pm

I used the same PAX config for all three tests:

Here's the config.

Code: Select all: # # PaX # CONFIG_PAX=y # # PaX Control # CONFIG_PAX_SOFTMODE=y CONFIG_PAX_EI_PAX=y CONFIG_PAX_PT_PAX_FLAGS=y # CONFIG_PAX_NO_ACL_FLAGS is not set CONFIG_PAX_HAVE_ACL_FLAGS=y # CONFIG_PAX_HOOK_ACL_FLAGS is not set # # Non-executable pages # CONFIG_PAX_NOEXEC=y CONFIG_PAX_PAGEEXEC=y CONFIG_PAX_MPROTECT=y CONFIG_PAX_NOELFRELOCS=y # # Address Space Layout Randomization # CONFIG_PAX_ASLR=y CONFIG_PAX_RANDUSTACK=y CONFIG_PAX_RANDMMAP=y

by **bplant** » Thu Jul 06, 2006 10:12 pm

The only difference between your PaX config and mine is that you have softmode, whereas I don't. The only other difference that I can think of is that you are running a 2.6.16.13 kernel, whereas I am using version 2.6.16.18.

Cheers,

Brad

by **terran** » Sun Sep 24, 2006 12:15 am

Hello,

Following up on the prevoius thread about trying to integrate grsecurity & PAX with xen, I made an attempt at apply grsecurity 2.1.9 to kernel 2.6.17.13 with xen 3.0.2-2 on an i386.

I've had some success, and have gotten the system to bootstrap as a domU, and run with stack randomization, and the grsecurity MAC system. However, if I'm still having some trouble getting PAX stack protection working.

I've been trying to debug the segmentation based protection as that seemed like the easier of the two stack protection mechanisms to get working. I've replaced most of the critical LDT, and GDT loads with the equivalent hyerpcalls to xen. And I think I've correctly merged head-xen.S and entry-xen.S

Currently, I can start up the kernel with "init=/bin/bash " and, depending on if I'm lucky, execute "ls" or something similarly small. However, after one or two invocations of ls, the kernel will crash due to some sort of mmap corruption.

I've been unable to figure out how to start diagnosing this failure. Does anyone have any suggestions?

You can grab my patches from here:
http://www.geocities.com/terran101us/index.htm

The interesting changes are probably in the "paxify" patch. I've also pasted an example of the console output during kernel crash below.

Thanks,
Albert

[357729.966743] Eeek! page_mapcount(page) went negative! (-1)
[357729.966747] page->flags = 14
[357729.966750] page->count = 0
[357729.966753] page->mapping = 00000000
[357729.966761] ------------[ cut here ]------------
[357729.966764] kernel BUG at mm/rmap.c:577!
[357729.966768] invalid opcode: 0000 [#1]
[357729.966771] SMP DEBUG_PAGEALLOC
[357729.966779] Modules linked in:
[357729.966785] CPU: 0
[357729.966787] EIP: 0061:[<c0158d2e>] Not tainted VLI
[357729.966788] EFLAGS: 00010286 (2.6.17.13-grsec #146)
[357729.966799] EIP is at page_remove_rmap+0xce/0xf0
[357729.966804] eax: ffffffff ebx: c102a880 ecx: fbf3f000 edx: fbf3f000
[357729.966809] esi: c1213100 edi: cf876384 ebp: d0439c08 esp: d0439bf0
[357729.976526] ds: 0069 es: 0069 ss: 0069
[357729.976661] Process bash (pid: 664, threadinfo=d0438000 task=cfdf4ac0)
[357729.976794] Stack: c0345ce2 00000000 cfdf4ac0 c102a880 00000000 c1165a00 d0439c74 c0150dfb
[357729.978082] c102a880 680e1000 0fe0e0e0 680e1000 00000000 00000001 680e2000 cf8b5680
[357729.979368] cf8d0e08 c03d43a0 c102a880 c1213100 c014b64e 00000000 fffffffd cf8d0e58
[357729.981113] Call Trace:
[357729.981373] <c0105a0d> show_stack_log_lvl+0x9d/0xd0 <c0105c43> show_registers+0x1b3/0x260 <c0105e5c> die+0x16c/0x360
[357729.982667] <c01060e8> do_trap+0x98/0xe0 <c0106961> do_invalid_op+0xa1/0xb0 <c010542b> error_code+0x2b/0x30
[357729.983959] <c0150dfb> unmap_vmas+0x2db/0x600 <c0154f53> exit_mmap+0x93/0x120 <c011e5a6> mmput+0x36/0xb0
[357730.068249] <c0174cb0> flush_old_exec+0x370/0xa80 <c019a1a2> load_elf_binary+0x342/0x1b2e <c0174156> search_binary_handler+0xb6/0x280
[357730.068283] <c0175ddc> do_execve+0x3ec/0x490 <c0103942> sys_execve+0x42/0xa0 <c0105267> syscall_call+0x7/0xb
[357730.068307] Code: 40 75 33 8b 42 04 c7 04 24 cb 5c 34 c0 89 44 24 04 e8 67 89 fc ff 8b 43 10 c7 04 24 e2 5c 34 c0 89 44 24 04 e8 54 89 fc ff eb 88 <0f> 0b ea a9 5c 34 c0 41 02 eb 83 8b 53 0c 8d 74 26 00 eb c4 8d
[357730.068423] EIP: [<c0158d2e>] page_remove_rmap+0xce/0xf0 SS:ESP 0069:d0439bf0

by **PaX Team** » Wed Sep 27, 2006 2:45 am

terran wrote:[357729.966743] Eeek! page_mapcount(page) went negative! (-1)

this suggests that there's a reference counting problem probably in vma mirroring, that is, mapcount wasn't incremented when establishing a mirror somewhere. maybe add a printk into unmap_vmas to see which vma it was (disable ASLR to get comparable numbers) and work it out from there.

by **squadra** » Fri Sep 29, 2006 10:44 pm

Hello everyone,

anybody here who can send me johns xen patchset, iÂ´ll put them on a permanent stable location. his links are dead since a few days

And. anyone tried to modify johns x64 patches for newer kernel versions, yet? (2.6.17/2.6.18)?

Thanks

Juergen

by **terran** » Sat Sep 30, 2006 6:17 pm

yeah, I started to think that it was the mirroring having issues. However, I only have a vague understanding of how the mirroring is supposed to work. Are there any docs I can read that would tell me when the mirrors are supposed to be introduced, and what invariants are expected?

Thanks,
Albert

by **PaX Team** » Mon Oct 02, 2006 6:10 pm

terran wrote:yeah, I started to think that it was the mirroring having issues. However, I only have a vague understanding of how the mirroring is supposed to work. Are there any docs I can read that would tell me when the mirrors are supposed to be introduced, and what invariants are expected?

Thanks,
Albert

http://pax.grsecurity.net/docs/segmexec.txt for a start and you can ask me if you have more questions.

by **terran** » Tue Oct 03, 2006 4:24 pm

Whoa. Those docs are awesome, and way more complete than what I was expecting to get. Thanks for pointing them out to me. I wish I had them a month ago when I started this

I'll see where I can get with this and drop you a note. For starters, I already know that there's only 1 gdt_table in head.S in the xen version, so that's looking like it's going to be a bug.

by **terran** » Thu Oct 05, 2006 1:38 am

Reading through the vma mirroring doc, it seems to imply that the mirror vmas are held in a special list and that there is a special hack in unmap_vma that handles double-counting the mirror vmas. However, the current implementation doesn't seem to do this anymore. Instead, it looks like the mirror vmas are initialized same as the original vmas, and that handles the reference counts correctly. Does that sound right?

by **PaX Team** » Thu Oct 05, 2006 7:32 am

terran wrote:Instead, it looks like the mirror vmas are initialized same as the original vmas, and that handles the reference counts correctly. Does that sound right?

correct. in general, the docs cover the 2.4 (and 2.2) implementation, in 2.6 i made a few changes in various parts that are not covered yet.

grsecurity forums

Xen Paravirtualization + grsecurity working!

Xen + Grsecurity + PAX on 2.6.17.13 for i386

Re: Xen + Grsecurity + PAX on 2.6.17.13 for i386