grsecurity forums

Architecture: amd64
kernel: vanilla-sources-2.6.24.7 patched with pax-linux-2.6.24.7-test53.patch COMPILED with "CONFIG_PAX=N"
compiler: gcc-4.3.1 with glibc 2.7
Program run to exhibit problem: vmware-server-1.0.6 build-91891
Operating system: Gentoo

Problem: dmesg indicates that pax is taking action even though "CONFIG_PAX=N". (see below.)
Question: With "CONFIG_PAX=N, shouldn't this kernel act like the corresponding unpatched vanilla kernel, which does run vmware-server without
obeservable error?

PAX: vmware-vmx:6433, uid/euid: 1000/1000, invalid execution attempt at ffffc2000007e010 RIP:
[<ffffc2000007e010>]
PGD 8000000000664063 PUD 21fc46067 PMD 21fc47063 PTE c09063
Oops: 0011 [1] SMP
CPU 1
Modules linked in: vmnet(P) vmmon(P) ipt_REJECT iptable_filter iptable_mangle iptable_raw ip_tables x_tables vboxdrv pwc nvidia(P) snd_hda_intel ohci1394 i2c_core ieee1394 loop
Pid: 6433, comm: vmware-vmx Tainted: P 2.6.24-vanilla-v7-pax-patched-nopax #4
RIP: 0010:[<ffffc2000007e010>] [<ffffc2000007e010>]
RSP: 0000:ffff81020c561c50 EFLAGS: 00013046
RAX: ffffc2000007e010 RBX: ffffc2000007e000 RCX: ffffc2000007e000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 00002b7067b586f0 R08: 0000000000000001 R09: ffff8102121feee8
R10: ffff81020c560000 R11: 0000000000000001 R12: ffff81021fc6ae40
R13: 00000000f3242b90 R14: 0000000000000000 R15: 0000000000000063
FS: 00002b7067b586f0(0000) GS:ffff81021fc6ae40(0063) knlGS:00000000f3242b90
CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffffc2000007e010 CR3: 000000020c49f000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vmware-vmx (pid: 6433, threadinfo ffff81020c560000, task ffff81021fd5b830)
Stack: ffffffff88bcef97 03800e06043c0400 0000000000000000 ffff810218c40000
0000000000003286 000000008005003b 00000000f3242de0 00000000000006e0
0000000000007440 ffff8066d000007f 452e006d0078ffff d800000480635a80
Call Trace:
[<ffffffff88bcef97>] :vmmon:Task_Switch_S1B1+0x2f7/0xab0
[<ffffffff88bdc2e6>] :vmmon:Vmx86_RunVM_S1B1+0xb6/0x210
[<ffffffff88bc564a>] :vmmon:init_module+0x11aa/0x4220
[<ffffffff88bc65f9>] :vmmon:init_module+0x2159/0x4220
[<ffffffff88bc8310>] :vmmon:init_module+0x3e70/0x4220
[<ffffffff802b6741>] compat_sys_ioctl+0x91/0x3e0
[<ffffffff80256aaa>] compat_sys_setitimer+0x14a/0x160
[<ffffffff802193a2>] ia32_sysret+0x0/0xa


Code: 48 8b 91 54 07 00 00 ff 34 24 48 89 54 24 08 48 8d 91 2c 07
RIP [<ffffc2000007e010>]
RSP <ffff81020c561c50>
CR2: ffffc2000007e010
---[ end trace 703c709996da97c6 ]---

I should add these warnings to my original post.

Warnings: (1) Sometimes the attempt to run vmware-server under these conditions damages the virtual machine so that it can't be powered on later, even with an unpatched
vanilla or a gentoo kernel. For this reason anyone wishing to experiment with this problem is advised to make a backup copy of their virtual machine first.

(2) under the vanilla-2.6.25 pax patched kernel, vmware-server crashes my host OS when I attempt to power on a virtual machine, even when CONFIG_PAX=N.

andrewo8lgeg19tov wrote:Problem: dmesg indicates that pax is taking action even though "CONFIG_PAX=N". (see below.)
Question: With "CONFIG_PAX=N, shouldn't this kernel act like the corresponding unpatched vanilla kernel, which does run vmware-server without obeservable error?

there're many changes introduced in PaX that are not .config dependent, in this case vmmon fell victim to some KERNEXEC related hardening (the normal vmalloc range is not executable, only the module area is). ideally i'd like to keep this setup and vmmon should be patched to use the module alloc code to get its executable page. i could turn this particular change off for the !KERNEXEC case but i'd rather not make a habit out of it...

Many thanks to the PaX TEAM for responding so quickly. As I have always been impressed with PaX and the the team's work on it, I would not like to second guess them by
recommending changes to it that they do not feel are advisable. I myself would not be able to make the suggested patches to the vmmon module but will try to find and contact
someone who could.

As I have already raised this question on the gentoo forums, may I add a posting quoting the pax teams response?
I suspect that most people believe that setting "CONFIG_PAX=N" on a hardened-sources kernel would allow them to use the grsecurity acl's without pax
and that similarly setting
KERNEXEC=N would allow them to use pax without the kernexec related features, so they might find it helpful to know that this is not the case for kernels >= 2.6.24.

The current solution for them would be to stick with the 2.6.23 series for the present.

Regarding the PaX teams comment "there're many changes introduced in PaX that are not .config dependent, in this case vmmon fell victim to some KERNEXEC related hardening (the normal vmalloc range is not executable, only the module area is). ideally i'd like to keep this setup and vmmon should be patched to use the module alloc code to get its executable page. i could turn this particular change off for the !KERNEXEC case but i'd rather not make a habit out of it...."

and my attempt to find someone to patch vmmon as suggested: As suggested in the README file accompanying the vmware-server source code vmware-server-distrib/lib/modules/source/vmmon-only, I emailed support@vmware.com to ask if they would be interested in making the suggested changes or if they could suggest someone who could. I have received no response and suspect that I won't.

If the PaX team is still interested in modifying PaX so that vmware-server can run under a PaX hardened kernel, configured with KERNEXEC=N,
I would be most appreciative. However for this to be useful, it would have to become a regular part of future PaX and Grsecurity releases.
Otherwise I would have to stop using grsecurity, something that I would regret after many years of using these impressive and excellent security items.

andrewo8lgeg19tov wrote:If the PaX team is still interested in modifying PaX so that vmware-server can run under a PaX hardened kernel, configured with KERNEXEC=N, I would be most appreciative. However for this to be useful, it would have to become a regular part of future PaX and Grsecurity releases.

i just gave it a bit more thought and i can probably add support for vmware's use of vmap without compromising KERNEXEC, but i'm not sure if that's all it takes to get vmware to work, feedback will be welcome.

I would most appreciate that. thanks very much for your helpfulness in this issue.
If you do this I would be most happy to test your patch for you.

I do have a question and a possible related suggestion.

From your perspective (and I openly acknowledge that I have no security expertise so users such as myself really do depend on your judgement)
would it be acceptable to have a configurable feature that allows PaX to be totally turned off in the grsecurity kernel? I suggest that users who use the ACL's and PaX
would at least still be able to use the ACL's, when some non-configurable feature of PaX prevents an essential application from running.

Thanks again for all of the wonderful work done by the PaX and Grsecurity teams. I have used both for many years.

andrewo8lgeg19tov wrote:would it be acceptable to have a configurable feature that allows PaX to be totally turned off in the grsecurity kernel?

it's basically not possible without undue burden on me. as i said in the past, there're several things that PaX does which are not under .config control. some are security related, some are just plain common sense changes and some are needed to implement other features on a better background.

for example, for many years i've had a change that made the i386 kernel boot into PAE mode directly, without going through the 2-level page table setup that vanilla kernels used until very recently (iirc, .25 or .26 finally got changed and i could reduce my own patch in turn). this is not a security related change per se (i would not even call it a fix), but it allowed me to simplify other code as i didn't have to care about having to deal with two different kinds of page table setups under PAE kernels. #ifdef'ing this code out is pretty much impossible without essentially duplicating everything affected - that's what i call undue burden, especially for long term maintenance.

another example, per-CPU GDTs. the 2.6 kernel creates a GDT for each CPU. it used to be a nice big array allocated statically somewhere in the kernel's image. at some point someone figured that since it's per-CPU, it should become a per-CPU variable, allocated dynamically and hence spread out all over kernel memory. i had to revert this change (and have maintained this mini-fork ever since) because this was a disaster for KERNEXEC. it's a whole lot easier to ensure that an important resource such as the GDT is read-only when it's in the same range as other read-only data are. now this change may break external modules if they try to access the GDT improperly, without using the proper wrappers.

yet another example, per-CPU data handling in general. on i386 it is based on the segmentation logic. the problem with the implementation was (and still is, in fact) that the data segment used for the per-CPU data covered the entire address space, not only the actual per-CPU data. that of course flew in the face of UDEREF and i had to change it so that the per-CPU segment covered only what it was supposed to. that's a change that can't be easily #ifdef'd either.

there're many more examples, these just came to my mind because i know how much time i spent on getting and keeping them to work so far and as i said at the beginning, i'm not going to make them configurable. with that said, there're several other kernel features where i go out of my way to keep them working as in vanilla, even if they're bad for security. think GNU_STACK support or READ_IMPLIES_EXEC. heck, i even fixed bugs in *them*.

Your answer make good sense to me. Thanks for explaining it so fully
and thanks again for all of the excellent work.

I'm just checking to see if the PaX team still hopes to make PaX (in some configuration)
compatible with vmware server on the AMD64.

Thanks

andrewo8lgeg19tov wrote:I'm just checking to see if the PaX team still hopes to make PaX (in some configuration)
compatible with vmware server on the AMD64.

i'd looked at the code a bit already but it didn't look like a 5 minute job so it's still on my TODO list. i'll get to it eventually, just don't know when...

PaX Team wrote:

andrewo8lgeg19tov wrote:I'm just checking to see if the PaX team still hopes to make PaX (in some configuration)
compatible with vmware server on the AMD64.

i'd looked at the code a bit already but it didn't look like a 5 minute job so it's still on my TODO list. i'll get to it eventually, just don't know when...

Not wanting to be pushy, but is this going to be looked at soon? I'm looking to run vmware-server on my hardened server.

Cheers.

Silverfish wrote:Not wanting to be pushy, but is this going to be looked at soon? I'm looking to run vmware-server on my hardened server.

not anytime soon, unless vmware pushes their code into vanilla, then it'd be inevitable .