Kernel panic with 2.6.24 under Xen, take two

[cormander]

Doing an interdiff on grsecurity-2.1.11-2.6.24.3-200803031942.patch and pax-linux-2.6.24.3-test33.patch

Most of it works, lots of offsets, but a few failures:

Code: Select all

Hunk #14 FAILED at 528.
1 out of 15 hunks FAILED -- saving rejects to file arch/x86/mm/fault_32.c.rej
Hunk #1 FAILED at 51.
Hunk #11 FAILED at 1524.
2 out of 11 hunks FAILED -- saving rejects to file fs/exec.c.rej
Hunk #2 FAILED at 155.
1 out of 3 hunks FAILED -- saving rejects to file fs/proc/task_mmu.c.rej
Hunk #1 FAILED at 21.
Hunk #6 FAILED at 211.
2 out of 11 hunks FAILED -- saving rejects to file mm/mprotect.c.rej


Similar failures for test 31 and 32. Different hunk #s but for the same files. Which exact PAX patch went into grsecurity-2.1.11-2.6.24.3-200803031942.patch ? That way I'll have to do less digging trying to undo those failed hunks.

Thanks for your help so far!

[PaX Team]

Similar failures for test 31 and 32. Different hunk #s but for the same files. Which exact PAX patch went into grsecurity-2.1.11-2.6.24.3-200803031942.patch ? That way I'll have to do less digging trying to undo those failed hunks.

normally it's the last PaX test patch, so it should have been test33 and indeed, they interdiff cleanly for me.

[xstasi]

....another update!

I found the file that messes everything up!

Here's what i've done:

I took a vanilla 2.6.24.3, unpacked it, patched it with pax
Then i patched -R (thanks splitdiff) only the file arch/x86/kernel/vmlinux_32.lds.S, and adjusted a few lines here and there to make it compile.

Obviously, the kernel doesn't run, but... it crashes on another part of the code

I launched gdb on it, inspected the memory.. and yes, the .data segment is there! kallsyms is full of nifty strings and data

So, the conclusion is that the file arch/x86/kernel/vmlinux_32.lds.S is the one who's messing with the segments, and making the kernel resulting full of zeroes under xen.

Obeying Murphy's laws, lds.S code is the only code type i've never read in a kernel, so it will take more than a long time for me to find what's offending there.

By the way, i thought it would have been a good idea to report my discover

Maybe pageexec, who's surely more experienced than me, will find a quicker solution

Cheers...

xstasi

[PaX Team]

I took a vanilla 2.6.24.3, unpacked it, patched it with pax
Then i patched -R (thanks splitdiff) only the file arch/x86/kernel/vmlinux_32.lds.S, and adjusted a few lines here and there to make it compile.

Obviously, the kernel doesn't run, but... it crashes on another part of the code

vmlinux_32.lds.S is the linker script that is used to produce vmlinux, you can't arbitrarily patch it out as it contains crucial changes related to certain PaX features (some outside of config options, say direct PAE boot without the temporary 2-level page tables), no wonder it didn't work . it also means that whatever xen doesn't like in the ELF produced by this linker script may turn out to be something to be taken care of in xen itself, not PaX. as i said before, it's time to ask the xen devs about how their ELF loader works, what it expects, etc then we can figure out a solution.

Obeying Murphy's laws, lds.S code is the only code type i've never read in a kernel, so it will take more than a long time for me to find what's offending there.

check binutils/ld info pages about linker scripts, they explain the syntax and everything else you need to know about them. you can also cut your teeth on the default linker scripts used to link userland executables, they're installed by binutils somewhere under /usr/lib/binutils/i686-pc-linux-gnu/2.18/ldscripts/ or similar.

[xstasi]

vmlinux_32.lds.S is the linker script that is used to produce vmlinux, you can't arbitrarily patch it out as it contains crucial changes related to certain PaX features (some outside of config options, say direct PAE boot without the temporary 2-level page tables), no wonder it didn't work .

I didn't expect it to work... I did that test just to make sure that the .lsd.S was the file to blame

it also means that whatever xen doesn't like in the ELF produced by this linker script may turn out to be something to be taken care of in xen itself, not PaX. as i said before, it's time to ask the xen devs about how their ELF loader works, what it expects, etc then we can figure out a solution.

Now I fully agree with you, and i've done a bug report, check it here:
http://bugzilla.xensource.com/bugzilla/ ... gi?id=1189

I will keep you informed!

Cheers

[cormander]

If anyone is interested, I took the grsecurity patch posted today and removed pax from it, and posted it here:

http://www.ravencore.com/grsec/

I have used the patch utility for years, but admittedly never used interdiff or otherwise. This diff was generated with the following steps:

Code: Select all

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.24.3.tar.gz
wget http://grsecurity.net/test/grsecurity-2.1.11-2.6.24.3-200803110829.patch
wget http://grsecurity.net/test/pax-linux-2.6.24.3-test34.patch
tar zxf linux-2.6.24.3.tar.gz
mv linux-2.6.24.3 linux-2.6.24.3-pax
patch -p0 -i pax-linux-2.6.24.3-test34.patch
tar zxf linux-2.6.24.3.tar.gz
patch -p0 -i grsecurity-2.1.11-2.6.24.3-200803110829.patch
mv linux-2.6.24.3 linux-2.6.24.3-grsec
diff -ruN linux-2.6.24.3-pax linux-2.6.24.3-grsec > temp.diff
tar zxf linux-2.6.24.3.tar.gz
cd linux-2.6.24.3
patch -N -p1 -i ../temp.diff
cd ..
mv linux-2.6.24.3 linux-2.6.24.3-grsec-nopax


Then I resolved the hunk failures in the following files (not a whole lot of changes):

Code: Select all

linux-2.6.24.3-grsec-nopax/arch/x86/Kconfig
linux-2.6.24.3-grsec-nopax/arch/x86/mm/fault_32.c
linux-2.6.24.3-grsec-nopax/fs/exec.c
linux-2.6.24.3-grsec-nopax/fs/proc/task_mmu.c
linux-2.6.24.3-grsec-nopax/kernel/sysctl.c
linux-2.6.24.3-grsec-nopax/mm/mprotect.c
linux-2.6.24.3-grsec-nopax/security/Kconfig


And then finally a diff on the vanilla and grsec-nopax directories

The vmlinuz image from this kernel boots under xen. I'm in the process of testing the TPE, RBAC, etc features

[ThomasR]

Hi Cormander,

did you make some progress on testing the grsec-nopax patch? I would be very interested in getting grsec to run with Xen as well.

Btw: Does anyone have experience with running unmodified linux guests under Xen (in terms of stability and performance)?

Best regards,

Thomas

[cormander]

Hello,

The grsec-nopax works great for me. No problems since I started using it (minus one bug report I filed, but it wasn't related to it being a xen domU). I also updated it to the March 23rd test patch this morning:

http://www.ravencore.com/grsec/

Though there is now a March 24th patch. Hard to keep up with the latest tests

What I do to save time on building new kernels for my xen domU images is change the .config file to have the network, filesystem (ext3 & reiserfs for me), and netfilter drivers to be built in, and do just 'make vmlinux'. It builds in around 10 minutes (whereas compiling all modules takes several hours).

As far as running unmodified kernels as paravirtulized guests... haven't done it beyond running it for a few days while I compared to to the grsecurity (with pax) on xen domU. Seems stable enough. Hell, if the kernel works just fine with the grsec-nopax patch, can't imagine there would be any problems with no patch at all.

Performance wise, if the other domU guests and the dom0 host OS are idle; running benchmark tests on one of the domUs, it almost feels like a real machine. They really did a good job with the paravirtualized interface for system CPU speed and I/O .

Stability wise, its pretty solid. I don't think redhat or novell would support it the way they do if it wasn't. The main stability factor for xen is what verson of the hypervisor you're running; I have 3.0.2 on my dom0 host OS, and it's rock solid. Only problem with it is you can't unmount xen images you've manually mounted from the dom0 under xen's mapper driver... which isn't a big issue anyway. From what I read 3.0.2 doesn't have suspend / resume support, but for me that's not a big deal. I compiled 3.2.1 xen hypervisor and over the weekend the host running it went offline.. haven't went to the datacenter to figure out why yet. So the latest version could be a mite unstable. I haven't tried 3.1

[ThomasR]

Hi,

thanks for the quick answer! I'll compile a test kernel myself this week.

As far as running unmodified kernels as paravirtulized guests... haven't done it beyond running it for a few days while I compared to to the grsecurity (with pax) on xen domU. Seems stable enough. Hell, if the kernel works just fine with the grsec-nopax patch, can't imagine there would be any problems with no patch at all.

What I actually meant is running a kernel without the DomU patch using Xen's hardware virtualization capabilities (if your hardware has VT or Pacifica support). If that would work nearly as well as using a paravirtualized guest, it should be possible to use the regular grsec+pax patch.

Best regards,

Thomas

[cormander]

What I actually meant is running a kernel without the DomU patch using Xen's hardware virtualization capabilities (if your hardware has VT or Pacifica support). If that would work nearly as well as using a paravirtualized guest, it should be possible to use the regular grsec+pax patch.

If I ever get any hardware that supports full virtualization (and I just might here in a few months), I'll be sure to run some benchmarks.

BTW I just posted another nopax patch (off of today's release):

http://www.ravencore.com/grsec/

[ralph]

im fiddling with this myself and cant get my kernels to boot as soon as i add grsec support to it.
these patches of yours, would you be able to put these out in .patch format, not rpms?
im using debian etch with my own 3.2.1 hypervisor and want to run my own customized 2.6.2x.x with grsec for security and flexibility.

[cormander]

You can get the src.rpm file and use rpm2cpio to extract the patches out of it; it contains not only the nopax patch, but pax updates to the grsecurity patch (if you ever use it on base hardware) and CVE patches applicable to 2.6.24.7. I also have about 300 backports in there - and the next one I'm going to post in the next week or so has an upward of 1400 patches from the upstream kernel that apply and build cleanly.

[cormander]

Actually I think I'll just post my git repository that I'm using for revision control, and it has all the patches in there.

I'll update my signature with the URL for it when that's done.

[ralph]

Thanks alot.

used alien to convert kernel-grsec-2.6.25.4-200806012125.src.rpm to .tgz and then patched my own kernel tree.
Just compiled and it worked at once.

Good work!

[spender]

As of the patches uploaded last night, 32bit xen domU should be compatible with grsecurity/PaX. Please let us know if you continue to have any issues.

-Brad