Page 1 of 2

grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Mon Mar 08, 2010 10:19 am
by A-N
Hi,

i am using "grsecurity-2.1.14-2.6.33-201003071645.patch". After I reboot my Server to start the new Kernel my Box rebooted all 2 minutes.

I cant see nothing in the logs or console.

Anyone a idea?

Kernel: 2.6.33
OS: Linux Enterprise Server 11 (x86_64)

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Mon Mar 08, 2010 10:01 pm
by Grach
Maybe fsck was started to check some fs? :)

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Tue Mar 09, 2010 1:58 am
by cormander
Could you post a link to your .config you used, as well as (if possible) your vmlinux image?

I know spender and the Pax Team will want these to be able to troubleshoot it.

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Tue Mar 09, 2010 3:56 am
by A-N

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Tue Mar 09, 2010 8:05 am
by PaX Team
i'll need vmlinux (the uncompressed kernel) and preferably a bzImage with kernel symbols (grsec disables them by default, you'll have to re-enable the option and also disable HIDESYM). from a quick qemu run it seems to be some NULL lock pointer but i can't tell from the raw disasm what code is at play, i'll need the symbols ;).

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sat Mar 13, 2010 12:54 pm
by Sadako
I'm having more or less the same issue, where my desktop box will typically oops at some random point during the boot process and reboot.
Maybe 10% of the time it'll actually boot successfully, but then oops during shutdown, I didn't leave the box up long enough to see if it would "just happen" while idle.

This is with the hardened gentoo 2.6.33 kernel with the 201003071645 patch, but I also tried the more current 201003112028 patch, same thing.

I've collected a couple of the oops messages via netconsole, and I've uploaded them along with the config, bzImage, System.map and bzip'd vmlinux.bz2, hopefully with all the debugging info you'll need.
They should all be available here; http://hamiltonshells.ca/~hopeless/grsec

This post seems to be about the same issue, curiously also with a 2.6.32.9 kernel, and all three of us so far seem to be running amd64.
I've only run up to .8 in the 2.6.32 series patches, I'll probably try 201003071225 to see if I have the same issue with it too.

Please let me know if I can provide anything else to help diagnose this.

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sun Mar 14, 2010 9:36 am
by spender
Does disabling KERNEXEC in the meantime work around the problem? The PaX team needs to replace a power supply on their development machine, so they won't be able to address the problem until later this week.

-Brad

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sun Mar 14, 2010 1:09 pm
by specs
@spender:
Does disabling KERNEXEC in the meantime work around the problem? The PaX team needs to replace a power supply on their development machine, so they won't be able to address the problem until later this week.

When disabling the KERNEXEC the system starts and seems to work with the grsecurity-2.6.32.9-201003112025 patch.

@Sadako:
This post seems to be about the same issue, curiously also with a 2.6.32.9 kernel, and all three of us so far seem to be running amd64.
I've only run up to .8 in the 2.6.32 series patches, I'll probably try 201003071225 to see if I have the same issue with it too.

Since I needed a stable machine I prefer the 2.6.32.9-series for now. There is a small chance that the bug was introduced in 2.6.32.9-patch and 2.6.33-patch at the same time. However when I read about the bug it seemed different since my pc reboots before finishing init.

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sun Mar 14, 2010 2:57 pm
by spender
There was some cleaning up of the KERNEXEC code recently (prompted by some changes due to the upcoming UDEREF support for x64) that likely introduced the bug. If you'd like to help with some more debugging, you could try booting a 32bit kernel with KERNEXEC enabled.

-Brad

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sun Mar 14, 2010 4:10 pm
by Grach
the upcoming UDEREF support for x64

You're just awesome, guys.

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Sun Mar 14, 2010 6:17 pm
by specs
I did not have any trouble upgrading an Atom N270 or an Intel Core2 Duo to 2.6.32.9-201003112025 (i386).
The compilation of 2.6.33 is still running, but I don't expect problems.

Unfortunately running an i386-kernel on the AMD64 is too much work (it would take a complete new installation).
Other than that I will try to keep the AMD64-kernel up-to-date.

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Mon Mar 15, 2010 5:34 pm
by PaX Team
Sadako wrote:This is with the hardened gentoo 2.6.33 kernel with the 201003071645 patch, but I also tried the more current 201003112028 patch, same thing.
could you post dmesg from this newer kernel please? i changed some KERNEXEC asm code in-between and would like to see what error this newer version produces. also an unrelated request: i saw in your dmesg that you have an amd64 CPU with GB pages support, so could you please send me the kmaps output of your box (kmaps.c is in my home dir, comment lines at the top have compiling instructions)?

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Thu Mar 18, 2010 4:10 pm
by Sadako
PaX Team wrote:
Sadako wrote:This is with the hardened gentoo 2.6.33 kernel with the 201003071645 patch, but I also tried the more current 201003112028 patch, same thing.
could you post dmesg from this newer kernel please? i changed some KERNEXEC asm code in-between and would like to see what error this newer version produces.
Here's three logs with the 201003112028 patch, each after an oops;
http://hamiltonshells.ca/~hopeless/grsec/201003112028/

PaX Team wrote:also an unrelated request: i saw in your dmesg that you have an amd64 CPU with GB pages support, so could you please send me the kmaps output of your box (kmaps.c is in my home dir, comment lines at the top have compiling instructions)?
I tried this, but I'm unsure what argument to pass to it...

`kmaps 201000` only produces one line of output before being killed ("pgd: 000 d90ecb38c0694ebd 0000000000000000"),
`kmaps 0` gives a lot of output, as does `kmaps 100000` and a few other values, but most return nothing.
And whenever it does give output, it's killed with the following reported in dmesg;
Code: Select all
kmaps: Corrupted page table at address 7fe739869000
PGD 110487067 PUD 1104c5067 PMD 11049a067 PTE 800f0000f000e235
Bad pagetable: 000d [#1] SMP

This is under a vanilla kernel with /dev/mem filtering disabled.

What value should I be passing to it, or how do I determine it?

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Thu Mar 18, 2010 4:37 pm
by PaX Team
Sadako wrote:I tried this, but I'm unsure what argument to pass to it...
if you run it without arguments it'll print usage ;). on amd64 you will have to look up init_level4_pgt from System.map or /proc/kallsyms, e.g.,
Code: Select all
# grep init_level4_pgt /proc/kallsyms -w
ffffffff82e17000 D init_level4_pgt
then you'll invoke kmaps as:
Code: Select all
kmaps 2e17000
it'll produce nice colored output ;).

Re: grsecurity-2.1.14-2.6.33-201003071645.patch - reboots my box

PostPosted: Thu Mar 18, 2010 5:08 pm
by Sadako
Ahh...

I saw the help output, but I had no idea what the value of init_level4_pgt should be or where to find it.

Anyways, here you go; http://hamiltonshells.ca/~hopeless/grsec/kmaps.txt.bz2

The file has the colour escape codes, so it'll look all crappy in a text editor but cat or less should show it as intended.