2.4.33-grsec hangs on boot

[Hal9000]

I had 2.4.32-grsec running without problems.
I downloaded 2.4.33, patched against grsecurity-2.1.9-2.4.33-rc2-200607111636.patch, copied the .config file over and then made following changes to grsec in the menuconfig:
Address Space Protection:
[*] Sanitize all freed memory
[*] Prevent invalid userland pointer dereference
Filesystem Protections:
[*] Restrict to user only

Now, when booting the server with 2.4.33-grsec this happens:

Code: Select all

Restarting system.

PXELINUX 3.11 2005-09-02  Copyright (C) 1994-2005 H. Peter Anvin
Booting from local disk...

LILO 22.6.1 boot:
Loading 2.4.33-grsec..................
BIOS data check successful

The system hangs after this, nothing happens.
I resetted the machine and booted from 2.4.32-grsec again...

Any ideas of what might have happened?
greets
hal

[Hal9000]

PS that log was taken from the serial console

[PaX Team]

[*] Prevent invalid userland pointer dereference

Now, when booting the server with 2.4.33-grsec this happens:

Code: Select all

Restarting system.

PXELINUX 3.11 2005-09-02  Copyright (C) 1994-2005 H. Peter Anvin
Booting from local disk...

LILO 22.6.1 boot:
Loading 2.4.33-grsec..................
BIOS data check successful

The system hangs after this, nothing happens.

what happens if you disable UDEREF?

[Hal9000]

I deactivated that option and prepared the new kernel, I will reboot as soon as I can (it's a production machine) and let you know.

[Hal9000]

Yes sir, the kernel boots now without UDEREF.
However, I don't have any virtualization... but I got this network boot or whatever it is called (PXELINUX)... perhaps this is causing problems?
hal

[PaX Team]

Yes sir, the kernel boots now without UDEREF.
However, I don't have any virtualization...

or so you thought... maybe your system has swallowed a trendy bluepill?

but I got this network boot or whatever it is called (PXELINUX)... perhaps this is causing problems?
hal

interesting, i don't think the source of the kernel image matters, if UDEREF hangs then it means that the main kernel image got executed but did something unexpected very early that caused the hang. have you got other systems booting the same kernel or at least via PXE? if so, do they have the same issue with UDEREF? it'd be nice if you could reproduce this on a non-production box else debugging it will be very time consuming...

[Hal9000]

Yes sir, the kernel boots now without UDEREF.
However, I don't have any virtualization...

or so you thought... maybe your system has swallowed a trendy bluepill?

I wish it had

but I got this network boot or whatever it is called (PXELINUX)... perhaps this is causing problems?
hal

interesting, i don't think the source of the kernel image matters, if UDEREF hangs then it means that the main kernel image got executed but did something unexpected very early that caused the hang. have you got other systems booting the same kernel or at least via PXE? if so, do they have the same issue with UDEREF? it'd be nice if you could reproduce this on a non-production box else debugging it will be very time consuming...

Unfortunatelly this is the only box I got. I will get a second one with similar setup (but Opteron instead of P4) in about two weeks, and could try UDEREF on that one.

[mtg]

We experienced exactly the same symptom with 2.4.33-grsec (2.1.9-200608131429) here on an old Dual PPro (HP LHPro).
No PXE - just local boot from a scsi-disk with lilo. With disabled UDEREF, it boots 2.4.33-grsec just fine.
On other machines (Athlon, P4/Xeon [SMP and Single]) no problems with enabled UDEREF.

[PaX Team]

We experienced exactly the same symptom with 2.4.33-grsec (2.1.9-200608131429) here on an old Dual PPro (HP LHPro).
No PXE - just local boot from a scsi-disk with lilo. With disabled UDEREF, it boots 2.4.33-grsec just fine.
On other machines (Athlon, P4/Xeon [SMP and Single]) no problems with enabled UDEREF.

can you do some debugging (will need a few reboots and patching the kernel with printk)? also, what happens if you configure and boot a UP kernel on the SMP box or just boot with one CPU on an SMP kernel (maxcpus=1)? i'd also like to see your .config, just in case.

[mtg]

We experienced exactly the same symptom with 2.4.33-grsec (2.1.9-200608131429) here on an old Dual PPro (HP LHPro).
No PXE - just local boot from a scsi-disk with lilo. With disabled UDEREF, it boots 2.4.33-grsec just fine.
On other machines (Athlon, P4/Xeon [SMP and Single]) no problems with enabled UDEREF.

can you do some debugging (will need a few reboots and patching the kernel with printk)? also, what happens if you configure and boot a UP kernel on the SMP box or just boot with one CPU on an SMP kernel (maxcpus=1)? i'd also like to see your .config, just in case.

Well, unfortunately it's a production machine, but I might be able to try some things in the evening (localtime Germany). I can definitely try the reboot with maxcpus=1 and with an UP-Kernel. Could you give me a hint where to put the printks?
Should I post the .config here? Or send an email? The whole thing? Or just the grsecurity part?

Thanks.

[Hal9000]

oh yeah, just for info, I am running an SMP kernel as well (have a single P4, but with HyperThreading enabled)
hal

[PaX Team]

Well, unfortunately it's a production machine, but I might be able to try some things in the evening (localtime Germany). I can definitely try the reboot with maxcpus=1 and with an UP-Kernel. Could you give me a hint where to put the printks?
Should I post the .config here? Or send an email? The whole thing? Or just the grsecurity part?

thanks for the help. let's first see the result of the single CPU boots, if that works, that already gives me a hint as to where to look. also, best is to email all your .config to me. as for the printk, do you have direct console access (so that writing to the video memory directly will be visible to you) or only remote serial? this code is so early in boot that realistically, only the first method works.

[mtg]

Well, unfortunately it's a production machine, but I might be able to try some things in the evening (localtime Germany). I can definitely try the reboot with maxcpus=1 and with an UP-Kernel. Could you give me a hint where to put the printks?
Should I post the .config here? Or send an email? The whole thing? Or just the grsecurity part?

thanks for the help. let's first see the result of the single CPU boots, if that works, that already gives me a hint as to where to look. also, best is to email all your .config to me. as for the printk, do you have direct console access (so that writing to the video memory directly will be visible to you) or only remote serial? this code is so early in boot that realistically, only the first method works.

Yes, I have console access. Is the email-address on pax.grsecurity.net the one?

As a sidenote: Here a 2.4.33-grsec SMP runs perfectly with UDEREF on a Single P4/HT.

[mtg]

oh yeah, just for info, I am running an SMP kernel as well (have a single P4, but with HyperThreading enabled)
hal

Do you have MD/Raid compiled into the kernel (CONFIG_MD*/CONFIG_BLK_DEV_MD)? That's the only configuration difference I could find.

[aldee]

I've been running into a similar problem (stops booting before it gets to open log files). The machine I was facing problems with is not a production machine, unfortunately I don't have a serial console or physical access. Otherwhise I'd be glad to help with debugging (I'm not a kernel hacker, however, I'd probably manage to apply some patches if someone told me where and what). Anyways, I fear that this would be no use without screen output.

I do not use a SMP kernel and this is not a multiprocessor or HT system either (Celeron 1200 actually). No, this is not a virtual server ;-).

Unlike for the previous posters, I had to disable both new features, MEMORY_SANITIZE and UDEREF for the machine to boot properly. Well, actually I didn't check if it works if only UDEREF is enabled. It didn't work with only MEMORY_SANITIZE being enabled. Tested on a vanilla 2.4.33 kernel with only grsec patches applied.