GrSecurity - Meltdown and Panic on Reboot

[MattF]

I had my GrSecurity kernel running fine a good while. Then on midnight of Thursday 13th November it stopped working correctly.

ov 14 00:17:35 cb kernel: PAX: terminating task: /bin/bash(sh):26692, uid/euid:
0/0, PC: 202af440, SP: 5bedefd4
Nov 14 00:17:35 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 a0 29 20
Nov 14 00:17:39 cb kernel: PAX: From 68.83.72.101: terminating task: /usr/sbin/e
xim(exim):9360, uid/euid: 47/0, PC: 23c6a440, SP: 58056244
Nov 14 00:17:39 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 50 c5 23
Nov 14 00:17:44 cb kernel: PAX: From 68.33.78.79: terminating task: /usr/sbin/ex
im(exim):29475, uid/euid: 47/0, PC: 2dde9440, SP: 5d62d1e4
Nov 14 00:17:44 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 40 dd 2d
Nov 14 00:17:45 cb kernel: PAX: From 213.106.167.244: terminating task: /usr/bin
/perl(eximstatspass):18644, uid/euid: 0/0, PC: 271b8440, SP: 5d8d63d4
Nov 14 00:17:45 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 30 1a 27
Nov 14 00:17:52 cb kernel: PAX: From 211.110.179.73: terminating task: /usr/sbin
/exim(exim):6416, uid/euid: 47/0, PC: 2d061440, SP: 5a21d484
Nov 14 00:17:52 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 c0 04 2d
Nov 14 00:17:59 cb kernel: PAX: From 213.106.167.244: terminating task: /bin/bas
h(system):29493, uid/euid: 0/0, PC: 2fb78440, SP: 5e2f1284
Nov 14 00:17:59 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 30 b6 2f
Nov 14 00:17:59 cb kernel: PAX: From 213.106.167.244: terminating task: /bin/bas
h(system):9421, uid/euid: 0/0, PC: 2b034440, SP: 59fb69e4
Nov 14 00:17:59 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 f0 01 2b
Nov 14 00:17:59 cb kernel: PAX: From 213.106.167.244: terminating task: /bin/bas
h(system):26680, uid/euid: 0/0, PC: 247f8440, SP: 5d1aecb4
Nov 14 00:17:59 cb kernel: PAX: bytes at PC: cd 80 c3 00 0e 00 00 00 b0 02 00 00
04 00 00 00 94 30 7e 24

The /var/log/messages is full of hundreds of these lines, I couldn't log into SSH, after asking for the username and password it returned a sig 9, presumably bash being killed off, also any service that authenicated such as email or FTP was not working correctly.

I ask the data center to investigate and reboot the server, they reboot the server however said it paniced on reboot, or didn't boot properly, so they reverted it back to their "approved stock kernel" - nasty performance and told me to leave it alone. I miss the GrSecurity kernel, however they've enforced a $75 reboot/kernel-selection fee now for any other kernel than their approved one so I am very warey about trying it again.

The thing is that cPanel/WHM (a control panel that runs on top of Red Hat) performs it updates just after midnight, the same time things went belly up, but from what I can make out it only updated Perl packages and no other core stuff, but it may have done.

Any ideas what could have caused this?

What options shall I select to completely disable PAX in kernel-config?

[PaX Team]

I had my GrSecurity kernel running fine a good while. Then on midnight of Thursday 13th November it stopped working correctly.

the bug you're experiencing was fixed by RedHat the day i had reported it to them (you should be subscribed to the grsec list!): https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=109918 and http://marc.theaimsgroup.com/?l=bugtraq&m=106876158031431&w=2. please upgrade to that package and then you can use grsec/PaX again.

What options shall I select to completely disable PAX in kernel-config?

if you really want to do that, disable the PAGEEXEC/SEGMEXEC/ASLR options.

[MattF]

I have the following at the moment:

glibc-common-2.3.2-27.9.7
glibc-devel-2.3.2-27.9.7
glibc-2.3.2-27.9.7

So I have rpm force downgrade to 2.3.2-27.9.6 then reboot into grSecuriy everything should be okay?

Or alternatively is there a later version of 2.3.2-27.9.7 which resolves this? Where can I get it?

[PaX Team]

glibc-common-2.3.2-27.9.7
glibc-devel-2.3.2-27.9.7
glibc-2.3.2-27.9.7

So I have rpm force downgrade to 2.3.2-27.9.6 then reboot into grSecuriy everything should be okay?

Or alternatively is there a later version of 2.3.2-27.9.7 which resolves this? Where can I get it?

those packages are the supposedly fixed ones (at least the original user who notified me is running with them), so i don't know how they can still fail on your system. in any case, do not downgrade to 2.3.2-27.9.6 because that was the first version of this security update which broke everything (and i thought you had installed that as well). all i can suggest is that you report your problems to RedHat.

[MattF]

I've done a bit more investigating and it seems 27.9.7 was released on the 13th by RedHat. At 00:02 on the 14th cPanel does it automatically system updated and updated to 27.9.7. And then the problem began.

For the time being I'm going to disable PAX and recompile, I hate this stock kernel and even though I haven't got the PAX features I've still got better security than a stock kernel I'd imagine.

Will the following safely disable PAX.

# Address Space Protection
#
CONFIG_GRKERNSEC_PAX_NOEXEC=n
CONFIG_GRKERNSEC_PAX_PAGEEXEC=n
CONFIG_GRKERNSEC_PAX_SEGMEXEC=n
CONFIG_GRKERNSEC_PAX_EMUTRAMP=n
CONFIG_GRKERNSEC_PAX_EMUSIGRT=n
CONFIG_GRKERNSEC_PAX_MPROTECT=n
CONFIG_GRKERNSEC_PAX_NOELFRELOCS=n
CONFIG_GRKERNSEC_PAX_ASLR=n
CONFIG_GRKERNSEC_PAX_RANDKSTACK=n
CONFIG_GRKERNSEC_PAX_RANDUSTACK=n
CONFIG_GRKERNSEC_PAX_RANDMMAP=n
CONFIG_GRKERNSEC_PAX_RANDEXEC=n
CONFIG_GRKERNSEC_KMEM=n
CONFIG_GRKERNSEC_IO=n
CONFIG_GRKERNSEC_PROC_MEMMAP=n
CONFIG_GRKERNSEC_HIDESYM=n

#

[PaX Team]

Will the following safely disable PAX.

yes but do it in menuconfig/xconfig, not .config directly.