crashing system since applying the new patch

[rayden]

Hello,

First of all i would like to thank the entire grs team for creating grsecurity, its extremely cool and usefull.

Last friday i upgraded some of my servers to Linux kernel 2.4.26, and it all worked fine, after that i applied the grsecurity patch. Since then ive been expierencing weird 'box freezings'

I have tested it on several boxes mostly Fedora Core 1, but it just keeps random crashing, no error of some sort is displayed. The box just freezes and needs a reboot to work properly again.

When i install a clean kernel (2.4.26 without grsec) it just works fine.

Did anyone of you had any similliar problems? Or have any solution for this problem?

Thanks in advance for any reply.

[Lam]

I have similar problems with 2.4.26-grsec- my production system crashed (or rather, freezed) two times since saturday, one time it oopsed in kswapd, continued working and only refused to reboot because swapoff hung. Besides that, many random programs started to take my whole CPU time in some infinite loop (ps -o wchan doesn't report anything under 2.4.26-grsec, so I can't really tell why do they hang).

I tried 2.4.26 with grsecurity 2.0 at first, then switched to 2.4.26 with grsecurity 1.9.15 with the same configuration which worked until saturday on 2.4.24-grsec-1.9.13, then tried to switch some options off (like kernel stack randomization), then upgraded glibc to Fedora testing's 2.3.3, then even changed some system hardware and run memtest86 on the memory. 2.4.24-grsec-1.9.13 was working perfectly fine, 2.4.26 without grsec is working fine right now, 2.4.26 with any grsecurity version seems broken.

Seems like a grsecurity problem to me. I'm also Red Hat user, maybe some of their glibc patches collide with newer grsecurity? (2.4.24+1.9.13 worked fine) Still, even if it's working on other distributions, I guess hard freezes and oopses are kernel bug, just triggered by something we have on our systems.

[PaX Team]

I have similar problems with 2.4.26-grsec- my production system crashed (or rather, freezed) two times since saturday, one time it oopsed in kswapd, continued working and only refused to reboot because swapoff hung. Besides that, many random programs started to take my whole CPU time in some infinite loop (ps -o wchan doesn't report anything under 2.4.26-grsec, so I can't really tell why do they hang).

can you post your .config (pax/grsec bits at least), the details of the oops (preferably decoded via ksymoops)? also, you can try to attach to the badly behaving processes with strace and see what they're doing.

[escriba]

Hey there, i had the same problem.

It happened already in two servers.This servers had 2.4.25-grsec and last weekend i had updated my servers to 2.4.26-grsec, and they crash every day since that update. And one curious think.. it happened only on redhat 7.3(Red Hat Linux release 7.3 (Valhalla)).

My Grsecurity kernel configuration:

#
# Grsecurity
#
CONFIG_GRKERNSEC=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_SHA256=y
# CONFIG_GRKERNSEC_LOW is not set
# CONFIG_GRKERNSEC_MID is not set
# CONFIG_GRKERNSEC_HI is not set
CONFIG_GRKERNSEC_CUSTOM=y

#
# PaX Control
#
CONFIG_GRKERNSEC_PAX_SOFTMODE=y
CONFIG_GRKERNSEC_PAX_EI_PAX=y
CONFIG_GRKERNSEC_PAX_PT_PAX_FLAGS=y
# CONFIG_GRKERNSEC_PAX_NO_ACL_FLAGS is not set
# CONFIG_GRKERNSEC_PAX_HAVE_ACL_FLAGS is not set
CONFIG_GRKERNSEC_PAX_HOOK_ACL_FLAGS=y

#
# Address Space Protection
#
CONFIG_GRKERNSEC_PAX_NOEXEC=y
CONFIG_GRKERNSEC_PAX_PAGEEXEC=y
CONFIG_GRKERNSEC_PAX_SEGMEXEC=y
CONFIG_GRKERNSEC_PAX_EMUTRAMP=y
CONFIG_GRKERNSEC_PAX_EMUSIGRT=y
CONFIG_GRKERNSEC_PAX_MPROTECT=y
# CONFIG_GRKERNSEC_PAX_NOELFRELOCS is not set
CONFIG_GRKERNSEC_PAX_ASLR=y
CONFIG_GRKERNSEC_PAX_RANDKSTACK=y
CONFIG_GRKERNSEC_PAX_RANDUSTACK=y
CONFIG_GRKERNSEC_PAX_RANDMMAP=y
CONFIG_GRKERNSEC_PAX_RANDEXEC=y
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_RTC=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_HIDESYM=y

#
# Role Based Access Control Options
#
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30

#
# Filesystem Protections
#
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
# CONFIG_GRKERNSEC_LINK is not set
# CONFIG_GRKERNSEC_FIFO is not set
# CONFIG_GRKERNSEC_CHROOT is not set

#
# Kernel Auditing
#
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
CONFIG_GRKERNSEC_EXECLOG=y
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_CHROOT_EXECLOG=y
CONFIG_GRKERNSEC_AUDIT_CHDIR=y
CONFIG_GRKERNSEC_AUDIT_MOUNT=y
CONFIG_GRKERNSEC_AUDIT_IPC=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_TIME=y
# CONFIG_GRKERNSEC_PROC_IPADDR is not set
# CONFIG_GRKERNSEC_AUDIT_TEXTREL is not set

#
# Executable Protections
#
# CONFIG_GRKERNSEC_EXECVE is not set
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_RANDPID=y
# CONFIG_GRKERNSEC_TPE is not set

#
# Network Protections
#
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_RANDISN=y
CONFIG_GRKERNSEC_RANDID=y
CONFIG_GRKERNSEC_RANDSRC=y
CONFIG_GRKERNSEC_RANDRPC=y
# CONFIG_GRKERNSEC_SOCKET is not set

#
# Sysctl support
#
CONFIG_GRKERNSEC_SYSCTL=y

#
# Logging options
#
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

i tryid to see if there were some errors on syslog log's, but nothing.

any furtor information needed just ask, i just want to put my servers ok. tks

[yaplik]

Hi,
I got similar problems with my 2.6.5-wolk3.0-rc2-grsec2 kernel, when i tried to boot it.
initrd works fine to nash handledevfs (its last message) then /sbin/init is going to executed but the box freeze, with grsec2 enabled i got (strange) CORE_LIMIT message for /sbin/hotplug by [kevent/0] , with grsec2 disabled nothing shows and box freeze too, so i fiddle with config and isolatethat problem with small config (no sound, graphic, etc. cards) and grsec2 disabled and
1. - working
#CONFIG_GRKERNSEC_PAX_PAGEEXEC
#CONFIG_GRKERNSEC_PAX_SEGMEXEC
#CONFIG_GRKERNSEC_PAX_ASLR
2. - working
CONFIG_GRKERNSEC_PAX_PAGEEXEC=y
#CONFIG_GRKERNSEC_PAX_SEGMEXEC
#CONFIG_GRKERNSEC_PAX_ASLR
3. - NOT working
#CONFIG_GRKERNSEC_PAX_PAGEEXEC
CONFIG_GRKERNSEC_PAX_SEGMEXEC=y
#CONFIG_GRKERNSEC_PAX_ASLR

so segmexec could be broken (on my i386) and I got CORE_LIMIT msg when /sbin/hotplug got SIGSEGV and wanna to set core filelimit before coredump
I hope it was already fixed in cvs
-- Yaplik
pubkey: 6970 3B7C 0F72 07DD 1825 B4FB CC27 E7D5 2CDE 2300

[PaX Team]

so, after some bug hunting i think you guys have different issues, so let's see them one by one. first the easy one, yaplik: it's a glibc bug where the AT_SYSINFO auxiliary vector is ignored and hence syscalls cause a general protection fault and the kernel kills the app. in theory enabling NOVSYSCALL would fix it were it not for another glibc bug (or at least in debian but i've just got a report from another distro as well) that will make it not work for another reason (glibc would fall back on the kernel provided sigreturn trampoline that happens to be put on the non-executable stack) - talk about catch-22 . so until your glibc is fixed, you can only use PAGEEXEC. suggested reading is at http://bugs.gentoo.org/show_bug.cgi?id=49014.

for the other users i don't have much to offer, since your problems are less systematic than the one above so i guess it's not related to this glibc behaviour. it'd be nice if you could disable various PaX features (RANDKSTACK comes to mind, it can very well cause these random lockups, although it won't help Lam who already tried that) and see if it helps. a question for escriba: any specific reason why you had to enable EMUSIGRT (just asking 'cos it's a security hazard, you'd be better off by not using it)?