PAX, Java and segfaults?

[JustMe]

We are using grsecurity-1.9.2 through 1.9.4 on both kernels 2.4.17 and 2.4.18 and it's been pretty stable and nice, but I'm wondering what suggestions you might have about some problems we've been having getting some Java compilers to work. Before we applied the patch, they worked fine, but we applied the patch on all our servers and none of the java compilers work and produce PAX errors now. We definitely want to keep using the patches, but we also have some user's complaining about not being able to compile even simply hello world applets.

If you run the java compiler, it (the compile process) will be killed or it will result in a segfault. Before I go on about things that are probably irrelevant, we use the kernels mentioned above, on both Redhat 6.2 and 7.2 and it has the same results. The memory and CPU vary, but are usually one or two CPU's that are at least 1 GHZ each, and 1 to 2 GIG's of RAM (I don't think that information matters though). Anyway, whenever someone tries to use the java compiler, here is the log in dmesg showing PAX and the error.

PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):28301, uid/euid: 886/886, EIP: 0811B350, ESP: BFFA821C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):30344, uid/euid: 886/886, EIP: 0811B350, ESP: BFFA19CC
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):6190, uid/euid: 886/886, EIP: 0811B350, ESP: BFFAEA6C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):15457, uid/euid: 886/886, EIP: 0811B350, ESP: BFF5C83C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):32262, uid/euid: 886/886, EIP: 0811B350, ESP: BFFB643C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):12335, uid/euid: 886/886, EIP: 0811B350, ESP: BFF70A9C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89
PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/javac(javac):20131, uid/euid: 0/0, EIP: 0811B790, ESP: BFFEB54C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 5c 67 0c 08 8b 47 08 89 68 18 89

You get the idea. I've tried running the chpax tool on it and I'm not sure if it helped or not. It did compile some simple hello world type of applets, but it still was killed and still segfaulted. I can't tell what's missing, to be honest, or what I need to do, since I'm not really too familiar with the java compiler -- but it did work without issue before the patch and I'm not sure what to do in regards to how PAX reacts to this. Any advice or guidance or links to any relevant information or experiences would be most helpful and appreciated. Thanks a lot, this is a great patch!

Regards,
Tim...

[PaX Team]

PAX: terminating task: /usr/local/jdk1.2/bin/i386/native_threads/java(java):32262, uid/euid: 886/886, EIP: 0811B350, ESP: BFFB643C
PAX: bytes at EIP: 55 89 e5 83 ec 1c c7 45 f0 54 62 0c 08 8b 47 08 89 68 18 89

You get the idea. I've tried running the chpax tool on it and I'm not sure if it helped or not. It did compile some simple hello world type of applets, but it still was killed and still segfaulted. I can't tell what's missing, to be honest, or what I need to do, since I'm not really too familiar with the java compiler -- but it did work without issue before the patch and I'm not sure what to do in regards to how PAX reacts to this.

this issue is a bit complex, let me explain it:

1. first of all, the java runtime (the libjvm.so library in particular) has at least two 'problems' that make it incompatible with PaX, therefore you have to 'chpax -p' it (the 'java' executable mentioned in the log itself, not that library). one problem is that libjvm.so has executable code in its data segment (some short FPU initialization code, i have no idea how that ended up in there), and the second one is that it generates executable code at runtime - fundamentally against what PaX would allow.

2. once you disable PaX on 'java', a new problem will pop up. actually, this is not that new as it occurs on a non-PaX kernel as well (someone else had reported this issue to me some time ago and yesterday i had time to track it down). this problem is a SIGSEGV which occurs in - where else - libjvm.so again (it is rather nondeterministic, probably a race condition). this time it is in real code, in a function called compiler_thread_loop__13CompileBroker (if you run 'java' from the commandline, you will see a nice report urging you to contact Sun with it).
although i tried all this on a newer JDK than what you seem to be using, i'm almost sure it is the same issue as yours and you can't do much about it besides trying a newer JDK or another JVM or maybe various command line options to disable JIT/multithreading/etc.

3. after the big VM ripout it took quite some time for the VM to stabilize (in particular, it used to have several race conditions, i remember that there were versions which behaved differently when the same PaX patch was applied to them). 2.4.17 was already better but still not perfect, the logs you reported show one such instance (i.e. they are not the problems i mentioned above in 1. and 2.). also, grsecurity 1.9.2 has an older version of PaX that uses invlpg (and therefore tracing as well) which have been known to screw up sometimes - no longer an issue as recent versions got rid of them.

in summary, my suggestion is to use 2.4.18 with a recent grsecurity patch and disable PaX on 'java', this will at least allow you to run java apps. however it won't be stable because of that race condition/SIGSEGV in libjvm.so.