attempted resource overstep

[PaX Team]

I knew you were going to ask that
Ok, already re-enabled randomization and it now fails again:

ok, i'm confused now . first you said that you'd disabled all but PaX in the kernel and it worked, and now you say that you re-enabled randomizaton (wasn't it already enabled?) and it fails... so can you summarize what PaX/grsec options were enabled in the kernel and on java and what worked/failed (something like a simple table would make it clear)?

[supermike]

Sorry, I meant I did as you suggested, disabled all but PAX, then also disabled randomization. After enabling the randomization options it failed.

Now I have re-enabled all my previous grsec options and PAX, but without the randomization and it works.
So the problem is caused by one or more of:
CONFIG_GRKERNSEC_PAX_ASLR=y
CONFIG_GRKERNSEC_PAX_RANDKSTACK=y
CONFIG_GRKERNSEC_PAX_RANDUSTACK=y
CONFIG_GRKERNSEC_PAX_RANDMMAP=y

[PaX Team]

Now I have re-enabled all my previous grsec options and PAX, but without the randomization and it works.
So the problem is caused by one or more of:
CONFIG_GRKERNSEC_PAX_ASLR=y
CONFIG_GRKERNSEC_PAX_RANDKSTACK=y
CONFIG_GRKERNSEC_PAX_RANDUSTACK=y
CONFIG_GRKERNSEC_PAX_RANDMMAP=y

so then it's at most 3 kernel recompilations and we'd know for sure... would you mind?

[supermike]

jeez I need a faster compter

well, a good guess:
disabled only CONFIG_GRKERNSEC_PAX_RANDUSTACK and it's still working

[PaX Team]

disabled only CONFIG_GRKERNSEC_PAX_RANDUSTACK and it's still working

so it must be RANDMMAP then. can you run java and the other one through strace -f -F and send the output in email please?

[MJatIFAD]

I too get the grsec attempted resource overstep with java using an off the shelf Mandrake 9.2 secure kernel 2.4.22-21mdk. The symptoms described in this thread sum up my problem very well. I am not an expert and I had difficulties to understand what the fix for the problem is, but as far as I understood I need to rebuild the kernel with the grsec patch and CONFIG_GRKERNSEC_PAX_RANDUSTACK disabled. Is this correct or is there a simpler solution?

[MJatIFAD]

Rebuild the kernel with CONFIG_GRKERNSEC_PAX_RANDUSTACK and it seemed to have some effect, but not enough to remove the "grsec: attempted resource overstep by requesting 4096 for RLIMIT_CORE against limit 0" error. Previously no java processes were started when I started tomcat but now some do get to live without the parent process, which is still killed. I also tried to disable CONFIG_GRKERNSEC_PAX_RANDMMAP without effect. I guess my problems are not exactly the same. What do I try next?

[MJatIFAD]

It seems that the grsecurity version in Mandrake 9.2 with the kernel source distribution 2.4.22-21mdk is different than the one previously discussed in this thread. I found that the compile options in the .config file were named differently, which I did not notice earlier because I use xconfig to disable/enable them, and here they have other but similar display names.

However, I found out that I had to disable the compile option CONFIG_GRKERNSEC_PROC_MEMMAP to get rid of my Java problem, but I now discovered other grsec errors on kdeinit, cleanup and procmail. In these cases it is hard to tell whether it has some visible effect on my system workings. It seems to me that the grsecurity patch in Mandrake 9.2 with the kernel source distribution 2.4.22-21mdk is not working very well with many standard system parts. Maybe I should try another kernel source distribution or maybe just I need to add some extra acl definitions? I am still new to this stuff, so I would apreciate if someone could give me some feedback on this.

[PaX Team]

Maybe I should try another kernel source distribution or maybe just I need to add some extra acl definitions? I am still new to this stuff, so I would apreciate if someone could give me some feedback on this.

what you should try is always the latest vanilla kernel and grsecurity. if that still gives you problems, then post as much info as you can find out, among others your .config, your ACLs, your relevant logs, etc.