Memory limitations using grsecurity

[Fill]

Hi everyone,

I am still kind of a noob in this unix world, and recently I've been running some tests on my home linux server, just to learn.

I have compiled my kernel with grsecurity with maximum security (i.e., HIGH), and maybe I've exaggerated. I am unable to run several apps, for example, javac or valgrind. These are the errors I get:

Code: Select all

filipe@BOX4:~$ javac teste.java
OpenJDK Client VM warning: Attempt to allocate stack guard pages failed.
Error occurred during initialization of VM
Could not reserve enough space for code cache
filipe@BOX4:~$


And, with valgrind...

Code: Select all

filipe@BOX4:~$ valgrind --memory-check=full main
==2450==
    Valgrind's memory management: out of memory:
       newSuperblock's request for 4194304 bytes failed.
       7401472 bytes have already been allocated.
    Valgrind cannot continue.  Sorry.

    There are several possible reasons for this.
    - You have some kind of memory limit in place.  Look at the
      output of 'ulimit -a'.  Is there a limit on the size of
      virtual memory or address space?
    - You have run out of swap space.
    - Valgrind has a bug.  If you think this is the case or you are
    not sure, please let us know and we'll try to fix it.
    Please note that programs can take substantially more memory than
    normal when running under Valgrind tools, eg. up to twice or
    more, depending on the tool.  On a 64-bit machine, Valgrind
    should be able to make use of up 32GB memory.  On a 32-bit
    machine, Valgrind should be able to use all the memory available
    to a single process, up to 4GB if that's how you have your
    kernel configured.  Most 32-bit Linux setups allow a maximum of
    3GB per process.

    Whatever the reason, Valgrind cannot continue.  Sorry.

filipe@BOX4:~$


So I was wondering if this is related to grsecurity policy? If there is some kind of limitation or something, and how can I disable it (sorry for my ignorance ).

Here are some details about my box:

Code: Select all

filipe@BOX4:~$ uname -a
Linux BOX4 2.6.32.43-grsec #4 SMP Mon Aug 22 14:21:01 WEST 2011 i686 GNU/Linux
filipe@BOX4:~$


Thanks in advance,
Fill

[Undine]

I'm wondering why your server is booting after security level = HIGH I always had OOM at boot time just before init with this config option ("Out of Memory and No Killable Processes..."). Maybe fixed now, I don't know.
Java badly worked with grsecurity, often with segfaults (thread deaths with speed about 1 thread per 2 secs, shown in dmesg).
I think developer tools are always need some exceptions to be made with paxctl or execstack. Look at dmesg for more info.
Sorry for my english

[PaX Team]

you can always try to strace -f the targets and see what syscalls failed, but my guess is that it's the MPROTECT restrictions from PaX that'll you have to disable on the affected executables (whose paths you'll see from the execve syscalls in the strace output).

[yobert]

In case anyone finds this thread... The command to set the PAX headers on java so it runs well with grsecurity is:

paxctl -c -m /usr/bin/java

I put this in /etc/rc.local, but really you only need to run it when you get a new version of java installed or something.