PaX and RLIMIT_AS

[Napfi]

Hi,
we are using Kernel 3.7.9 with grsec patch. Now we experience problems with an application that sets RLIMIT_AS. In "top" or /proc/PID/status a VMSIZE of around 80 MB is reported. However, if we start with pax_softmode=1, size is reported with about 9 MB. Even though the documentation states that RLIMIT_AS refers to the virtual memory space used, I expect that it's more the highest address allowed to use. Due to address space layout randomization in PaX, memory areas can very well lie beyond address [RLIMIT_AS]. I kernel and tools expect that address space is used linear from lowest to highest address, this might explain, why VMSIZE is reported wrongly with PaX enabled. Could you confirm my assumption? If not, what could cause these weird VMSIZE reportings?

Regards, Martin

[PaX Team]

RLIMIT_AS specifies the amount of virtual memory that a process can map, not the highest mappable address (the latter cannot be controlled by userland, it's a kernel invariant). as for RLIMIT_AS and ASLR interference, you should try a newer kernel as we've fixed some issues since.