mmap failure with grsec 4.3.5

[benoit@scylladb.com]

Hi,

I am running grsecurity for personal convenience on my workstation.

While trying to run scylladb the product made by my current customer I got the following trace.

The setup is an fedora 23 lxc container running on an ubuntu 4.3.5 grsec host.

mprotect(0x348c50da000, 12288, PROT_READ) = 0
mprotect(0x348c52de000, 4096, PROT_READ) = 0
mprotect(0x1c2d000, 4096, PROT_READ) = 0
mprotect(0x348c5500000, 4096, PROT_READ) = 0
munmap(0x348c54f1000, 37083) = 0
set_tid_address(0x348c54db750) = 9874
set_robust_list(0x348c54db760, 24) = 0
rt_sigaction(SIGRTMIN, {0x348c13bdb20, [], SA_RESTORER|SA_SIGINFO, 0x348c13c8430}, NULL, = 0
rt_sigaction(SIGRT_1, {0x348c13bdbb0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x348c13c8430}, NULL, = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x38d24f612e0) = -1 ENOENT (No such file or directory)
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=49077591, f_bfree=41930274, f_bavail=39431510, f_files=12476416, f_ffree=11989493, f_fsid={861819776, -1156077281}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
mmap(NULL, 35184372088832, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, = 0
tgkill(9874, 9874, SIGABRT) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=9874, si_uid=0} ---
+++ killed by SIGABRT (core dumped) +++
Abandon (core dumped)

So the Scylla database is failing to mmap anonymous memory and core dumping.

Any idea why mmap is failling this way ?

Best regards

Benoît Canet

[PaX Team]

does this fail on a vanilla kernel? other than that, we don't restrict mmap sizes or rlimits or overcommit ourselves, so maybe check those first.

edit: actually there's one restriction for such big sizes, if you have UDEREF enabled then the userland address space size will be reduced to 42 bits (from 47) thus a 45 bit allocation won't fit.

[benoit@scylladb.com]

Ok I will test vanilla and lxc on friday.
(I am currently on travel an jailed in the container).

and also removing uderef.

Thanks

Best regards

Benoît

[PaX Team]

if you know you had UDEREF in the kernel then i'm pretty sure that's the limit you ran into.

[benoit@scylladb.com]

I previously just disabled my grsecurity kernel and now a few month later I tried again with 4.6.4
while removing UDEREF.

Same behavior.

[PaX Team]

on amd64 KERNEXEC also reduces the userland address space size (as it's a side effect of the underlying per-cpu pgd feature).

[benoit@scylladb.com]

Thanks a lot it work when removing KERNEXEC and UDEREF from the kernel config.