'go' race tests fail

[wizeman]

Hi,

I'm running grsec kernel patch 2.9.1-3.2.52-201311182331, and ran into failures in the 'go' race tests:

Code: Select all

$ go test -race -short std
(...)
==5784== ERROR: Failed to allocate 0x400000 (4194304) bytes at address 0x130800000000 (12)
==5784== ERROR: Failed to allocate 0x80000 (524288) bytes at address 0x600000000000 (12)
FATAL: ThreadSanitizer can not mmap thread trace
(...)


I filed a bug with the go developers thinking they weren't using MAP_FIXED, but it turns out they are using it.
Here's an strace of the mmap calls in one of the tests:

Code: Select all

Process 5784 attached
[pid  5784] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d0423c000
[pid  5784] mmap(NULL, 2208688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x38d03e02000
[pid  5784] mmap(0x38d04018000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x38d04018000
[pid  5784] mmap(0x38d0401a000, 13232, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x38d0401a000
[pid  5784] mmap(NULL, 3865920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x38d03a52000
[pid  5784] mmap(0x38d03df8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a6000) = 0x38d03df8000
[pid  5784] mmap(0x38d03dfe000, 15680, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x38d03dfe000
[pid  5784] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d0423b000
[pid  5784] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d0423a000
[pid  5784] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d04239000
[pid  5784] mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d04219000
[pid  5784] mmap(NULL, 268480448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38cf3a47000
[pid  5784] mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d04209000
[pid  5784] mmap(0xc000000000, 65536, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc000000000
[pid  5784] mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041f9000
[pid  5784] mmap(0xc200000000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d040f9000
[pid  5784] mmap(0xc200000000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc200000000
[pid  5784] mmap(0xc1ffff0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041e9000
[pid  5784] mmap(0xc1ffff0000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xc1ffff0000
[pid  5784] mmap(0x130800000000, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid  5784] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041f5000
[pid  5784] mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041d9000
[pid  5784] mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041c9000
[pid  5784] mmap(NULL, 147456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041a5000
[pid  5784] mmap(0x600000000000, 524288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
[pid  5784] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041a1000
[pid  5784] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x38d041a1000
[pid  5784] +++ exited with 1 +++
[pid  4133] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=5784, si_status=1, si_utime=0, si_stime=0} ---
==5784== ERROR: Failed to allocate 0x400000 (4194304) bytes at address 0x130800000000 (12)
==5784== ERROR: Failed to allocate 0x80000 (524288) bytes at address 0x600000000000 (12)
FATAL: ThreadSanitizer can not mmap thread trace
FAIL    time    0.018s


I didn't see anything strange in the kernel logs.

So apparently, mmap() is failing with ENOMEM when mapping a 4MB region (and later a 512 KB region) with MAP_FIXED, even though MAP_NORESERVE is also specified.

Any idea why mmap() is failing (and how can I solve this problem)?

[PaX Team]

you're probably using UDEREF which on amd64 restricts the userland address space size to 42 bits (down from 47) and those addresses fall outside of it. thread sanitizer is not the only one that hardcodes such addresses and the fix should be to make these assumptions about the address space size configurable, but i don't think earlier attempts by gentoo guys went anywhere.

[wizeman]

you're probably using UDEREF which on amd64 restricts the userland address space size to 42 bits (down from 47) and those addresses fall outside of it. thread sanitizer is not the only one that hardcodes such addresses and the fix should be to make these assumptions about the address space size configurable, but i don't think earlier attempts by gentoo guys went anywhere.

Unfortunately it's not so easy to patch the 'go' runtime to modify the hardcoded addresses, as the ThreadSanitizer code seems to be distributed as a binary blob even in go's source code tarball. Apparently the ThreadSanitizer code actually comes from LLVM... and also unfortunately, they don't even seem to document which tag/commit it came from.

I suppose it wouldn't be possible to disable UDEREF for a given binary for compatibility reasons (like the PaX flags), i.e. to increase the address space size to 47 bits just for that process, but without compromising UDEREF protections when running other processes, right?

I'm sorry for not being very familiar with how UDEREF works...

Thanks!

[PaX Team]

i thought that all the sanitizer code was open source, somehow i doubt that the gcc guys would accept blobs . as for the UDEREF side, it's a kernel self-protection feature, not something you can (or even want to) control on a per-process basis.