grsec 2.6.30.7-200909151852 problem with old_mmap?

[sfaerber]

Hi,

i just discovered another problem with grsec 2.6.30.7-200909151852, involving a MySQL-Server which can't start and exits with
"InnoDB: Error: pthread_create returned 11".
The MySQL-Server works perfectly on 2.6.30.7 without grsecurity and on 2.6.27.7 with grsecurity 200811291903.
I tried comparing straces to see where the problem is and found some (in my opinion) odd behaviour.

On 2.6.20.7 with 200909151852:

Code: Select all

7094  old_mmap(0xb3560000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa52c5000
7094  munmap(0xa52c5000, 131072)        = 0
7094  old_mmap(0xb3540000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa52c5000
7094  munmap(0xa52c5000, 131072)        = 0
7094  old_mmap(0xb3520000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa52c5000
7094  munmap(0xa52c5000, 131072)        = 0
7094  old_mmap(0xb3500000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa52c5000
7094  munmap(0xa52c5000, 131072)        = 0
... a few thousand calls and ends with
7094  kill(7093, SIGRTMIN)              = 0
7094  poll( <unfinished ...>
7093  <... rt_sigsuspend resumed> )     = ? ERESTARTNOHAND (To be restarted)
7093  --- SIGRTMIN (Unknown signal 32) @ 0 (0) ---
7093  sigreturn()                       = ? (mask now [HUP INT QUIT PIPE ALRM TERM TSTP RTMIN])
7093  write(2, "InnoDB: Error: pthread_create re"..., 42) = 42



On 2.6.27 with grsec it looks like this:

Code: Select all

6904  old_mmap(0x587e0000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x587e0000
6904  mprotect(0x587e0000, 4096, PROT_NONE) = 0
... startup continues


Any hints in the right direction, do you need more debug information?

Sebastian

[PaX Team]

On 2.6.20.7 with 200909151852:

Code: Select all

7094  old_mmap(0xb3560000, 131072, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xa52c5000
7094  munmap(0xa52c5000, 131072)        = 0

Any hints in the right direction, do you need more debug information?

wow, that's the stone age old linuxthreads trying to allocate a thread stack with its fucked up algorithm (something firefox 3.5.x users came to enjoy recently, i might add). since the beginning of ASLR i had a special case just to allow this scheme to work but i figured that, after so many years of NPTL out in the wild, there can't be anyone using linuxthreads, especially with a 2.6 kernel, so i removed that code for good and i have no plans to put it back. if you really can't upgrade to a newer mysql using a more recent glibc, then you'll have to turn ASLR off on the mysql binaries.