Can't load innodb plugin - segfault

[closet geek]

Hi,

I'm using MySQL 5.1.56 on a 32bit 2.6.32.15 kernel with grsec + pax.

When I try and follow the instructions here: http://dev.mysql.com/doc/refman/5.1/en/ ... nnodb.html I always get a segmentation fault:

Code: Select all

110711 19:31:45 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=419430400
read_buffer_size=2097152
max_used_connections=0
max_threads=200
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3794186 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x22)[0x849fef2]
/usr/sbin/mysqld(handle_segfault+0x3ac)[0x81f2dac]
[0xa9720400]
/usr/lib/mysql/plugin/ha_innodb_plugin.so[0x80256e73]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x156)[0x82d79a6]
/usr/sbin/mysqld[0x835a11a]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x520)[0x835dc80]
/usr/sbin/mysqld[0x81f4ce7]
/usr/sbin/mysqld(main+0x5e5)[0x81f8885]
/lib/tls/libc.so.6(__libc_start_main+0xd3)[0xa9578e93]
/usr/sbin/mysqld[0x813d511]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

here is a strace of what is happening:

Code: Select all

29478 open("/usr/lib/mysql/plugin/ha_innodb_plugin.so", O_RDONLY) = 3
29478 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\257\4\0004\0\0\0"..., 512) = 512
29478 fstat64(3, {st_mode=S_IFREG|0755, st_size=5866072, ...}) = 0
29478 old_mmap(NULL, 1449312, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x8485e000
29478 old_mmap(0x849b9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15b000) = 0x849b9000
29478 old_mmap(0x849bb000, 19808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x849bb000
29478 close(3)                          = 0     
29478 mprotect(0x8485e000, 1421312, PROT_READ|PROT_WRITE) = 0
29478 mprotect(0x8485e000, 1421312, PROT_READ|PROT_EXEC) = 0
29478 brk(0x881d000)                    = 0x881d000
29478 open("./mysql/plugin.frm", O_RDONLY|O_LARGEFILE) = 3
29478 read(3, "\376\1\t\t\3\0\0\20\1\0\0000\0\0i\1A\2\0\0\0\0\0\0\0\0\0\2!\0\10\0"..., 64) = 64
29478 _llseek(3, 64, [64], SEEK_SET)    = 0
29478 read(3, "//\0\0 \0\0", 7)         = 7     
29478 _llseek(3, 4096, [4096], SEEK_SET) = 0
29478 read(3, "\1\1\0\0\n\0\2\0\300\0\1\0\0\0\1\200\2\0\0\1\0\300\0\377PRIMARY\377"..., 33) = 33
29478 pread64(3, "\0\0\6\0MyISAM\0\0\0\0\0\0", 16, 5034) = 16
29478 pread64(3, "\377                               "..., 577, 4457) = 577
29478 _llseek(3, 8192, [8192], SEEK_SET) = 0
29478 read(3, "\212\1\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 288) = 288
29478 read(3, ">\0\3\2\2\24)                         "..., 106) = 106
29478 close(3)                          = 0     
29478 getcwd("/var/lib/mysql", 4096)    = 15   
29478 lstat64("/var/lib/mysql/mysql", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
29478 lstat64("/var/lib/mysql/mysql/plugin.MYI", {st_mode=S_IFREG|0600, st_size=1024, ...}) = 0
29478 lstat64("./mysql/plugin.MYI", {st_mode=S_IFREG|0600, st_size=1024, ...}) = 0
29478 open("/var/lib/mysql/mysql/plugin.MYI", O_RDWR|O_LARGEFILE) = 3
29478 read(3, "\376\376\7\1\0\2\1[\0\260\0d\0\304\0\1\0\0\1\0S\1\0\0", 24) = 24
29478 readlink("./mysql/plugin.MYI", 0xbc290810, 511) = -1 EINVAL (Invalid argument)
29478 readlink("./mysql/plugin.MYD", 0xbc290610, 511) = -1 EINVAL (Invalid argument)
29478 _llseek(3, 0, [0], SEEK_SET)      = 0     
29478 read(3, "\376\376\7\1\0\2\1[\0\260\0d\0\304\0\1\0\0\1\0S\1\0\0\0\0\0\377\0\0\0\0"..., 347) = 347
29478 lstat64("./mysql/plugin.MYD", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
29478 open("./mysql/plugin.MYD", O_RDWR|O_LARGEFILE) = 4
29478 access("./mysql/plugin.TRG", F_OK) = -1 ENOENT (No such file or directory)
29478 gettimeofday({1310407754, 335775}, NULL) = 0
29478 _llseek(4, 0, [0], SEEK_CUR)      = 0     
29478 _llseek(4, 0, [0], SEEK_END)      = 0     
29478 close(3)                          = 0     
29478 close(4)                          = 0     
29478 --- SIGSEGV (Segmentation fault) @ 0 (0) ---

Any ideas?

Thanks.

[specs]

I wonder if it's really necessary to use strace to find the source of the problem.
In normal cases the syslog (or dmesg) should produce enough information to track down the problem (use google using the message found with dmesg).

Could you look at the output from "dmesg" and the output from "paxctl -v /usr/bin/mysql".
Using paxctl I find (hardened gentoo, without configuration):

Code: Select all

$ /sbin/paxctl -v /usr/bin/mysql     
PaX control v0.5
Copyright 2004,2005,2006,2007 PaX Team <pageexec@freemail.hu>

- PaX flags: -------x-e-- [/usr/bin/mysql]
        RANDEXEC is disabled
        EMUTRAMP is disabled


The above pc is no production machine.

After you solved the problem with mysql you should decide for yourselves if it is worth the trouble of upgrading the kernel and some utils when you take the trouble to installing an up-to-date version of mysql.
The grsecurity-2.1.14-2.6.32.15-201006011506.patch seems a bit old, even gradm currently ships as version 2.2.2

[closet geek]

Hi,

Well I offer a strace as in the past it's helped make sense of what part of pax/grsec was causing the issue. This time around it's not so clear to me, so hopefully someone else can spot the problem (if it is indeed a grsec/pax problem).

There is nothing in dmesg. The only thing logged to /var/log/messages is:

lsb_log_message: Manager of pid-file quit without updating file. failed
lsb_log_message: MySQL manager or server PID file could not be found! failed

Not all that helpful. After conversion the mysql binary just has the same permissions as you'd expect (the default ones). I'm loathe to change stuff on the binary via paxctl because every upgrade is going to break MySQL until paxctl is run again.

One thing to note - as of MySQL 5.5 the InnoDB plugin is going to replace the built in InnoDB, if the plugin gets loaded in the same manner as I'm trying all grsec/pax machines will not work once upgraded to MySQL 5.5 if the problem is with grsec/pax.

Thanks.

[specs]

Perhaps you should try paxctl on this for once.
It should tell you if the paxsettings are the problem.

Brad should be able to tell you if gradm can be used instead of paxctl, without having to check each program after each update.

As for dmesg: most of the time when pax or grsecurity is the problem you can expect some message in dmesg.

As for mysql:

Code: Select all

# grep -A 2 \\\[mysqld\\\] /etc/mysql/my.cnf
[mysqld]
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
# /etc/init.d/mysql start
 * Starting  ...
 * Starting  (/etc/mysql/my.cnf)                                                                                  [ ok ]
# tail /var/log/mysql/mysqld.err
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.5
110712 18:25:45  InnoDB: Initializing buffer pool, size = 16.0M
110712 18:25:45  InnoDB: Completed initialization of buffer pool
110712 18:25:45  InnoDB: highest supported file format is Barracuda.
110712 18:25:45 InnoDB Plugin 1.0.15 started; log sequence number 44247
110712 18:25:45 [Note] Event Scheduler: Loaded 0 events
110712 18:25:45 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.56-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo Linux mysql-5.1.56


As far as I can see it runs like it should here (up-to-date software, but no production-pc or real test of mysql).
Are you sure you isolated the problem correctly?

Have you looked if the Wiki gave any hints?
http://en.wikibooks.org/wiki/Grsecurity/Reporting_Bugs

Success.

[PaX Team]

the first thing to try is a newer kernel/grsec , and second, you should enable coredumping and look at what code crashed exactly.