Apache2

[dflt]

Apache2 dies on me quite often, with the following:

System: Debian Squeeze x86_64 2.6.32.50-grsec
Config is here: http://pastebin.com/1sDju4fm

Code: Select all

Feb  1 14:08:37 wintermute kernel: [3580947.388082] grsec: From 66.249.66.56: Segmentation fault occurred at 0000002100006a90 in /usr/lib/apache2/mpm-prefork/apache2[apache2:27280] uid/euid:33/33 gid/egid:33/33, parent /usr/lib/apache2/mpm-prefork/apache2[apache2:30408] uid/euid:0/0 gid/egid:0/0
Feb  1 14:08:37 wintermute kernel: [3580947.388172] grsec: From 66.249.66.56: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/apache2/mpm-prefork/apache2[apache2:27280] uid/euid:33/33 gid/egid:33/33, parent /usr/lib/apache2/mpm-prefork/apache2[apache2:30408] uid/euid:0/0 gid/egid:0/0


I have a coredump, but gdb cannot find the proper symbols (apache2 and php5-gdb installed), because in squeeze the gdb is fckd up, at least that's what I'm reading. Any ideas, what can I do to find out what causes this rather annoying disturbance?

[PaX Team]

even without debug symbols you can post stuff like register info and disassembly around the faulting code then we'll see it from there.

[dflt]

Here is what I get from gdb. Guide my hand, I'm not sure with this stuff :O.

Code: Select all

gdb /usr/sbin/apache2 /tmp/core-jan20
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...

warning: The current binary is a PIE (Position Independent Executable), which
GDB does NOT currently support.  Most debugger features will fail if used
in this session.

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2-mpm-prefork...done.
(no debugging symbols found)...done.
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x000002b5f0270e78 in ?? ()
(gdb) bt full
#0  0x000002b5f0270e78 in ?? ()
No symbol table info available.
#1  0x0000005da65e66d0 in ?? ()
No symbol table info available.
#2  0x000002b5f027143b in ?? ()
No symbol table info available.
#3  0x000002b5f094e248 in ?? ()
No symbol table info available.
#4  0x0000005da65e66b8 in ?? ()
No symbol table info available.
#5  0x0000005da6554db0 in ?? ()
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.


[PaX Team]

x/8i $pc
x/8x $sp
info reg

[dflt]

Code: Select all

(gdb) x/8i $pc
0x2b5f0270e78:  Cannot access memory at address 0x2b5f0270e78
(gdb) x/8x $sp
0x3a3c51dd230:  0xa65e66d0      0x0000005d      0xf027143b      0x000002b5
0x3a3c51dd240:  0xf094e248      0x000002b5      0xa65e66b8      0x0000005d
(gdb) info reg
rax            0x0      0
rbx            0x5da6298e80     402219699840
rcx            0xa      10
rdx            0x5da74441a8     402238226856
rsi            0x5da74441a8     402238226856
rdi            0x5da6298e80     402219699840
rbp            0x5da7444180     0x5da7444180
rsp            0x3a3c51dd230    0x3a3c51dd230
r8             0x5da74441a8     402238226856
r9             0x5da656ecf8     402222673144
r10            0x2b5f3570e90    2980494904976
r11            0x0      0
r12            0x5da7444158     402238226776
r13            0x5da74441a8     402238226856
r14            0x50     80
r15            0x5da5f352f8     402216145656
rip            0x2b5f0270e78    0x2b5f0270e78
eflags         0x206    [ PF IF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
fctrl          0x27f    639
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x2b5    693
fioff          0xf024f210       -266014192
foseg          0x3a3    931
fooff          0xc51dabc0       -987911232
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]


[PaX Team]

ok, we'll need the code dumped into the core as well. you'll have to adjust /proc/<pid>/coredump_filter for the apache processes to include MMF_DUMP_MAPPED_PRIVATE, i.e., if the defaults are ...0033 then you'll want ...0037 in there.

[dflt]

Ok I'll get it tonight.

[dflt]

Here you go:

Code: Select all

 gdb /usr/sbin/apache2 core
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...

warning: The current binary is a PIE (Position Independent Executable), which
GDB does NOT currently support.  Most debugger features will fail if used
in this session.

Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2-mpm-prefork...done.
(no debugging symbols found)...done.
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000031b2ede5261 in ?? ()
(gdb) bt full
#0  0x0000031b2ede5261 in ?? ()
No symbol table info available.
#1  0x0000005e1f9f9d28 in ?? ()
No symbol table info available.
#2  0x0000031b2edfe818 in ?? ()
No symbol table info available.
#3  0x0000005e1f9f9860 in ?? ()
No symbol table info available.
#4  0x0000005e1eadc000 in ?? ()
No symbol table info available.
#5  0x0000005e1ea7c4b0 in ?? ()
No symbol table info available.
#6  0x0000031b2edf12af in ?? ()
No symbol table info available.
#7  0x0000005e1f9f9860 in ?? ()
No symbol table info available.
#8  0x0000031b2ede5299 in ?? ()
No symbol table info available.
#9  0x0000031b2f549528 in ?? ()
No symbol table info available.
#10 0x0000031b2edfe4b2 in ?? ()
No symbol table info available.
#11 0x0000031b2f549528 in ?? ()
No symbol table info available.
#12 0x0000031b2f548d98 in ?? ()
No symbol table info available.
#13 0x0000005e1ea7c4b0 in ?? ()
No symbol table info available.
#14 0x0000031b2edfe738 in ?? ()
No symbol table info available.
#15 0x0000031b2f5493a0 in ?? ()
No symbol table info available.
#16 0x0000031b2ede58ae in ?? ()
No symbol table info available.
#17 0x0000005e1ea50d80 in ?? ()
No symbol table info available.
#18 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) x/8i $pc
0x31b2ede5261:  mov    (%rdi),%rbx
0x31b2ede5264:  mov    0x10(%rbx),%eax
0x31b2ede5267:  sub    $0x1,%eax
0x31b2ede526a:  test   %eax,%eax
0x31b2ede526c:  mov    %eax,0x10(%rbx)
0x31b2ede526f:  jne    0x31b2ede52a8
0x31b2ede5271:  mov    0x73c118(%rip),%rax        # 0x31b2f521390
0x31b2ede5278:  add    $0x8,%rax
(gdb) x/8x $sp
0x3971abb15d0:  0x1f9f9d28      0x0000005e      0x2edfe818      0x0000031b
0x3971abb15e0:  0x1f9f9860      0x0000005e      0x1eadc000      0x0000005e
(gdb) info reg
rax            0x31b2ede5260    3415285322336
rbx            0x5e1f9f9d28     404257479976
rcx            0x31b32b98590    3415350019472
rdx            0x31b2edf1290    3415285371536
rsi            0x31b32b9bc90    3415350033552
rdi            0x0      0
rbp            0x5e1f9f98b0     0x5e1f9f98b0
rsp            0x3971abb15d0    0x3971abb15d0
r8             0x689e9edc1a0b13b7       7538637494431716279
r9             0x5e1eae49e0     404241664480
r10            0x31b3216ce90    3415339355792
r11            0x31b31e8d5ae    3415336342958
r12            0x0      0
r13            0x0      0
r14            0xffffffff00000001       -4294967295
r15            0x5e1e4652f8     404234851064
rip            0x31b2ede5261    0x31b2ede5261
eflags         0x206    [ PF IF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
fctrl          0x27f    639
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x31b    795
fioff          0x2edefef0       786366192
foseg          0x397    919
fooff          0x1abaef70       448458608
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]
(gdb)


[PaX Team]

so it's a NULL deref, now you'll have to find out in which library/function it occured. probably the fastest way is to 'objdump -drw' all libraries loaded into an apache process and grep for a hopefully unique pattern, say '0x73c118(%rip),%rax'. once you have that information, you can probably bug your upstream about it.

[dflt]

So, if I lsof one of the apache processes that would give me all the libs loaded, then I objdump them and look for this particular pattern or any of the patterns gdb gave me in the last dump? Or I'm doomed without proper asm knowledge?

[dflt]

Ahh, I've found it! It's inside the libapache2-mod-php5 package, inside /usr/lib/apache2/modules/libphp5.so

Code: Select all

0000000000354260 <_zval_ptr_dtor>:
  354260:       53                      push   %rbx
  354261:       48 8b 1f                mov    (%rdi),%rbx
  354264:       8b 43 10                mov    0x10(%rbx),%eax
  354267:       83 e8 01                sub    $0x1,%eax
  35426a:       85 c0                   test   %eax,%eax
  35426c:       89 43 10                mov    %eax,0x10(%rbx)
  35426f:       75 37                   jne    3542a8 <_zval_ptr_dtor+0x48>
  354271:       48 8b 05 18 c1 73 00    mov    0x73c118(%rip),%rax        # a90390 <vtbl_pass+0x13ab0>
  354278:       48 83 c0 08             add    $0x8,%rax
  35427c:       48 39 c3                cmp    %rax,%rbx
  35427f:       74 37                   je     3542b8 <_zval_ptr_dtor+0x58>
  354281:       48 f7 43 18 fc ff ff ff         testq  $0xfffffffffffffffc,0x18(%rbx)
  354289:       75 55                   jne    3542e0 <_zval_ptr_dtor+0x80>
  35428b:       80 7b 14 03             cmpb   $0x3,0x14(%rbx)
  35428f:       76 08                   jbe    354299 <_zval_ptr_dtor+0x39>
  354291:       48 89 df                mov    %rbx,%rdi
  354294:       e8 3f 35 d7 ff          callq  c77d8 <_zval_dtor_func@plt>
  354299:       48 89 df                mov    %rbx,%rdi
  35429c:       5b                      pop    %rbx
  35429d:       e9 d6 3e d7 ff          jmpq   c8178 <_efree@plt>
  3542a2:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  3542a8:       83 f8 01                cmp    $0x1,%eax
  3542ab:       74 13                   je     3542c0 <_zval_ptr_dtor+0x60>
  3542ad:       0f b6 43 14             movzbl 0x14(%rbx),%eax
  3542b1:       83 e8 04                sub    $0x4,%eax
  3542b4:       3c 01                   cmp    $0x1,%al
  3542b6:       76 17                   jbe    3542cf <_zval_ptr_dtor+0x6f>
  3542b8:       5b                      pop    %rbx
  3542b9:       c3                      retq
  3542ba:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  3542c0:       0f b6 43 14             movzbl 0x14(%rbx),%eax
  3542c4:       c6 43 15 00             movb   $0x0,0x15(%rbx)
  3542c8:       83 e8 04                sub    $0x4,%eax
  3542cb:       3c 01                   cmp    $0x1,%al
  3542cd:       77 e9                   ja     3542b8 <_zval_ptr_dtor+0x58>
  3542cf:       48 89 df                mov    %rbx,%rdi
  3542d2:       5b                      pop    %rbx
  3542d3:       e9 f0 4a d7 ff          jmpq   c8dc8 <gc_zval_possible_root@plt>
  3542d8:       0f 1f 84 00 00 00 00 00         nopl   0x0(%rax,%rax,1)
  3542e0:       48 89 df                mov    %rbx,%rdi
  3542e3:       e8 e0 65 d7 ff          callq  ca8c8 <gc_remove_zval_from_buffer@plt>
  3542e8:       eb a1                   jmp    35428b <_zval_ptr_dtor+0x2b>
  3542ea:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  3542f0:       41 57                   push   %r15
  3542f2:       41 56                   push   %r14
  3542f4:       41 55                   push   %r13
  3542f6:       41 54                   push   %r12
  3542f8:       55                      push   %rbp
  3542f9:       53                      push   %rbx
  3542fa:       48 83 ec 18             sub    $0x18,%rsp
  3542fe:       4c 8b 3d 8b c0 73 00    mov    0x73c08b(%rip),%r15        # a90390 <vtbl_pass+0x13ab0>
  354305:       49 8b bf 20 03 00 00    mov    0x320(%r15),%rdi
  35430c:       48 8b 07                mov    (%rdi),%rax
  35430f:       4c 8b 60 f8             mov    -0x8(%rax),%r12
  354313:       4c 8d 70 f8             lea    -0x8(%rax),%r14
  354317:       45 85 e4                test   %r12d,%r12d
  35431a:       7e 48                   jle    354364 <_zval_ptr_dtor+0x104>
  35431c:       4c 8d 6c 24 08          lea    0x8(%rsp),%r13
  354321:       4c 89 f3                mov    %r14,%rbx
  354324:       31 ed                   xor    %ebp,%ebp
  354326:       41 83 ec 01             sub    $0x1,%r12d
  35432a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  354330:       48 83 eb 08             sub    $0x8,%rbx
  354334:       4c 89 ef                mov    %r13,%rdi
  354337:       83 c5 01                add    $0x1,%ebp
  35433a:       48 8b 03                mov    (%rbx),%rax
  35433d:       48 89 44 24 08          mov    %rax,0x8(%rsp)
  354342:       48 c7 03 00 00 00 00    movq   $0x0,(%rbx)
  354349:       e8 9a 60 d7 ff          callq  ca3e8 <_zval_ptr_dtor@plt>
  35434e:       41 39 ec                cmp    %ebp,%r12d
  354351:       79 dd                   jns    354330 <_zval_ptr_dtor+0xd0>
  354353:       45 89 e4                mov    %r12d,%r12d
  354356:       49 8b bf 20 03 00 00    mov    0x320(%r15),%rdi
  35435d:       49 f7 d4                not    %r12
  354360:       4f 8d 34 e6             lea    (%r14,%r12,8),%r14
  354364:       48 8d 47 18             lea    0x18(%rdi),%rax
  354368:       49 39 c6                cmp    %rax,%r14
  35436b:       74 12                   je     35437f <_zval_ptr_dtor+0x11f>
  35436d:       4c 89 37                mov    %r14,(%rdi)
  354370:       48 83 c4 18             add    $0x18,%rsp
  354374:       5b                      pop    %rbx
  354375:       5d                      pop    %rbp
  354376:       41 5c                   pop    %r12
  354378:       41 5d                   pop    %r13
  35437a:       41 5e                   pop    %r14
  35437c:       41 5f                   pop    %r15
  35437e:       c3                      retq
  35437f:       48 8b 47 10             mov    0x10(%rdi),%rax
  354383:       49 89 87 20 03 00 00    mov    %rax,0x320(%r15)
  35438a:       e8 e9 3d d7 ff          callq  c8178 <_efree@plt>
  35438f:       eb df                   jmp    354370 <_zval_ptr_dtor+0x110>
  354391:       66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00    nopw   %cs:0x0(%rax,%rax,1)


What do you suggest, shall I send them this thread, so they can fix it? Oh and thanks for the help, I owe you a beer or anything, you name it .

[PaX Team]

so _zval_ptr_dtor got a NULL ptr argument that it never expected, so i guess you'll have to resolve the backtrace the same way to find out who managed to call this function with a NULL and where that NULL came from. then should probably talk to your distro/php guys to see what's up .

[dflt]

Uh oh . Can you point me towards a set of books/tutorials, so I can get this done, properly... in the future ?

[PaX Team]

i think google's your friend here . you'll have to learn intel assembly, file formats (ELF), how the toolchain works, how to recognize and debug various problems, etc. be prepared that it won't be an overnight read and i don't really know of any place that has comprehensive materials on all these subjects.

[tjh]

Are you using a PHP Opcode Cache? I've seen APC and XCache cause these issues.