GPF in 3.0.4

[moseleymark]

It's time for my bi-annual GPF fault post

I know PAX isn't triggering this but is just reporting it. I'm just hoping that there's something useful in this traceback that might indicate some issue. Or at least that I can rule out grsec/pax before moving on to LKML (where I probably won't get a reply anyway). This is from 3.0.4 with grsecurity-2.2.2-3.0.4-201109011725.patch and no other patches. I just rolled this earlier today and I've gotten a handful of crashes today alone. Traceback always seems to indicate the same spot. Similar to a previous post of mine, I've got no idea why isofs_lookup would be called (some sort of profiling thing?), since we've got no CDs mounted anywhere.

As always, I'm happy to send along anything useful and to try out any debugging steps that would be helpful. The fact that the offending binary is always php might be interesting but isn't super surprising since this is primarily a server dedicated to CGI (shared web hosting). I've got exec logging on but the GPF kills the system before it logs what the process was that the PID in the traceback reports (and console falls offscreen well before the PID).

Thanks!


PAX: suspicious general protection fault: 0000 [#1] SMP
[21474.894576] Modules linked in: ip_queue dcdbas evdev joydev hed i7core_edac dm_mod sr_mod cdrom [last unloaded: scsi_wait_scan]
[21474.917768]
[21474.920800] Pid: 12041, comm: php Not tainted 3.0.4-nx #1
[21475.263429] Dell Inc. PowerEdge R610/0XDN97
[21475.272036] EIP: 0060:[<005f9195>] EFLAGS: 00210046 CPU: 5
[21475.283046] EIP is at __mutex_unlock_slowpath+0x45/0x140
[21475.293708] EAX: 00000100 EBX: 00000000 ECX: 40000200 EDX: df09f5e8
[21475.306277] ESI: 00000044 EDI: 00000048 EBP: ef2c7ecc ESP: ef2c7ebc
[21475.318847] DS: 0068 ES: 0068 FS: 00d8 GS: 007b SS: 0068
[21475.329683] Process php (pid: 12041, ti=f218ae3c task=f218aae0 task.ti=f218ae3c)
[21475.344573] Stack:
[21475.348642] 00200246 fffffffe df09f5e8 eb45d784 ef2c7ed4 005f929d ef2c7ee8 0012d829
[21475.364311] 00000000 df09f5e8 ef2c7efc ef2c7f6c 00130c1b ef2c7f5c 00000000 00000000
[21475.379984] f0fb16c0 df09f540 00004452 00000001 d98e9051 00000000 f1392b28 eb45d784
[21475.395657] Call Trace:
[21475.400601] [<00200246>] ? isofs_lookup+0x2b6/0x480
[21475.410571] [<005f929d>] mutex_unlock+0xd/0x10
[21475.419675] [<0012d829>] vfs_rmdir+0x99/0xc0
[21475.428430] [<00130c1b>] do_rmdir+0x10b/0x160
[21475.437361] [<00004452>] ? do_segment_not_present+0x32/0x90
[21475.448718] [<0000dbba>] ? syscall_trace_enter+0x16a/0x170
[21475.459900] [<00130cd5>] sys_rmdir+0x15/0x20
[21475.468650] [<005fbcd1>] syscall_call+0x7/0xb
[21475.477577] [<0020007b>] ? isofs_lookup+0xeb/0x480
[21475.487373] [<00200246>] ? isofs_lookup+0x2b6/0x480
[21475.497340] [<00210202>] ? nfs_parse_mount_options+0xa32/0xb40
[21475.509217] [<00200292>] ? isofs_lookup+0x302/0x480
[21475.519186] Code: 89 c6 85 db 75 13 64 a1 ec 04 00 00 f7 40 10 00 ff ff 07 0f 85 d5 00 00 00 9c 8f 45 f0 fa e8 23 5e ac ff 8d 7e 04 b8 00 01 00 00 <f
[21475.557836] EIP: [<005f9195>] __mutex_unlock_slowpath+0x45/0x140 SS:ESP 0068:ef2c7ebc
[21475.573959] ---[ end trace f97589a2d56c3c10 ]---


Here's another one from just a little while later:

PAX: suspicious general protection fault: 0000 [#1]
[ 860.117493] SMP
[ 860.121227] Modules linked in: ip_queue hed dcdbas joydev evdev i7core_edac dm_mod sr_mod cdrom [last unloaded: scsi_wait_scan]
[ 860.144397]
[ 860.147424] Pid: 1263, comm: php Not tainted 3.0.4-nx #1 Dell Inc. PowerEdge R610/0XDN97
[ 860.163736] EIP: 0060:[<005f9195>] EFLAGS: 00010046 CPU: 0
[ 860.174791] EIP is at __mutex_unlock_slowpath+0x45/0x140
[ 860.185456] EAX: 00000100 EBX: 00000000 ECX: 40000200 EDX: ecaffa80
[ 860.198026] ESI: 00000044 EDI: 00000048 EBP: ebda3e2c ESP: ebda3e1c
[ 860.210594] DS: 0068 ES: 0068 FS: 00d8 GS: 007b SS: 0068
[ 860.221430] Process php (pid: 1263, ti=e7b4ae3c task=e7b4aae0 task.ti=e7b4ae3c)
[ 860.236146] Stack:
[ 861.020624] 00000246 fffffffe ecaffa80 cdd52a7c ebda3e34 005f929d ebda3e48 0012d829
[ 861.036297] 00000000 ecaffa80 ebda3e5c ebda3ecc 00130c1b ebda3ebc 00000000 00000000
[ 861.051970] ed051780 ecaff5e8 00004452 00000001 f0f56051 00000000 f1371540 cdd52a7c
[ 861.067643] Call Trace:
[ 861.072588] [<005f929d>] mutex_unlock+0xd/0x10
[ 861.081691] [<0012d829>] vfs_rmdir+0x99/0xc0
[ 861.090439] [<00130c1b>] do_rmdir+0x10b/0x160
[ 861.099369] [<00004452>] ? do_segment_not_present+0x32/0x90
[ 861.110723] [<0000dbba>] ? syscall_trace_enter+0x16a/0x170
[ 861.121904] [<00130cd5>] sys_rmdir+0x15/0x20
[ 861.130658] [<005fbcd1>] syscall_call+0x7/0xb
[ 861.139587] [<005f007b>] ? tg3_get_invariants+0xd8e/0x32fc
[ 861.150772] [<00010282>] ? perf_misc_flags+0x32/0x80
[ 861.160914] [<0011f532>] ? filp_close+0x52/0x80
[ 861.170191] [<002cdc88>] ? trace_hardirqs_on_thunk+0xc/0x10
[ 861.181547] [<005fbd0b>] ? restore_all+0xf/0xf
[ 861.190644] [<005f007b>] ? tg3_get_invariants+0xd8e/0x32fc
[ 861.201824] [<00010282>] ? perf_misc_flags+0x32/0x80
[ 861.211962] Code: 89 c6 85 db 75 13 64 a1 ec 04 00 00 f7 40 10 00 ff ff 07 0f 85 d5 00 00 00 9c 8f 45 f0 fa e8 23 5e ac ff 8d 7e 04 b8 00 01 00 00 <f
[ 861.250511] EIP: [<005f9195>] __mutex_unlock_slowpath+0x45/0x140 SS:ESP 0068:ebda3e1c
[ 861.266632] ---[ end trace 48962aeb03e3452c ]---

[spender]

Can you post or mail your vmlinux file?

-Brad

[moseleymark]

I stupidly had recompiled the kernel in the meantime, so I don't know if that invalidates the addresses I posted. So I grabbed a new traceback with a new kernel, so addresses would match:

PAX: suspicious general protection fault: 0000 [#1] SMP
[ 2712.504288] Modules linked in: ip_queue evdev joydev dcdbas hed i7core_edac dm_mod sr_mod cdrom [last unloaded: scsi_wait_scan]
[ 2712.561676]
[ 2712.564709]
[ 2712.567746] Pid: 24292, comm: php Not tainted 3.0.4-nx #1 Dell Inc. PowerEdge R610/0XDN97
[ 2712.584251] EIP: 0060:[<005f9195>] EFLAGS: 00010046 CPU: 3
[ 2712.595264] EIP is at __mutex_unlock_slowpath+0x45/0x140
[ 2712.605925] EAX: 00000100 EBX: 00000000 ECX: 40000200 EDX: f7599c78
[ 2712.618496] ESI: 00000044 EDI: 00000048 EBP: ea39fec4 ESP: ea39feb4
[ 2712.631070] DS: 0068 ES: 0068 FS: 00d8 GS: 007b SS: 0068
[ 2712.790864] Process php (pid: 24292, ti=ee1a835c task=ee1a8000 task.ti=ee1a835c)
[ 2713.063917] Stack:
[ 2713.102024] 00000246 fffffffe f7599c78 d824dfb4 ea39fecc 005f929d ea39fee0 0012d829
[ 2713.117696] 00000000 f7599c78 ea39fef4 ea39ff64 00130c1b ea39ff54 00000000 00000000
[ 2713.133366] f17c49c0 dc7c9dc8 00004452 00000001 f221b04f 00000000 f1178930 d824dfb4
[ 2713.149037] Call Trace:
[ 2713.153980] [<005f929d>] mutex_unlock+0xd/0x10
[ 2713.163085] [<0012d829>] vfs_rmdir+0x99/0xc0
[ 2713.171836] [<00130c1b>] do_rmdir+0x10b/0x160
[ 2713.180765] [<00004452>] ? do_segment_not_present+0x32/0x90
[ 2713.226335] [<0000dbba>] ? syscall_trace_enter+0x16a/0x170
[ 2713.237520] [<00130cd5>] sys_rmdir+0x15/0x20
[ 2713.246282] [<005fbcd1>] syscall_call+0x7/0xb
[ 2713.255213] [<00010282>] ? perf_misc_flags+0x32/0x80
[ 2713.265355] Code: 89 c6 85 db 75 13 64 a1 ec 04 00 00 f7 40 10 00 ff ff 07 0f 85 d5 00 00 00 9c 8f 45 f0 fa e8 23 5e ac ff 8d 7e 04 b8 00 01 00 00 <f
[ 2713.303944] EIP: [<005f9195>] __mutex_unlock_slowpath+0x45/0x140 SS:ESP 0068:ea39feb4
[ 2713.320087] ---[ end trace 64579da9710f33bf ]---

I'm apparently not clever enough to figure out how to attach a file to this post Where should I mail that vmlinux to?

[PaX Team]

Where should I mail that vmlinux to?

spender and me

[moseleymark]

Just fyi, I sent the vmlinux off to PAX Team but I don't have Brad's email.

[taaroa]

Just fyi, I sent the vmlinux off to PAX Team but I don't have Brad's email.

http://en.wikibooks.org/wiki/Grsecurity ... s#Contacts

[moseleymark]

Ah, good to know, thanks.

spender: I'll send that along, hopefully not hitting your inbox twice.

[PaX Team]

the failing code is:

Code: Select all

lock xadd %ax,(%edi)

i.e., it's a NULL deref, probably on

Code: Select all

mutex_unlock(&dentry->d_inode->i_mutex);

in vfs_rmdir. since we don't really play with dentry/inode lifetimes, this could be a vanilla kernel bug. could you try to reproduce it without grsec or if that fails, figure out if there's a grsec option that causes it?

[moseleymark]

I'll play around with it. Unfortunately, the systems in question are also the systems where I'd rather chew my own arm off than run without grsec/PAX compiled-in and with full ACL enabled, so turning those off isn't an option in this case

[moseleymark]

I went out of town a few days after I opened this thread and only had time to goof around a little with it but getting back to the real world (and having a chance to look through LKML), it looks like a vanilla bug posted in the meantime, from commit 64252c75a2196a0cf1e0d3777143ecfe0e3ae650. I definitely appreciate you guys taking a look, just to rule out grsec/pax.

[moseleymark]

Incidentally, how do you guys go about pinpointing the location of the code? Do you dump the assembly code from vmlinux and try to correlate the comments in it with the source? I'd love to hear what you guys use.

[spender]

objdump from vmlinux paired with reverse engineering backgrounds

-Brad