Kernel errors with 3.4.4 and 3.3.8

[moseleymark]

Getting this in 3.4.4 and 3.3.8. Got something similar with 3.2.21 too but grsec wasn't mentioned in traceback that I could see.

I can post kernel .config as needed, but wanted to see if this was a known thing. One way to trigger is by disabling and reenabling the acl system repeatedly, but also doing a reload. The box is a Dell PE 620, dual 8-core with 128gig. The box isn't doing anything (will eventually run mysql) beyond just booting up. The 'kvmhost' in the kernel name signifies that there are some PAX features turned off, but I'm getting the same thing in kernels with full PAX enablement. After the bug trace, I get mountains of "BUG: soft lockup" errors and the terminal is unresponsive. Let me know if there's anything useful I can send over.

This is 3.4.4:

[ 330.010849] BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
[ 330.010851] IP: [<ffffffff812d9ea3>] lookup_acl_obj_label+0x13/0x90
[ 330.010857] PGD 0
[ 330.010858] Oops: 0000 [#1] SMP
[ 330.010860] CPU 18
[ 330.010860] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc coretemp joy3
[ 330.010873]
[ 330.010874] Pid: 20558, comm: syslog-ng Tainted: G C 3.4.4-kvmhost #1 Dell Inc. PowerEdge R620/0KCKR5
[ 330.010876] RIP: 0010:[<ffffffff812d9ea3>] [<ffffffff812d9ea3>] lookup_acl_obj_label+0x13/0x90
[ 330.010879] RSP: 0018:ffff881fe2f1fbf8 EFLAGS: 00010206
[ 330.010880] RAX: ffff881fe8828000 RBX: 0000000000000000 RCX: 0000000202808000
[ 330.010881] RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000404
[ 330.010883] RBP: ffff881fe2f1fbf8 R08: 0000000000000000 R09: ffff880fe71de005
[ 330.010884] R10: 000000006c6c756e R11: 0000000000000004 R12: ffff881fe8c08240
[ 330.010885] R13: 0000000000000005 R14: 0000000000000404 R15: ffff881fe8c08000
[ 330.010886] FS: 00007f3518585b00(0000) GS:ffff880fffd20000(0000) knlGS:0000000000000000
[ 330.010888] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 330.010889] CR2: 00000000000002c0 CR3: 0000001fe5c3e000 CR4: 00000000000407f0
[ 330.010890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 330.010891] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 330.010893] Process syslog-ng (pid: 20558, threadinfo ffff881fe3311ca8, task ffff881fe3311780)
[ 330.010894] Stack:
[ 330.010894] ffff881fe2f1fc88 ffffffff812dabde ffff881fe2f1fc8c 00000001e2f1fd18
[ 330.010898] 0000000000000000 0000000000000000 ffff880fe723a120 ffff881fe8c08240
[ 330.010901] ffff880fe723a120 ffff880fe723a100 01ff881fe2f1fcb8 0000000000000000
[ 330.010903] Call Trace:
[ 330.010906] [<ffffffff812dabde>] __chk_obj_label+0x13e/0x5a0
[ 330.010909] [<ffffffff812dbd5e>] gr_search_file+0x5e/0x190
[ 330.010912] [<ffffffff81171b65>] ? do_lookup+0x55/0x330
[ 330.010915] [<ffffffff812e0483>] gr_acl_handle_hidden_file+0x33/0xb0
[ 330.010917] [<ffffffff81172ac8>] do_last+0x178/0x970
[ 330.010919] [<ffffffff81173d35>] path_openat+0xd5/0x450
[ 330.010921] [<ffffffff812e36ed>] ? gr_log_resource+0x2d/0x140
[ 330.010923] [<ffffffff811741d8>] do_filp_open+0x48/0xa0
[ 330.010927] [<ffffffff81182b09>] ? expand_files+0x49/0x200
[ 330.010929] [<ffffffff81182d12>] ? alloc_fd+0x52/0x130
[ 330.010932] [<ffffffff811632fe>] do_sys_open+0x15e/0x230
[ 330.010935] [<ffffffff812f20c6>] ? __put_user_4+0x16/0x20
[ 330.010937] [<ffffffff81163411>] sys_open+0x21/0x30
[ 330.010941] [<ffffffff81613fd9>] system_call_fastpath+0x18/0x1d
[ 330.010942] Code: c9 0f 1f 80 00 00 00 00 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 49 89 d0 48 6
[ 330.010954] RIP [<ffffffff812d9ea3>] lookup_acl_obj_label+0x13/0x90
[ 330.010956] RSP <ffff881fe2f1fbf8>
[ 330.010957] CR2: 00000000000002c0
[ 330.010958] ---[ end trace acdd7066ca56a91e ]---




Similar error in 3.3.8:

[ 121.244686] BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
[ 121.244688] IP: [<ffffffff813069f3>] lookup_acl_obj_label+0x13/0x90
[ 121.244697] PGD 0
[ 121.244698] Oops: 0000 [#1] SMP
[ 121.244700] CPU 19
[ 121.244701] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_LOG xt_l3
[ 121.244717]
[ 121.244718] Pid: 9137, comm: syslog-ng Tainted: G C 3.3.8-kvmhost #1 Dell Inc. PowerEdge R620/0KCKR5
[ 121.244721] RIP: 0010:[<ffffffff813069f3>] [<ffffffff813069f3>] lookup_acl_obj_label+0x13/0x90
[ 121.244724] RSP: 0018:ffff880fe22b5c08 EFLAGS: 00010206
[ 121.244725] RAX: ffff881fe93e0000 RBX: 0000000000000000 RCX: 0000000202808000
[ 121.244726] RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000404
[ 121.244728] RBP: ffff880fe22b5c08 R08: 0000000000000000 R09: 0000000000000000
[ 121.244729] R10: 0000000000000100 R11: 0000000000000246 R12: ffff881fe8c00240
[ 121.244730] R13: 0000000000000005 R14: 0000000000000404 R15: ffff881fe8c00000
[ 121.244732] FS: 00007f3fa849db00(0000) GS:ffff88203fd20000(0000) knlGS:0000000000000000
[ 121.244733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 121.244734] CR2: 00000000000002c0 CR3: 0000000fe11d3000 CR4: 00000000000406f0
[ 121.244736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 121.244737] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 121.244738] Process syslog-ng (pid: 9137, threadinfo ffff880c53180528, task ffff880c53180000)
[ 121.244739] Stack:
[ 121.244740] ffff880fe22b5c98 ffffffff81307c6e 0000000000000000 0000000100000000
[ 121.244744] 0000000000000000 0000000000000000 ffff880fe6c86220 ffff881fe8c00240
[ 121.244746] ffff880fe6c86220 ffff880fe6c86200 0100000000000000 0000000000000000
[ 121.244749] Call Trace:
[ 121.244752] [<ffffffff81307c6e>] __chk_obj_label+0x13e/0x5b0
[ 121.244755] [<ffffffff81308c5e>] gr_search_file+0x5e/0x190
[ 121.244758] [<ffffffff8130d533>] gr_acl_handle_hidden_file+0x33/0xb0
[ 121.244762] [<ffffffff811957bc>] do_last+0x17c/0x980
[ 121.244764] [<ffffffff81196a05>] path_openat+0xd5/0x450
[ 121.244766] [<ffffffff8131082d>] ? gr_log_resource+0x2d/0x140
[ 121.244768] [<ffffffff81196ea8>] do_filp_open+0x48/0xa0
[ 121.244772] [<ffffffff811a5d49>] ? expand_files+0x49/0x200
[ 121.244774] [<ffffffff811a6029>] ? alloc_fd+0x129/0x150
[ 121.244777] [<ffffffff81185b02>] do_sys_open+0x152/0x1e0
[ 121.244779] [<ffffffff81185bd1>] sys_open+0x21/0x30
[ 121.244784] [<ffffffff8165f230>] system_call_fastpath+0x18/0x1d
[ 121.244785] Code: c9 0f 1f 80 00 00 00 00 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 49 89 d0 48 6
[ 121.244798] RIP [<ffffffff813069f3>] lookup_acl_obj_label+0x13/0x90
[ 121.244800] RSP <ffff880fe22b5c08>
[ 121.244801] CR2: 00000000000002c0
[ 121.244803] ---[ end trace 8790f6f97eb30404 ]---
[ 121.587254] BUG: unable to handle kernel NULL pointer dereference at 0000000000000014
[ 121.595993] IP: [<ffffffff8130470e>] gr_check_protected_task+0x1e/0x50
[ 121.603289] PGD 0
[ 121.605535] Oops: 0000 [#2] SMP
[ 121.609146] CPU 8
[ 121.611187] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_LOG xt_l3
[ 121.649055]
[ 121.650711] Pid: 1438, comm: syslog-ng Tainted: G D C 3.3.8-kvmhost #1 Dell Inc. PowerEdge R620/0KCKR5
[ 121.661884] RIP: 0010:[<ffffffff8130470e>] [<ffffffff8130470e>] gr_check_protected_task+0x1e/0x50
[ 121.671895] RSP: 0018:ffff881fe5109de8 EFLAGS: 00010286
[ 121.677809] RAX: ffff881fe5e65e00 RBX: ffff880c53180000 RCX: ffff880c531903c0
[ 121.685754] RDX: 0000000000000000 RSI: 000000000000000f RDI: ffff880c53180000
[ 121.693705] RBP: ffff881fe5109de8 R08: e200000000000000 R09: 00000000000023b1
[ 121.701650] R10: ffffffff81c877c0 R11: 0000000000000246 R12: 000000000000000f
[ 121.709600] R13: 000000000000000f R14: ffff880c53180000 R15: 0000000000000000
[ 121.717553] FS: 00007f3fa849db00(0000) GS:ffff880fffc80000(0000) knlGS:0000000000000000
[ 121.726566] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 121.732962] CR2: 0000000000000014 CR3: 0000001fe7036000 CR4: 00000000000406f0
[ 121.740912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 121.748856] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 121.756809] Process syslog-ng (pid: 1438, threadinfo ffff881fe5e66328, task ffff881fe5e65e00)
[ 121.766302] Stack:
[ 121.768538] ffff881fe5109e08 ffffffff81303dd5 ffff881fe5109ea8 ffff880c53180000
[ 121.776812] ffff881fe5109e38 ffffffff81068139 0000000000000002 ffff880fe6533a00
[ 121.785086] ffff881fe5109ea8 000000000000000f ffff881fe5109e68 ffffffff8106a44e
[ 121.793359] Call Trace:
[ 121.796094] [<ffffffff81303dd5>] gr_handle_signal+0x55/0x80
[ 121.802415] [<ffffffff81068139>] check_kill_permission+0xf9/0x210
[ 121.809305] [<ffffffff8106a44e>] group_send_sig_info+0x2e/0x90
[ 121.815897] [<ffffffff8106a4ef>] kill_pid_info+0x3f/0x60
[ 121.821908] [<ffffffff8106a6e4>] sys_kill+0x94/0x190
[ 121.827548] [<ffffffff811cf474>] ? sys_epoll_ctl+0x134/0x890
[ 121.833971] [<ffffffff810784ff>] ? sys_clock_gettime+0xef/0x160
[ 121.840667] [<ffffffff8165f230>] system_call_fastpath+0x18/0x1d
[ 121.847353] Code: c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 f6 05 18 4a 6e 00 01 74 36 48 85 ff 74 31 48 8
[ 121.868913] RIP [<ffffffff8130470e>] gr_check_protected_task+0x1e/0x50
[ 121.876292] RSP <ffff881fe5109de8>
[ 121.880172] CR2: 0000000000000014
[ 121.883877] ---[ end trace 8790f6f97eb30405 ]---

[PaX Team]

[ 330.010849] BUG: unable to handle kernel NULL pointer dereference at 00000000000002c0
[ 330.010851] IP: [<ffffffff812d9ea3>] lookup_acl_obj_label+0x13/0x90

this is RBAC code, spender will take a look at it i guess when he's awake .

[spender]

Hi,

Could you make your vmlinux available to me? My email is spender@grsecurity.net.

Thanks,
-Brad

[moseleymark]

Sent a PM with a link, since it was still 50meg even after bzip2'ing.