chroot crash

[palettentreter]

Hello,

I'm running a hardened gentoo system with pv_ops on a XEN server. The system is mainly building packages in several non-hardened chroots. With 2.6.34-hardened-r12 everything was fine, but now after upgrading to 2.6.36-hardened-r9 I get this:

Code: Select all

[128981.599823] BUG: unable to handle kernel NULL pointer dereference at 0000000000000658
[128981.599842] IP: [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[128981.599857] PGD 794fe067 PUD 7d539067 PMD 0
[128981.599867] Oops: 0000 [#1] SMP
[128981.599875] last sysfs file: /sys/devices/vbd-51712/block/xvda/dev
[128981.599884] CPU 0
[128981.599887] Modules linked in: ipv6
[128981.599897]
[128981.599902] Pid: 11401, comm: udevadm Not tainted 2.6.36-hardened-r9_xenU #1 /
[128981.599910] RIP: e030:[<ffffffff812e7ab7>]  [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[128981.599922] RSP: e02b:ffff88007caafbc8  EFLAGS: 00010246
[128981.599929] RAX: 0000000000000000 RBX: ffff88007df24240 RCX: 0000000000000000
[128981.599936] RDX: 0000000000000019 RSI: 0000000000000000 RDI: ffff88007cc01c00
[128981.599944] RBP: ffff88007cc01c00 R08: 0000000000000000 R09: ffffffff819e0dc0
[128981.599952] R10: ffff88007caafe68 R11: 0000000000000019 R12: 0000000000000000
[128981.599959] R13: ffff88007caafca4 R14: ffff88007caafdb8 R15: ffff88007df24240
[128981.599971] FS:  00007f985d6dc700(0000) GS:ffff88000204e000(0000) knlGS:0000000000000000
[128981.599980] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[128981.599987] CR2: 0000000000000658 CR3: 000000007d445000 CR4: 0000000000002660
[128981.599995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[128981.600005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[128981.600017] Process udevadm (pid: 11401, threadinfo ffff88007caae000, task ffff88007df24240)
[128981.600030] Stack:
[128981.600035]  0000000000000002 ffff88007d72c000 0000000000000000 ffffffff814e9ef9
[128981.600047] <0> 0000000000000041 ffffffff00000072 0000000000000019 ffff88007caafe68
[128981.600061] <0> 0000000000000001 ffff88007caafee8 ffff88007caafee8 7fffffffffffffff
[128981.600077] Call Trace:
[128981.600086]  [<ffffffff814e9ef9>] ? unix_find_other+0x1b9/0x270
[128981.600096]  [<ffffffff814ebd84>] ? unix_dgram_sendmsg+0x454/0x5d0
[128981.600105]  [<ffffffff81442c77>] ? sock_sendmsg+0xe7/0x120
[128981.609703]  [<ffffffff8100515c>] ? pte_pfn_to_mfn+0x2c/0x60
[128981.609703]  [<ffffffff81005da9>] ? xen_set_pte_at+0x79/0x100
[128981.609703]  [<ffffffff810c103a>] ? __do_fault+0x46a/0x550
[128981.609703]  [<ffffffff81006e12>] ? check_events+0x12/0x20
[128981.609703]  [<ffffffff81441989>] ? copy_from_user+0x89/0x110
[128981.609703]  [<ffffffff81442e3c>] ? sys_sendto+0x13c/0x1a0
[128981.609703]  [<ffffffff81103441>] ? d_alloc+0x21/0x1c0
[128981.609703]  [<ffffffff81534af5>] ? _raw_spin_lock+0x5/0x10
[128981.609703]  [<ffffffff810ed706>] ? alloc_file+0x26/0xd0
[128981.609703]  [<ffffffff810099c2>] ? system_call_fastpath+0x16/0x1b
[128981.609703] Code: 14 b8 01 00 00 00 48 8b 5c 24 08 48 8b 6c 24 10 48 83 c4 18 c3 48 c7 c7 00 50 78 81 e8 f3 d0 24 00 31 f6 48 89 ef e8 b9 2a d8 ff <48> 8b 90 58 06 00 00 48 39 93 58 06 00 00 75 19 f0 ff 05 32 d5
[128981.609703] RIP  [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[128981.609703]  RSP <ffff88007caafbc8>
[128981.609703] CR2: 0000000000000658
[128981.609703] ---[ end trace 556e3d12dc80dd95 ]---

Anything known about this? Could it be due to the chroots being non-hardened?

[palettentreter]

I just realized there may be something else involved. I also experienced crashes like this one on a physical hardened gentoo server when compiling software in a chroot. The strange part is that it was only one specific chroot that caused these crashes, while others worked fine. So I decided to rebuild the problematic chroot from scratch on a different server, which now also crashes like above. The chroot environments differ mainly in their CFLAGS and CXXFLAGS settings. Now the current chroot consistently crashes with the same call trace as above. Here's another one:

Code: Select all

[14424.322746] BUG: unable to handle kernel NULL pointer dereference at 0000000000000658
[14424.322768] IP: [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[14424.322787] PGD 2877f067 PUD 16789067 PMD 0
[14424.322800] Oops: 0000 [#1] SMP
[14424.322810] last sysfs file: /sys/devices/vbd-51744/block/xvdc/dev
[14424.322820] CPU 0
[14424.322825] Modules linked in: ipv6
[14424.322838]
[14424.322846] Pid: 8076, comm: udevadm Not tainted 2.6.36-hardened-r9_xenU #1 /
[14424.322858] RIP: e030:[<ffffffff812e7ab7>]  [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[14424.322876] RSP: e02b:ffff880068f41bc8  EFLAGS: 00010246
[14424.322884] RAX: 0000000000000000 RBX: ffff88007d7d0000 RCX: 0000000000000000
[14424.322896] RDX: 0000000000000019 RSI: 0000000000000000 RDI: ffff88007df39000
[14424.322906] RBP: ffff88007df39000 R08: 0000000000000000 R09: ffffffff819e0dc0
[14424.322917] R10: ffff880068f41e68 R11: 0000000000000019 R12: 0000000000000000
[14424.322927] R13: ffff880068f41ca4 R14: ffff880068f41db8 R15: ffff88007d7d0000
[14424.322943] FS:  00007f2b33caf700(0000) GS:ffff88000204e000(0000) knlGS:0000000000000000
[14424.322956] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[14424.322965] CR2: 0000000000000658 CR3: 0000000026c25000 CR4: 0000000000002660
[14424.322976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14424.322987] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14424.322999] Process udevadm (pid: 8076, threadinfo ffff880068f40000, task ffff88007d7d0000)
[14424.323009] Stack:
[14424.323016]  0000000000000002 ffff88007fd7c000 0000000000000000 ffffffff814e9ef9
[14424.323032] <0> 0000000000000041 ffffffff00000072 0000000000000019 ffff880068f41e68
[14424.323052] <0> 0000000000000001 ffff880068f41ee8 ffff880068f41ee8 7fffffffffffffff
[14424.323075] Call Trace:
[14424.323088]  [<ffffffff814e9ef9>] ? unix_find_other+0x1b9/0x270
[14424.323101]  [<ffffffff814ebd84>] ? unix_dgram_sendmsg+0x454/0x5d0
[14424.323114]  [<ffffffff81442c77>] ? sock_sendmsg+0xe7/0x120
[14424.323127]  [<ffffffff8100515c>] ? pte_pfn_to_mfn+0x2c/0x60
[14424.323140]  [<ffffffff81005da9>] ? xen_set_pte_at+0x79/0x100
[14424.323153]  [<ffffffff810c103a>] ? __do_fault+0x46a/0x550
[14424.323165]  [<ffffffff81006e12>] ? check_events+0x12/0x20
[14424.323179]  [<ffffffff81441989>] ? copy_from_user+0x89/0x110
[14424.323191]  [<ffffffff81442e3c>] ? sys_sendto+0x13c/0x1a0
[14424.323205]  [<ffffffff81103441>] ? d_alloc+0x21/0x1c0
[14424.323218]  [<ffffffff81534af5>] ? _raw_spin_lock+0x5/0x10
[14424.323231]  [<ffffffff810ed706>] ? alloc_file+0x26/0xd0
[14424.323245]  [<ffffffff810099c2>] ? system_call_fastpath+0x16/0x1b
[14424.323255] Code: 14 b8 01 00 00 00 48 8b 5c 24 08 48 8b 6c 24 10 48 83 c4 18 c3 48 c7 c7 00 50 78 81 e8 f3 d0 24 00 31 f6 48 89 ef e8 b9 2a d8 ff <48> 8b 90 58 06 00 00 48 39 93 58 06 00 00 75 19 f0 ff 05 32 d5
[14424.323400] RIP  [<ffffffff812e7ab7>] gr_handle_chroot_unix+0x57/0xb0
[14424.323414]  RSP <ffff880068f41bc8>
[14424.323421] CR2: 0000000000000658
[14424.323432] ---[ end trace 5e86dbc0f8ae2609 ]---

Note that other chroots on the same server work fine. The crashing one uses these settings:
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -msse -mmmx -msse2"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="-O2 -pipe -msse -mmmx -msse2"

While other ones work fine, for example with:
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -msse -mmmx -msse2 -march=core2 -mssse3"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="-O2 -pipe -msse -mmmx -msse2 -march=core2 -mssse3"

I really don't know where to look, so I'd be grateful for any hint...

[spender]

There's only one thing it could possibly be which will be changed in the next patch, unless this was some bug fixed earlier (2.6.36 is 6 months old or so).

-Brad