Grsecurity and Systemtap

[moseleymark]

Hi. I'm curious if anyone's had grsecurity and systemtap working. I was playing with Systemtap (tried 0.8 as well as the current git repo) today and it's incredibly powerful. Using a vanilla 2.6.27.6 kernel, I was able to get it up and running but when the grsec patch was applied, using basically the same kernel .config (obviously with the addition of the pax/grsec settings), I get a nice kernel oops then panic. This is just with the patch, not any running ACL nor any sysctl settings except for exec_logging turned on. BTW, systemtap needs a slight patch to accommodate the module struct name changes that the grsec patch makes:

--- systemtap-0.8-original/runtime/print.c 2008-11-13 15:29:23.000000000 -0500
+++ systemtap-0.8/runtime/print.c 2008-11-18 18:26:42.000000000 -0500
@@ -280,9 +280,9 @@
printk(KERN_DEBUG "%s: systemtap: %s, base: %p, memory: %lu+%lu+%u+%u+%u data+text+ctx+net+alloc, probes: %d\n",
THIS_MODULE->name,
vstr,
- THIS_MODULE->module_core,
- (unsigned long) (THIS_MODULE->core_size - THIS_MODULE->core_text_size),
- (unsigned long) THIS_MODULE->core_text_size,
+ THIS_MODULE->module_core_rx,
+ (unsigned long) (THIS_MODULE->core_size_rw - THIS_MODULE->core_size_rx),
+ (unsigned long) THIS_MODULE->core_size_rx,
ctx,
_stp_allocated_net_memory,
_stp_allocated_memory - _stp_allocated_net_memory,
@@ -291,9 +291,9 @@
printk(KERN_DEBUG "%s: systemtap: %s, base: %p, memory: %lu+%lu+%u+%u data+text+ctx+net, probes: %d\n",
THIS_MODULE->name,
vstr,
- THIS_MODULE->module_core,
- (unsigned long) (THIS_MODULE->core_size - THIS_MODULE->core_text_size),
- (unsigned long) THIS_MODULE->core_text_size,
+ THIS_MODULE->module_core_rx,
+ (unsigned long) (THIS_MODULE->core_size_rw - THIS_MODULE->core_size_rx),
+ (unsigned long) THIS_MODULE->core_size_rx,
ctx,
_stp_allocated_net_memory,
num_probes);


Anybody else have any success with this? It could just be that Systemtap is doing too much fiddling with the guts of the kernel to be compatible with grsec. It's also a legitimate argument that it provides way too much power but it'd be handy for debugging. And setting disable_modules in sysctl would be enough to disable it in production (since it basically compiles a new module and loads/unloads that for your trace).

I'll be happy to supply the two kernel configs if it'll help.

Here's the oops:

[ 1039.230662] BUG: unable to handle kernel paging request at 000984e0
[ 1039.240627] IP: [<0040ae17>] kprobe_exceptions_notify+0x177/0x410
[ 1039.240627] *pde = 00000000
[ 1039.240627] Oops: 0000 [#42] SMP
[ 1039.240627] Modules linked in: stap_ae0ed6e2d43d55530aa1a49955e8a2cb_10955 ide_scsi dm_snapshot dm_mirror dm_log dm_mod tg3 libphy reiserfs joydev in
[ 1039.240627]
[ 1039.240627] Pid: 5641, comm: getty Tainted: G D (2.6.27.6-grsec #1)
[ 1039.240627] EIP: 0060:[<0040ae17>] EFLAGS: 00010002 CPU: 1
[ 1039.240627] EIP is at kprobe_exceptions_notify+0x177/0x410
[ 1039.240627] EAX: 00000000 EBX: 000984e0 ECX: e667bf40 EDX: 00000000
[ 1039.240627] ESI: e667bf7c EDI: 00000000 EBP: e667bee4 ESP: e667bec0
[ 1039.240627] DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
[ 1039.240627] Process getty (pid: 5641, ti=e667a000 task=e6676540 task.ti=e667a000)
[ 1039.240627] Stack: c14254d0 e6676540 00000046 00000046 00000000 00000046 c1418020 ffffffff
[ 1039.240627] 00000000 e667bf04 0040c01c e667bf40 00000002 00000000 c1417080 e667bf40
[ 1039.240627] 00000002 e667bf28 0040c0c2 ffffffff 00000000 00000000 0040c060 e667bf7c
[ 1039.240627] Call Trace:
[ 1039.240627] [<0040c01c>] ? notifier_call_chain+0x3c/0x80
[ 1039.240627] [<0040c0c2>] ? __atomic_notifier_call_chain+0x62/0x90
[ 1039.240627] [<0040c060>] ? __atomic_notifier_call_chain+0x0/0x90
[ 1039.240627] [<0040c10f>] ? atomic_notifier_call_chain+0x1f/0x30
[ 1039.240627] [<00044aa2>] ? notify_die+0x32/0x40
[ 1039.240627] [<0040a603>] ? do_int3+0x43/0xa0
[ 1039.240627] [<00409b17>] ? int3+0x37/0x40
[ 1039.240627] [<000984e1>] ? sys_write+0x1/0x70
[ 1039.240627] [<00004f12>] ? syscall_call+0x7/0xb
[ 1039.240627] =======================
[ 1039.240627] Code: 8b 14 10 85 d2 0f 84 f2 fe ff ff 8b 51 0c 89 f0 e8 df fd ff ff 85 c0 0f 84 e0 fe ff ff b8 01 80 00 00 e9 d8 fe ff ff 8b 5e 2c 4b <
[ 1039.240627] EIP: [<0040ae17>] kprobe_exceptions_notify+0x177/0x410 SS:ESP 0068:e667bec0
[ 1039.240627] ---[ end trace 8c4f7d5de847a0c5 ]---
[ 1040.396383] BUG: unable to handle kernel paging request at 00098470
[ 1040.405976] IP: [<0040ae17>] kprobe_exceptions_notify+0x177/0x410
[ 1040.405976] *pde = 00000000
[ 1040.405976] Oops: 0000 [#43] SMP
[ 1040.405976] Modules linked in: stap_ae0ed6e2d43d55530aa1a49955e8a2cb_10955 ide_scsi dm_snapshot dm_mirror dm_log dm_mod tg3 libphy reiserfs joydev in
[ 1040.405976]
[ 1040.405976] Pid: 1, comm: init Tainted: G D (2.6.27.6-grsec #1)
[ 1040.405976] EIP: 0060:[<0040ae17>] EFLAGS: 00010002 CPU: 1
[ 1040.405976] EIP is at kprobe_exceptions_notify+0x177/0x410
[ 1040.405976] EAX: 00000000 EBX: 00098470 ECX: f6837f40 EDX: 00000000
[ 1040.405976] ESI: f6837f7c EDI: 00000000 EBP: f6837ee4 ESP: f6837ec0
[ 1040.405976] DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
[ 1040.405976] Process init (pid: 1, ti=f6836000 task=f6838000 task.ti=f6836000)
[ 1040.405976] Stack: 00000046 f6838000 00000000 00000001 00000000 00000046 c1418020 ffffffff
[ 1040.405976] 00000000 f6837f04 0040c01c f6837f40 00000002 00000000 c1417080 f6837f40
[ 1040.405976] 00000002 f6837f28 0040c0c2 ffffffff 00000000 00000000 0040c060 f6837f7c
[ 1040.405976] Call Trace:
[ 1040.405976] [<0040c01c>] ? notifier_call_chain+0x3c/0x80
[ 1040.405976] [<0040c0c2>] ? __atomic_notifier_call_chain+0x62/0x90
[ 1040.405976] [<0040c060>] ? __atomic_notifier_call_chain+0x0/0x90
[ 1040.405976] [<0040c10f>] ? atomic_notifier_call_chain+0x1f/0x30
[ 1040.405976] [<00044aa2>] ? notify_die+0x32/0x40
[ 1040.405976] [<0040a603>] ? do_int3+0x43/0xa0
[ 1040.405976] [<00409b17>] ? int3+0x37/0x40
[ 1040.405976] [<00098471>] ? sys_read+0x1/0x70
[ 1040.405976] [<00004f12>] ? syscall_call+0x7/0xb
[ 1040.405976] =======================
[ 1040.405976] Code: 8b 14 10 85 d2 0f 84 f2 fe ff ff 8b 51 0c 89 f0 e8 df fd ff ff 85 c0 0f 84 e0 fe ff ff b8 01 80 00 00 e9 d8 fe ff ff 8b 5e 2c 4b <
[ 1040.405976] EIP: [<0040ae17>] kprobe_exceptions_notify+0x177/0x410 SS:ESP 0068:f6837ec0
[ 1040.405976] ---[ end trace 8c4f7d5de847a0c5 ]---
[ 1041.557257] Kernel panic - not syncing: Attempted to kill init!

[PaX Team]

BTW, systemtap needs a slight patch to accommodate the module struct name changes that the grsec patch makes:

note that under PaX core_size_rw and core_size_rx are exclusive regions, don't substract them .

[ 1039.230662] BUG: unable to handle kernel paging request at 000984e0
[ 1039.240627] IP: [<0040ae17>] kprobe_exceptions_notify+0x177/0x410

thanks, should be fixed in test21, can you try it? note that systemtap may do other things behind the kernel's back, so it may very well need its own pax_open/close_kernel annotations. also for the future, can you capture full lines please? at least the one reporting the code bytes got truncated, fortunately it was still enough here to help me find the problem, but it's better to have full info.

[moseleymark]

Hi. test21 seems to have done the trick. Thanks for the ridiculously speedy fix!

Sorry about the oops paste. The oops and subsequent panic wasn't getting logged to the normal locations, so I had to get it from a serial console. It must be chopping off long lines (or maybe just copied poorly). It might be a case of syslog-ng still buffering when the kernel panicked. I repeated it several times without getting anything in the logs.

That's good to know too about the core_size_rw and core_size_rx addresses. I had just done a straight substitution for the old and new names. Luckily all that it is just some debugging output -- though obviously wrong in the case of my patch

BTW, for anyone following this thread because they're interested in systemtap, the little patch I put in the top message shouldn't be trusted (or at least you should ignore the output printed by _stp_print_kernel_info)--not to mention I'm not remotely competent in C, much less kernel code. The function it's in appears to just be for printing out debugging output, so the big printk in _stp_print_kernel_info (in runtime/print.c) could probably more safely turned into just:

<snip>
printk(KERN_DEBUG "%s: systemtap: %s, probes: %d\n",
THIS_MODULE->name,
vstr,
num_probes);
</snip>