Kernel crashing in busybox/uClibc environment

[ptr]

I've a problem with grsecurity inside a busybox/uClibc environment (compiled with buildroot). The kernel crashes when doing certain things:

1) Run a program that generates a floating point exception (e.g. divide by zero), crash.
This does not happen on a Debian/glibc system with grsecurity or a busybox/uClibc without grsecurity.
(screenshots: busybox/uClibc with grsec, busybox/uClibc without grsec)

2) Try to write to /dev/mem, crash.
This also does not happen on a Debian/glibc system with grsecurity or a busybox/uClibc without grsecurity.
(screenshots: debian/glibc, busybox/uClibc)

Is this a problem in uClibc, grsecurity or both or did I forgot to enable a option somewhere?
Have other people noticed this and is there a patch or workaround available?

Thanks in advance.

[PaX Team]

1) Run a program that generates a floating point exception (e.g. divide by zero), crash.
This does not happen on a Debian/glibc system with grsecurity or a busybox/uClibc without grsecurity.
(screenshots: busybox/uClibc with grsec, busybox/uClibc without grsec)

could you please decode the first oops via ksymoops? without symbols it's kinda hard to tell what and where went wrong.

2) Try to write to /dev/mem, crash.
This also does not happen on a Debian/glibc system with grsecurity or a busybox/uClibc without grsecurity.
(screenshots: debian/glibc, busybox/uClibc)

same request for the decoded oops, and i'd also like to see the PaX options you enabled.

[ptr]

Ok, here are the decodes oopses (the oopses are different from the screenshots and also handtyped so I hope they are correct):

# dd if=/dev/random of=/dev/mem:

Code: Select all

peter@debian:/usr/src/linux$ ksymoops -K -v vmlinux -m System.map -L -O < oops1.txt
ksymoops 2.4.9 on i686 2.4.32-grsec.20051117.  Options used
     -v vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m System.map (specified)

Oops: 0002
CPU:   0
EIP:   0010:[<c0323eaa>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000  ebx: c3d6fc40  ecx: 00000400  edx: 00000000
esi: 00000007  edi: 00000000  ebp: 00000000  esp: c3657d94
ds: 0018   es: 0018  ss: 0018
Process dd (pid: 66, stackpage=c3657000)
Stack: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
       00000000 00000000 00000000 00000000 00000000 00000000 c037f7d9 c3d6fc40
       00000007 c033566b 00000001 c03243b2 00000001 00000000 00000000 00000000
Call Trace:    [<c03243b2>] [<c0155fa5>] [<c0156a56>] [<c0156bcb>] [<c01424b8>]
  [<c020cf95>] [<c03229d1>] [<c0202bb8>] [<c016c169>] [<c0130203>]
Code: f3 ab bb 00 e0 ff ff 21 e3 8b 8b ac 06 00 00 85 c9 0f 85 34


>>EIP; c0323eaa <gr_log_start+7a/2f0>   <=====

Trace; c03243b2 <gr_log_varargs+62/e24>
Trace; c0155fa5 <pax_mirror_fault+55/1e0>
Trace; c0156a56 <do_no_page+156/240>
Trace; c0156bcb <handle_mm_fault+8b/1e0>
Trace; c01424b8 <do_page_fault+148/820>
Trace; c020cf95 <random_read+e5/140>
Trace; c03229d1 <gr_handle_mem_write+21/30>
Trace; c0202bb8 <write_mem+8/20>
Trace; c016c169 <sys_write+99/110>
Trace; c0130203 <system_call+33/50>

Code;  c0323eaa <gr_log_start+7a/2f0>
00000000 <_EIP>:
Code;  c0323eaa <gr_log_start+7a/2f0>   <=====
   0:   f3 ab                     repz stos %eax,%es:(%edi)   <=====
Code;  c0323eac <gr_log_start+7c/2f0>
   2:   bb 00 e0 ff ff            mov    $0xffffe000,%ebx
Code;  c0323eb1 <gr_log_start+81/2f0>
   7:   21 e3                     and    %esp,%ebx
Code;  c0323eb3 <gr_log_start+83/2f0>
   9:   8b 8b ac 06 00 00         mov    0x6ac(%ebx),%ecx
Code;  c0323eb9 <gr_log_start+89/2f0>
   f:   85 c9                     test   %ecx,%ecx
Code;  c0323ebb <gr_log_start+8b/2f0>
  11:   0f 85 34 00 00 00         jne    4b <_EIP+0x4b>

Divide by zero:

Code: Select all

peter@debian:/usr/src/linux$ ksymoops -K -v vmlinux -m System.map -L -O < oops2.txt
ksymoops 2.4.9 on i686 2.4.32-grsec.20051117.  Options used
     -v vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m System.map (specified)

Oops: 0002
CPU:    0
EIP:    0010:[<c0323eaa>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000  ebx: c360a000  ecx: 00000400  edx: 00000000
esi: 00000018  edi: 00000000  ebp: 00000000  esp: c360bc74
ds: 0018   es: 0018   ss: 0018
Process test (pid: 126, stackpage=c360b000)
Stack: 08049000 08049000 080494a0 c3e181c0 c360a000 c02c2e00 00001812 00000000
       c360a000 00000000 08049590 080494a0 080484a0 00000000 c037f7d9 c360a000
       00000018 c033566b 00000001 c03243b2 00000001 c3c06300 00000001 00000000
Call Trace:    [<c02c2e00>] [<c03243b2>] [<c016474b>] [<c016474b>] [<c01568ce>]
  [<c01568ce>] [<c031f1d0>] [<c014ea5b>] [<c0326df2>] [<c0155fa5>] [<c0325222>]
  [<c031f1d0>] [<c0176a0e>] [<c014f900>] [<c014f9a5>] [<c0130000>] [<c0206840>]
  [<c017d221>] [<c013021d>] [<c01308f0>] [<c0130254>]
Code: f3 ab bb 00 e0 ff ff 21 e3 8b 8b ac 06 00 00 85 c9 0f 85 34


>>EIP; c0323eaa <gr_log_start+7a/2f0>   <=====

Trace; c02c2e00 <ip_local_deliver_finish+0/170>
Trace; c03243b2 <gr_log_varargs+62/e24>
Trace; c016474b <__alloc_pages+6b/280>
Trace; c016474b <__alloc_pages+6b/280>
Trace; c01568ce <do_anonymous_page+fe/130>
Trace; c01568ce <do_anonymous_page+fe/130>
Trace; c031f1d0 <gr_learn_resource+40/160>
Trace; c014ea5b <update_one_process+6b/140>
Trace; c0326df2 <rb_insert_color+d2/f0>
Trace; c0155fa5 <pax_mirror_fault+55/1e0>
Trace; c0325222 <gr_log_resource+72/90>
Trace; c031f1d0 <gr_learn_resource+40/160>
Trace; c0176a0e <do_coredump+ce/1ed>
Trace; c014f900 <collect_signal+b0/f0>
Trace; c014f9a5 <dequeue_signal+65/d0>
Trace; c0130000 <do_signal+220/2c7>
Trace; c0206840 <tty_ioctl+300/570>
Trace; c017d221 <sys_ioctl+c1/2c3>
Trace; c013021d <system_call+4d/50>
Trace; c01308f0 <do_divide_error+0/100>
Trace; c0130254 <signal_return+14/20>

Code;  c0323eaa <gr_log_start+7a/2f0>
00000000 <_EIP>:
Code;  c0323eaa <gr_log_start+7a/2f0>   <=====
   0:   f3 ab                     repz stos %eax,%es:(%edi)   <=====
Code;  c0323eac <gr_log_start+7c/2f0>
   2:   bb 00 e0 ff ff            mov    $0xffffe000,%ebx
Code;  c0323eb1 <gr_log_start+81/2f0>
   7:   21 e3                     and    %esp,%ebx
Code;  c0323eb3 <gr_log_start+83/2f0>
   9:   8b 8b ac 06 00 00         mov    0x6ac(%ebx),%ecx
Code;  c0323eb9 <gr_log_start+89/2f0>
   f:   85 c9                     test   %ecx,%ecx
Code;  c0323ebb <gr_log_start+8b/2f0>
  11:   0f 85 34 00 00 00         jne    4b <_EIP+0x4b>


Here are all PaX related options. I'm using Grsecurity with security level "high".

Code: Select all

CONFIG_GRKERNSEC_PAX_RANDUSTACK=y
CONFIG_GRKERNSEC_PAX_ASLR=y
CONFIG_GRKERNSEC_PAX_RANDMMAP=y
CONFIG_GRKERNSEC_PAX_NOEXEC=y
# CONFIG_GRKERNSEC_PAX_PAGEEXEC is not set
# CONFIG_GRKERNSEC_PAX_NOELFRELOCS is not set
CONFIG_GRKERNSEC_PAX_MPROTECT=y
# CONFIG_GRKERNSEC_PAX_ETEXECRELOCS is not set
# CONFIG_GRKERNSEC_PAX_SOFTMODE is not set
CONFIG_GRKERNSEC_PAX_EI_PAX=y
CONFIG_GRKERNSEC_PAX_PT_PAX_FLAGS=y
CONFIG_GRKERNSEC_PAX_HAVE_ACL_FLAGS=y
CONFIG_GRKERNSEC_PAX_RANDKSTACK=y
CONFIG_GRKERNSEC_PAX_SEGMEXEC=y
# CONFIG_GRKERNSEC_PAX_EMUTRAMP is not set
# CONFIG_GRKERNSEC_PAX_EMUSIGRT is not set

I'm happy to provide further details when you need them.

Thanks.

[PaX Team]

Ok, here are the decodes oopses (the oopses are different from the screenshots and also handtyped so I hope they are correct):

thanks a lot for the info. i see what happened but i can't explain why . gr_log_start() does a memset on either of two boot-time allocated buffers (kernel panics if those fail), however in your case the buffer pointer was NULL, kinda not possible unless there's some memory corruption coming from another place. since you seem to be able to reproduce it at will, one quick way of finding the corruption would be to enable KERNEXEC and declare gr_audit_log_fmt and gr_alert_log_fmt as char * const so that they'd go into .rodata and with KERNEXEC that would actually be read-only, any overwrite attempt will oops and we can catch the culprit.

[spender]

I noticed the version on your kernel is not the standard -grsec. Is the kernel you are running a vanilla kernel patched with grsec only?

-Brad

[ptr]

thanks a lot for the info. i see what happened but i can't explain why . gr_log_start() does a memset on either of two boot-time allocated buffers (kernel panics if those fail), however in your case the buffer pointer was NULL, kinda not possible unless there's some memory corruption coming from another place. since you seem to be able to reproduce it at will, one quick way of finding the corruption would be to enable KERNEXEC and declare gr_audit_log_fmt and gr_alert_log_fmt as char * const so that they'd go into .rodata and with KERNEXEC that would actually be read-only, any overwrite attempt will oops and we can catch the culprit.

Ok, but KERNEXEC is only available in 2.6 right? I've tried a 2.6 kernel and I'm seeing the same oopses. I've changed the formats to char * const but the oopses still look the same. I've also changed the format/buffer to static array but that all doesn't seem to help. I'm going to install busybox on a harddisk now (all testing I did was from a bootable CD), that should ease debugging.

I noticed the version on your kernel is not the standard -grsec. Is the kernel you are running a vanilla kernel patched with grsec only?

It's a vanilla 2.4.32 kernel with the latest grsecurity (11/12) applied. I've changed the version to indicate when the kernel was built.