grsecurity forums

Hi Grsec,

I'm not too familiar with what to do when I get a kernel oops (I'm reading into it). For now, I'm posting what I've got below.

We got this kernel oops with the following two patches:
grsecurity-2.1.0-2.6.10-200501081640.patch
linux-2.6.10-secfix-200501071130.patch

(We also had linux-2.6.10-NFS_ALL.dif from nfs.sourceforge.net).

I've been trying to get the current patches working, but they aren't yet playing nice with the nfs patches.
(ie grsecurity-2.1.1-2.6.10-as2-200501242254.patch,
patch-2.6.10-as2
linux-2.6.10-NFS_ALL.dif)

In any case, the problem looks like it's between proftpd and gr_log_resource.

If you need anything else, let me know.
Jason


> Jan 30 14:20:26 hillmont kernel: Oops: 0000 [#1]
> Jan 30 14:20:26 hillmont kernel: SMP
> Jan 30 14:20:26 hillmont kernel: CPU: 2
> Jan 30 14:20:26 hillmont kernel: EIP:
> 0060:[gr_log_resource+70/160] Not tainted VLI
> Jan 30 14:20:26 hillmont kernel: EFLAGS: 00010002
> (2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189)
> Jan 30 14:20:26 hillmont kernel: EFLAGS: 00010002
> (2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189)
> Jan 30 14:20:26 hillmont kernel: EIP is at gr_log_resource+0x46/0xa0
> Jan 30 14:20:26 hillmont kernel: eax: 00000000 ebx: 00000000 ecx:
> 00000000 edx: 00000000
> Jan 30 14:20:26 hillmont kernel: esi: dae5b020 edi: 00000000 ebp:
> 00000000 esp: e8485c80
> Jan 30 14:20:26 hillmont kernel: ds: 007b es: 007b ss: 0068
> Jan 30 14:20:26 hillmont kernel: Process proftpd (pid: 9599,
> threadinfo=e8484000 task=dae5b020)
> Jan 30 14:20:26 hillmont kernel: Stack: 00000001 00000000 00000005
> c0275dc0 dae5b020 00000000 00000000 00000001
> Jan 30 14:20:26 hillmont kernel: 00000001 dae5b020 00000005
> 00000000 f7d32934 c015c0c7 dae5b020 00000000
> Jan 30 14:20:26 hillmont kernel: 00000000 00000001 00000001
> 00000000 00004021 e8485d70 c015c194 dae5b020

Ok, on a different machine, I was able to catch it over the serial console this time. This time I was able to get the Call Trace as well.

Below I'll give both the raw output, as well as running it through ksymoops.

(Again, if I'm doing something wrong, if I need to be giving more information, or if I need to provide something else of relevance, please let me know, I'll be more than happy to comply.)

The raw output.

Unable to handle kernel NULL pointer dereference at virtual address 00000080
printing eip:
c027adf6
*pgd = c040fa1800000000
*pmd = 0000000000000000
Oops: 0000 [#1]
SMP
CPU: 0
EIP: 0060:[<c027adf6>] Not tainted VLI
EFLAGS: 00010002 (2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189)
EIP is at gr_log_resource+0x46/0xa0
eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 00000000
esi: eff62540 edi: 00000000 ebp: 00000000 esp: c6569dac
ds: 007b es: 007b ss: 0068
Process bash (pid: 7016, threadinfo=c6568000 task=eff62540)
Stack: 00000001 00000000 00000002 c0275dc0 eff62540 00000000 00000000 00000001
00000001 eff62540 00000002 00000000 0009fe80 c015c0c7 eff62540 00000000
00000000 00000001 00000001 00000000 00000008 ec4cac80 c015c194 eff62540
Call Trace:
[<c0275dc0>] gr_learn_resource+0x40/0x158
[<c015c0c7>] update_one_process+0x53/0x100
[<c015c194>] update_process_times+0x20/0x34
[<c014b5aa>] smp_apic_timer_interrupt+0xc6/0xd4
[<c01411ec>] apic_timer_interrupt+0x1c/0x30
[<c0392078>] nf_hook_slow+0x0/0xe8
[<c039e2fb>] ip_rcv+0x3c7/0x414
[<c039e50c>] ip_rcv_finish+0x0/0x204
[<c02d19a5>] e1000_clean_rx_irq+0x411/0x420
[<c0389451>] netif_receive_skb+0x195/0x1cc
[<c038950d>] process_backlog+0x85/0x114
[<c038961e>] net_rx_action+0x82/0x11c
[<c015882a>] __do_softirq+0x6a/0xd4
[<c01588bc>] do_softirq+0x28/0x30
[<c01687b5>] irq_exit+0x2d/0x30
[<c0142770>] do_IRQ+0x20/0x28
[<c014113a>] common_interrupt+0x1a/0x20
[<c01565cc>] exit_notify+0x694/0x6e4
[<c01569a4>] do_exit+0x388/0x3b8
[<c01569f9>] sys_exit+0xd/0x10
[<c0140217>] syscall_call+0x7/0xb
Code: 00 20 01 75 73 83 fb 08 75 09 f6 86 a1 01 00 00 40 75 65 83 7c 24 1c 00 74 1c 8b 86 7c 04 00 00 8d 14 dd 00 00 00 00 89 c1 89 d0 <3b> bc 08 80 00 00 00 77 1c eb 42 8b 86 7c 04 00 00 8d 14 dd 00
<0>Kernel panic - not syncing: Fatal exception in interrupt



Run through ksymoops:

bourbon: 10:32am# ksymoops -m /boot/System.map-2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189 -o /lib/modules/2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr2b-v6.189/ /root/oops
ksymoops 2.4.5 on i686 2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr2b-v6.189. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr2b-v6.189/ (specified)
-m /boot/System.map-2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189 (specified)

Error (regular_file): read_ksyms stat /proc/ksyms failed
ksymoops: No such file or directory
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Unable to handle kernel NULL pointer dereference at virtual address 00000080
c027adf6
*pgd = c040fa1800000000
Oops: 0000 [#1]
CPU: 0
EIP: 0060:[<c027adf6>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010002 (2.6.10-grsec+gg3+e+fhs6b+nfs+gr0501+++p4+c4a+gr6b-v6.189)
eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 00000000
esi: eff62540 edi: 00000000 ebp: 00000000 esp: c6569dac
ds: 007b es: 007b ss: 0068
Stack: 00000001 00000000 00000002 c0275dc0 eff62540 00000000 00000000 00000001
00000001 eff62540 00000002 00000000 0009fe80 c015c0c7 eff62540 00000000
00000000 00000001 00000001 00000000 00000008 ec4cac80 c015c194 eff62540
[<c0275dc0>] gr_learn_resource+0x40/0x158
[<c015c0c7>] update_one_process+0x53/0x100
[<c015c194>] update_process_times+0x20/0x34
[<c014b5aa>] smp_apic_timer_interrupt+0xc6/0xd4
[<c01411ec>] apic_timer_interrupt+0x1c/0x30
[<c0392078>] nf_hook_slow+0x0/0xe8
[<c039e2fb>] ip_rcv+0x3c7/0x414
[<c039e50c>] ip_rcv_finish+0x0/0x204
[<c02d19a5>] e1000_clean_rx_irq+0x411/0x420
[<c0389451>] netif_receive_skb+0x195/0x1cc
[<c038950d>] process_backlog+0x85/0x114
[<c038961e>] net_rx_action+0x82/0x11c
[<c015882a>] __do_softirq+0x6a/0xd4
[<c01588bc>] do_softirq+0x28/0x30
[<c01687b5>] irq_exit+0x2d/0x30
[<c0142770>] do_IRQ+0x20/0x28
[<c014113a>] common_interrupt+0x1a/0x20
[<c01565cc>] exit_notify+0x694/0x6e4
[<c01569a4>] do_exit+0x388/0x3b8
[<c01569f9>] sys_exit+0xd/0x10
[<c0140217>] syscall_call+0x7/0xb
Code: 00 20 01 75 73 83 fb 08 75 09 f6 86 a1 01 00 00 40 75 65 83 7c 24 1c 00 74 1c 8b 86 7c 04 00 00 8d 14 dd 00 00 00 00 89 c1 89 d0 <3b> bc 08 80 00 00 00 77 1c eb 42 8b 86 7c 04 00 00 8d 14 dd 00


>>EIP; c027adf6 <gr_log_resource+46/a0> <=====

>>esi; eff62540 <pg0+2f922540/3f9be400>
>>esp; c6569dac <pg0+5f29dac/3f9be400>

Code; c027adcb <gr_log_resource+1b/a0>
00000000 <_EIP>:
Code; c027adcb <gr_log_resource+1b/a0>
0: 00 20 add %ah,(%eax)
Code; c027adcd <gr_log_resource+1d/a0>
2: 01 75 73 add %esi,0x73(%ebp)
Code; c027add0 <gr_log_resource+20/a0>
5: 83 fb 08 cmp $0x8,%ebx
Code; c027add3 <gr_log_resource+23/a0>
8: 75 09 jne 13 <_EIP+0x13> c027adde <gr_log_resource+2e/a0>
Code; c027add5 <gr_log_resource+25/a0>
a: f6 86 a1 01 00 00 40 testb $0x40,0x1a1(%esi)
Code; c027addc <gr_log_resource+2c/a0>
11: 75 65 jne 78 <_EIP+0x78> c027ae43 <gr_log_resource+93/a0>
Code; c027adde <gr_log_resource+2e/a0>
13: 83 7c 24 1c 00 cmpl $0x0,0x1c(%esp,1)
Code; c027ade3 <gr_log_resource+33/a0>
18: 74 1c je 36 <_EIP+0x36> c027ae01 <gr_log_resource+51/a0>
Code; c027ade5 <gr_log_resource+35/a0>
1a: 8b 86 7c 04 00 00 mov 0x47c(%esi),%eax
Code; c027adeb <gr_log_resource+3b/a0>
20: 8d 14 dd 00 00 00 00 lea 0x0(,%ebx,8),%edx
Code; c027adf2 <gr_log_resource+42/a0>
27: 89 c1 mov %eax,%ecx
Code; c027adf4 <gr_log_resource+44/a0>
29: 89 d0 mov %edx,%eax
Code; c027adf6 <gr_log_resource+46/a0> <=====
2b: 3b bc 08 80 00 00 00 cmp 0x80(%eax,%ecx,1),%edi <=====
Code; c027adfd <gr_log_resource+4d/a0>
32: 77 1c ja 50 <_EIP+0x50> c027ae1b <gr_log_resource+6b/a0>
Code; c027adff <gr_log_resource+4f/a0>
34: eb 42 jmp 78 <_EIP+0x78> c027ae43 <gr_log_resource+93/a0>
Code; c027ae01 <gr_log_resource+51/a0>
36: 8b 86 7c 04 00 00 mov 0x47c(%esi),%eax
Code; c027ae07 <gr_log_resource+57/a0>
3c: 8d .byte 0x8d
Code; c027ae08 <gr_log_resource+58/a0>
3d: 14 dd adc $0xdd,%al

<0>Kernel panic - not syncing: Fatal exception in interrupt

1 error issued. Results may not be reliable.
bourbon: 10:32am#

And also, I'm not using the ACLs on this machine at all.

CONFIG_GRKERNSEC=y
# CONFIG_GRKERNSEC_LOW is not set
# CONFIG_GRKERNSEC_MEDIUM is not set
# CONFIG_GRKERNSEC_HIGH is not set
CONFIG_GRKERNSEC_CUSTOM=y
# CONFIG_GRKERNSEC_KMEM is not set
# CONFIG_GRKERNSEC_IO is not set
# CONFIG_GRKERNSEC_PROC_MEMMAP is not set
# CONFIG_GRKERNSEC_BRUTE is not set
# CONFIG_GRKERNSEC_HIDESYM is not set
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
# CONFIG_GRKERNSEC_PROC is not set
# CONFIG_GRKERNSEC_LINK is not set
# CONFIG_GRKERNSEC_FIFO is not set
# CONFIG_GRKERNSEC_CHROOT is not set
CONFIG_GRKERNSEC_AUDIT_GROUP=y
CONFIG_GRKERNSEC_AUDIT_GID=1002
CONFIG_GRKERNSEC_EXECLOG=y
CONFIG_GRKERNSEC_RESLOG=y
# CONFIG_GRKERNSEC_CHROOT_EXECLOG is not set
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
# CONFIG_GRKERNSEC_AUDIT_MOUNT is not set
# CONFIG_GRKERNSEC_AUDIT_IPC is not set
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
# CONFIG_GRKERNSEC_TIME is not set
CONFIG_GRKERNSEC_PROC_IPADDR=y
# CONFIG_GRKERNSEC_EXECVE is not set
# CONFIG_GRKERNSEC_SHM is not set
# CONFIG_GRKERNSEC_DMESG is not set
# CONFIG_GRKERNSEC_RANDPID is not set
# CONFIG_GRKERNSEC_TPE is not set
CONFIG_GRKERNSEC_RANDNET=y
CONFIG_GRKERNSEC_RANDISN=y
CONFIG_GRKERNSEC_RANDID=y
CONFIG_GRKERNSEC_RANDSRC=y
CONFIG_GRKERNSEC_RANDRPC=y
# CONFIG_GRKERNSEC_SOCKET is not set
# CONFIG_GRKERNSEC_SYSCTL is not set
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

Can you reproduce the problem with 2.1.1?

-Brad

Hi Brad!

Actually, we need the NFS patches from nfs.sourceforge.net (Trond Myklebust's patches), otherwise our system freezes and loads shoot through the roof.

Unfortunately, the as2 patch apparently incorporates some but not all of Trond's patches. There are 28 failed / rejected hunks when applying both the as2 and Trond patches.

I don't really have the time right now to do hand patch those failed hunks, so this will probably have to wait until stable 2.6.11 is released. My apologies about that.

Thanks for your help though! I'll let you know if it fails in 2.6.11!
Jason