2.6.37.3 crashes when btrfs is used for root partition

[nannou9]

Kernels 2.6.37.2, 2.6.27.3 (and possibly others but not tested) are crashing when doing:

gradm -E
gradm -F -L /x.log

on btrfs root partition.

Problem occurs always and is causing absolute system crash.

Unfortunately i do not have time right now to rewrite the kernel dump. Will do that later if needed.

[spender]

I'll need some kind of oops to debug this. You could boot with netconsole or try to get some info out from sysrq.

BTW, did you mean when you run *either* gradm -E or gradm -F -L learn.log? The sequence of gradm -E followed by gradm -F -L learn.log shouldn't be allowed (since gradm -F -L learn.log implies enabling).

-Brad

[nannou9]

I do not mean *either* commands but *one of*

Note that it is a host machine, not virtual one.
This problem was successfully repeated on pure funtoo stage3 unpacked system with same kernel.
Another kernel builded with same .config file works great on reiserfs.

The Kernel OOPS:

Mar 17 11:52:46 XXXXXXXX kernel: [11985.702015] ------------[ cut here ]------------
Mar 17 11:52:46 XXXXXXXX kernel: [11985.702166] kernel BUG at grsecurity/gracl.c:1924!
Mar 17 11:52:46 XXXXXXXX kernel: [11985.702314] invalid opcode: 0000 [#1] SMP
Mar 17 11:52:46 XXXXXXXX kernel: [11985.702540] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sr0/dev
Mar 17 11:52:46 XXXXXXXX kernel: [11985.702795] CPU 3
Mar 17 11:52:46 XXXXXXXX kernel: [11985.702838] Modules linked in: radeon ttm drm_kms_helper cfbcopyarea cfbimgblt cfbfillrect xt_mark vboxnetflt vboxnetadp vboxdrv e1000e fuse xfs exportfs jfs ext2 scsi_wait_scan
Mar 17 11:52:46 XXXXXXXX kernel: [11985.704105]
Mar 17 11:52:46 XXXXXXXX kernel: [11985.704248] Pid: 6446, comm: gradm Not tainted 2.6.37.3-grsec #1
Mar 17 11:52:46 XXXXXXXX kernel: [11985.704398] Dell Inc. OptiPlex 980 /0D441T
Mar 17 11:52:46 XXXXXXXX kernel: [11985.704705] RIP: 0010:[<ffffffff812a8c82>] [<ffffffff812a8c82>] chk_subj_label+0x180/0x18f
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705003] RSP: 0018:ffff88010cb1dd88 EFLAGS: 00010246
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705151] RAX: 0000000000000000 RBX: ffff88011f83d300 RCX: 000000000000000d
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705303] RDX: ffff88011c663380 RSI: 000000000000000e RDI: 000000000000010e
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705455] RBP: ffff88010cb1ddb8 R08: 0000000000000ff5 R09: 00000000ffffff02
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705606] R10: 00000000ffffff01 R11: 0000000000000246 R12: ffff88011eee0200
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705758] R13: ffff88011c663000 R14: 0000000000000000 R15: 0000000000000000
Mar 17 11:52:46 XXXXXXXX kernel: [11985.705963] FS: 0000632ca3262700(0000) GS:ffff8800db4c0000(0000) knlGS:0000000000000000
Mar 17 11:52:46 XXXXXXXX kernel: [11985.706216] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 17 11:52:46 XXXXXXXX kernel: [11985.706366] CR2: 000000000064ed60 CR3: 000000011ea18000 CR4: 00000000000006f0
Mar 17 11:52:46 XXXXXXXX kernel: [11985.706517] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 17 11:52:46 XXXXXXXX kernel: [11985.706669] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 17 11:52:46 XXXXXXXX kernel: [11985.706821] Process gradm (pid: 6446, threadinfo ffff88010cb1c000, task ffff8800bbf86f90)
Mar 17 11:52:46 XXXXXXXX kernel: [11985.707074] Stack:
Mar 17 11:52:46 XXXXXXXX kernel: [11985.707218] 0000000000000000 0000000000000000 ffff88011fc60000 ffff88011fc60000
Mar 17 11:52:46 XXXXXXXX kernel: [11985.707628] ffff88011fc60000 ffff88010cb1dde0 ffff88010cb1de18 ffffffff812ab1b1
Mar 17 11:52:46 XXXXXXXX kernel: [11985.708092] ffff88011edfca80 0000000000000000 ffff88010cb1dde8 ffff88011fc601f8
Mar 17 11:52:46 XXXXXXXX kernel: [11985.708502] Call Trace:
Mar 17 11:52:46 XXXXXXXX kernel: [11985.708648] [<ffffffff812ab1b1>] gr_set_acls+0x12c/0x2ea
Mar 17 11:52:46 XXXXXXXX kernel: [11985.708798] [<ffffffff812acd11>] gracl_init+0x72b/0x794
Mar 17 11:52:46 XXXXXXXX kernel: [11985.708957] [<ffffffff812acff9>] write_grsec_handler+0x27f/0x976
Mar 17 11:52:46 XXXXXXXX kernel: [11985.709108] [<ffffffff810dfc8e>] vfs_write+0x111/0x171
Mar 17 11:52:46 XXXXXXXX kernel: [11985.709258] [<ffffffff810dfda4>] sys_write+0x45/0x69
Mar 17 11:52:46 XXXXXXXX kernel: [11985.709407] [<ffffffff81002a7b>] system_call_fastpath+0x16/0x1b
Mar 17 11:52:46 XXXXXXXX kernel: [11985.709557] Code: ff f0 ff 05 d1 5d 5f 00 71 09 f0 ff 0d c8 5d 5f 00 cd 04 48 89 45 d8 e8 b7 da e4 ff fe 05 07 86 5c 00 48 8b 45 d8 48 85 c0 75 04 <0f> 0b eb fe 48 83 c4 18 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 54
Mar 17 11:52:46 XXXXXXXX kernel: [11985.712639] RIP [<ffffffff812a8c82>] chk_subj_label+0x180/0x18f
Mar 17 11:52:46 XXXXXXXX kernel: [11985.712829] RSP <ffff88010cb1dd88>
Mar 17 11:52:46 XXXXXXXX kernel: [11985.712983] ---[ end trace 4e98fbd6b8749f54 ]---

So any help will be appreciated.
Thanks in advance!

[spender]

Ok, you triggered the BUG() I inserted in the code for when the inode/dev for "/" generated in userland can't be found when doing a traversal in the kernel. I think someone else had reported this in an earlier kernel and I fixed their problem at that time. I need to know if there's anything else unique about this system other than btrfs. Are you using containers? Can you add -DGRADM_DEBUG to the CFLAGS in the gradm makefile, comment out the call to "transmit_to_kernel" in gradm_arg.c on or around line 437, and then email me the output of a gradm -E on your system? Also provide me with the output of the "stat /" command.

-Brad

[nannou9]

What do you mean as a containers?

[spender]

Vserver or some other kind of filesystem "virtualization". If you don't know what containers are, then I imagine you probably aren't using them

-Brad

[nannou9]

Well, I have made those changes.
The system have not crashed this time- it was still fully usable (as it supposed to be).
Here is the output from gradm -E (hope it's helpful):

ROLE: :::kernel::: type:special uid/gid:1
TRANSITIONS: :::kernel:::
SUBJECT: / dev:15 inode:256 mode:32803 c_raise:ffffffff c_drop:0
OBJECT: /etc/grsec dev:15 inode:1364262 mode:0
OBJECT: / dev:15 inode:256 mode:927
ROLE: default type:default uid/gid:0
TRANSITIONS: admin
SUBJECT: /usr/bin/xauth dev:15 inode:52692 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /usr/bin/xauth dev:15 inode:52692 mode:25
OBJECT: /home dev:15 inode:477145 mode:17
SUBJECT: /sbin/init dev:15 inode:477164 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /sbin/init dev:15 inode:477164 mode:25
OBJECT: /var/log/wtmp dev:15 inode:20973 mode:22
SUBJECT: /sbin/getty dev:0 inode:268435479 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /var/log/wtmp dev:15 inode:20973 mode:22
SUBJECT: /usr/bin/sudo dev:0 inode:268435478 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /bin/su dev:15 inode:270 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /bin/su dev:15 inode:270 mode:25
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /bin/login dev:15 inode:320 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /bin/login dev:15 inode:320 mode:25
OBJECT: /var/log/faillog dev:0 inode:268435462 mode:2147484055
OBJECT: /var/log/wtmp dev:15 inode:20973 mode:22
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /usr/sbin/crond dev:0 inode:268435476 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /usr/sbin/cron dev:15 inode:107517 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /usr/sbin/cron dev:15 inode:107517 mode:25
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /usr/sbin/rsyslogd dev:0 inode:268435475 mode:2147516416 c_raise:ffffffff c_drop:0
SUBJECT: /sbin/syslog-ng dev:0 inode:268435474 mode:2147516416 c_raise:ffffffff c_drop:0
SUBJECT: /sbin/klogd dev:0 inode:268435473 mode:2147516416 c_raise:ffffffff c_drop:0
SUBJECT: /usr/bin/exim dev:0 inode:268435472 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /usr/bin/postgres dev:15 inode:2974253 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /usr/bin/postgres dev:15 inode:2993400 mode:25
OBJECT: /usr/lib64/eselect-postgresql/binwrapper dev:15 inode:2974253 mode:25
OBJECT: /dev/log dev:17 inode:9157 mode:23
SUBJECT: /usr/bin/ssh dev:15 inode:2708146 mode:32768 c_raise:ffffffff c_drop:0
OBJECT: /usr/bin/ssh dev:15 inode:2708146 mode:25
OBJECT: /etc/ssh/ssh_config dev:15 inode:2708022 mode:17
SUBJECT: /usr/X11R6/bin/XFree86 dev:0 inode:268435471 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /dev/mem dev:17 inode:4204 mode:23
SUBJECT: /usr/X11R6/bin/Xorg dev:0 inode:268435470 mode:2147516416 c_raise:ffffffff c_drop:0
OBJECT: /dev/mem dev:17 inode:4204 mode:23
SUBJECT: /usr/sbin/sshd dev:15 inode:2708153 mode:33828 c_raise:250400c3 c_drop:dafbff3c
OBJECT: /usr/sbin/sshd dev:15 inode:2708153 mode:25
OBJECT: /var/run/.nscd_socket dev:0 inode:268435469 mode:2147483671
OBJECT: /var/run/utmpx dev:0 inode:268435461 mode:2147483671
OBJECT: /var/run/utmp dev:15 inode:21013 mode:23
OBJECT: /var/run/sshd dev:0 inode:268435467 mode:2147483664
OBJECT: /var/run dev:15 inode:21003 mode:16
OBJECT: /var/log/wtmp dev:15 inode:20973 mode:22
OBJECT: /var/log/lastlog dev:15 inode:20984 mode:23
OBJECT: /var/mail dev:15 inode:480915 mode:16
OBJECT: /var/log dev:15 inode:20970 mode:16
OBJECT: /usr/share/zoneinfo dev:15 inode:166326 mode:17
OBJECT: /usr/lib64 dev:15 inode:108613 mode:25
OBJECT: /usr/lib32 dev:15 inode:107803 mode:25
OBJECT: /selinux dev:0 inode:268435460 mode:2147483665
OBJECT: /proc/sys/kernel/ngroups_max dev:3 inode:6413 mode:17
OBJECT: /proc/sys dev:3 inode:4026531852 mode:0
OBJECT: /proc/kcore dev:3 inode:4026532050 mode:0
OBJECT: /proc dev:3 inode:1 mode:17
OBJECT: /root dev:15 inode:477367 mode:16
OBJECT: /lib64 dev:15 inode:477533 mode:25
OBJECT: /lib32 dev:15 inode:477381 mode:25
OBJECT: /home dev:15 inode:477145 mode:16
OBJECT: /etc/grsec dev:15 inode:1364262 mode:0
OBJECT: /etc dev:15 inode:391 mode:17
OBJECT: /dev/tty dev:17 inode:4126 mode:23
OBJECT: /dev/pts dev:9 inode:1 mode:23
OBJECT: /dev/ptmx dev:17 inode:4120 mode:23
OBJECT: /dev/null dev:17 inode:4119 mode:23
OBJECT: /dev/urandom dev:17 inode:4133 mode:17
OBJECT: /dev/random dev:17 inode:4122 mode:17
OBJECT: /dev/log dev:17 inode:9157 mode:23
OBJECT: /dev dev:17 inode:3537 mode:0
OBJECT: /bin/bash dev:15 inode:284 mode:24
OBJECT: / dev:15 inode:256 mode:16
SUBJECT: / dev:15 inode:256 mode:32768 c_raise:7390c9df c_drop:8c6f3620
OBJECT: /usr/sbin/sshd dev:15 inode:2708153 mode:16
OBJECT: /etc/ssh dev:15 inode:459 mode:0
OBJECT: /lib/modules dev:15 inode:478116 mode:4194304
OBJECT: /proc/kallsyms dev:3 inode:4026532030 mode:0
OBJECT: /proc/modules dev:3 inode:4026532029 mode:0
OBJECT: /proc/slabinfo dev:3 inode:4026532037 mode:0
OBJECT: /proc/kcore dev:3 inode:4026532050 mode:0
OBJECT: /etc/grsec dev:15 inode:1364262 mode:0
OBJECT: /dev/port dev:17 inode:2698 mode:0
OBJECT: /dev/mem dev:17 inode:4204 mode:0
OBJECT: /dev/kmem dev:17 inode:2694 mode:0
OBJECT: /dev/grsec dev:17 inode:9533 mode:0
OBJECT: /boot dev:8388614 inode:2 mode:0
OBJECT: /var/log dev:15 inode:20970 mode:17
OBJECT: /var/tmp dev:15 inode:21034 mode:407
OBJECT: /var dev:15 inode:2232 mode:415
OBJECT: /tmp dev:15 inode:2227 mode:407
OBJECT: /root dev:15 inode:477367 mode:17
OBJECT: /proc/sys dev:3 inode:4026531852 mode:17
OBJECT: /proc dev:3 inode:1 mode:31
OBJECT: /etc dev:15 inode:391 mode:25
OBJECT: /usr/src dev:15 inode:54094 mode:0
OBJECT: /usr dev:15 inode:52167 mode:25
OBJECT: /lib64 dev:15 inode:477533 mode:25
OBJECT: /lib32 dev:15 inode:477381 mode:25
OBJECT: /sbin dev:15 inode:477148 mode:25
OBJECT: /bin dev:15 inode:258 mode:25
OBJECT: /dev/cdrom dev:0 inode:268435465 mode:2147483665
OBJECT: /dev/fd0 dev:0 inode:268435464 mode:2147483665
OBJECT: /dev/initctl dev:17 inode:3033 mode:23
OBJECT: /dev/mixer dev:17 inode:2804 mode:23
OBJECT: /dev/dsp dev:17 inode:4037 mode:23
OBJECT: /dev/ptmx dev:17 inode:4120 mode:23
OBJECT: /dev/pts dev:9 inode:1 mode:23
OBJECT: /dev/tty dev:17 inode:4126 mode:23
OBJECT: /dev/console dev:17 inode:4115 mode:23
OBJECT: /dev/null dev:17 inode:4119 mode:23
OBJECT: /dev/psaux dev:0 inode:268435463 mode:2147483671
OBJECT: /dev/input dev:17 inode:3958 mode:23
OBJECT: /dev/zero dev:17 inode:4134 mode:23
OBJECT: /dev/random dev:17 inode:4122 mode:17
OBJECT: /dev/urandom dev:17 inode:4133 mode:17
OBJECT: /dev dev:17 inode:3537 mode:16
OBJECT: /mnt dev:15 inode:1484 mode:23
OBJECT: /home dev:15 inode:477145 mode:415
OBJECT: /opt dev:15 inode:1490 mode:25
OBJECT: / dev:15 inode:256 mode:17
SUBJECT: /sbin/grlearn dev:15 inode:3155424 mode:132132 c_raise:0 c_drop:ffffffff
CONNECT 0.0.0.0/0:0-0
BIND 0.0.0.0/0:0-0
OBJECT: /sbin/grlearn dev:15 inode:3155424 mode:25
OBJECT: / dev:15 inode:256 mode:0
SUBJECT: /sbin/gradm_pam dev:15 inode:3155422 mode:164896 c_raise:20004000 c_drop:dfffbfff
CONNECT 0.0.0.0/0:2049-2049 dgram udp
BIND 0.0.0.0/0:0-0
OBJECT: /sbin/gradm_pam dev:15 inode:3155422 mode:25
OBJECT: /usr/lib dev:15 inode:54093 mode:25
OBJECT: /usr/lib64 dev:15 inode:108613 mode:25
OBJECT: /lib dev:15 inode:2703856 mode:25
OBJECT: /lib64 dev:15 inode:477533 mode:25
OBJECT: /dev/null dev:17 inode:4119 mode:23
OBJECT: /dev/log dev:17 inode:9157 mode:23
OBJECT: /var/log/faillog dev:0 inode:268435462 mode:2147483671
OBJECT: /var/run/utmpx dev:0 inode:268435461 mode:2147483671
OBJECT: /var/run/utmp dev:15 inode:21013 mode:23
OBJECT: /var/run dev:15 inode:21003 mode:16
OBJECT: /dev/pts dev:9 inode:1 mode:23
OBJECT: /dev/tty dev:17 inode:4126 mode:23
OBJECT: /dev dev:17 inode:3537 mode:16
OBJECT: /selinux dev:0 inode:268435460 mode:2147483665
OBJECT: /proc/filesystems dev:3 inode:4026532038 mode:17
OBJECT: /proc dev:3 inode:1 mode:16
OBJECT: /dev/urandom dev:17 inode:4133 mode:17
OBJECT: /etc/nsswitch.conf dev:15 inode:2698856 mode:17
OBJECT: /usr/share/zoneinfo dev:15 inode:166326 mode:17
OBJECT: /etc/security dev:15 inode:1325 mode:17
OBJECT: /etc/pam.conf dev:0 inode:268435459 mode:2147483665
OBJECT: /etc/pam.d dev:15 inode:1033 mode:17
OBJECT: /etc/shadow dev:15 inode:3141469 mode:17
OBJECT: /etc/passwd dev:15 inode:3141468 mode:17
OBJECT: /etc/protocols dev:15 inode:1466 mode:17
OBJECT: /etc/localtime dev:15 inode:3056443 mode:17
OBJECT: /etc/ld.so.preload dev:0 inode:268435457 mode:2147483665
OBJECT: /etc/ld.so.cache dev:15 inode:3141475 mode:17
OBJECT: / dev:15 inode:256 mode:0
OBJECT: /dev/grsec dev:17 inode:9533 mode:22
SUBJECT: /sbin/gradm dev:15 inode:3155420 mode:164896 c_raise:4000 c_drop:ffffbfff
CONNECT 0.0.0.0/0:2049-2049 dgram udp
BIND 0.0.0.0/0:0-0
OBJECT: /sbin/gradm_pam dev:15 inode:3155422 mode:24
OBJECT: /sbin/gradm dev:15 inode:3155420 mode:25
OBJECT: /usr/lib dev:15 inode:54093 mode:25
OBJECT: /usr/lib64 dev:15 inode:108613 mode:25
OBJECT: /lib dev:15 inode:2703856 mode:25
OBJECT: /lib64 dev:15 inode:477533 mode:25
OBJECT: /dev/urandom dev:17 inode:4133 mode:17
OBJECT: /etc/protocols dev:15 inode:1466 mode:17
OBJECT: /etc/ld.so.preload dev:0 inode:268435457 mode:2147483665
OBJECT: /etc/ld.so.cache dev:15 inode:3141475 mode:17
OBJECT: / dev:15 inode:256 mode:0
OBJECT: /dev/grsec dev:17 inode:9533 mode:22
ROLE: admin type:special uid/gid:0
TRANSITIONS: admin
SUBJECT: / dev:15 inode:256 mode:167939 c_raise:ffffffff c_drop:0
OBJECT: / dev:15 inode:256 mode:1023

[nannou9]

I have forgotten about the stat command.

Here you are my stat / output:

Plik: `/'
rozmiar: 226 blokow: 8 bloki I/O: 4096 katalog
Urzadzenie: fh/15d inody: 256 dowiazan: 1
Dostep: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Dostep: 2011-03-16 13:01:56.450999922 +0100
Modyfikacja: 2011-03-16 12:24:43.796000007 +0100
Zmiana: 2011-03-16 12:24:43.796000007 +0100
Utworzenie: -

Unfortunately it is in Polish Language. But i believe you do not need translation

[spender]

BTW, it only didn't crash because I instructed you to comment out the line of code that actually sends the policy to the kernel and enables RBAC

-Brad

[nannou9]

IC

BTW. Thank you for making this world even better- safer I mean

[spender]

Tonight I'll give you a list of changes to the RBAC system's kernel code (and I will probably add these to a debugging option in the next patch) so we can figure out what's going on. The policy and inode/device numbers from userland all look fine.

-Brad

[spender]

Ok, I've uploaded a new patch for 2.6.37.4. Can you echo "CONFIG_GRKERNSEC_RBAC_DEBUG=y" to your .config and build/boot it? I'll need the info it prints before the crash. Also, change your gradm back to the way it was previously.

-Brad

[nannou9]

Sorry for late answer. I was very bussy for last few days.
Unfortunately i have very stupid question. How to compile kernel without replacing existing .config file? I am using funtoo (gentoo) for years and i am always using genkernel. But even w/o genkernel simple execution of 'make' command is replacing already existing .config file. In other words i have no idea how to compile the kernel with handy customized .config file.

Please help :/

[spender]

make shouldn't be overwriting your .config in the build directory with something other than the config options that are already there (possibly minus changes required due to some consistency checking). I routinely used a prepared config copied into the build path as .config and build it fine with just make. Perhaps you're making some other mistake in something you haven't mentioned?

-Brad

[nannou9]

1. Extracted kernel source
2. Applied patch -p1
3. make mrproper
4. placed custom config with additional line as you described
5. running make which prints lots of lines when few are interesting:

HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --silentoldconfig Kconfig
#
# configuration written to .config
#

And at the very beginning i have rewritten .config file without CONFIG_GRKERNSEC_RBAC_DEBUG=y

I've been even trying to 'make silentoldconfig' and then add CONFIG_GRKERNSEC_RBAC_DEBUG=y to .config file. But after the kernel was builded there was no additional debug info. Just same kernel stacktrace as earlier.

What am i doing wrong?