RPI not booting due to randomize layout plugin

[N8Fear]

Hello,

I know that my error is most likely not due to the grsec patch itself but due to some ideosyncracies of the Raspberry Pi.
Some time (during 3.13.0 and 3.14.1 (i think it has to be around 3.13.5)) my rpi refused to boot grsec-patched kernels. I tracked the error down to a line in the randomize_layout_plugin. If I apply the following patch (disabling the call to relayout_decl) the rpi boots even with newer patches.

Code: Select all

+++ b/tools/gcc/randomize_layout_plugin.c
@@ -453,7 +453,7 @@ static void randomize_layout_finish_decl(void *event_data, void *data)
        if (!lookup_attribute("randomize_performed", TYPE_ATTRIBUTES(type)))
                return;
 
-       relayout_decl(decl);
+//     relayout_decl(decl);
 }


Since I'm in no way an expert for gcc plugins I'm more or less guessing that this is the line that actually does "the magic". If not: would you be so nice and try to explain what that line does and in what way it is important?
Do you have any hints how to debug my issue further?

[spender]

Hi,

It's surprising that that would work for you, since without that line you shouldn't be able to load modules properly due to a problem with gcc that could only be worked around by this finish_decl hook. It could be that there's some ARM-specific ops structure that can't be randomized (maybe it's used in assembly without some auto-generated list of offsets like used elsewhere).

Could you add the following line of code just before the line you removed and give me the output it creates (sort & uniq it first) when compiling your kernel?

Code: Select all

fprintf(stderr, "Would have randomized decl for type %s\n", ORIG_TYPE_NAME(type));

Thanks,
-Brad

[N8Fear]

Actually I'm building myself monolithic kernels whereever I can. Therefore it's no wonder that I don't get hit by that bug.

The output you requested:

Code: Select all

Would have randomized decl for type address_space_operations
Would have randomized decl for type btrfs_compress_op
Would have randomized decl for type btrfs_free_space_op
Would have randomized decl for type cpu_user_fns
Would have randomized decl for type cred
Would have randomized decl for type ctl_table
Would have randomized decl for type dentry_operations
Would have randomized decl for type dev_pm_ops
Would have randomized decl for type dwc_otg_hcd_function_ops
Would have randomized decl for type dwc_otg_pcd_function_ops
Would have randomized decl for type ethtool_ops
Would have randomized decl for type export_operations
Would have randomized decl for type extent_io_ops
Would have randomized decl for type fast_hash_ops
Would have randomized decl for type fatent_operations
Would have randomized decl for type file_lock
Would have randomized decl for type file_operations
Would have randomized decl for type firmware_ops
Would have randomized decl for type flow_cache_ops
Would have randomized decl for type fsnotify_ops
Would have randomized decl for type fs_struct
Would have randomized decl for type gre_protocol
Would have randomized decl for type group_info
Would have randomized decl for type header_ops
Would have randomized decl for type hid_ll_driver
Would have randomized decl for type inode_operations
Would have randomized decl for type ipc_namespace
Would have randomized decl for type ipc_ops
Would have randomized decl for type ip_set_type_variant
Would have randomized decl for type k_clock
Would have randomized decl for type kernfs_ops
Would have randomized decl for type kset_uevent_ops
Would have randomized decl for type linux_binfmt
Would have randomized decl for type lock_manager_operations
Would have randomized decl for type mmc_bus_ops
Would have randomized decl for type mmc_host_ops
Would have randomized decl for type mm_struct
Would have randomized decl for type neigh_table
Would have randomized decl for type net
Would have randomized decl for type net_device_ops
Would have randomized decl for type od_ops
Would have randomized decl for type outer_cache_fns
Would have randomized decl for type path
Would have randomized decl for type pci_ops
Would have randomized decl for type pid_namespace
Would have randomized decl for type proc_dir_entry
Would have randomized decl for type proc_ns_operations
Would have randomized decl for type proto
Would have randomized decl for type psci_operations
Would have randomized decl for type Qdisc_class_ops
Would have randomized decl for type rb_augment_callbacks
Would have randomized decl for type rchan_callbacks
Would have randomized decl for type sdhci_ops
Would have randomized decl for type security_operations
Would have randomized decl for type seq_operations
Would have randomized decl for type signal_struct
Would have randomized decl for type skb_checksum_ops
Would have randomized decl for type sock_iocb
Would have randomized decl for type super_operations
Would have randomized decl for type sysfs_ops
Would have randomized decl for type task_struct
Would have randomized decl for type thermal_zone_device_ops
Would have randomized decl for type tty_operations
Would have randomized decl for type tty_port_operations
Would have randomized decl for type uart_ops
Would have randomized decl for type usb_ep_ops
Would have randomized decl for type usb_gadget_ops
Would have randomized decl for type user_namespace
Would have randomized decl for type user_struct
Would have randomized decl for type uts_namespace
Would have randomized decl for type vm_area_struct
Would have randomized decl for type vm_operations_struct
Would have randomized decl for type xfrm_replay


Am I correct that the best way to debug it further would be to cross check this output with the corresponding output from e.g. a x86 kernel build, remove the structs from the list and take a look at what's left afterwards?
Is there some compiler magic like __no_const to exclude specific structs from being randomized?

[spender]

Yes, there's __no_randomize_layout

Can you add it to cpu_user_fns in arch/arm/include/asm/page.h ?

-Brad

[N8Fear]

Sorry for the late reply. I tried that but it didn't help. At the moment I'm trying to add it to more and more of the structs to see if that will get anywhere nearer to a solution.

[N8Fear]

I apologize again for taking that much time. It seems like there are actually two structs that break booting on the rpi:
From ipc/util.h: ipc_ops
and from fs/proc/internal.h: proc_dir_entry (which originally even has attribute __randomize_layout).

Each of the two break booting individually for me unless changed to __no_randomize_layout. Do you have any ideas how I can debug the issue further?

[spender]

In what way does it not boot? Can you get us some serial console logs? I don't see immediately how randomization of ipc_ops would affect anything on ARM specifically.

-Brad

[N8Fear]

It doesn't even get to console. It just shows the "debug-screen" (http://elinux.org/images/9/9e/Debug-screen.jpg, which essentially means that something fails very early...
(The colored screen is actually part of the loaded GPU firmware and normally replaced by the console more or less instantly)

Update: I'll try to setup u-boot to see if I get some error that hints on the actual problem that way...

[UojNodshh0Vt1UaY]

I appear to be having the same problem, only I am using a beaglebone black.

Making the same modification as N8fear's first post, it is possible to successfully boot.

Additionally, I had no issues with 3.11.x, where I am now trying on 3.14.x and having this issue.

It boots with u-boot, and there is absolutely no output on the serial console when it attempts to boot the kernel. u-boot prints its start up messages then gets to "starting kernel..." and hangs.

[N8Fear]

Could you try out if the following patch fixes the issue for you on the beaglebone black:

Code: Select all

diff --git a/fs/proc/internal.h b/fs/proc/internal.h
index b26174e..5c1d8f5 100644
--- a/fs/proc/internal.h
+++ b/fs/proc/internal.h
@@ -49,7 +49,7 @@ struct proc_dir_entry {
    u8 restricted; /* a directory in /proc/net that should be restricted via GRKERNSEC_PROC */
    u8 namelen;
    char name[];
-} __randomize_layout;
+} __no_randomize_layout;
 
 union proc_op {
    int (*proc_get_link)(struct dentry *, struct path *);
diff --git a/ipc/util.h b/ipc/util.h
index 9c47d6f..3fd276d 100644
--- a/ipc/util.h
+++ b/ipc/util.h
@@ -81,7 +81,7 @@ struct ipc_ops {
    int (*getnew) (struct ipc_namespace *, struct ipc_params *);
    int (*associate) (struct kern_ipc_perm *, int);
    int (*more_checks) (struct kern_ipc_perm *, struct ipc_params *);
-};
+} __no_randomize_layout;
 
 struct seq_file;
 struct ipc_ids;


On my Raspberry both these changes are somehow needed (you'll even get a security benefit because except for these structs you get the benefit of randomize layout - if it works...)

[UojNodshh0Vt1UaY]

Thanks heaps n8fear! That worked for me too!:-)

[N8Fear]

Hmm - in that case there seem to be something ARM-specific (at least a subset of ARM), that breaks with this options enabled...
Maybe I can rig up some kind of serial console for debugging output using an Arduino..

[spender]

Hi,

Could you try compiling the kernel with unmodified RANDSTRUCT natively on the ARM device? I'd like to rule out cross-compilation with the plugin as the source of the error. Also, were you cross-compiling from amd64 or i386?

Thanks,
-Brad

[N8Fear]

I'm crosscompiling on amd64, on Gentoo, toolchain created via crossdev:

Code: Select all

cross-armv6j-hardfloat-linux-gnueabi/binutils-2.24-r3
cross-armv6j-hardfloat-linux-gnueabi/gcc-4.8.3
cross-armv6j-hardfloat-linux-gnueabi/gcc-4.9.0
cross-armv6j-hardfloat-linux-gnueabi/glibc-2.19-r1
cross-armv6j-hardfloat-linux-gnueabi/linux-headers-3.15


I can try it on the Raspberry but that will take some time (as in "it's slow ;D )...

[N8Fear]

I just finished testing it with kernel compiled on the Raspberry itself. The result is the same. So it's either some interaction with the plugin and arm gcc in general or it must have other causes.

Unrelated to that I got a lot warnings during compilation (on 3.15.3 but I can remember seeing them also on 3.15.1) like this:

Code: Select all

In file included from include/linux/atomic.h:127:0,
                 from include/linux/spinlock.h:402,
                 from include/linux/mmzone.h:7,
                 from include/linux/bootmem.h:7,
                 from mm/percpu.c:57:
include/asm-generic/atomic-long.h:228:20: note: expected ‘struct atomic_long_t *’ but argument is of type ‘struct atomic_long_unchecked_t *’
 static inline long atomic_long_read(atomic_long_t *l)


The following patch fixes that for me:

Code: Select all

diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h
index 1887948..36c2f13 100644
--- a/include/linux/vmstat.h
+++ b/include/linux/vmstat.h
@@ -228,18 +228,18 @@ static inline void __mod_zone_page_state(struct zone *zone,
 
 static inline void __inc_zone_state(struct zone *zone, enum zone_stat_item item)
 {
-       atomic_long_inc(&zone->vm_stat[item]);
-       atomic_long_inc(&vm_stat[item]);
+       atomic_long_inc_unchecked(&zone->vm_stat[item]);
+       atomic_long_inc_unchecked(&vm_stat[item]);
 }
 
 static inline void __dec_zone_state(struct zone *zone, enum zone_stat_item item)
 {
-       atomic_long_dec(&zone->vm_stat[item]);
-       if (item == NR_FILE_DIRTY && unlikely(atomic_long_read(&zone->vm_stat[item]) < 0))
-               atomic_long_set(&zone->vm_stat[item], 0);
-       atomic_long_dec(&vm_stat[item]);
-       if (item == NR_FILE_DIRTY && unlikely(atomic_long_read(&vm_stat[item]) < 0))
-               atomic_long_set(&vm_stat[item], 0);
+       atomic_long_dec_unchecked(&zone->vm_stat[item]);
+       if (item == NR_FILE_DIRTY && unlikely(atomic_long_read_unchecked(&zone->vm_stat[item]) < 0))
+               atomic_long_set_unchecked(&zone->vm_stat[item], 0);
+       atomic_long_dec_unchecked(&vm_stat[item]);
+       if (item == NR_FILE_DIRTY && unlikely(atomic_long_read_unchecked(&vm_stat[item]) < 0))
+               atomic_long_set_unchecked(&vm_stat[item], 0);
 }
 
 static inline void __inc_zone_page_state(struct page *page,


Edit: Looks like I fixed that for 3.15.1 also, because without fixing it it breaks the build...