grsecurity forums

Hi,

When using grsec 3.0-3.13.9-201404062127 and ZFS (an out-of-tree module), I seem to be hitting this size overflow:

Code: Select all

[   38.550690] PAX: size overflow detected in function spa_sync_nvlist.isra.22 /tmp/nix-build-zfs-0.6.2-3.13.9.drv-0/zfs-0.6.2/module/zfs/../../module/zfs/spa.c:5787 cicus.362_65 min, count: 2
[   38.553712] CPU: 0 PID: 1631 Comm: txg_sync Tainted: P           O 3.13.9-grsec #1-NixOS
[   38.555168] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   38.556123]  ffffffff816fd887 0000000000000000 ffffffffa0690df8 ffff880009437c98
[   38.557409]  ffffffff81482bcb ffffffffa069d3c0 ffff880009437cc8 ffffffff8117ae1b
[   38.558697]  0000000000004000 ffff880009bfe578 ffff880017a43358 000000000000001b
[   38.559978] Call Trace:
[   38.560432]  [<ffffffffa0690df8>] ? zfs_fill_zplprops+0xf70d/0x3fe0f [zfs]
[   38.561594]  [<ffffffff81482bcb>] dump_stack+0x45/0x56
[   38.562460]  [<ffffffffa069d3c0>] ? zfs_fill_zplprops+0x1bcd5/0x3fe0f [zfs]
[   38.563616]  [<ffffffff8117ae1b>] report_size_overflow+0x3b/0x50
[   38.564622]  [<ffffffffa061daa8>] spa_sync_nvlist.isra.22+0x1f8/0x200 [zfs]
[   38.565832]  [<ffffffffa061e132>] spa_sync+0x3e2/0xae0 [zfs]
[   38.566804]  [<ffffffff810a5bf8>] ? ktime_get_ts+0x48/0xe0
[   38.567752]  [<ffffffffa062f5c6>] txg_sync_thread+0x316/0x660 [zfs]
[   38.568821]  [<ffffffffa0695590>] ? zfs_fill_zplprops+0x13ea5/0x3fe0f [zfs]
[   38.570006]  [<ffffffffa062f2b0>] ? txg_init+0x290/0x290 [zfs]
[   38.571005]  [<ffffffffa04d63ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[   38.572160]  [<ffffffffa04d6370>] ? __thread_exit+0xa0/0xa0 [spl]
[   38.573168]  [<ffffffff81073982>] kthread+0xd2/0xf0
[   38.573965]  [<ffffffff810738b0>] ? insert_kthread_work+0x40/0x40
[   38.574970]  [<ffffffff8148a1c4>] ret_from_fork+0x74/0xa0
[   38.575913]  [<ffffffff810738b0>] ? insert_kthread_work+0x40/0x40


The code in question seems to be:

Code: Select all

5771 static void
5772 spa_sync_nvlist(spa_t *spa, uint64_t obj, nvlist_t *nv, dmu_tx_t *tx)
5773 {
5774         char *packed = NULL;
5775         size_t bufsize;
5776         size_t nvsize = 0;
5777         dmu_buf_t *db;
5778
5779         VERIFY(nvlist_size(nv, &nvsize, NV_ENCODE_XDR) == 0);
5780
5781         /*
5782          * Write full (SPA_CONFIG_BLOCKSIZE) blocks of configuration
5783          * information.  This avoids the dbuf_will_dirty() path and
5784          * saves us a pre-read to get data we don't actually care about.
5785          */
5786         bufsize = P2ROUNDUP((uint64_t)nvsize, SPA_CONFIG_BLOCKSIZE);
5787         packed = vmem_alloc(bufsize, KM_PUSHPAGE);
5788
5789         VERIFY(nvlist_pack(nv, &packed, &nvsize, NV_ENCODE_XDR,
5790             KM_PUSHPAGE) == 0);
5791         bzero(packed + nvsize, bufsize - nvsize);


Specifically, the panic seems to indicate the error is in line 5787 (or around that point?).

SPA_CONFIG_BLOCKSIZE is defined to be 16384:

Code: Select all

#define     SPA_CONFIG_BLOCKSIZE    (1ULL << 14)


The macro in line 5786 is defined as:

Code: Select all

#define P2ROUNDUP(x, align)>    (-(-(x) & -(align)))


(... and it's used in dozens of places all over the ZFS code).

Here's a disassembly of where *I think* is the problem:

Code: Select all

   4e901:       48 8b 4c 24 08          mov    0x8(%rsp),%rcx
   4e906:       31 d2                   xor    %edx,%edx
   4e908:       48 f7 d9                neg    %rcx
   4e90b:       48 81 e1 00 c0 ff ff    and    $0xffffffffffffc000,%rcx
   4e912:       48 89 c8                mov    %rcx,%rax
   4e915:       48 f7 d8                neg    %rax
   4e918:       48 83 d2 00             adc    $0x0,%rdx
   4e91c:       48 89 c3                mov    %rax,%rbx
   4e91f:       48 f7 da                neg    %rdx
   4e922:       48 89 d1                mov    %rdx,%rcx
   4e925:       48 09 c1                or     %rax,%rcx
   4e928:       0f 85 5b 01 00 00       jne    4ea89 <spa_sync_nvlist.isra.22+0x1d9>
...
   4ea89:       48 c7 c1 00 00 00 00    mov    $0x0,%rcx
   4ea90:       48 c7 c2 00 00 00 00    mov    $0x0,%rdx
   4ea97:       be 9b 16 00 00          mov    $0x169b,%esi
   4ea9c:       48 c7 c7 00 00 00 00    mov    $0x0,%rdi
   4eaa3:       e8 00 00 00 00          callq  4eaa8 <spa_sync_nvlist.isra.22+0x1f8>
   4eaa8:       0f 1f 84 00 00 00 00    nopl   0x0(%rax,%rax,1)
   4eaaf:       00


I think (but I'm not sure) the code in 4ea89..4eaa8 is a call to report_size_overflow().
Notice how the constant in 4e90b seems to be equal to -SPA_CONFIG_BLOCKSIZE.

How can I fix this, as a user?

Hi,

Could you apply this patch, please:

Code: Select all

--- module/zfs/spa.c.orig       2014-04-09 22:02:39.368026785 +0200
+++ module/zfs/spa.c    2014-04-09 22:04:14.092026781 +0200
@@ -5793,6 +5793,7 @@
         * saves us a pre-read to get data we don't actually care about.
         */
        bufsize = P2ROUNDUP((uint64_t)nvsize, SPA_CONFIG_BLOCKSIZE);
+       printk(KERN_ERR "bufsize: %lx nvsize: %lx\n", bufsize, nvsize);
        packed = vmem_alloc(bufsize, KM_PUSHPAGE);
 
        VERIFY(nvlist_pack(nv, &packed, &nvsize, NV_ENCODE_XDR,


And run make with EXTRA_CFLAGS="-fdump-ipa-all -fdump-tree-all" on module/zfs/spa.c and please send me the results (module/zfs/spa.c.*).

Hi ephox,

When I applied your patch, it still crashed but it didn't print anything, indicating that the overflow happened before the printk().

I have added this patch instead:

Code: Select all

diff --git a/module/zfs/spa.c b/module/zfs/spa.c
index 65f78b7..a88e84f 100644
--- a/module/zfs/spa.c
+++ b/module/zfs/spa.c
@@ -73,6 +73,7 @@
 #include <sys/pool.h>
 #include <sys/sysdc.h>
 #include <sys/zone.h>
+#include <linux/kernel.h>
 #endif /* _KERNEL */
 
 #include "zfs_prop.h"
@@ -5783,7 +5784,13 @@ spa_sync_nvlist(spa_t *spa, uint64_t obj, nvlist_t *nv, dmu_tx_t *tx)
         * information.  This avoids the dbuf_will_dirty() path and
         * saves us a pre-read to get data we don't actually care about.
         */
+#ifdef _KERNEL
+       printk(KERN_ERR "nvsize: %lx\n", nvsize);
+#endif
        bufsize = P2ROUNDUP((uint64_t)nvsize, SPA_CONFIG_BLOCKSIZE);
+#ifdef _KERNEL
+       printk(KERN_ERR "bufsize: %lx nvsize: %lx\n", bufsize, nvsize);
+#endif
        packed = vmem_alloc(bufsize, KM_PUSHPAGE);
 
        VERIFY(nvlist_pack(nv, &packed, &nvsize, NV_ENCODE_XDR,


Which led to this crash:

Code: Select all

[   61.570847] SPL: using hostid 0xc16d4bf5
[   61.581839] nvsize: 3ac
[   61.582351] PAX: size overflow detected in function spa_sync_nvlist.isra.22 /tmp/nix-build-zfs-0.6.2-3.13.9.drv-6/zfs-0.6.2/module/zfs/../../module/zfs/spa.c:5792 cicus.362_68 min, count: 2
[   61.585389] CPU: 0 PID: 1730 Comm: txg_sync Tainted: P           O 3.13.9-grsec #1-NixOS
[   61.586831] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   61.587855]  ffffffff816fd887 0000000000000000 ffffffffa0744e38 ffff88000884bc80
[   61.589249]  ffffffff81482bcb ffffffffa075140f ffff88000884bcb0 ffffffff8117ae1b
[   61.590572]  0000000000004000 ffff880008ffc640 ffff88001c56d368 000000000000001b
[   61.591917] Call Trace:
[   61.592360]  [<ffffffffa0744e38>] ? zfs_fill_zplprops+0xf70d/0x3fe3f [zfs]
[   61.593521]  [<ffffffff81482bcb>] dump_stack+0x45/0x56
[   61.594413]  [<ffffffffa075140f>] ? zfs_fill_zplprops+0x1bce4/0x3fe3f [zfs]
[   61.595586]  [<ffffffff8117ae1b>] report_size_overflow+0x3b/0x50
[   61.596607]  [<ffffffffa06d1ae3>] spa_sync_nvlist.isra.22+0x233/0x240 [zfs]
[   61.597801]  [<ffffffffa06d2172>] spa_sync+0x3e2/0xae0 [zfs]
[   61.598763]  [<ffffffff810a5bf8>] ? ktime_get_ts+0x48/0xe0
[   61.599706]  [<ffffffffa06e3606>] txg_sync_thread+0x316/0x660 [zfs]
[   61.600762]  [<ffffffffa07495d0>] ? zfs_fill_zplprops+0x13ea5/0x3fe3f [zfs]
[   61.601937]  [<ffffffffa06e32f0>] ? txg_init+0x290/0x290 [zfs]
[   61.602926]  [<ffffffffa05113ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[   61.604063]  [<ffffffffa0511370>] ? __thread_exit+0xa0/0xa0 [spl]
[   61.605118]  [<ffffffff81073982>] kthread+0xd2/0xf0
[   61.605962]  [<ffffffff810738b0>] ? insert_kthread_work+0x40/0x40
[   61.607047]  [<ffffffff8148a1c4>] ret_from_fork+0x74/0xa0
[   61.607941]  [<ffffffff810738b0>] ? insert_kthread_work+0x40/0x40


Notice how nvsize was 0x3ac.

You can download the spa.c.* files, along with the patched spa.c file, in this URL:

https://dl.dropboxusercontent.com/u/731705/zfs-debug.tar.xz

Thanks!

Hi ephox,

Did you make any progress fixing this issue?

At the moment I simply have disabled the size overflow plugin on my machines, but it would be nice to re-enable it...
If there is any more useful information I can provide, please let me know...

Thanks,
Ricardo

Hi,

Sorry, I haven't had enough time to deal with it yet.

Thanks for the report. This bug will be fixed in the next PaX version.

Awesome, thanks!