PAX: size overflow detected in function evtchn_read drivers/

[gaima]

Hi,

I recently moved from Gentoos hardened-sources 3.18.8, to 4.0.6-r2 (then 4.0.7-r1), and have hit the following size overflow since running Xen VMs.

Code: Select all

Jul  6 22:12:02 minixen2 kernel: [   70.506894] PAX: size overflow detected in function evtchn_read drivers/xen/evtchn.c:232 cicus.194_307 min, count: 18
Jul  6 22:12:02 minixen2 kernel: [   70.506949] CPU: 0 PID: 4217 Comm: xenstored Not tainted 4.0.6-hardened-r2 #1
Jul  6 22:12:02 minixen2 kernel: [   70.506951] Hardware name: MSI MS-7677/H61I-E35 (MS-7677), BIOS V6.2 09/10/2012
Jul  6 22:12:02 minixen2 kernel: [   70.506952]  0000000000000000 0000000000000001 ffffffff817216b0 ffff88005afdc840
Jul  6 22:12:02 minixen2 kernel: [   70.506954]  ffffffff8119a879 ffff88005afdc888 ffffffff814bc723 0000000000000004
Jul  6 22:12:02 minixen2 kernel: [   70.506956]  0000734ac07df994 0000734ac07df998 ffff880062a33ffc ffff880058a5e300
Jul  6 22:12:02 minixen2 kernel: [   70.506958] Call Trace:
Jul  6 22:12:02 minixen2 kernel: [   70.506964]  [<ffffffff817216b0>] ? dump_stack+0x40/0x50
Jul  6 22:12:02 minixen2 kernel: [   70.506967]  [<ffffffff8119a879>] ? report_size_overflow+0x29/0x40
Jul  6 22:12:02 minixen2 kernel: [   70.506970]  [<ffffffff814bc723>] ? evtchn_read+0x563/0x590
Jul  6 22:12:02 minixen2 kernel: [   70.506972]  [<ffffffff811a8796>] ? do_vfs_ioctl+0x456/0x780
Jul  6 22:12:02 minixen2 kernel: [   70.506975]  [<ffffffff811d5acb>] ? fsnotify+0x2eb/0x420
Jul  6 22:12:02 minixen2 kernel: [   70.506979]  [<ffffffff8100a13f>] ? xen_restore_fl_direct_reloc+0x4/0x4
Jul  6 22:12:02 minixen2 kernel: [   70.506980]  [<ffffffff814bcce0>] ? evtchn_ioctl+0x1c0/0x680
Jul  6 22:12:02 minixen2 kernel: [   70.506982]  [<ffffffff81194bd9>] ? vfs_read+0x129/0x230
Jul  6 22:12:02 minixen2 kernel: [   70.506984]  [<ffffffff81194d29>] ? SyS_read+0x49/0xc0
Jul  6 22:12:02 minixen2 kernel: [   70.506987]  [<ffffffff81727475>] ? system_call_fastpath+0x12/0x17
Jul  6 22:12:02 minixen2 kernel: [   70.506990]  [<ffffffff8101d691>] ? pax_randomize_kstack+0x51/0x70
Jul  6 22:12:02 minixen2 kernel: [   70.506992]  [<ffffffff8172749c>] ? ret_from_sys_call+0x22/0x64


Had this happen on two entirely unrelated servers too.
Naturally disabling CONFIG_PAX_SIZE_OVERFLOW stops the overflow failure.
CONFIG_XEN_DEV_EVTCHN=y

This looks similar to this;
https://forums.grsecurity.net/viewtopic.php?f=3&t=4045
However my knowledge of C and kernel programming is a million miles off sufficient to make any useful attempts at resolving it.


Mike

[ephox]

Hi,

Could you please send me the results (drivers/xen/evtchn.*) of make drivers/xen/evtchn.o EXTRA_CFLAGS="-fdump-tree-all -fdump-ipa-all" and apply this patch:

Code: Select all

--- drivers/xen/evtchn.c.orig   2015-07-12 23:44:07.849930909 +0200
+++ drivers/xen/evtchn.c        2015-07-12 23:49:16.817917134 +0200
@@ -225,6 +225,7 @@
                goto unlock_out;
 
        u->ring_cons += (bytes1 + bytes2) / sizeof(evtchn_port_t);
+       printk(KERN_ERR "PAX bytes1: %x, bytes2: %x, count: %lx\n", bytes1, bytes2, count);
        rc = bytes1 + bytes2;
 
  unlock_out:


[gaima]

Hi,

Could you please send me the results (drivers/xen/evtchn.*) of make drivers/xen/evtchn.o EXTRA_CFLAGS="-fdump-tree-all -fdump-ipa-all" and apply this patch:

Code: Select all

--- drivers/xen/evtchn.c.orig   2015-07-12 23:44:07.849930909 +0200
+++ drivers/xen/evtchn.c        2015-07-12 23:49:16.817917134 +0200
@@ -225,6 +225,7 @@
                goto unlock_out;
 
        u->ring_cons += (bytes1 + bytes2) / sizeof(evtchn_port_t);
+       printk(KERN_ERR "PAX bytes1: %x, bytes2: %x, count: %lx\n", bytes1, bytes2, count);
        rc = bytes1 + bytes2;
 
  unlock_out:


Sure thing.
Patch applied, make run, resulting files here;
https://www.dropbox.com/s/ke1um8vfae5k6 ... ar.gz?dl=0

[PaX Team]

also don't forget the new logs when you trigger the problem with the patch .

[gaima]

also don't forget the new logs when you trigger the problem with the patch .

Righto, log below.

Code: Select all

[    4.064706] xen:xen_evtchn: Event-channel device installed
...
[   57.125214] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125222] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125229] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125238] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125413] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125684] PAX bytes1: 4, bytes2: 0, count: 4
[   57.125729] PAX bytes1: 4, bytes2: 0, count: 4
[   57.126049] PAX bytes1: 4, bytes2: 0, count: 4
[   57.126362] PAX bytes1: 4, bytes2: 0, count: 4
[   57.126808] PAX bytes1: 4, bytes2: 0, count: 4
[   57.127182] PAX bytes1: 4, bytes2: 0, count: 4
[   57.127479] PAX bytes1: 4, bytes2: 0, count: 4
[   57.127571] PAX bytes1: 4, bytes2: 0, count: 4
[   57.127905] PAX bytes1: 4, bytes2: 0, count: 4
[   57.128429] PAX bytes1: 4, bytes2: 0, count: 4
[   57.128730] PAX bytes1: 4, bytes2: 0, count: 4
[   57.128732] PAX: size overflow detected in function evtchn_read drivers/xen/evtchn.c:233 cicus.215_308 min, count: 28
[   57.128799] CPU: 2 PID: 3997 Comm: xenstored Not tainted 4.0.7-hardened-r1 #3
[   57.128803] Hardware name: MSI MS-7677/H61I-E35 (MS-7677), BIOS V6.2 09/10/2012
[   57.128807]  0000000000000000 0000000000000004 ffffffff817254b3 ffff8800621c50c0
[   57.128811]  ffffffff8119af59 ffff8800621c5108 ffffffff814c0494 00000000ffffff02
[   57.128812]  000071408d9276d8 000071408d9276d4 0000000000000004 ffff8800656a8ffc
[   57.128815] Call Trace:
[   57.128821]  [<ffffffff817254b3>] ? dump_stack+0x40/0x50
[   57.128825]  [<ffffffff8119af59>] ? report_size_overflow+0x29/0x40
[   57.128827]  [<ffffffff814c0494>] ? evtchn_read+0x594/0x5c0
[   57.128830]  [<ffffffff811e5fd7>] ? fcntl_setlk+0x197/0x400
[   57.128832]  [<ffffffff814c0740>] ? evtchn_ioctl+0x1c0/0x680
[   57.128834]  [<ffffffff811952b9>] ? vfs_read+0x129/0x230
[   57.128836]  [<ffffffff81195409>] ? SyS_read+0x49/0xc0
[   57.128839]  [<ffffffff81004952>] ? xen_mc_flush+0x172/0x1b0
[   57.128842]  [<ffffffff8172b235>] ? system_call_fastpath+0x12/0x17
[   57.128844]  [<ffffffff810037ce>] ? xen_load_sp0+0x7e/0x140
[   57.128846]  [<ffffffff8172b25c>] ? ret_from_sys_call+0x22/0x64
[   59.814099] PAX bytes1: 4, bytes2: 0, count: 4
[   59.814238] PAX bytes1: 4, bytes2: 0, count: 4
[   59.850396] PAX bytes1: 4, bytes2: 0, count: 4
[   59.885676] PAX bytes1: 4, bytes2: 0, count: 4
[   59.885792] PAX bytes1: 4, bytes2: 0, count: 4
[   59.970925] PAX bytes1: 4, bytes2: 0, count: 4
[   59.971036] PAX bytes1: 4, bytes2: 0, count: 4
[   60.978368] PAX bytes1: 4, bytes2: 0, count: 4
[   60.978577] PAX bytes1: 4, bytes2: 0, count: 4
[   61.036822] PAX bytes1: 4, bytes2: 0, count: 4
[   61.036845] PAX bytes1: 4, bytes2: 0, count: 4
[   61.051250] PAX bytes1: 4, bytes2: 0, count: 4
[   61.076874] PAX bytes1: 4, bytes2: 0, count: 4
[   61.076899] PAX bytes1: 4, bytes2: 0, count: 4


Code: Select all

minixen2 ~ # dmesg | egrep "PAX bytes1:" | awk -F" " '{ $1="";  $2=""; print}' | sort | uniq -c
   1630   PAX bytes1: 4, bytes2: 0, count: 4
minixen2 ~ #


This particular machine starts 4 VMs at startup.

[ephox]

Could you please test this patch:

Code: Select all

--- drivers/xen/evtchn.c 2014-03-31 12:48:25.863129794 +0200
+++ drivers/xen/evtchn.c        2015-07-13 22:26:13.834256111 +0200
@@ -201,8 +201,8 @@

        /* Byte lengths of two chunks. Chunk split (if any) is at ring wrap. */
        if (((c ^ p) & EVTCHN_RING_SIZE) != 0) {
-               bytes1 = (EVTCHN_RING_SIZE - EVTCHN_RING_MASK(c)) *
-                       sizeof(evtchn_port_t);
+               bytes1 = EVTCHN_RING_SIZE - EVTCHN_RING_MASK(c);
+               bytes1 *= sizeof(evtchn_port_t);
                bytes2 = EVTCHN_RING_MASK(p) * sizeof(evtchn_port_t);
        } else {
                bytes1 = (p - c) * sizeof(evtchn_port_t);



[gaima]

Could you please test this patch:

Code: Select all

--- drivers/xen/evtchn.c 2014-03-31 12:48:25.863129794 +0200
+++ drivers/xen/evtchn.c        2015-07-13 22:26:13.834256111 +0200
@@ -201,8 +201,8 @@

        /* Byte lengths of two chunks. Chunk split (if any) is at ring wrap. */
        if (((c ^ p) & EVTCHN_RING_SIZE) != 0) {
-               bytes1 = (EVTCHN_RING_SIZE - EVTCHN_RING_MASK(c)) *
-                       sizeof(evtchn_port_t);
+               bytes1 = EVTCHN_RING_SIZE - EVTCHN_RING_MASK(c);
+               bytes1 *= sizeof(evtchn_port_t);
                bytes2 = EVTCHN_RING_MASK(p) * sizeof(evtchn_port_t);
        } else {
                bytes1 = (p - c) * sizeof(evtchn_port_t);



I have literally no idea what that does, but it seems to have solved the problem.

"PAX bytes1: 4, bytes2: 0, count: 4" is overflowing dmesg merrily.
No processes are being terminated though.
VM operations are completely normal again.


Thanks!
Mike

[PaX Team]

what happens behind the scenes is a series of transformations by the compiler that ends up with an intentional integer overflow which the plugin instrumented code catches later at runtime (i.e., it's a known false positive case that we have to handle somehow). the story goes something like this: the original source code line can be summarized as (small_const1 - variable)*small_const2. later gcc turns this into small_const1*small_const2 - small_const2*variable and then into small_const3 + large_const2*variable. the latter term with large_const2 is what triggers the overflow detector eventually (in non-plugin generated code it'd simply end up with integer truncation and produce the correct result). what the 'fix' really does is prevent this transformation and thus no more overflow detection at runtime. i used quotes because this is unfortunately a fragile solution, if a later compiler version develops more smartness in recognizing these opportunities for transformation then we'd have to do something else (unfortunately our hands are tied with the plugin approach so this is the best we can do for now, short of disabling the plugin, that is).