size overflow in function virtnet_receive

[jdoe]

Hey,

With 4.3.4 and grsecurity-3.1-4.3.4-201601231215.patch I get a panic almost immediately on boot:

Code: Select all

PAX: size overflow detected in function virtnet_receive drivers/net/virtio_net.c:354 cicus.800_438 max, count: 123, decl: page_to_skb; num: 4; context: fndecl;


[ephox]

Could you please apply this patch and send me the result from dmesg?

Code: Select all

--- drivers/net/virtio_net.c.orig       2016-01-24 21:18:00.515405396 +0100
+++ drivers/net/virtio_net.c    2016-01-24 21:22:02.971394587 +0100
@@ -351,12 +351,15 @@
        struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
        u16 num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
        struct page *page = virt_to_head_page(buf);
-       int offset = buf - page_address(page);
+       int offset;
        unsigned int truesize = max(len, mergeable_ctx_to_buf_truesize(ctx));
+       struct sk_buff *head_skb, *curr_skb;
 
-       struct sk_buff *head_skb = page_to_skb(vi, rq, page, offset, len,
+       printk(KERN_ERR "PAX overflow: buf: %p page_address(page): %p\n", buf, page_address(page));
+       offset = buf - page_address(page);
+       head_skb = page_to_skb(vi, rq, page, offset, len,
                                               truesize);
-       struct sk_buff *curr_skb = head_skb;
+       curr_skb = head_skb;
 
        if (unlikely(!curr_skb))
                goto err_skb;


[jdoe]

I realized after submitting that this is the same issue I reported before. As before, I am unable to reproduce with your patch applied.

[jdoe]

This is still an issue with grsecurity-3.1-4.4.2-201602182048.patch.

With the stock grsec patch I will trigger this reliably on every boot before I get a login prompt on the console.

Code: Select all

[   25.672156] PAX: size overflow detected in function virtnet_receive drivers/net/virtio_net.c:360 cicus.794_435 max, count: 123, decl: page_to_skb; num: 4; context: fndecl;
[   25.673760] CPU: 1 PID: 463 Comm: ip Not tainted 4.4.2-grsec-guest #1
[   25.675416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[   25.677428]  53d0c28798196ab6 53d0c28798196ab6 0000000000000000 ffffffff8c2e52a7
[   25.679639]  ffff880333349805 53d0c28798196ab6 ffffffff8c6bdfed ffffffff8c167fd1
[   25.681926]  0000000000000092 ffff88033fc43ec0 0000000000000600 0000000000000000
[   25.684169] Call Trace:
[   25.684857]  <IRQ>  [<ffffffff8c2e52a7>] ? 0xffffffff8c2e52a7
[   25.686346]  [<ffffffff8c167fd1>] ? 0xffffffff8c167fd1
[   25.688185]  [<ffffffff8c3e082e>] ? 0xffffffff8c3e082e
[   25.689596]  [<ffffffff8c0aca67>] ? 0xffffffff8c0aca67
[   25.690971]  [<ffffffff8c3e10d8>] ? 0xffffffff8c3e10d8
[   25.692189]  [<ffffffff8c3e10d8>] ? 0xffffffff8c3e10d8
[   25.693256]  [<ffffffff8c45947e>] ? 0xffffffff8c45947e
[   25.694426]  [<ffffffff8c0814ee>] ? 0xffffffff8c0814ee
[   25.695593]  [<ffffffff8c570bfc>] ? 0xffffffff8c570bfc
[   25.696744]  <EOI>  [<ffffffff8c08165c>] ? 0xffffffff8c08165c
[   25.698076]  [<ffffffff8c0816d2>] ? 0xffffffff8c0816d2
[   25.699244]  [<ffffffff8c3e0542>] ? 0xffffffff8c3e0542
[   25.700309]  [<ffffffff8c45a5f4>] ? 0xffffffff8c45a5f4
[   25.701382]  [<ffffffff8c45a873>] ? 0xffffffff8c45a873
[   25.702658]  [<ffffffff8c45a92d>] ? 0xffffffff8c45a92d
[   25.703718]  [<ffffffff8c46a453>] ? 0xffffffff8c46a453
[   25.704777]  [<ffffffff8c46b4ca>] ? 0xffffffff8c46b4ca
[   25.705835]  [<ffffffff8c46b2a4>] ? 0xffffffff8c46b2a4
[   25.707052]  [<ffffffff8c46b25b>] ? 0xffffffff8c46b25b
[   25.708165]  [<ffffffff8c2d8e5f>] ? 0xffffffff8c2d8e5f
[   25.709370]  [<ffffffff8c085dbd>] ? 0xffffffff8c085dbd
[   25.710481]  [<ffffffff8c4691cd>] ? 0xffffffff8c4691cd
[   25.711548]  [<ffffffff8c478d69>] ? 0xffffffff8c478d69
[   25.712633]  [<ffffffff8c46901d>] ? 0xffffffff8c46901d
[   25.713754]  [<ffffffff8c47c910>] ? 0xffffffff8c47c910
[   25.715337]  [<ffffffff8c46900b>] ? 0xffffffff8c46900b
[   25.716800]  [<ffffffff8c47ab6a>] ? 0xffffffff8c47ab6a
[   25.717918]  [<ffffffff8c47b7a7>] ? 0xffffffff8c47b7a7
[   25.719351]  [<ffffffff8c43b9cd>] ? 0xffffffff8c43b9cd
[   25.720649]  [<ffffffff8c43cddc>] ? 0xffffffff8c43cddc
[   25.721823]  [<ffffffff8c43e289>] ? 0xffffffff8c43e289
[   25.722990]  [<ffffffff8c43e289>] ? 0xffffffff8c43e289
[   25.724140]  [<ffffffff8c56f4f0>] ? 0xffffffff8c56f4f0
[   25.725283] Kernel panic - not syncing: Aiee, killing interrupt handler!
[   25.727118] Kernel Offset: 0xb000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   25.729517] ---[ end Kernel panic - not syncing: Aiee, killing interrupt handler!


With your debugging patch applied I am no longer able to trigger the above. The vm boots successfully, login prompt is displayed, and I get a bunch of log spam from the debug printk. However, when I try to SSH into the vm I trigger this panic reliably:

Code: Select all

[   53.791837] PAX: size overflow detected in function virtnet_receive drivers/net/virtio_net.c:408 cicus.754_395 max, count: 93, decl: skb_add_rx_frag; num: 4; context: fndecl;
[   53.794975] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.2-grsec-guest #2
[   53.796448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[   53.798565]  dbc8fb5f43ba9861 dbc8fb5f43ba9861 0000000000000000 ffffffffbd2e52a7
[   53.800516]  0000000000000900 dbc8fb5f43ba9861 ffffffffbd6bdfed ffffffffbd167fd1[   53.802275] PAX overflow: buf: ffff88032f27aa00 page_address(page): ffff88032f278000

[   53.803973]
[   53.804553]  ffff8800bb9b6f00 ffff88033fc83e80 ffff8800bb9b6f00 0000000000004200
[   53.806497] Call Trace:
[   53.807117]  <IRQ>  [<ffffffffbd2e52a7>] ? 0xffffffffbd2e52a7
[   53.808455]  [<ffffffffbd167fd1>] ? 0xffffffffbd167fd1
[   53.809595]  [<ffffffffbd3e0bdc>] ? 0xffffffffbd3e0bdc
[   53.810725]  [<ffffffffbd3e111e>] ? 0xffffffffbd3e111e
[   53.811858]  [<ffffffffbd3e111e>] ? 0xffffffffbd3e111e
[   53.812990]  [<ffffffffbd4594c4>] ? 0xffffffffbd4594c4
[   53.814136]  [<ffffffffbd0814ee>] ? 0xffffffffbd0814ee
[   53.815268]  [<ffffffffbd081792>] ? 0xffffffffbd081792
[   53.816398]  [<ffffffffbd004562>] ? 0xffffffffbd004562
[   53.817532]  [<ffffffffbd570013>] ? 0xffffffffbd570013
[   53.818666]  <EOI>  [<ffffffffbd00a0d6>] ? 0xffffffffbd00a0d6
[   53.820004]  [<ffffffffbd032776>] ? 0xffffffffbd032776
[   53.821142]  [<ffffffffbd00a0db>] ? 0xffffffffbd00a0db
[   53.822273]  [<ffffffffbd0ad476>] ? 0xffffffffbd0ad476
[   53.822359] PAX overflow: buf: ffff88032f71aa00 page_address(page): ffff88032f718000
[   53.822362] PAX: size overflow detected in function virtnet_receive drivers/net/virtio_net.c:408 cicus.754_395 max, count: 93, decl: skb_add_rx_frag; num: 4; context: fndecl;
[   53.829525]  [<ffffffffbd02b473>] ? 0xffffffffbd02b473
[   53.830895]  [<ffffffffbd029d63>] ? 0xffffffffbd029d63
[   53.832262] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.4.2-grsec-guest #2
[   53.832263] Kernel panic - not syncing: Aiee, killing interrupt handler!
[   53.834159] Kernel Offset: 0x3c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   53.837762] ---[ end Kernel panic - not syncing: Aiee, killing interrupt handler!


The location of the panic has moved to the "buf - page_address(page);" in the main loop. After applying a similar printk before that I get even more logspam, but I am no longer able to trigger either panic. Removing the printks cause both to return. Do you have any suggestions for tracking this down more usefully?

[PaX Team]

what happens if you just change the type of 'offset' to long?

[jdoe]

Both long and unsigned int (to match the prototypes for page_to_skb and skb_add_rx_frag) panic at line 360.

[jdoe]

This is still an issue on 4.5.7 with grsecurity-3.1-4.5.7-201606080852.patch

It triggers reliably on a Debian Stretch KVM guest virtual machine when the system tries to bring up network interfaces on boot.

Code: Select all

[   13.875018] PAX: size overflow detected in function virtnet_receive drivers/net/virtio_net.c:360 cicus.802_426 max, count: 123, decl: page_to_skb; num: 4; context: fndecl;
[   13.878829] CPU: 3 PID: 377 Comm: ip Not tainted 4.5.7-grsec #2
[   13.880161]  fbb6a65efc345c8d fbb6a65efc345c8d 0000000000000286 ffffffffb82c4751
[   13.882553]  ffffffffb8638fb4 fbb6a65efc345c8d ffffffffb8638a2e 0000000000000168
[   13.884741]  ffffffffb81836e9 0000000000000048 ffff88023fd83ec0 0000000000000600
[   13.887243] Call Trace:
[   13.887862]  <IRQ>  [<ffffffffb82c4751>] ? 0xffffffffb82c4751
[   13.889156]  [<ffffffffb81836e9>] ? 0xffffffffb81836e9
[   13.890174]  [<ffffffffb83a7277>] ? 0xffffffffb83a7277
[   13.891245]  [<ffffffffb80ac0f5>] ? 0xffffffffb80ac0f5
[   13.892283]  [<ffffffffb851a28a>] ? 0xffffffffb851a28a
[   13.893316]  [<ffffffffb80a58fd>] ? 0xffffffffb80a58fd
[   13.894342]  [<ffffffffb83a7b24>] ? 0xffffffffb83a7b24
[   13.895392]  [<ffffffffb83a7b24>] ? 0xffffffffb83a7b24
[   13.898239]  [<ffffffffb83f3ab7>] ? 0xffffffffb83f3ab7
[   13.899259]  [<ffffffffb8086331>] ? 0xffffffffb8086331
[   13.900359]  [<ffffffffb851c23c>] ? 0xffffffffb851c23c
[   13.912481]  <EOI>  [<ffffffffb80864ab>] ? 0xffffffffb80864ab
[   13.914582]  [<ffffffffb8086529>] ? 0xffffffffb8086529
[   13.915855]  [<ffffffffb83a6f73>] ? 0xffffffffb83a6f73
[   13.917382]  [<ffffffffb83f4f1f>] ? 0xffffffffb83f4f1f
[   13.918816]  [<ffffffffb83f51da>] ? 0xffffffffb83f51da
[   13.919875]  [<ffffffffb83f52a3>] ? 0xffffffffb83f52a3
[   13.921223]  [<ffffffffb8406018>] ? 0xffffffffb8406018
[   13.922555]  [<ffffffffb8407221>] ? 0xffffffffb8407221
[   13.923591]  [<ffffffffb8406fd4>] ? 0xffffffffb8406fd4
[   13.925591]  [<ffffffffb8406f8b>] ? 0xffffffffb8406f8b
[   13.927419]  [<ffffffffb82b7a4f>] ? 0xffffffffb82b7a4f
[   13.929631]  [<ffffffffb808ae56>] ? 0xffffffffb808ae56
[   13.931272]  [<ffffffffb8404c3e>] ? 0xffffffffb8404c3e
[   13.935952]  [<ffffffffb841575b>] ? 0xffffffffb841575b
[   13.937171]  [<ffffffffb84169f9>] ? 0xffffffffb84169f9
[   13.938496]  [<ffffffffb8404a76>] ? 0xffffffffb8404a76
[   13.939739]  [<ffffffffb841a8b8>] ? 0xffffffffb841a8b8
[   13.943495]  [<ffffffffb8404a54>] ? 0xffffffffb8404a54
[   13.944730]  [<ffffffffb841898f>] ? 0xffffffffb841898f
[   13.946041]  [<ffffffffb8419616>] ? 0xffffffffb8419616
[   13.947342]  [<ffffffffb83d458f>] ? 0xffffffffb83d458f
[   13.948723]  [<ffffffffb83d5a45>] ? 0xffffffffb83d5a45
[   13.955835]  [<ffffffffb83d706b>] ? 0xffffffffb83d706b
[   13.956905]  [<ffffffffb83d706b>] ? 0xffffffffb83d706b
[   13.958062]  [<ffffffffb851ab30>] ? 0xffffffffb851ab30
[   13.959133] Kernel panic - not syncing: Aiee, killing interrupt handler!
[   13.960654] Kernel Offset: 0x37000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   13.962679] ---[ end Kernel panic - not syncing: Aiee, killing interrupt handler!


[PaX Team]

does the bug still disappear when you try to print out those values?

[spender]

Hi,

Could you follow up on this? We would like to work with you to have this fixed before releasing a 4.6 patch.

Thanks,
-Brad

[jdoe]

Hi, sorry. I just retested with today's patch, grsecurity-3.1-4.5.7-201606282216.patch.

Same as before, without debug printks (the two "buf - page_address(page);"s around virtio_net.c:360 and virtio_net.c:405) I get more-or-less instant panics when the network is brought up. When I add the debug printks (and make no other changes), I get a massive amount of log spam but no panic.