PAX: refcount overflow detected in: drbd_r_kvm:8417, uid/eui

[schiffi]

Hi,

on a new installed cluster I just discovered the following refcount overflow detected by PAX.

Is this a real bug (in drb) or a false positive in PAX?

Kernel: gentoo-hardened (3.10.1-hardened-r1) which includes grsecurity patch 2.9.1-3.10.1-201307181236

EDIT: Where can I upload/send the kernel image and System.map?

Thanks!

Code: Select all

[38365.392331] PAX: refcount overflow detected in: drbd_r_kvm:8417, uid/euid: 0/0
[38365.392332] CPU: 4 PID: 8417 Comm: drbd_r_kvm Not tainted 3.10.1-hardened-r1 #3
[38365.392333] Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.0 07/05/2013
[38365.392335] task: ffff880852ea1080 ti: ffff880852ea14e0 task.ti: ffff880852ea14e0
[38365.392336] RIP: 0010:[<ffffffffa0bd7713>]  [<ffffffffa0bd7713>] bm_page_io_async+0x219/0x228 [drbd]
[38365.392340] RSP: 0018:ffff880854789c28  EFLAGS: 00000a12
[38365.392341] RAX: ffff88085318d408 RBX: 0000000000000008 RCX: ffff8808533a9350
[38365.392343] RDX: ffff880854789ba8 RSI: 0000000000000000 RDI: ffff88085318d408
[38365.392344] RBP: 0000000000007467 R08: ffff88085318d000 R09: 0000000000000008
[38365.392345] R10: 0000000000000007 R11: 0000000000000001 R12: ffff88085780c740
[38365.392346] R13: ffff880852861800 R14: ffff880855c0d940 R15: ffff880852f4a140
[38365.392348] FS:  0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000
[38365.392349] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[38365.392350] CR2: 00000f312ee7b040 CR3: 000000000141f000 CR4: 00000000000407f0
[38365.392352] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[38365.392353] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[38365.392353] Stack:
[38365.392354]  ffff880852861800 ffffea001d0e7ae0 00000001d19fffb0 00000001a0bef113
[38365.392356]  0000000000000008 0000000000000000 ffff880852ea1080 ffff880852ea14e0
[38365.392361]  ffff880852f4a140 0000000000000000 ffff880852861800 ffff880855c0d940
[38365.392364] Call Trace:
[38365.392367]  [<ffffffffa0bd7968>] ? bm_rw+0x246/0x583 [drbd]
[38365.392372]  [<ffffffffa0bf84c8>] ? drbd_md_clear_flag+0x24/0x24 [drbd]
[38365.392375]  [<ffffffffa0c0a706>] ? page_chain_add.part.23+0x3cf0/0xefe8 [drbd]
[38365.392379]  [<ffffffffa0bf8758>] ? drbd_bmio_set_n_write+0x290/0x2b1 [drbd]
[38365.392383]  [<ffffffffa0bf84c8>] ? drbd_md_clear_flag+0x24/0x24 [drbd]
[38365.392387]  [<ffffffffa0bf83c6>] ? drbd_bitmap_io+0x6e/0x90 [drbd]
[38365.392392]  [<ffffffffa0be2365>] ? receive_state+0x1029/0x1582 [drbd]
[38365.392395]  [<ffffffffa0bdf94d>] ? drbd_recv+0x22/0x168 [drbd]
[38365.392400]  [<ffffffffa0bea53a>] ? drbdd_init+0x13b/0x1c5 [drbd]
[38365.392404]  [<ffffffffa0bf2518>] ? drbd_thread_setup+0x44/0xff [drbd]
[38365.392408]  [<ffffffffa0bf24d4>] ? conn_destroy+0x86/0x86 [drbd]
[38365.392411]  [<ffffffff81071408>] ? kthread+0xad/0xb5
[38365.392413]  [<ffffffff81400000>] ? intel_pstate_cpu_init+0x72/0x2db
[38365.392416]  [<ffffffff8107135b>] ? kthread_freezable_should_stop+0x3b/0x3b
[38365.392418]  [<ffffffff814093a2>] ? ret_from_fork+0x72/0xa0
[38365.392421]  [<ffffffff8107135b>] ? kthread_freezable_should_stop+0x3b/0x3b
[38365.392422] Code: 24 48 4c 89 e6 49 c7 44 24 40 65 73 bd a0 8b 7c 24 1c e8 f0 87 62 e0 f0 41 01 9d d0 04 00 00 71 0a f0 41 29 9d d0 04 00 00 cd 04 <48> 83 c4 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 41 57 41 89 cf 41

[tjh]

Just a friendly pointer to the Reporting Bugs wiki page. It'll help get it resolved for you

[schiffi]

Thanks...

[spender]

Sorry about the delay, the PaX Team and I have been away (H2HC) but discussed your post. It'll be looked into and resolved soon.

Thanks,
-Brad

[PaX Team]

after having looked at the drbd code a bit i think this could be a real bug in drbd but only upstream can tell for sure so you'll have to contact them. you can show them the following that i figured out so far:

the refcount overflow was detected in drivers/block/drbd/drbd_bitmap.c:bm_page_io_async at the

atomic_add(len >> 9, &mdev->rs_sect_ev)

statement. rs_sect_ev is an atomic_t in struct drbd_conf declared in drivers/block/drbd/drbd_int.h (i'll note here that i think the rs_sect_in field is simiarly affected by this problem).

based on the code, these two fields don't look like refcounts, nor are they free-running counters or statistics either (the usual cases for false positives). instead they're some sector counts that get reset on certain events (the details of which i can't tell as i don't know the drbd code). therefore my feeling is that these counts are not supposed to overflow as they'd otherwise lead to incorrect calculations in drbd_rs_should_slow_down and drbd_rs_controller (the latter reads rs_sect_in into an unsigned int btw, this is mixing up signed/unsigned integers, that can't be good...).

so what happened to you is that somehow rs_sect_ev reached 2G (that corresponds to about 1TB of traffic between two counter resets or 'events') and the signed overflow detection triggered on it (if that's too unrealistic traffic for drbd then there was some other problem calculating the sector counts that resulted in some big enough value to trigger a signed overflow, though at the moment of the overflow 'len' had a value of 8 only). in any case it looks that an atomic_t is not enough to store real life sector counts and will have to be enlarged probably (or the counters will have to be reset more frequently).

[schiffi]

With gentoo kernel 3.14.18-hardened-r2 I had this issue again and this time I contacted the drbd devs. They prooved that this must be false positive:

> >Well, yes, why would it not overflow.
> >It is *not* a refcount.
> >It is an atomic counter.
> >It is meant to overflow.

So can you please add this to the positive list for CONFIG_PAX_REFCOUNT?

Thanks and sorry for this very late answer

[PaX Team]

thanks, is there some public record of this discussion? also what about rs_sect_in?

[schiffi]

Please see here:

http://article.gmane.org/gmane.linux.ke ... devel/2640

[PaX Team]

thanks, next time perhaps cc me too, it'd save on roundtrip times between this forum and their mailing list . as for his question about how refcount overflow detection/prevention works, we posted about this in the past on this forum, but the quick answer is that the protection does *not* allow the actual overflow to occur and will let the code continue until the triggering task is about to return to userland when the forced SIGKILL will be processed (in an exploit scenario this would very likely be the attacking process so killing it is the best course of action along with grsec's bruteforce detection and lockout feature).

[schiffi]

If I got it right you added that drbd to the whitelist, right? Which version will include the fix then?

And many thanks for PaX!

-Marc

[PaX Team]

yes, i've turned these non-refcount uses into atomic_unchecked_t types and the next patches will have the fix.