size overflow detected in function generic_file_read_iter

[GetinakS]

Hello,
I'm running Hardened Gentoo with kernel 4.4.2-hardened. Same problem was with 4.3.3-hardened-r4
And i've got an overflow everytime, when i'm trying to use parted/fdisk on a very large drive (16.3TB RAID 50 Unit), and every startup, when kernel trying to read partition table.

Code: Select all

[    9.650154] PAX: size overflow detected in function generic_file_read_iter mm/filemap.c:1541 cicus.566_336 max, count: 31, decl: page_cache_async_readahead; num: 5; context: fndecl;
[    9.650880] CPU: 0 PID: 1435 Comm: systemd-udevd Not tainted 4.4.2-hardened.fsck-u-1.7.5-r3 #1
[    9.650885] Hardware name: Intel    /SDS2, BIOS SDS21.86B.0052.P03.0301181422    01/18/2003
[    9.650891]  00000000 00000000 f3b0fe10 0051e139 00000286 c284acf9 f3b0fe38 0018466b
[    9.650905]  c2865458 c284ad48 c284acf9 00000605 c2840f54 f4020900 ed9f0000 0000105e
[    9.650915]  f3b0febc 0012590d c2840f54 ffff4111 ffff4111 ffff4111 ffff4111 ffff4111
[    9.650926] Call Trace:
[    9.650954]  [<0051e139>] dump_stack+0x42/0x59
[    9.650965]  [<0018466b>] report_size_overflow+0x6b/0x80
[    9.650976]  [<0012590d>] generic_file_read_iter+0x7fd/0x830
[    9.650986]  [<001bd5e3>] blkdev_read_iter+0x63/0xa0
[    9.650996]  [<0017dbaa>] __vfs_read+0xaa/0xe0
[    9.651003]  [<0017e213>] vfs_read+0x93/0x200
[    9.651026]  [<0017ef07>] SyS_read+0x47/0xa0
[    9.651037]  [<00002c5f>] do_syscall_32_irqs_on+0x5f/0x180
[    9.651049]  [<00c0decb>] entry_INT80_32+0x3b/0x47
[    9.651061]  [<0001007b>] ? convert_from_fxsr+0x13b/0x160


That's don't happens, when i'm trying to use parted on small drive (4Gb /160 Gb).
My hardware config: Intel SDS2, 6Gb RAM, 3Ware 9550SX RAID ( that "disk" is located on 3ware controller)

[PaX Team]

can you print out the value of *ppos in do_generic_file_read like this:

Code: Select all

--- a/mm/filemap.c       2016-01-29 17:04:53.377255733 +0100
+++ b/mm/filemap.c        2016-02-28 10:40:19.689919097 +0100
@@ -1538,6 +1538,7 @@
        unsigned int prev_offset;
        int error = 0;

+       printk("PAX: ppos:%Lx\n", *ppos);
        index = *ppos >> PAGE_CACHE_SHIFT;
        prev_index = ra->prev_pos >> PAGE_CACHE_SHIFT;
        prev_offset = ra->prev_pos & (PAGE_CACHE_SIZE-1);


if i'm not mistaken, this is a very serious problem throughout the vm/vfs where volumes/filesystems larger than 16TB cannot be handled by 32 bit kernels and you should not be able to create them in the first place. what likely happens here (we'll see from the above debug message for sure) is that when the kernel tries to turn a 64 bit file offset into a page cache index, the result of the shift gets truncated (index is 32 bit only on a 32 bit kernel which can hold values of file offsets up to 44 bits) and the kernel will access the beginning of the volume instead of above the 16TB offset as intended. needless to say that this will result in data corruption all over the place. unfortunately blockdev doesn't catch the out-of-bounds access at a higher layer either as it (correctly) uses full 64 bit offsets. i don't know if it's fixable at all as pgoff_t would have to be changed into a 64 bit type which has a cascading effect everywhere.

[GetinakS]

Thanks for assistance!
I've got this sequence:
0, 0, 200, 400, 600, 800, a00, c00, e00, [growing up everytime by 0x200], 1e00, 00, [repeating 00 seven times], 10000, 00 and after that get size overflow.

[PaX Team]

can you post the dmesg itself?

[GetinakS]

Here is part of dmesg (full dmesg is >65Kb, http://pastebin.com/raw/j6M0XKqZ)

Code: Select all

[   33.527578] program parted is using a deprecated SCSI ioctl, please convert it to SG_IO
[   37.288545] PAX: ppos:0
[   37.290484] PAX: ppos:0
[   37.291956] PAX: ppos:200
[   37.293425] PAX: ppos:400
[   37.294895] PAX: ppos:600
[   37.296365] PAX: ppos:800
[   37.297828] PAX: ppos:a00
[   37.299295] PAX: ppos:c00
[   37.300759] PAX: ppos:e00
[   37.302231] PAX: ppos:1000
[   37.303750] PAX: ppos:1200
[   37.305273] PAX: ppos:1400
[   37.306757] PAX: ppos:1600
[   37.308243] PAX: ppos:1800
[   37.309724] PAX: ppos:1a00
[   37.311201] PAX: ppos:1c00
[   37.312675] PAX: ppos:1e00
[   37.314159] PAX: ppos:0
[   37.315641] PAX: ppos:0
[   37.316932] PAX: ppos:0
[   37.318011] PAX: ppos:0
[   37.319072] PAX: ppos:0
[   37.320088] PAX: ppos:0
[   37.320938] PAX: ppos:0
[   37.321674] PAX: ppos:0
[   37.322300] PAX: ppos:10000
[   37.323211] PAX: ppos:0
[   37.323852] PAX: size overflow detected in function generic_file_read_iter mm/filemap.c:1541 cicus.566_338 max, count: 31, decl: page_cache_async_readahead; num: 5; context: fndecl;
[   37.327112] CPU: 0 PID: 1195 Comm: parted Not tainted 4.4.2-hardened.fsck-u-1.7.5-r3 #2
[   37.330222] Hardware name: Intel    /SDS2, BIOS SDS21.86B.0052.P03.0301181422    01/18/2003
[   37.334109]  00000000 00000000 f3823da4 0051e159 00000296 c284acf9 f3823dcc 0018468b
[   37.338852]  c2865468 c284ad48 c284acf9 00000605 c2840f54 f4020540 f4502f00 f3823e78
[   37.344410]  f3823e4c 001258d3 c2840f54 ffff4111 ffff4111 ffff4111 ffff4111 ffff4111
[   37.350773] Call Trace:
[   37.354038]  [<0051e159>] dump_stack+0x42/0x59
[   37.354050]  [<0018468b>] report_size_overflow+0x6b/0x80
[   37.354059]  [<001258d3>] generic_file_read_iter+0x7c3/0x850
[   37.354065]  [<001bd603>] blkdev_read_iter+0x63/0xa0
[   37.354073]  [<0017dbca>] __vfs_read+0xaa/0xe0
[   37.354077]  [<0017e233>] vfs_read+0x93/0x200
[   37.354081]  [<0017ef27>] SyS_read+0x47/0xa0
[   37.354089]  [<00002c5f>] do_syscall_32_irqs_on+0x5f/0x180
[   37.354097]  [<00c0deeb>] entry_INT80_32+0x3b/0x47
[   37.354102]  [<00c0def7>] ? entry_INT80_32+0x47/0x47
[   47.600820] PAX: ppos:0
[   47.604882] PAX: ppos:34
[   47.608620] PAX: ppos:174
[   47.612185] PAX: ppos:0


At 33s i've run parted /dev/sdb (16.3 TB disk), at 37s i've type "print" command, that cause overflow

[PaX Team]

hmm, are you sure you properly applied the patch? thing is, the size overflow message line number should have changed to 1542 but it still shows 1541...

[GetinakS]

Yes,yes,yes. I've insert printk after, it's my mistake, here is a correct version:

Code: Select all

[   25.639708] program parted is using a deprecated SCSI ioctl, please convert it to SG_IO
[   28.682446] PAX: ppos:0
[   28.684390] PAX: ppos:0
[   28.685861] PAX: ppos:200
[   28.687333] PAX: ppos:400
[   28.688803] PAX: ppos:600
[   28.690274] PAX: ppos:800
[   28.691742] PAX: ppos:a00
[   28.693213] PAX: ppos:c00
[   28.694683] PAX: ppos:e00
[   28.696151] PAX: ppos:1000
[   28.697669] PAX: ppos:1200
[   28.699188] PAX: ppos:1400
[   28.700666] PAX: ppos:1600
[   28.702154] PAX: ppos:1800
[   28.703656] PAX: ppos:1a00
[   28.705136] PAX: ppos:1c00
[   28.706608] PAX: ppos:1e00
[   28.708092] PAX: ppos:0
[   28.709571] PAX: ppos:0
[   28.710860] PAX: ppos:0
[   28.711933] PAX: ppos:0
[   28.712994] PAX: ppos:0
[   28.714015] PAX: ppos:0
[   28.714861] PAX: ppos:0
[   28.715596] PAX: ppos:0
[   28.716220] PAX: ppos:10000
[   28.717144] PAX: ppos:0
[   28.717785] PAX: ppos:105eed9ff000
[   28.718481] PAX: size overflow detected in function generic_file_read_iter mm/filemap.c:1542 cicus.566_338 max, count: 31, decl: page_cache_async_readahead; num: 5; context: fndecl;
[   28.721824] CPU: 0 PID: 1194 Comm: parted Not tainted 4.4.2-hardened.fsck-u-1.7.5-r3 #3
[   28.724987] Hardware name: Intel    /SDS2, BIOS SDS21.86B.0052.P03.0301181422    01/18/2003
[   28.728935]  00000000 00000000 f471ddbc 0051e159 00000286 c284acf9 f471dde4 0018467b
[   28.733736]  c2865468 c284ad48 c284acf9 00000606 c2840f54 f4020540 f3cdb180 f471de90
[   28.739335]  f471de64 001258c3 c2840f54 ed9ff000 0000105e ffff4111 ffff4111 ffff4111
[   28.745739] Call Trace:
[   28.749014]  [<0051e159>] dump_stack+0x42/0x59
[   28.752439]  [<0018467b>] report_size_overflow+0x6b/0x80
[   28.755990]  [<001258c3>] generic_file_read_iter+0x7b3/0x840
[   28.759667]  [<001bd5f3>] blkdev_read_iter+0x63/0xa0
[   28.763431]  [<0017dbba>] __vfs_read+0xaa/0xe0
[   28.767270]  [<0017e223>] vfs_read+0x93/0x200
[   28.771183]  [<0017ef17>] SyS_read+0x47/0xa0
[   28.775150]  [<00002c5f>] do_syscall_32_irqs_on+0x5f/0x180
[   28.779252]  [<00c0deeb>] entry_INT80_32+0x3b/0x47
[   28.779260]  [<00c00000>] ? l2tp_ip6_sendmsg+0x390/0x7b0
[   28.779272]  [<00040033>] ? can_boost+0x3/0x190
[   28.779282]  [<00010206>] ? fpregs_get+0xc6/0x160
[   50.180329] PAX: ppos:0
[   50.187525] PAX: ppos:34
[   50.187633] PAX: ppos:174
[   50.187758] PAX: ppos:0


Full dmesg output after parted got killed: http://pastebin.com/b4hMTKt4

[PaX Team]

much better (in a sense anyway ), 105eed9ff000 is a 45 bit value thus the integer truncation problem i described above. now it's your turn to report all this to the kernel developers so that they can figure out what to do.

[PaX Team]

from twitter it seems that the problem was recognized in 2009 already but apparently nothing was done ever since. my best advice is to not use such big devices with a 32 bit kernel.

[GetinakS]

Thank you! I'll report it to kernel developers. Anyway, i've remember, that on vanilla kernel (version 3.1.x, i don't remember exact version) i've been able to work with this array, so if i didn't make a decision to upgrade system... someday i'll ended with damaged data (this box is planned to be a backup server, and a damaged backup is a very bad thing).