Consul triggering Kernel crash
Posted: Thu Feb 19, 2015 5:30 am
- Code: Select all
Feb 19 07:11:04 kernel: list_del corruption. prev->next should be ffff88007801e580, but was fefefefefefefefe
Feb 19 07:11:04 kernel: ------------[ cut here ]------------
Feb 19 07:11:04 kernel: kernel BUG at lib/list_debug.c:87!
Feb 19 07:11:04 kernel: invalid opcode: 0000 [#1] SMP
Feb 19 07:11:04 kernel: Modules linked in: netconsole configfs ipv6 ppdev microcode pcspkr e1000 parport_pc parport i2c_piix4 i2c_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif crct10dif_common sr_mod cdrom floppy mptsas mptscsih mptbase scsi_transport_sas pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod
Feb 19 07:11:04 kernel: CPU: 1 PID: 30919 Comm: consul Not tainted 3.14.32-100.el6.x86_64 #1
Feb 19 07:11:04 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Feb 19 07:11:04 kernel: task: ffff88007803c360 ti: ffff88007803cad8 task.ti: ffff88007803cad8
Feb 19 07:11:04 kernel: RIP: 0010:[<ffffffff812f44ac>] [<ffffffff812f44ac>] __list_del_entry_debug+0x7c/0xa0
Feb 19 07:11:04 kernel: RSP: 0018:ffffc9000e5a3d80 EFLAGS: 00010083
Feb 19 07:11:04 kernel: RAX: 0000000000000054 RBX: ffff88007801e580 RCX: 0000000000000000
Feb 19 07:11:04 kernel: RDX: ffff88007fd0cf00 RSI: ffff88007fd0b508 RDI: 0000000000000046
Feb 19 07:11:04 kernel: RBP: ffffc9000e5a3d88 R08: 0000000000000092 R09: 0000000002000000
Feb 19 07:11:04 kernel: R10: 00000000000004c7 R11: ffffea0000db9f40 R12: ffff88007801e568
Feb 19 07:11:04 kernel: R13: 0000000000000282 R14: ffff88007961a280 R15: 0000000000000003
Feb 19 07:11:04 kernel: FS: 000002d2b1f2b700(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
Feb 19 07:11:04 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 19 07:11:04 kernel: CR2: ffffffffff600400 CR3: 0000000001647000 CR4: 00000000000007f0
Feb 19 07:11:04 kernel: Stack:
Feb 19 07:11:04 kernel: ffff88007801e580 ffffc9000e5a3da0 ffffffff812f4580 ffff88007801e580
Feb 19 07:11:04 kernel: ffffc9000e5a3db8 ffffffff812f45b0 ffff88007814be80 ffffc9000e5a3de0
Feb 19 07:11:04 kernel: ffffffff810a7b09 ffff88007801e550 ffff880036ebfb40 ffff880036ebfb80
Feb 19 07:11:04 kernel: Call Trace:
Feb 19 07:11:04 kernel: [<ffffffff812f4580>] __list_del_entry+0x10/0x30
Feb 19 07:11:04 kernel: [<ffffffff812f45b0>] list_del+0x10/0x30
Feb 19 07:11:04 kernel: [<ffffffff810a7b09>] remove_wait_queue+0x29/0x50
Feb 19 07:11:04 kernel: [<ffffffff811fefba>] ep_unregister_pollwait.isra.8+0x3a/0x60
Feb 19 07:11:04 kernel: [<ffffffff811ff002>] ep_remove+0x22/0xd0
Feb 19 07:11:04 kernel: [<ffffffff8120046c>] SyS_epoll_ctl+0x41c/0xa30
Feb 19 07:11:04 kernel: [<ffffffff816389ff>] system_call_fastpath+0x16/0x1b
Feb 19 07:11:04 kernel: [<ffffffff81638a36>] ? sysret_check+0x2d/0x6c
Feb 19 07:11:04 kernel: [<ffffffff8162f4cc>] ? retint_swapgs+0x13/0x16
Feb 19 07:11:04 kernel: Code: e8 9f b6 32 00 0f 0b 48 89 de 48 c7 c7 88 6b db 81 31 c0 e8 8c b6 32 00 0f 0b 48 89 de 48 c7 c7 48 6b db 81 31 c0 e8 79 b6 32 00 <0f> 0b 48 89 de 48 c7 c7 10 6b db 81 31 c0 e8 66 b6 32 00 0f 0b
Feb 19 07:11:04 kernel: RIP [<ffffffff812f44ac>] __list_del_entry_debug+0x7c/0xa0
Feb 19 07:11:04 kernel: RSP <ffffc9000e5a3d80>
Feb 19 07:11:04 kernel: ---[ end trace 6a8fa99325ee51c4 ]---
Feb 19 07:11:04 kernel: grsec: banning user with uid 515 until system restart for suspicious kernel crash
In this case uid 515 is the user running the consul daemon on those servers. Reproducible every 24h or so...