KVM and grsecurity

[jdoe]

Hi all.

I was asking in #grsecurity the other day if anyone had reported problems with KVM and recent grsec patches enabled on both guest and host. My guest will consistently crash after a day or two with the following failure:

Code: Select all

KVM internal error. Suberror: 1
emulation failure
RAX=ffffffff8101848e RBX=000000004e2537ef RCX=0000000000000020 RDX=ffffffff81526510
RSI=000000004e2537ef RDI=0000000000000380 RBP=00000004e257e694 RSP=ffff8802168dbde0
R8 =0000000000000001 R9 =00000b48a017e21d R10=00000b48a017e21d R11=00000b48a017e21d
R12=0000000000000000 R13=00000b4d81ea4000 R14=ffff88021fd8c280 R15=000000000000c280
RIP=ffffffff81018490 RFL=00010092 [--S-A--] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00000000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0000 0000000000000000 ffffffff 00000000
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff88021fd80000 ffffffff 00000000
LDT=0000 0000000000000000 ffffffff 00000000
TR =0040 ffffffff816124c0 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff813ed000 0000007f
IDT=     ffffffff813ef040 00000fff
CR0=8005003b CR2=000002d891507200 CR3=00000000013d9000 CR4=001406b0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=24 3f c3 90 57 9d 0f 1f 44 00 00 48 0f ba 2c 24 3f c3 89 ff <89> b7 00 b0 5f ff 48 0f ba 2c 24 3f c3 89 ff 8b 87 00 b0 5f ff 48 0f ba 2c 24 3f c3 48 8b


So "mov DWORD PTR [rdi-0xa05000],esi" where rdi = 0x380.

- This has happened with all versions of the 3.2.55 grsec patch I've tested over the last week or two.
- It has occurred at least 4 times in the same place, with the same value of rdi. (possibly more, I just lost my old libvirt logs when downgrading.)
- The host kernel never crashes/BUGs, only the guest.
- Rolling KVM back from 1.7 to 1.1.2 had no effect.
- There are no useful debugging messages anywhere in the guest that I've found, even if a console is left open.
- I can't reproduce the crashes with a generic debian kernel in the guest.
- Matching the grsec config on the broken guest/host with a working guest/host had no effect. I'm not religious about keeping up to date though, that was kernel 3.2.50 and different host hardware. Same version of KVM.
- Turning off a few of the more obviously-potentially-breakage-causing features (RANDKSTACK, the new RANDSTRUCT stuff) had no effect.

The next step, I guess, is copying the 'known good' 3.2.50 guest kernel over and seeing if it dies. It will only fail every 1-3 days though, so that could take a while -- any suggestions for things I can do in the meantime?

host kernel config and guest kernel config.

[jdoe]

Code: Select all

ffffffff8101843a T setup_IO_APIC_irq_extra
ffffffff810184cc t arch_local_irq_restore
ffffffff810184da t native_apic_mem_write
ffffffff810184e9 t native_apic_mem_read


... based on System.map I guess it's dying in native_apic_mem_write, which looks like the asm I see in the KVM error. Since it and arch_local_irq_restore get inlined, I assume it's coming through setup_IO_APIC_irq_extra, but that only gets called on boot. I don't know how it could cause a crash days later.

[jdoe]

No luck, the 'known good' guest config that's been running on another host for several months crashed within a day, with the same KVM virtualization failure.

[PaX Team]

sorry for the belated reply, but is this still reproducible? i've looked at the code but i don't see why native_apic_mem_write(APIC_TMICT, ...) would suddenly fail...