Strange traces when booting.

[hammerzeit]

Hi.

I'm using grsec with CentOS 6.3 and I am seeing some strange issues when booting the grsec kernel.
I have attached the error below.

Now, on some machines, it throws this trace during boot, but happily continues booting afterward. On some machines it doesn't, and on some it doesn't throw this error at all. It happens on both the Dell below, and on a Supermicro, so I can be reasonably sure it's not hardware-related.
\
Googling on the various pieces of the trace has led me to believe this is a timer-related issue, however I'm not 100% sure.
There's some mention on kernel.org about this bug being introduced/fixed around 2.6.24 or thereabouts, so I think I have a different issue.

I have tried the following:
-Checked if the time in bios was OK, it is.
-Disabled HPET in the bios, no difference.
-Tried hpet=disable and noapic kernel options, this doesn't seem to make a difference either.

Now, the stock CentOS kernel boots fine.

So I'm puzzled as to why this is happening, and I hope you guys may shed some light on this.
Apologies if this came up earlier, I've used the forum search but could only find topics on compilation problems.

Code: Select all

------------[ cut here ]------------
WARNING: at kernel/time/clockevents.c:112 clockevents_program_event+0x81/0xa0()
Hardware name: PowerEdge R410
Modules linked in: ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Not tainted 2.6.32.60-1.el6.grsec.custom.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff81061b1b>] warn_slowpath_common+0x7b/0xc0
 [<ffffffff81061b74>] warn_slowpath_null+0x14/0x20
 [<ffffffff81097851>] clockevents_program_event+0x81/0xa0
 [<ffffffff81098dc5>] tick_dev_program_event+0x65/0xc0
 [<ffffffff81098e9a>] tick_program_event+0x2a/0x30
 [<ffffffff8108dd94>] hrtimer_interrupt+0x1c4/0x260
 [<ffffffff814d023c>] smp_apic_timer_interrupt+0x6c/0x9c
 [<ffffffff8100bc83>] apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff812d0a1d>] ? acpi_idle_enter_c1+0xa3/0xc1
 [<ffffffff812d09fc>] ? acpi_idle_enter_c1+0x82/0xc1
 [<ffffffff813d6d24>] ? cpuidle_idle_call+0xa4/0x130
 [<ffffffff81009e66>] ? cpu_idle+0xb6/0x110
 [<ffffffff814c2304>] ? start_secondary+0x1fb/0x23e
---[ end trace 013cefb09051f8b6 ]---
hrtimer: interrupt took 599 ns
WARNING: at kernel/time/clockevents.c:112 clockevents_program_event+0x81/0xa0()
Hardware name: PowerEdge R410
Modules linked in: ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 0, comm: swapper Tainted: G        W  2.6.32.60-1.el6.grsec.custom.x86_64 #1
Call Trace:
 <IRQ>  [<ffffffff81061b1b>] warn_slowpath_common+0x7b/0xc0
 [<ffffffff81061b74>] warn_slowpath_null+0x14/0x20
 [<ffffffff81097851>] clockevents_program_event+0x81/0xa0
 [<ffffffff81098dc5>] tick_dev_program_event+0x65/0xc0
 [<ffffffff81098e9a>] tick_program_event+0x2a/0x30
 [<ffffffff8108dd94>] hrtimer_interrupt+0x1c4/0x260
 [<ffffffff814d023c>] smp_apic_timer_interrupt+0x6c/0x9c
 [<ffffffff8100bc83>] apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff812d0a1d>] ? acpi_idle_enter_c1+0xa3/0xc1
 [<ffffffff812d09fc>] ? acpi_idle_enter_c1+0x82/0xc1
 [<ffffffff813d6d24>] ? cpuidle_idle_call+0xa4/0x130
 [<ffffffff81009e66>] ? cpu_idle+0xb6/0x110
 [<ffffffff814b175a>] ? rest_init+0x7a/0x80
 [<ffffffff84b90ded>] ? start_kernel+0x401/0x40d
 [<ffffffff84b902fa>] ? x86_64_start_reservations+0x125/0x129
 [<ffffffff84b903f8>] ? x86_64_start_kernel+0xfa/0x109
---[ end trace 013cefb09051f8b7 ]---

[PaX Team]

can you test with a vanilla 2.6.32.x kernel as well? just to see if it's really not present in mainline.

[hammerzeit]

Hello,
Sorry for the delay in posting.

So anyway I built a vanilla kernel (2.6.32.60) without the grsec patches and it indeed shows the same issues. Using udevdebug and modprobedebug I've traced it down a bit further, it seems to go wrong at the udev stage, but I'm not entirely sure where exactly. The timing issue is suspect but it may be a red herring. When it does go wrong there's no real error message, it just hangs forever.

Interestingly enough it seems that leaving the server overnight it will continue booting at some point.

I've tried various combinations of options like notsc, nohpet hpet=off, hpet=force, clocksource=[acpi_pm,tsc,hpet] etc etc... No luck so far.

But like you say, it seems to be occuring with the mainline kernel also.