Hanging kernels after 2.6.3x

[renton]

Hi,

Background:
https://lkml.org/lkml/2013/2/9/63

http://forums.grsecurity.net/viewtopic.php?f=3&t=3230#p12480

Now I'm writing a new topic as there appear gr_ functions in this trace. The reason of kernel hanging is maybe grsec patch, can it be possible? And do you happen to know how to fix it?
Thanks.

2.6.35.7 trace:
https://www.dropbox.com/s/4wdxy0jgtuikr ... inlock.txt

3.7.3 trace:
https://www.dropbox.com/s/0mju5kc0pwb3t ... lockup.txt

[spender]

The issue shown in the 2.6.35.7 kernel has already been fixed (calling gr_learn_resource in interrupt context). Any grsec patch from at least a year ago would have had the fix. The 3.7.3 kernel issue is something different. Please give me a kernel log from a current grsecurity patch, as I recently fixed an RBAC bug with similar stack traces.

Please don't bother the upstream developers with grsec problems. If you provide the information I ask for, I can resolve any issue.

-Brad

[renton]

Thank you for your answer!

What do you mean by "kernel log"? Is it what syslog reads from /proc/kmsg and writes for example in /var/log/messages?

[spender]

I mean the same kinds of oops/call trace/etc reports that you submitted upstream. I need that information to find the source of the problem (if it still exists).

-Brad

[renton]

Except for this trace:
https://www.dropbox.com/s/0mju5kc0pwb3t59/3.7.3_softlockup.txt
I didn't post anything in LKML.
Will it be helpfull?

[renton]

May be you meant to upgrade to grsecurity-2.9.1-3.8.2-201303072205.patch and if it hangs then to send you a trace?

[spender]

Yes.

-Brad

[renton]

It worked only for 2 days and once again - softlockup.
Unfortunately netconsole didn't give any trace nevertheless that I took kernel config of previous one.

grsecurity-2.9.1-3.8.2-201303091131.patch

Mar 15 00:51:38 l25 [86616.019966] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4303602251, last pi
ng 4303602752, now 4303603253
Mar 15 00:51:51 l25 [86628.833984] BUG: soft lockup - CPU#1 stuck for 22s! [vsftpd:17026]
Mar 15 00:51:51 l25 [86628.843985] BUG: soft lockup - CPU#2 stuck for 22s! [httpd:18926]
Mar 15 00:51:51 l25 [86628.853982] BUG: soft lockup - CPU#3 stuck for 22s! [httpd:25116]
Mar 15 00:51:51 l25 [86628.873971] BUG: soft lockup - CPU#4 stuck for 22s! [httpd:27745]
Mar 15 00:51:51 l25 [86628.883969] BUG: soft lockup - CPU#5 stuck for 22s! [httpd:5182]
Mar 15 00:51:51 l25 [86628.893963] BUG: soft lockup - CPU#6 stuck for 22s! [umount:25288]
Mar 15 00:51:51 l25 [86628.913955] BUG: soft lockup - CPU#7 stuck for 22s! [httpd:5154]
Mar 15 00:51:51 l25 [86629.083885] BUG: soft lockup - CPU#8 stuck for 22s! [httpd:25005]
Mar 15 00:51:51 l25 [86629.093870] BUG: soft lockup - CPU#9 stuck for 22s! [httpd:2472]
Mar 15 00:51:51 l25 [86629.103866] BUG: soft lockup - CPU#10 stuck for 22s! [php:25289]
Mar 15 00:51:51 l25 [86629.123866] BUG: soft lockup - CPU#11 stuck for 22s! [haproxy_aux3_po:4606]
Mar 15 00:51:51 l25 [86629.133852] BUG: soft lockup - CPU#12 stuck for 22s! [httpd:17505]
Mar 15 00:51:51 l25 [86629.153852] BUG: soft lockup - CPU#13 stuck for 22s! [haproxy_aux2_po:1505]
Mar 15 00:51:51 l25 [86629.163837] BUG: soft lockup - CPU#14 stuck for 22s! [httpd:18908]
Mar 15 00:51:51 l25 [86629.173826] BUG: soft lockup - CPU#15 stuck for 22s! [httpd:7193]
Mar 15 00:51:51 l25 [86629.193827] BUG: soft lockup - CPU#16 stuck for 22s! [mysqld_aux3:20122]
Mar 15 00:51:51 l25 [86629.203821] BUG: soft lockup - CPU#17 stuck for 22s! [httpd:17507]
Mar 15 00:51:51 l25 [86629.223808] BUG: soft lockup - CPU#18 stuck for 22s! [httpd:10114]
Mar 15 00:51:51 l25 [86629.233807] BUG: soft lockup - CPU#19 stuck for 22s! [zabbix_agentd:25966]
Mar 15 00:51:51 l25 [86629.243801] BUG: soft lockup - CPU#20 stuck for 22s! [suexec:25290]
Mar 15 00:51:51 l25 [86629.263794] BUG: soft lockup - CPU#21 stuck for 22s! [httpd:27244]
[skip]
Mar 15 00:52:19 l25 [86657.370709] BUG: soft lockup - CPU#30 stuck for 22s! [vsftpd:27135]
Mar 15 00:52:19 l25 [86657.390710] BUG: soft lockup - CPU#31 stuck for 22s! [httpd:14616]
Mar 15 00:52:28 l25 [86665.956723] INFO: rcu_sched self-detected stall on CPU
Mar 15 00:52:28 l25 [86665.956726] INFO: rcu_sched self-detected stall on CPU
[skip]
Mar 15 00:52:55 l25 [86693.373971] BUG: soft lockup - CPU#31 stuck for 22s! [httpd:14616]
Mar 15 00:53:00 l25 [86698.062554] BUG: spinlock lockup suspected on CPU#24, httpd/25199
Mar 15 00:53:00 l25 [86698.063755] BUG: spinlock lockup suspected on CPU#29, httpd/10340
Mar 15 00:53:00 l25 [86698.063757] lock: ffffffff826ec308, .magic: dead4ead, .owner: tar/507, .owner_cpu: 0
Mar 15 00:53:00 l25 [86698.064262] BUG: spinlock lockup suspected on CPU#25, httpd/16675
Mar 15 00:53:00 l25 [86698.064265] lock: ffffffff826ec308, .magic: dead4ead, .owner: tar/507, .owner_cpu: 0
Mar 15 00:53:00 l25 [86698.064686] BUG: spinlock lockup suspected on CPU#26, httpd/5223

[spender]

Can you mail me your .config and a link to your vmlinux (not vmlinuz) file from the kernel source tree?

-Brad

[renton]

sent to spender@