grsec: halting the system due to suspicious kernel crash

[timbgo]

There are points that I made, and while it is true that on gNewSense they don't censor, my message that I posted is not promptly available because of the break in presentation on the web interface at the turn of the months August and September of 2013.

The points that I made on Debian and Grsecurity on gNewSense mailing list (Debian, but I am mentioning it here because only in this thread I reported on my thread on gNewsense, sorry for the inconvenience), are best presented here:

https://lists.nongnu.org/archive/html/g ... 00001.html

Mind e.g. this snippet:

PASTING START
###############################################################
### that is the crux of the matter in this whole story:
### most users, if they knew what SELinux was, wouldn't want to
### "harden" their GNU/Linuces with it
###############################################################
PASTING END

I have had yet more attacks on my SOHO, and I have yet more proofs of Grsecurity/Pax superiority over other (spyware laden/inferior) security software of GNU Linux of today, and I have more scathing langauge on those preventing the furtherence of security in GNU Linux, those oposing our reqonquest of the freedom of the Internet by the users, who, some of them, seem to be sitting pretty in various projects and places GNU Linux, such as, very apparently, amongst the leaders of (actually only) nominally (since spyware endorsing) "free" software that Debian GNU Linux seems to me to have become...
Sad but true, sad but true, sad but true!
Miroslav Rovis,
Zagreb, Croatia
http://www.CroatiaFidelis.hr

I do have more proofs but, alas, it's such a huge task to present it!...

[[
but pls. find more about it in the Debian side of this Debian/Gentoo twin topic, here:
coming these minutes, the link, it's in the
"grsec: halting the system... kernel crash, the Debian side"
topic:
viewtopic.php?f=3&t=3712&p=13821#p13821
]]

[timbgo]

New development, but in line with the old:
http://www.croatiafidelis.hr/gnu/grsec/ ... 80x960.jpg



Pls. compare the manually typed textual version, in the next post, with this image and report typoes I might have made.

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[timbgo]

Yes, that's what we have computers for, isn't that so, all you FONs [1] who just need to sell your users, and just can't do proper programming for common good, because you only have your own very twisted "good" as your miserable aim and fleeting advantage which is gone before you even know it!

Starting from the dear leader Linus, great friend of Schmoog.

Yes, that's what we have computers for: to type manually from images what the computers don't show us in text, yes, it's hidden from us, because they are programmed not to save it for us in our logs! Previously I used to see those crashes documented in Call Traces in my /var/log/messages, in Gentoo, and in Debian in both /var/log/messages and /var/log/kern.log. Not any more!

Why was this message with the code and all that developers need not in the /var/log/messages?

Because there are guaranteed ways in recent kernels and environment (someone more advanced would need to tell precisely and in correct technical terms) to hide what is happening in the kernel and generally in your computer, dear poor user (an expression that I use and which is something like avarage Joe user, except nicer, because it is instead inspired by the most read Book on the Planet)... And some programmers are bent on hiding things from users (and worse).

Because there are guaranteed ways to hide what is happening in the kernel and generally in your machine, dear poor user like me!

So the following is manual typing. Pls. report errors if you find any, comparing the text with the image posted above.

Code: Select all

[ 1420.202469] BUG: unable to handle kernel NULL pointer dereference at 0000000000e8
[ 1420.202535] IP: [<ffffffff81761abc>] ffffffff81761abc
[ 1420.202574] PGD 42138b000
[ 1420.202596] Thread overrun stack, or stack corrupted
[ 1420.202632] Oops: 0000 [#1] SMP
[ 1420.202659] Modules linked in: ipv6 r8169 mii
[ 1420.202700] CPU: 0 PID: 3054 Comm: mkdir Not tainted 3.16.3-hardened-140921-01 #5
[ 1420.202753] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./970 Extreme4, BIOS P2.60 11/11/2013
[ 1420.202822] task: ffff88042d2d9340 ti: ffff88042d2d9ad8 task.ti: ffff88042d2d9ad8
[ 1420.202875] RIP: 0010:[<ffffffff81761abc>]  [<ffffffff81761abc>] ffffffff81761abc
[ 1420.202956] RSP: 0018:ffffc9001a423e60  EFLAGS: 00010247
[ 1420.202993] RAX: 0000000000000000 RBX: ffff8800b11b6b60 RCX: ffffe8ffffc010b6
[ 1420.203043] RDX: 0000000000000013 RSI: ffff8800b11b6b60 RDI: ffff88041e19af78
[ 1420.203093] RBP: ffffc9001a423e70 R08: 0000000000000073 R09: 0000000000000000
[ 1420.203143] R10: 0000000000000004 R11: 0000000000000000 R12: ffff88041e19af78
[ 1420.203193] R13: 00000000000001ed R14: 0000000000000000 R15: 000003cc0b389ad7
[ 1420.203243] FS:  0000033068ec8700(0000) GS:ffff99043fc00000(0000) knlGS:0000000000000000
[ 1420.203299] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1420.203340] CR2: 00000000000000e8 CR3: 0000000001771000 CR4: 00000000000007f0
[ 1420.203389] Stack:
[ 1420.203404]  ffff88041e19af78 ffff8800b11b6b60 ffffc9001a423e98 ffffffff8136474b
[ 1420.203461]  ffff88042d79cba8 ffff8800b11b6b60 0000000000000002 ffffc9001a423ee8
[ 1420.203519]  ffffffff8113b879 ffffff9cffff4111 ffff88042c8cf820 ffff88042d79cba8
[ 1420.203576] Call Trace:
[ 1420.203600]  [<ffffffff8136474b>] gr_handle_create+0x52/0x65
[ 1420.203643]  [<ffffffff8113b079>] SYSC_mkdirat+0xa4/0xe8
[ 1420.203683]  [<ffffffff8113da4e>] SyS_mkdir+0x1e/0x29
[ 1420.203727]  [<ffffffff81769148>] system_call_fastpath+0x16
[ 1420.203774]  [<ffffffff81769178>] ? sysret_check+0x26/0x65
[ 1420.203813] Code: 41 5e 41 5f 5d 48 0f ba 2c 24 3f c3 55 48 89 e5 53 48 89 f3 50 48 89 7d f0 8b 90 1c
 04 00 00 48 8b 43 20 <48> 8b b0 e8 00 00 00 e8 90 fd ff ff 5a 5b 5d 48 0f ba 2c 24 3f
[ 1420.204056] RIP  [<ffffffff81761abc>] ffffffff81761abc
[ 1420.204094]  RSP <ffffc9001a423e60>
[ 1420.204119] CR2: 00000000000000e8
[ 1420.212331] Kernel panic - not syncing: grsec: halting the system due to suspicious kernel crash caused by root
[ 1420.212436] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[ 1420.212516] drm_dms_helper: panic occurred, switching back to text console


What happened in my box, in my non-expert understanding, and confirmation will follow or not by people with authority in these matters, is probably the same after-free bug as previously, and sure those who made these probable intrusion attempts possible, by selling exploits to subjects that attempted these intrusion attempts into my SOHO, which look in the same line as previous attempts that I reported here a year ago, and are presented in this topic, and some other related topis, were doing the work soo much better than previously!

Namely, as I work slowly, need to study these gNSAschmoog-Linux [2] issues at length, need to read documentation repeatedly to get it right, I have only these recent days finally found enough time to study and build policy and enable gradm onto my Gentoo grsec-hardened kernel, and as you can probably guess from the above, I enabled it successfully.

But all the time since back when I reported the first kernel panics in this topic, I was using grsecurity without enabling gradm. The reason lies in the systemd and related programs that were imposed on me like they were imposed on most poor users of the gNSAschmoog-Linux, and that made the enabling of gradm so difficult.

If you take a look at the:

Air-Gapped Gentoo Install, Tentative
http://forums.gentoo.org/viewtopic-t-987268.html

and pages linked to from there (it's all a very hefty read), you will see how much I struggled, but was finally on the verge of making it: to have a poetteringware-free Gentoo system [3].

My story of how to get the right choices of installation of schlinux is not over yet, I'm still fighting for my Debian boxes, where actually more work is needed to deploy gradm properly then in my potteringware-free Gentoo, and surely I don't know how much my Gentoo box was damaged, and actually if it was or ont, and if I will have to go over from scratch with it or will be able to keep upgrading it.

I was saying that this time around the would be intruders crept in much more silently, and were only exposed now that I finally went through the gradm learning a few times and edited correctly the /etc/grsec/policy for gradm and enabled it.

And I can really see how hard it has become to keep your system secure, due to the sad Orwellian state of our world.

Newbies need not accept my statements above as truth ahead of time. I am not an expert. I hope there will be confirmation whether it is or not a case of free-after bug here.

I thank again grsecurity and PaX developers. Sure I will gladly send the vmlinuz in question to them, and whatever may be necessary. (I not only work slowly but also communicate from the SOHO in air-gapped with machines that go online, patience needed.)

I hope this helps, marginally though it may be, in the advancement of this great program.

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr
--
[1] FON = Friend Of N..
Listen what BSD developer Poul-Heening Kamp talks on those:
http://video.fosdem.org/2014/Janson/Sun ... eport.webm

[2] why "gNSAschmoog-Linux"? read:

The future with Systemd
http://forums.debian.net/viewtopic.php? ... 90#p553405

[3] For Gentoo I went this way:

Uninstalling dbus and *kits (to Unfacilitate Remote Seats)
http://forums.gentoo.org/viewtopic-t-992146.html

For Debian I'm still researching what can be done:

How to avoid stealth installation of systemd?
http://forums.debian.net/viewtopic.php?f=20&t=116770

These links above I have posted in good faith, for the benefit of other users, because other, more advanced users than me, have also complained how it is much harder to get gradm enabled with proper policy with poetteringware. And grsecurity protection is not complete without gradm properly configured and enabled.

[spender]

Hi,

Can you email spender@grsecurity.net with a link to your vmlinux (not vmlinuz) file? I'll need it to properly decode the oops. In the next grsecurity patch we will also resolve an issue that was preventing the symbol for RIP to be displayed.

Thanks,
-Brad

[timbgo]

Hi,

Can you email spender@grsecurity.net with a link to your vmlinux (not vmlinuz) file? I'll need it to properly decode the oops. In the next grsecurity patch we will also resolve an issue that was preventing the symbol for RIP to be displayed.

Thanks,
-Brad

Brad, thanks for replying.

Pls. excuse me for late reply, I did check back, but only for a few hours.

Because I'm overwhelmed (will describe more on what happened from the user's point of view, as soon as I find strength, after I send what is necessary to devs though).

I'll follow what you ask as best I can, very soon. Only censorship could stop me now.

Because you and PaX Team are the most important hope in FOSS Linux of today.

Miroslav Rovis
Zagreb, Croatia
www.CroatiaFidelis.hr

[timbgo]

Just in case:
I didn't send anything to your email, but here in your forums post.
Saying that for clarity just in case.
Miro

[spender]

So I've taken a look at the vmlinux you sent, but I have some more questions. It's triggering on a NULL dereference of the created dentry's d_sb field, which shouldn't be possible (it's always set in __d_alloc()). Is the problem reproducible? What filesystem did the mkdir occur on? (e.g. NFS, ext3, etc) With what patch did you first experience the problem? Is it reproducible with the latest 3.16.3? Does the problem go away if you replace the occurrence of "dentry->d_sb->s_dev" in grsecurity/gracl.c with "dentry->d_inode->i_sb->s_dev"?

Thanks,
-Brad

[timbgo]

Just woke up (Europe here).
I'm really glad I can try and be of some help.

The problem occurred when I connected from my online Gentoo (only IPTV on that Gentoo, but I finally get it that IPTV is as online as anything else...) to my first time gradm enabled (after long time --and a few tries-- policy learning) master Gentoo.

All my computers are on my SOHO, and I connect with NFS between them, when I move files around.

Sadly I was very surprised and don't recall what exactly I did that triggered the kernel panic, to be able to reply to you whether I tried to do the mkdir from the IPTV online computer or from the first time gradm enabled master Gentoo.

(And the other elements mentioned in your question I now have to first try and understand the meaning of.) But yes, I recall copying files from the IPTV-online system onto my master Gentoo, the command started on the command line, from one of them, don't remember one or the other.

However I have the system that I connected from (the IPTV-online system) stowed away all its /var/log directory in the state just after the crash it introduced on the master Gentoo, and I will try and see what information I can recover from it.

And I also have the entire master Gentoo, but entire, stowed away because I dumped the system partitions, which is what I do to be able to clone that system onto three other boxes Gentoo boxes that I have, and those dd-dumps are the state right after the crash, IIRC (maybe I tried and do something minor, I suffer from poor memory to tell for sure).

I'll mount those and try and see if I can answer your questions more fully.

Let me now try and study your requests to be able to reply a little more fully.

I guess it's still night and you U.S. Americans are sleeping right now. So good morning when you are back to read this from me!

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[timbgo]

So I've taken a look at the vmlinux you sent, but I have some more questions.

I'll try and go slowly over those.

It's triggering on a NULL dereference of the created dentry's d_sb field, which shouldn't be possible (it's always set in __d_alloc()).

Have to skip what I understand only too vaguely, if at all, like the above.

Is the problem reproducible?

I'm willing to try and see if it is, however it broke at least another, old, Gentoo system that was on the SOHO at that time, so I have to thread carefully.

It broke it in the sense that it now gets shut down without much load, with just some minimally high loads.

That is exactly the problem that I had experienced previously on the now IPTV-online Gentoo system until, and at least for a while even after, I cloned it from scratch. The shutting down, under just above minimal loads, wouldn't go away, as I just said, even after cloning the master Gentoo system on it, possibly because something, through intrusion, was stored somewhere other than in filesystems (because I left none whatsoever of the old system data, I zeroed out all and any that can be dd-zeroed out, before cloning the master Gentoo system on it). The shutting down of that now IPTV-online system (and previously also with previous old Gentoo install that was my only IPTV-online system), I haven't yet checked if it still persists with the cloned new no-poetteringware Gentoo which is currently on it.

The old system on my back-then and also-now IPTV-online system (now it is a no-poetteringware system, so no-systemd no-dbus no-policykit no-polkit no-pulseaudio system)... back then it was a systemd-policykit-dbus-pulseaudio system, so at least prone to surveille-rootkits (if not already spyware by virtue of those mere programs)...

And so is the old system a potteringware one, on this other box which is now shutting down under some not high loads at all, and so it is unuseable for any, say, ffmpeg video processing (talking about the so called "obsolete" but in effect real FFmpeg), which can not be deployed on that system in full at all, because it triggers shutdowns, as does other heavier processing. Within minutes of starting ffmpeg processing of my videos, the system shuts down.

That system is some 200 days not updated system (that's how slowly I work in some areas), but never experienced stubborn shutdowns like now after this some kind of breakage that three days ago has occurred. It only got broken down, apparently, because it was connected to the SOHO and had the suspected rootkits (see above; I'm writing for not just the devs but for current and future users of grsecurity; be informed, gentle readers!).

So I'm willing to go through trying to reproduce the problems described, with 3.16.2, the current one, and with 3.16.3 kernel, but I have to thread carefully to not lose more of my systems. Because, judging by the effects, it must have been an intrusion.

I know what, say, memory errors from bad system RAM look like, I was able to understand it was those a few months ago, because I always check the sums when I move files, and I sometimes move lots of files, and pretty large files (such as, say, the dd-dumps that measure in tens of GB). An so I knew that it must have been RAM memory errors, back when I had those peculiar problems a few months ago, of very different kind than these described in this topic, because I saw that files didn't retain the same SHA256 sums, some of the many files just couldn't be copied with integrity. And without further ado I checked RAM memory with running sysresccd and that program, whatever the name, can't recall, mem86+ or somesuch, to be chosen at boot, and there were errors in RAM. Memory replaced, no more errors.

But I have to thread carefully now.

My only defence is what I call poor users' security and it consists of having clean backups and restoring the system partitions in the previous state. Which works but is so lengthy, everything all over again every time...

Let me just note that I don't see any apparent breakage in the other systems protected with grsecurity. Not even the Debian which was on-the-SOHO. This Debian that I write this on and with which I post to grsecurity forums is not that one, no, this one is strictly air-gapped-off from the SOHO. On this one I only experienced technically non-related, but socially, yes, related noise, or should I call it suspected peculiar attention, when I was connecting after I posted these recent posts in this topic, and especially I couldn't connect to sftp the (surely ) encrypted kernel file, the vmlinux to grsecurity devs for quite a while. My sftp'ing to my hoster of my domain http://www.CroatiaFidelis.hr was simply blocked for a while.

The old systems, one extinct, the other still to replace, I mentioned to explain what happened, I don't think they deserve much thought otherwise.

What filesystem did the mkdir occur on? (e.g. NFS, ext3, etc)

I explained it was NFS, and I looked for the /var/log dir from the IPTV-online system before cloning clean system on it that I thought I stowed away, and I think I may have either misplaced it, or not having it at all, haven't been able to find it.
I will next look up the gradm-first-time-enabled master Gentoo dd-dumps, and will be able to tell more.

With what patch did you first experience the problem?

This is the first time I experience this problem. The patch is what Gentoo devs prepare, so it is was goes with the hardened sources that they do. I haven't looked into those and don't know what patch they use. I could try, gladly, and ask them. I think I will on Gentoo Forums...

Should I rather try and use the testing patches like I use on average once a month for my Debian boxes and then post the packages for other Debianers to download?

I believe I could be helpful with my reports either if I keep to stock Gentoo grsecurity hardened sources, as well as it I try and compile the kernel with the latest grsecurity testing patch myself... Can't decide, but it also depends what support I can muster for this issue on Gentoo Forums, because I will also, hopefully, post there (see this report around the string "social", in this very post, to understand that I might be censored; well I hope I won't be, but that was so very likely a tiny bit of censorship there; remember that while globally the NSA is bad enough in intrusion, but we have a very bad, though sometimes very stupid as well, tiny internet leviathan called UDBA in Croatia...).

Is it reproducible with the latest 3.16.3?

I'm also willing to try and report on that.
However, it's updating my local mirror that I then first need to do, to be able to update my Gentoo master machine.
I don't simply go online with it. I update it from my private mirror, as I described in:

Air-Gapped Gentoo Install, Tentative
https://forums.gentoo.org/viewtopic-t-987268.html

Does the problem go away if you replace the occurrence of "dentry->d_sb->s_dev" in grsecurity/gracl.c with "dentry->d_inode->i_sb->s_dev"?

I found the file and that occurence. I'll test it and report it.
If it turns out I work too slowly, I report what I tested on the next 3.16.x kernel

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[timbgo]

I just posted on Gentoo Forums, a scathing article, but I believe still without exaggerating.

That is what I believe. However, not necessarily the moderators there will find it such as well.

It is very much about grsecurity

So, just the links, and will, in case I'm not impeded, I explain more later:

Why is Gentoo not switching to systemd?
https://forums.gentoo.org/viewtopic-p-7 ... ml#7624042
https://forums.gentoo.org/viewtopic-p-7 ... ml#7624044

It did keep opening for me for some two days under above links, but might not continue.
The regular links, the sole links that will need to keep opening if the article is not removed are these:

EDIT START 2014-09-30 12h CET
Why is Gentoo not switching to systemd?
https://forums.gentoo.org/viewtopic-t-9 ... ml#7624042
https://forums.gentoo.org/viewtopic-t-9 ... ml#7624044
EDIT END

The post seems to be standing there, and that it will not be removed.
I'd hope that we still have chances for our free computing.

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[timbgo]

Possibly related to this issue (say maybe it's tripping something under some of the code that grsec is to patch onto the kernel, and making grsec look bad (just guessing, I'm not a dev, but I have seen bad games against grsec before)...

[Possibly related to this issue], esp. my not being able to find anything whatsoever in the logs on the issues that I had, and which made me a little angry, see here:

(same topic)
viewtopic.php?f=3&t=3709&start=15#p14457

is what I described I still have in my system despite the very careful air-gapped near-reinstall:

Syslog-ng from Delay in Logging to Broken Pipe and no Loggin
https://forums.gentoo.org/viewtopic-t-1001994.html

EDIT START 2014-10-25 I was thinking, and hope it's not too late to say it here: what could that be? The September version of syslog-ng made huge delays in logs, and even lost logging for hours, and made Grsecurity possibly misbehave...

Did that syslog-ng version 3.5.6 have issues in SELinux hardened kernel? In non-hardened kernel?

If I were able to dedicate serious time to the issue, I'd search for bugs with the syslog-ng maintainer, but no time. Not that capable... So just pointing to the issue...
EDIT END

Let's see if anything clarifies on the issue...

Miroslav Rovis
Zagreb, Croatia
http://www.CroatiaFidelis.hr

[Orfheo]

It looks like, from the trace, I had almost the same kernel panic pointing to grsec and mkdir,
so I'm trying to jump on this thread.

On an updated, stable, gentoo, kernel 3.15.10-hardened-r1, for two nights, at cron.daily
running time my firewall crashed with this panic, one recorded into the log files

Code: Select all

Oct 30 03:10:02 alecto kernel: [117982.502192] BUG: unable to handle kernel NULL pointer dereference at 00000028
Oct 30 03:10:02 alecto kernel: [117982.505409] IP: [<c1593a13>] c1593a13
Oct 30 03:10:02 alecto kernel: [117982.505830] *pdpt = 0000000032743001 *pde = 0000000000000000
Oct 30 03:10:02 alecto kernel: [117982.506160] Oops: 0000 [#1] PREEMPT SMP
Oct 30 03:10:02 alecto kernel: [117982.506541] Modules linked in: vmw_vsock_vmci_transport vsock vmw_balloon vmw_vmci
Oct 30 03:10:02 alecto kernel: [117982.507157] CPU: 0 PID: 21911 Comm: mkdir Not tainted 3.15.10-hardened-r1 #1
Oct 30 03:10:02 alecto kernel: [117982.507323] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/15/2011
Oct 30 03:10:02 alecto kernel: [117982.507568] task: f66b3300 ti: f66b3768 task.ti: f66b3768
Oct 30 03:10:02 alecto kernel: [117982.508867] EIP: 0060:[<c1593a13>] EFLAGS: 00010246 CPU: 0
Oct 30 03:10:02 alecto kernel: [117982.510002] EAX: f4e52180 EBX: cf322100 ECX: 00000014 EDX: 00000000
Oct 30 03:10:02 alecto kernel: [117982.511219] ESI: f4e52180 EDI: 00000000 EBP: ed021efc ESP: ed021efc
Oct 30 03:10:02 alecto kernel: [117982.512342]  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
Oct 30 03:10:02 alecto kernel: [117982.513488] CR0: 8005003b CR2: 00000028 CR3: 22ce3000 CR4: 000006f0
Oct 30 03:10:02 alecto kernel: [117982.514678] Stack:
Oct 30 03:10:02 alecto kernel: [117982.515622]  ed021f10 c11be0e7 f64e8e10 cf322100 00000002 ed021f38 c10ed80b ffffff9c
Oct 30 03:10:02 alecto kernel: [117982.516865]  580222e8 01ed51f8 f64e8e10 f619be00 580222e8 580222d2 58022034 ed021f4c
Oct 30 03:10:02 alecto kernel: [117982.518015]  c10ed84d ffffff9c 580222e8 000001ed f66b3768 c1599e38 580222e8 000001ed
Oct 30 03:10:02 alecto kernel: [117982.519241] Call Trace:
Oct 30 03:10:02 alecto kernel: [117982.521351]  [<c11be0e7>] gr_handle_create+0x4d/0x75
Oct 30 03:10:02 alecto kernel: [117982.522634]  [<c10ed80b>] SyS_mkdirat+0x8f/0xbf
Oct 30 03:10:02 alecto kernel: [117982.523781]  [<c10ed84d>] SyS_mkdir+0x12/0x14
Oct 30 03:10:02 alecto kernel: [117982.524889]  [<c1599e38>] syscall_call+0x7/0x7
Oct 30 03:10:02 alecto kernel: [117982.526032]  [<c11d5fc9>] ? debug_smp_processor_id+0x12/0x15
Oct 30 03:10:02 alecto kernel: [117982.530403]  [<c1008af3>] ? pax_randomize_kstack+0x2d/0x6e
Oct 30 03:10:02 alecto kernel: [117982.532941]  [<c1599e53>] ? restore_all_pax+0x7/0x7
Oct 30 03:10:02 alecto kernel: [117982.535652] Code: 0f 85 8f fe ff ff e9 72 fe ff ff 8b 5a 04 85 db 8b 42 08 75 9d eb 89 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 8b 4a 54 8b 49 08 8b 52 20 <8b> 52 28 e8 f5 fd ff ff 5d c3 55 89 e5 8b 00 0f c8 5d c3 55 89
Oct 30 03:10:02 alecto kernel: [117982.545453] EIP: [<c1593a13>] do_handle_create.isra.10+0xc/0x16 SS:ESP 0068:ed021efc
Oct 30 03:10:02 alecto kernel: [117982.548655] CR2: 0000000000000028
Oct 30 03:10:02 alecto kernel: [117982.573662] ---[ end trace d0799a58a2a06ee5 ]---
Oct 30 03:10:02 alecto kernel: [117982.575828] note: mkdir[21911] exited with preempt_count 2


You may find the "vmlinux" kernel image at http://www.vitillaro.org/curr/vmlinux

The filesystems mounted on the machine are

Code: Select all

rootfs on / type rootfs (rw)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=220820,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on / type ext3 (rw,relatime,errors=continue,barrier=1,data=writeback)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,size=206980k,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
/dev/sda1 on /boot type ext3 (rw)
/dev/sda3 on /tmp type ext3 (rw)
/dev/sda6 on /var type ext3 (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nodev,noexec,nosuid)


Any idea?

[Orfheo]

Think I've done a step forward debugging this kernel Oops.

The filesystem type from which the BUG panic come is "openrc":

Code: Select all

openrc                 0     0         0    - /sys/fs/cgroup/openrc


To get the OOps, with the current kernel and with the previous one I used for months
without any problem, now, with grsec enabled and suitable acl, this simple command is
enough

Code: Select all

mkdir -p /sys/fs/cgroup/openrc/acct


The problem is reproducible and deterministic on my system.

cron.daily run was the source each night because, on my firewall, each
night restart the "acct" service for rotating the accounting /var/account/pacct
file.

I can't understand what changed in the last gentoo emerge/update of a couple of days
ago, but I doubt the system is compromised: it has been always protected by grsec/pax
and is not directly exposed to the Net.

The only significant update I can see from my emerge logs, done on Oct 28, is the bash
shell, but just in the case this a "grep" of my emerge logs of the last update:

Code: Select all

>>> Emerging (1 of 22) sys-libs/timezone-data-2014g
>>> Emerging (2 of 22) sys-devel/make-4.0-r1
>>> Emerging (3 of 22) dev-libs/popt-1.16-r2
>>> Emerging (4 of 22) dev-libs/libgpg-error-1.12
>>> Emerging (5 of 22) app-shells/bash-4.2_p53
>>> Emerging (6 of 22) sys-apps/man-pages-3.72
>>> Emerging (7 of 22) dev-lang/perl-5.18.2-r2
>>> Emerging (8 of 22) perl-core/File-Temp-0.230.0
>>> Emerging (9 of 22) virtual/perl-File-Temp-0.230.0-r1
>>> Emerging (10 of 22) sys-kernel/gentoo-sources-3.16.5
>>> Emerging (11 of 22) sys-apps/help2man-1.45.1
>>> Emerging (12 of 22) dev-libs/libgcrypt-1.5.4-r1
>>> Emerging (13 of 22) net-libs/libpcap-1.6.2-r1
>>> Emerging (14 of 22) dev-libs/openssl-1.0.1j
>>> Emerging (15 of 22) sys-apps/hwids-20141010
>>> Emerging (16 of 22) net-misc/curl-7.37.1
>>> Emerging (17 of 22) net-analyzer/tcpdump-4.6.2
>>> Emerging (18 of 22) dev-vcs/git-2.0.4
>>> Emerging (19 of 22) dev-lang/python-3.4.1
>>> Emerging (20 of 22) sys-apps/portage-2.2.8-r2
>>> Emerging (21 of 22) app-editors/vim-core-7.4.273
>>> Emerging (22 of 22) app-editors/vim-7.4.273


Just testing a previous version of bash, 4.1_p17, to see if it makes any difference.

[spender]

Hi Orfheo,

Sorry I missed your previous post -- this particular thread is quite long.

About the bug, it seems to be the same case above of a NULL dentry->d_sb.

Can you try the following patch and send me the debug information it provides?

https://grsecurity.net/~spender/dsb.diff

Thanks,
-Brad

[Orfheo]

Nice to meet you Brad.

I understand, this thread is long and quite complex, probably I made
a mistake. My fault, apologies.

I tested the patch you sent me without luck: I can't get any printk
even over a virtual serial console (fortunately this is a virtual machine).

But i've been a C programmer in my youth, so I tried to add some printk
by myself, hoping the output may be useful.

You may find the DEBUG gracl.c code at the URL;

http://www.vitillaro.org/curr/gracl.c.DEBUG

and this is the info I got just before the Oops

Code: Select all

[  123.848607] dentry = f50c1a80, mnt = f6466510
[  123.852470] matchn =    (nil)
[  123.853378] dentry = f515ca00, mnt = f6466510
[  123.853384] matchn =    (nil)
[  123.854139] dentry = f515ce00, mnt = f6466510
[  123.854145] matchn =    (nil)
[  137.758093] dentry = f5139f00, mnt = f6466c10
[  137.762098] matchn = f36c0420
[  137.764858] do_handle_create dentry f5139f00
[  137.766077] BUG: unable to handle kernel NULL pointer dereference at 0000001c
[  137.766153] IP: [<c1593a8f>] c1593a8f
[  137.766301] *pdpt = 0000000032809001 *pde = 0000000000000000
[  137.766469] Oops: 0000 [#1] PREEMPT SMP
[  137.766862] Modules linked in: vmw_vsock_vmci_transport vsock vmw_balloon vmw_vmci
[  137.767196] CPU: 0 PID: 5268 Comm: mkdir Not tainted 3.15.10-hardened-r1 #7
[  137.767238] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/15/2011
[  137.767321] task: f67a9320 ti: f67a9788 task.ti: f67a9788
[  137.767434] EIP: 0060:[<c1593a8f>] EFLAGS: 00010246 CPU: 0
[  137.767456] EAX: 00000020 EBX: f5139f00 ECX: 00000003 EDX: 00000000
[  137.767478] ESI: f36c0420 EDI: 00000000 EBP: f4d29f60 ESP: f4d29f4c
[  137.767503]  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
[  137.767534] CR0: 8005003b CR2: 0000001c CR3: 32808000 CR4: 000006f0
[  137.767621] Stack:
[  137.767735]  c1722d78 f5139f00 f36c0420 f5139f00 00000000 f4d29f70 c11be149 f5139f00
[  137.767754]  00000002 f4d29f98 c10ed80b ffffff9c 5dca3493 01edffff f6466c10 f61a2580
[  137.767778]  5dca3493 5dca347d 5dca3204 f4d29fac c10ed84d ffffff9c 5dca3493 000001ed
[  137.767817] Call Trace:
[  137.769009]  [<c11be149>] gr_handle_create+0x66/0x90
[  137.769076]  [<c10ed80b>] SyS_mkdirat+0x8f/0xbf
[  137.769094]  [<c10ed84d>] SyS_mkdir+0x12/0x14
[  137.769119]  [<c1599ed8>] syscall_call+0x7/0x7
[  137.769499] Code: 5a 04 85 db 8b 42 08 75 9d eb 89 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 53 89 c6 89 d3 52 68 78 2d 72 c1 e8 89 da ff ff 8b 53 20 <ff> 72 1c 52 ff 73 1c 68 96 2d 72 c1 e8 75 da ff ff 8b 43 20 8b
[  137.769691] EIP: [<c1593a8f>] do_handle_create.isra.10+0x18/0x4c SS:ESP 0068:f4d29f4c
[  137.769734] CR2: 000000000000001c
[  137.770078] ---[ end trace b03e47682a3ec49a ]---
[  137.770282] note: mkdir[5268] exited with preempt_count 2


As you can see the first printk in do_handle_create can get its way to the serial console

do_handle_create dentry f5139f00

while the second one, I guess, generate the Oops.

It looks the dentry pointer is not NULL, but it is not pointing to a good area anymore.

Hope it will help and I just haven't done a mess, Orfheo.