suspend-to-RAM broke between 3.4.5 and 3.5.9 for my system

[christian.]

I am using a Gentoo Hardened system for some time now (two or more years). Part of my installation is a Grsecurity enhanced kernel (package: sys-kernel/hardened-sources). Since an upgrade from 3.4.5 to 3.5.4 the suspend-to-RAM mechanism broke -- which previously worked flawlessly.

The suspend mode can be engaged, but when I wake up the computer, the monitor stays off and the suspend LED is still on. After a few seconds (usually, the computer resumes immediately) the computer reboots. BTW: The computer is a IBM ThinkPad X41.

In my effort to narrow it down I tried a non-hardened 3.5.4 kernel, which suspends without any problems. Then I tried a hardened kernel with Grsecurity/PAX disabled and the problem reappeared; blank screen, suspend LED still on, this time the fan is being activated (might be random) and then the system reboots -- no kernel panic, nothing.

I tried to use the suspend debug technique described in
linux/Documentation/power/basic-pm-debugging.txt
to no avail.

A few weeks ago, I decided to take a look at the latest test patch set 3.6.9 -- downloaded it straight from grsecurity.net. Here another bug occured additionally, but this time I was able to narrow both bugs down:

1) When enabling PAX_MEMORY_UDEREF, I get a kernel panic in init as soon as init is started.

2) When enabling PAX_MPROTECT suspend-to-RAM stops working with the forementioned symptoms.

A few days ago I tried the latest patchset
grsecurity-2.9.1-3.6.10-201212101818.patch

Problem 1) disappeared (I now can use PAX_MEMORY_UDEREF without a kernel panic), but bug 2) still persists.

How can I provide information to narrow that thing down? As I said: I tried to use the CMOS-clock debugging technique for suspend-to-RAM described in the kernel documentation without success.

My kernel config, which provides working suspend, is here:
http://pastebin.com/VfNWWsK5

And this is a "diff" to the kernel configuration, with which my computer does not resume anymore.

Code: Select all

--- 3/config    2012-12-12 22:41:23.785640073 +0100
+++ 4/config    2012-12-13 16:01:31.075705056 +0100
@@ -2776,6 +2776,7 @@
 # Grsecurity
 #
 CONFIG_ARCH_TRACK_EXEC_LIMIT=y
+CONFIG_PAX_PER_CPU_PGD=y
 CONFIG_PAX_USERCOPY_SLABS=y
 CONFIG_GRKERNSEC=y
 # CONFIG_GRKERNSEC_CONFIG_AUTO is not set
@@ -2811,8 +2812,9 @@
 CONFIG_PAX_MPROTECT=y
 # CONFIG_PAX_MPROTECT_COMPAT is not set
 # CONFIG_PAX_ELFRELOCS is not set
-# CONFIG_PAX_KERNEXEC is not set
+CONFIG_PAX_KERNEXEC=y
 CONFIG_PAX_KERNEXEC_PLUGIN_METHOD=""
+CONFIG_PAX_KERNEXEC_MODULE_TEXT=4
 
 #
 # Address Space Layout Randomization


Thanks for your time, Christian

[PaX Team]

is this still a problem (with 3.10)?

[christian.]

is this still a problem (with 3.10)?

Sorry, I forgot to tick the "notify me" checkbox. Thanks for considering to look into it.

At the moment I am running the Gentoo hardened kernel 3.11.2. The following patch is part of that kernel "4420_grsecurity-2.9.1-3.11.2-201309281103.patch". I hope that relates in any way to your patch version numbering scheme.

I activated CONFIG_PAX_KERNEXEC option again and the problem still persists exactly as described before: Fan activates, black screen, suspend LED still active and a few seconds later a reboot.

If you have any idea on how I could assist in narrowing it down, I am all ears. Though it is not critical to me: I think this particular system of mine will survive without the option being enabled. I just cannot live without suspend-to-ram.

Thanks again for your time, Christian

[PaX Team]

a few things would be useful to know/have:

1. a more recent config (for 3.11.x)
2. test your kernel in qemu and see if it fails there as well
3. how do you initiate suspend?
4. dmesg from a working suspend/resume cycle

[vincent.]

I believe I am having the same issue. Suspend to RAM worked flawlessly for me on a number of x86 systems with 3.2 grsec kernels, and no longer works on recent kernels (including the latest 3.13.5 grsec patch). Suspend seems to work fine (as far as I can tell, i.e. the suspend LED is blinking), but when trying to resume, the screen nevers comes up and it looks like the kernel panics silently.

I have been able to narrow it down to PAX_KERNEXEC : resume does not work with KERNEXEC active, but works with KERNEXEC deactivated, regardless of any other option I have tried. Furthermore, I've done some quick and dirty bissecting, which shows that :
- resume worked with KERNEXEC on any grsec kernel up to and including 3.4.7-201208021520
- resume does not work with KERNEXEC on any grsec kernel I've tested starting from 3.5.0-201208052125
Since I did not notice any significant change in KERNEXEC between those two versions, I strongly suspect the breakage was caused by changes in the mainline kernel. My primary suspect would be the 'x86, realmode: Move ACPI wakeup to unified realmode code' commit :
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c9b77ccb52a5c77233b0e557b7d4417b00ef4012
which introduced significant changes in the realmode wakeup code, though I haven't tried reverting that specific patch yet. I'm not quite sure that the new wakeup code is correct with KERNEXEC active, e.g. the 'ljmpl $__KERNEL_CS, $pa_startup_32' at the end of arch/x86/realmode/rm/wakeup_asm.S looks like it should be different with KERNEXEC, but my understanding of KERNEXEC is too limited to be sure about that.

Extra info :
- I haven't seen any significant difference in the dmesg between kernexec / non-kernexec versions, but I can provide those dmesg if needed
- I've run the basic PM debugging tests with the /sys/power/pm_test interface, which all work on a KERNEXEC 3.13.5 kernel, i.e. echo core > /sys/power/pm_test; echo mem > /sys/power/state does an almost-suspend/resume cycle without a problem.

[vincent.]

This issue is still present with the latest 3.14.1. After some further investigation, my understanding of the resume code is the following :
- resume starts by executing wakeup_start in arch/x86/realmode/rm/wakeup_asm.S (part of the 'realmode blob' introduced in 3.5.0), which sets up temporary segment descriptors, restores %cr3, %cr4 and various MSR, and ends up executing
(1) ljmpl $__KERNEL_CS, $pa_startup_32
- this then jumps to startup_32 in arch/x86/realmode/rm/trampoline_32.S (still in the realmode blob), which executes
(2) jmp *%eax
- this in turn jumps to wakeup_pmode_return in arch/x86/kernel/acpi/wakeup_32.S (outside of the realmode blob), which restores the LDT, IDT and generic registers.

This works OK on any 32 bit config without KERNEXEC (regardless of other PaX/grsec options), but fails when KERNEXEC is activated.

I've tried to follow the execution in the failing case by setting up blinking keyboard LEDs at various points in the code (since I don't have access to any other I/O at that point except perhaps the PC speaker). When KERNEXEC is activated, I see the following behavior :
- execution reaches the ljmpl $__KERNEL_CS, ... instruction in (1) above
- however, it never reaches (2) jmp *%eax

This seems to confirm that (1) is the faulting instruction. Am I correct in assuming that $pa_startup_32 (in the realmode blob, which apparently ends up in the .init.data section of the kernel image) is not reachable through __KERNEL_CS with KERNEXEC ?

[PaX Team]

(1) ljmpl $__KERNEL_CS, $pa_startup_32

thanks for the report, can you change the above to:

Code: Select all

ljmpl   *pmode_entry

and see if you can resume?

This seems to confirm that (1) is the faulting instruction. Am I correct in assuming that $pa_startup_32 (in the realmode blob, which apparently ends up in the .init.data section of the kernel image) is not reachable through __KERNEL_CS with KERNEXEC ?

yes, that code is mapped at runtime outside of KERNEL_CS in the low 1MB. the above patch will fix it hopefully.

[vincent.]

thanks for the report, can you change the above to:

ljmpl *pmode_entry

and see if you can resume?

Yes, that fixes it. Thank you very much.

[christian.]

Thanks guys. I just compiled an current 3.15.8 with grsecurity and KERN_EXEC enabled, and suspend to RAM works. Again, thank you.