No sound, unbootable system

[Christoph]

Hello,

I applied the grsecurity patch to my kernel sources, set almost all of the PAX and GrSecurity options to "yes", compiled the kernel and installed it. After rebooting and logging in into a KDE session, I realized that I had no sound. To find the reason for that, I set about the half of the PAX and GRSecurity options to "no" and recompiled the kernel. When I restarted my computer, Linux did not boot anymore. Only the tasks in the initial ramdisk were executed, but at the moment when control should be passed to the normal root directory on the harddisk, nothing happened. So, I rebooted with the normal kernel of the distribution. Then, I recompiled the grsecurity-patched sources with all PAX and GRSecurity options set to "no". But the resulting kernel also did not boot. So, I have two questions:

1. What causes the lack of sound in KDE?
2. Why does my computer boot if almost all options are enabled, but not if only some or no options are enabled?

Regards
Christoph

[PaX Team]

I applied the grsecurity patch to my kernel sources

which versions (of kernel/grsec)?

1. What causes the lack of sound in KDE?

you would have to see what exactly fails there. does the sound driver load/recognize the card (check in dmesg and /proc/alsa maybe)? if the card is properly found, is the volume set up properly? maybe alsactl or whatever is run during boot fails due to text relocations, you should be able to find that out if you manually run it.

2. Why does my computer boot if almost all options are enabled, but not if only some or no options are enabled?

probably your distro isn't prepared for some of the hardening options, that could be due to text relocations in libraries or maybe some bad GNU_STACK markings or compat VDSO, hard to say without having more information about your system. what you can try to debug it is to boot a fully enabled kernel (which you say works) then once in a shell, disable PaX features on a binary and see at which point it fails to run. then you should strace that and post or send me the log (also watch the syslog, maybe grsec logs the reason as well).

[Christoph]

Hello,

which versions (of kernel/grsec)?

Kernel 2.6.18.5, grsecurity 2.1.9-2.6.18-200610021833

you would have to see what exactly fails there.

I fear that I am not able to see that. I have sound, I can play sound files from a text console and hear the sound from the boxes. But no sound in KDE. I tried to find anything in my log files that gives me a hint about the reason for the lack of sound in KDE, but I was not successful.

probably your distro isn't prepared for some of the hardening options

But all of the hardening options are disabled?

that could be due to text relocations in libraries or maybe some bad GNU_STACK markings or compat VDSO, hard to say without having more information about your system.

Debian etch, Kernel 2.6.18-5-k7. I took the Debian source of the kernel, applied the grsecurity patch, copied the default configuration of the Debian kernel to .config, disabled COMPAT_VSDO (the glibc version in etch is new enough), disabled all grsecurity and PAX options, compiled the kernel, installed it and rebooted.

what you can try to debug it is to boot a fully enabled kernel (which you say works) then once in a shell, disable PaX features on a binary and see at which point it fails to run. then you should strace that and post or send me the log (also watch the syslog, maybe grsec logs the reason as well).

I cannot find the problem in another application. It is a special problem that occurs when control is passed from the root of the initrd to the real root filesystem. It is difficult to start strace or other debugging tools at that point.

Regards
Christoph

[PaX Team]

Kernel 2.6.18.5, grsecurity 2.1.9-2.6.18-200610021833

that's a very old and unsupported version, please upgrade to 2.6.23 (and soon .24) if possible and see if it still fails.

[Alexei.Sheplyakov]

Hi,

you would have to see what exactly fails there.

I fear that I am not able to see that.

Could you please install the 'prelink' package (apt-get install prelink)
and run the following script:

Code: Select all

#!/bin/sh
# find-pax-offending.sh
#
# Scan for PT_GNU_STACK RWX libraries and binaries at the locations specified
# on the command-line (or "standard" ones if none is given. Requires
# the `exectack' utility from the `prelink' package to operate properly.
# Doesn't depend on PaX/grsecurity functionality, so if the patched kernel
# is unbootable for some reason, the distro-supplied one is OK.
#
# ISSUES:
# Brain-dead libraries/binaries which
# 1. assume any writable memory mapping to be executable
# 2. are not marked with proper PT_GNU_STACK flag
# are NOT detected

if [ `id -u` -eq 0 ]; then
   echo "Running this script as root is a BAD idea"
fi

if [  -z "$@" ]; then
   dirs="/lib /usr/lib /bin /usr/bin"
else
   dirs_xx="$@"
   dirs=""
   for d in $dirs_xx; do
      if [ -d $d ]; then dirs="$d $dirs"; fi
   done
fi


for F in `find $dirs -type f -name '*.so' -or -perm 755`; do
   if file $F | grep -e '\<ELF\>' | grep -q -e '\(\<shared\>\)\|\(\<executable\>\)' ; then
      execstack $F
   fi   
done | grep -v '^-'


It is a special problem that occurs when control is passed from
the root of the initrd to the real root filesystem. It is difficult
to start strace or other debugging tools at that point.

Could you please unpack the initramfs image and scan it with the script above?
I.e.

Code: Select all

mkdir foo
zcat /boot/initrd-image-whatever-your-kernel-version-is | \
{ cd foo; cpio --extract ; }
find-pax-offending.sh `pwd`/foo


[Alexei.Sheplyakov]

Kernel 2.6.18.5, grsecurity 2.1.9-2.6.18-200610021833

that's a very old and unsupported version, please upgrade to 2.6.23

First of all, this seems to be a userspace problem, so, I doubt upgrading
the kernel will help. Secondly, people who need a secure system don't run
the lastest development versions of software. So, having to upgrade
the kernel every month (or so) kind of denies the whole purpose of PaX
and grsecurity.

[PaX Team]

First of all, this seems to be a userspace problem, so, I doubt upgrading
the kernel will help.

sometimes kernel bugs manifest in a failing userland (at least i can think of a few in PaX in the past). of course something like GNU_STACK or text relocations are userland problems, but then they should produce an error printed from userland as well which didn't happen here, hence my guess that it's likely some lower level kernel bug that in turn could have been fixed since.

Secondly, people who need a secure system don't run the lastest development versions of software. So, having to upgrade the kernel every month (or so) kind of denies the whole purpose of PaX and grsecurity.

oh boy, you're preaching to the choir here, tell that to the mainline kernel devs instead .

[Alexei.Sheplyakov]

sometimes kernel bugs manifest in a failing userland (at least i can think
of a few in PaX in the past). of course something like GNU_STACK or text
relocations are userland problems,

I've scaned my own box (which runs Debian too) and found out klibc libraries
and binaries (which are typically used in the initramfs images) have
PT_GNU_STACK RWX:

Code: Select all

~/bin/find-pax-offending.sh
X /lib/klibc-HTvrSvZXEQwxnmV8HLm2r15Q8yI.so
X /usr/lib/diet/bin/elftrunc
X /usr/lib/diet/bin/dnsd
X /usr/lib/klibc/bin/cat
X /usr/lib/klibc/bin/chroot
X /usr/lib/klibc/bin/dd
X /usr/lib/klibc/bin/false
X /usr/lib/klibc/bin/fstype
X /usr/lib/klibc/bin/gunzip
X /usr/lib/klibc/bin/gzip
X /usr/lib/klibc/bin/halt
X /usr/lib/klibc/bin/insmod
X /usr/lib/klibc/bin/ipconfig
X /usr/lib/klibc/bin/kill
X /usr/lib/klibc/bin/kinit
X /usr/lib/klibc/bin/kinit.shared
X /usr/lib/klibc/bin/ln
X /usr/lib/klibc/bin/minips
X /usr/lib/klibc/bin/mkdir
X /usr/lib/klibc/bin/mkfifo
X /usr/lib/klibc/bin/mknod
X /usr/lib/klibc/bin/mount
X /usr/lib/klibc/bin/nfsmount
X /usr/lib/klibc/bin/nuke
X /usr/lib/klibc/bin/pivot_root
X /usr/lib/klibc/bin/poweroff
X /usr/lib/klibc/bin/readlink
X /usr/lib/klibc/bin/reboot
X /usr/lib/klibc/bin/resume
X /usr/lib/klibc/bin/run-init
X /usr/lib/klibc/bin/sh.shared
X /usr/lib/klibc/bin/sleep
X /usr/lib/klibc/bin/true
X /usr/lib/klibc/bin/umount
X /usr/lib/klibc/bin/uname
X /usr/lib/klibc/bin/zcat


I never noticed this before, since my initramfs uses glibc based utilities
(due to LVM and mdadm).

but then they should produce an error printed from userland as well
which didn't happen here,

We don't really know. If the init binary (one in the initramfs image)
itself is PT_GNU_STACK RWX the box will reboot instantly. That happens
even before (framebuffer) console modules get loaded, so no error messages
are printed.

OK, that's just a theory, let's wait for more data from Christoph.

oh boy, you're preaching to the choir here, tell that to the mainline
kernel devs instead .

Anyway, your decision to support only the lastest kernel (instead of 2.6.16
or something like that) is a bit strange. Securing something which can't
be secure by definition... What is the point of that?

[Christoph]

Hi,

Could you please install the 'prelink' package (apt-get install prelink)
and run the following script:

The output is:

X ./bin/cat
X ./bin/chroot
X ./bin/dd
X ./bin/false
X ./bin/fstype
X ./bin/gunzip
X ./bin/halt
X ./bin/insmod
X ./bin/ipconfig
X ./bin/kill
X ./bin/kinit.shared
X ./bin/ln
X ./bin/minips
X ./bin/mkdir
X ./bin/mkfifo
X ./bin/mknod
X ./bin/mount
X ./bin/nfsmount
X ./bin/nuke
X ./bin/pivot_root
X ./bin/poweroff
X ./bin/readlink
X ./bin/reboot
X ./bin/resume
X ./bin/run-init
X ./bin/sh.shared
X ./bin/sleep
X ./bin/true
X ./bin/umount
X ./bin/uname
X ./bin/zcat
X ./lib/klibc-rOj3PRLKBA9FcF5ZuoqKQLmOWcA.so

Regards
Christoph

[Alexei.Sheplyakov]

Hello,

The output is:

X ./bin/cat
X ./bin/chroot
X ./bin/dd

(skipped)

So, the early userspace is definitely crippled. Fix the binaries/libraries
with execstack -c, repack the initramfs image, and try to boot with that.
This script below might help you. It needs the `fakeroot' utility
(apt-get install fakeroot).

Code: Select all

#!/bin/sh -e
if [ `id -u` -eq 0 ]; then
   echo "Running this script as root is a BAD idea"
   exit 1
fi

if [ -z "$1" ]; then
   echo "`basename $0`: attempt fix crippled klibc early userspace."
   echo "Usage: `basename $0` /path/to/initramfs/image"
   exit 1
fi

INITRAMFS="$1"
if [ ! -f $INITRAMFS_IMG ]; then
   echo "`basename $0`: $1: no such file"
   exit 1
fi
NEW_INIRAMFS=`basename $INITRAMFS`.new
STAGE=${NEW_INIRAMFS}.tmp
if [ -d $STAGE ]; then
   rm -rf $STAGE
fi
mkdir -p $STAGE

zcat "$INITRAMFS" | { cd $STAGE ; cpio --extract || exit 1 ; }

{
for F in `find $STAGE -type f -name '*.so' -or -perm 755`; do
   if file $F | grep -e '\<ELF\>' | grep -q -e '\(\<shared\>\)\|\(\<executable\>\)' ; then
      execstack $F
   fi   
done | grep -v '^-' ; } | sed -e 's/^\<X\>//' | \
while read FILE; do
   execstack -c $FILE
done

{ cd $STAGE ; find . | fakeroot cpio --quiet --dereference -o -H newc ; } | \
gzip -9 > ${NEW_INIRAMFS}

echo "Now run the following commands as root:"
echo "cp -a $INITRAMFS ${INITRAMFS}.bak"
echo "cp -a $NEW_INIRAMFS $INITRAMFS"



Good luck!

P.S.
Unfortunately, the version of grsec patch you happen to use has
several bugs, so you'll really need to upgrade the kernel

[Christoph]

Hello,

Good luck!

Unfortunately, executing your script did not help, though execstack could successfully clear the executable stack flag of the binaries. Still, control could not be passed to the real root file system.

P.S.
Unfortunately, the version of grsec patch you happen to use has
several bugs

What are these bugs?

so you'll really need to upgrade the kernel

That is not a good option for me, for the reason you mentioned above.

Regards
Christoph

[Alexei.Sheplyakov]

Unfortunately, executing your script did not help, though execstack could
successfully clear the executable stack flag of the binaries. Still, control
could not be passed to the real root file system.

There might be several reasons for that.

1. The boot loader uses the old initramfs image. (Do you use lilo?)
2. There's some PT_GNU_STACK stuff left in the *real* root filesystem.
3. Some PaX-related *kernel* bug preventing your hardware from working properly.

You might try to compile in all drivers necessary to mount root to sort
out initramfs related issues. If that won't help, you *really* have
to upgrade the kernel ... or give up.

What are these bugs?

See e.g. viewtopic.php?p=6791

These are definitely DoS'able, and even exploitable. (I didn't bother
to write an exploit, so I can't know for sure).

That is not a good option for me, for the reason you mentioned above.

That means grsecurity is not a good option for you.

[PaX Team]

Anyway, your decision to support only the lastest kernel (instead of 2.6.16
or something like that) is a bit strange.

it's the best way to spend my time on tracking 2.6 development.

Securing something which can't be secure by definition... What is the point of that?

what makes you think that "2.6.16 or something like that" is secure?

[Alexei.Sheplyakov]

what makes you think that "2.6.16 or something like that" is secure?

Obviously, it can't be absolutely secure, but

- The rate of changes is not that high as in mainline.
- No new features are added, hence, no new bugs (of course, that doesn't mean
all of old bugs are known).
- Finally, upgrade to new (bugfix) version doesn't break anything. Many people
refrain from upgrading (and run vulnerable versions of software) exactly
because of new versions come not only with a bugfix, but with lots of other
(unwanted, irrelevant) changes.

[Christoph]

Hello,

with kernel 2.6.24 and the newest stable version of grsecurity, the initrd problem has disappeared. But I still have problems with sound in KDE. As I was now able to check which kernel configuration option causes the lack of sound, I found out that it occurs when "Restrict mprotect" is enabled.

Does anybody know in which binary I must turn the MPROTECT flag off? Or how can I find that out myself?

Regards
Christoph