It is possible to install grsecurity on a Desktop, I have it working here on stable (Lenny) and unstable (Sid) pc's.
I haven't heard of specific problems with desktop managers yet. Gdm works without problem here, but I don't use the Gnome desktop manager myself.
A few hints to start:
- Before starting take the time to read all the comments in "make menuconfig" for all options. You probably need to select "custom" instead of "high security" for that purpose.
- If you don't read all the information make sure to at least read the QuickStart Guide from a to z (
http://www.grsecurity.org/papers.php)
- Did you select "Disable privileged I/O"? (disable it for X).
- Did you change the group numbers? (The default group numbers used by Debian start at 1000, the special groups in grsecurity also start at 1000, I normally choose the special groups in the 200x numbers).
- Have you installed paxctl already (debian package available in usual repositories)?
- Check all the grsecurity options in sysctl, they are enabled by default but they can be altered until you set kernel.grsecurity.grsec_lock = 1 (last line in /etc/sysctl when you have a stable system).
If nothing works disable the starting of gdm in runlevel 2 (/etc/rc2.d), start the pc and start gdm from the prompt as root.
When it fails check dmesg and /var/log/Xorg.0.log for hints.
Grsecurity is known to possibly break applications. I suggest you configure your bootloader to start different versions of the kernel and start with low or medium security first. Instead of "high security" I usually choose "custom", but I think most high security options are enabled on my systems.