using grsecurity-2.9.1-3.7.1-201212271953.patch and CONFIG_MODULES unset ends up with:
In file included from drivers/video/vesafb.c:12:0:
include/linux/moduleloader.h: In function 'apply_relocate':
include/linux/moduleloader.h:60:63: error: dereferencing pointer to incomplete type
make[2]: *** [drivers/video/vesafb.o] Error 1
make[1]: *** [drivers/video] Error 2
make: *** [drivers] Error 2
my fix ended up being http://pastebin.com/SG6wtiHq
the first hunk is optional and should go upstream I guess?