grsecurity forums

Hi all,

I'm experiencing a bandwidth problem with a router (ubuntu running on a 3.14.17 grsec kernel with netns ans ovs). The bandwidth between 2 vm on 2 different networks (2 vlans tagged with ovs basically) is really slow (80kb/s = red color) whereas when rebooting my router on a regular kernel everything is ok (green color = 3.87Gb/s). Tests were done with iperf.

Here is a picture representing the architecture :
http://postimg.org/image/dcs55r7nl/

Please tell me if you need more infos

grsec config :

CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_JIT_HARDEN=y
CONFIG_GRKERNSEC_PERF_HARDEN=y
CONFIG_GRKERNSEC_RAND_THREADSTACK=y
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_KSTACKOVERFLOW=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODHARDEN=y
CONFIG_GRKERNSEC_HIDESYM=y
CONFIG_GRKERNSEC_RANDSTRUCT=y
CONFIG_GRKERNSEC_RANDSTRUCT_PERFORMANCE=y
CONFIG_GRKERNSEC_KERN_LOCKOUT=y
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USERGROUP=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_SYMLINKOWN=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_SYSFS_RESTRICT=y
CONFIG_GRKERNSEC_DEVICE_SIDECHANNEL=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_MOUNT=y
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_CHMOD=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_CAPS=y
CONFIG_GRKERNSEC_CHROOT_INITRD=y
CONFIG_GRKERNSEC_RESLOG=y
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_TIME=y
CONFIG_GRKERNSEC_PROC_IPADDR=y
CONFIG_GRKERNSEC_RWXMAP_LOG=y
CONFIG_GRKERNSEC_DMESG=y
CONFIG_GRKERNSEC_HARDEN_PTRACE=y
CONFIG_GRKERNSEC_PTRACE_READEXEC=y
CONFIG_GRKERNSEC_SETXID=y
CONFIG_GRKERNSEC_HARDEN_IPC=y
CONFIG_GRKERNSEC_TPE=y
CONFIG_GRKERNSEC_BLACKHOLE=y
CONFIG_GRKERNSEC_NO_SIMULT_CONNECT=y
CONFIG_GRKERNSEC_DENYUSB=y
CONFIG_GRKERNSEC_SYSCTL=y
CONFIG_GRKERNSEC_SYSCTL_ON=y

1. what about the PaX part of the config?
2. can you run perf on the host kernel while the two VMs are commucating to see where the CPU cycles are spent?

Sorry, I forgot the PaX part :

#
# Security options
#

#
# Grsecurity
#
CONFIG_PAX_KERNEXEC_PLUGIN=y
CONFIG_PAX_PER_CPU_PGD=y
CONFIG_TASK_SIZE_MAX_SHIFT=42
CONFIG_PAX_USERCOPY_SLABS=y
CONFIG_GRKERNSEC=y
CONFIG_GRKERNSEC_CONFIG_AUTO=y
# CONFIG_GRKERNSEC_CONFIG_CUSTOM is not set
CONFIG_GRKERNSEC_CONFIG_SERVER=y
# CONFIG_GRKERNSEC_CONFIG_DESKTOP is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_NONE is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_GUEST=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_HOST is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_EPT=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_SOFT is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_XEN is not set
# CONFIG_GRKERNSEC_CONFIG_VIRT_VMWARE is not set
CONFIG_GRKERNSEC_CONFIG_VIRT_KVM=y
# CONFIG_GRKERNSEC_CONFIG_VIRT_VIRTUALBOX is not set
CONFIG_GRKERNSEC_CONFIG_PRIORITY_PERF=y
# CONFIG_GRKERNSEC_CONFIG_PRIORITY_SECURITY is not set

#
# Default Special Groups
#
CONFIG_GRKERNSEC_PROC_GID=1001
CONFIG_GRKERNSEC_TPE_UNTRUSTED_GID=1005
CONFIG_GRKERNSEC_SYMLINKOWN_GID=1006

#
# Customize Configuration
#
#
# PaX
#
CONFIG_PAX=y

#
# PaX Control
#
# CONFIG_PAX_SOFTMODE is not set
CONFIG_PAX_EI_PAX=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_XATTR_PAX_FLAGS=y
# CONFIG_PAX_NO_ACL_FLAGS is not set
CONFIG_PAX_HAVE_ACL_FLAGS=y
# CONFIG_PAX_HOOK_ACL_FLAGS is not set

#
# Non-executable pages
#
CONFIG_PAX_NOEXEC=y
CONFIG_PAX_PAGEEXEC=y
CONFIG_PAX_EMUTRAMP=y
CONFIG_PAX_MPROTECT=y
# CONFIG_PAX_MPROTECT_COMPAT is not set
# CONFIG_PAX_ELFRELOCS is not set
CONFIG_PAX_KERNEXEC=y
CONFIG_PAX_KERNEXEC_PLUGIN_METHOD_BTS=y
CONFIG_PAX_KERNEXEC_PLUGIN_METHOD="bts"

#
# Address Space Layout Randomization
#
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDKSTACK=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y

#
# Miscellaneous hardening features
#
# CONFIG_PAX_MEMORY_SANITIZE is not set
# CONFIG_PAX_MEMORY_STACKLEAK is not set
# CONFIG_PAX_MEMORY_STRUCTLEAK is not set
# CONFIG_PAX_MEMORY_UDEREF is not set
CONFIG_PAX_REFCOUNT=y
CONFIG_PAX_CONSTIFY_PLUGIN=y
CONFIG_PAX_USERCOPY=y
# CONFIG_PAX_USERCOPY_DEBUG is not set
CONFIG_PAX_SIZE_OVERFLOW=y
CONFIG_PAX_LATENT_ENTROPY=y

#
# Memory Protections
#
CONFIG_GRKERNSEC_KMEM=y
CONFIG_GRKERNSEC_IO=y
CONFIG_GRKERNSEC_JIT_HARDEN=y
CONFIG_GRKERNSEC_PERF_HARDEN=y


I'll run iperf on the host kernel and keep you in touch asap. FYI, the host is also running the same kernel with grsec.

I'll try to describe you more clearly what I have :


VM-a Router VM-b
------ -> ------ -> ------
Host 1 Host 2 Host 3

VM-a (no grsec kernel) is running on host 1 (with grsec/Pax), Router (grsec kernel) uses namespaces netns and ovs like a tenant router. Router is running on Host2 with Grsec kernel. VM-b (no grsec) is running on Host3 (Grsec kernel).

When VM-a is communicating with VM-b via router the traffic is about 80kb/s when router is running on a grsec kernel. When I reboot router under a regular kernel, traffic is becoming normal (about 3.87Gb/s).
Iperf between host 1, 2 and 3 is ok (9.89Gb/s).
So I suspect something to be wrong with my grsec kernel on the router...

so run perf stat on the router and then we'll know where it spends its time (if it's cpu bound at all, that is). you can also try to do a binary search for the config option that causes this, i'd start with SIZE_OVERFLOW/LATENT_ENTROPY/KERNEXEC/USERCOPY/REFCOUNT in that order.

Is there a buffer added somewhere ?

LRO is off on my physical NICS

we don't touch networking code this deep, so i have no idea why the cpu is basically idle instead of transmitting packets. so i guess the next step is disabling config options until the culprit is found then we can do some debugging.

PS: i edited the console output above to use the code tag, feel free to do so yourself in the future

Ok, I'll compile a new kernel with SIZE_OVERFLOW/LATENT_ENTROPY/KERNEXEC/USERCOPY/REFCOUNT disabled to see if it fixes the issue, if yes I'll try to disable them one by one in the order you told me.

also try a newer kernel please, yours is quite old now...

yes I'm compiling the 3.14.34, I keep you in touch...thx

another idea i just had is that maybe some procfs/sysfs restriction prevents your userland from setting a knob somewhere that'd be needed for proper network performance. maybe try to disable them too and/or check if all your interface/network settings end up being the same between the grsec and normal kernel.

You mean CONFIG_GRKERNSEC_SYSFS_RESTRICT ?

yes though on a second thought this would affect non-root users only and i guess you're not doing any setup on the router as such.

Ok SIZE_OVERFLOW/LATENT_ENTROPY/KERNEXEC/USERCOPY/REFCOUNT are not set now in my new 3.14.34 kernel and no changes in the behaviour... same problem...
I'll try setting CONFIG_GRKERNSEC_SYSFS_RESTRICT off ... but I'm starting to be pessimistic.