grsec fork rate limiting

Discuss usability issues, general maintenance, and general support issues for a grsecurity-enabled system.

grsec fork rate limiting

Postby harrygittens » Tue Mar 20, 2007 7:39 am

hi
do grsec have any fork rate limiting any more. i always thought it did.

reason is because we have shell access for users. they are limited to 15 procs through /etc/security/limits.conf. grsec is enabled and on high too.

the normal bash fork bombs and c fork() don't work, but this cmd is killing our system:

Code: Select all
python -c 'while 1: __import__("os").fork()'


httpd locks up and new users cannot ssh in. machine can still be pinged but thats about it

we have banned ip but he keeps coming back cus he is using tor.. we have removed execute permission on python but how can this fok bomb make load avg go 50+
harrygittens
 
Posts: 21
Joined: Fri Feb 16, 2007 2:20 pm

Postby tosh » Tue Mar 20, 2007 3:49 pm

Hello. I don't use PAM but have limits set in /etc/limits, the above code do not kill my system. I can log in as other user and system is responsive. Also I found lots of that in logs:
Code: Select all
Mar 20 20:42:15 pilzz kernel: grsec: From 10.0.0.2: denied resource overstep by requesting 50 for RLIMIT_NPROC against
limit 50 for /usr/bin/python[python:19491] uid/euid:1007/1007 gid/egid:100/100, parent /sbin/init[init:1] uid/euid:0/0
gid/egid:0/0
Mar 20 20:42:15 pilzz kernel: grsec: more alerts, logging disabled for 10 seconds
This is from console where above code was run:
Code: Select all
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 11] Resource temporarily unavailable
So are you sure your limits settings are enforced and grsec is configured to limit fork?
Last edited by tosh on Tue Mar 20, 2007 4:52 pm, edited 1 time in total.
tosh
 
Posts: 19
Joined: Mon Apr 10, 2006 9:13 pm

Postby harrygittens » Tue Mar 20, 2007 4:38 pm

hi
is this what you mean

Code: Select all
CONFIG_GRKERNSEC_FORKFAIL=y


here is my other config

Code: Select all
CONFIG_GRKERNSEC=y
# CONFIG_GRKERNSEC_LOW is not set
# CONFIG_GRKERNSEC_MEDIUM is not set
CONFIG_GRKERNSEC_HIGH=y
# CONFIG_GRKERNSEC_CUSTOM is not set
CONFIG_GRKERNSEC_KMEM=y
# CONFIG_GRKERNSEC_IO is not set
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODSTOP=y
CONFIG_GRKERNSEC_HIDESYM=y
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30
CONFIG_GRKERNSEC_PROC=y
# CONFIG_GRKERNSEC_PROC_USER is not set
CONFIG_GRKERNSEC_PROC_USERGROUP=y
CONFIG_GRKERNSEC_PROC_GID=1001
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_FIFO=y
# CONFIG_GRKERNSEC_CHROOT is not set
CONFIG_GRKERNSEC_CHROOT_MOUNT=y
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_CHMOD=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_CAPS=y
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
# CONFIG_GRKERNSEC_EXECLOG is not set
CONFIG_GRKERNSEC_RESLOG=y
# CONFIG_GRKERNSEC_CHROOT_EXECLOG is not set
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
CONFIG_GRKERNSEC_AUDIT_MOUNT=y
# CONFIG_GRKERNSEC_AUDIT_IPC is not set
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
CONFIG_GRKERNSEC_TIME=y
# CONFIG_GRKERNSEC_PROC_IPADDR is not set
# CONFIG_GRKERNSEC_AUDIT_TEXTREL is not set
CONFIG_GRKERNSEC_EXECVE=y
CONFIG_GRKERNSEC_SHM=y
CONFIG_GRKERNSEC_DMESG=y
# CONFIG_GRKERNSEC_TPE is not set
CONFIG_GRKERNSEC_RANDNET=y
# CONFIG_GRKERNSEC_SOCKET is not set
# CONFIG_GRKERNSEC_SYSCTL is not set
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4
harrygittens
 
Posts: 21
Joined: Fri Feb 16, 2007 2:20 pm

Postby tosh » Tue Mar 20, 2007 5:02 pm

harrygittens wrote:is this what you mean

Code: Select all
CONFIG_GRKERNSEC_FORKFAIL=y

No the above logs only . You need CONFIG_GRKERNSEC_EXECVE to limit forks (which you have set in your config) and finally
Code: Select all
sysctl -a | grep execve
should return:
Code: Select all
kernel.grsecurity.execve_limiting = 1


You wrote that bash fork bombs don't work so I would suggest checking if limits in PAM config do work.
tosh
 
Posts: 19
Joined: Mon Apr 10, 2006 9:13 pm

Postby harrygittens » Wed Mar 21, 2007 4:52 am

hi
i get this

Code: Select all
root@themany:/home/encore# sysctl -a | grep execve
error: "Success" reading key "dev.parport.parport0.autoprobe3"
error: "Success" reading key "dev.parport.parport0.autoprobe2"
error: "Success" reading key "dev.parport.parport0.autoprobe1"
error: "Success" reading key "dev.parport.parport0.autoprobe0"
error: "Success" reading key "dev.parport.parport0.autoprobe"
error: "Operation not permitted" reading key "net.ipv6.route.flush"
error: "Operation not permitted" reading key "net.ipv4.route.flush"
root@themany:/home/encore#


it does not seem to be there
harrygittens
 
Posts: 21
Joined: Fri Feb 16, 2007 2:20 pm

Re: grsec fork rate limiting

Postby Alexei.Sheplyakov » Fri Mar 23, 2007 3:04 pm

harrygittens wrote:hi
do grsec have any fork rate limiting any more. i always thought it did.


Yes, to some extent. CONFIG_GRKERNSEC_BRUTE:

"attempts to bruteforce exploits against forking
daemons such as apache or sshd will be deterred. When a child of a
forking daemon is killed by PaX or crashes due to an illegal
instruction, the parent process will be delayed 30 seconds upon every
subsequent fork until the administrator is able to assess the
situation and restart the daemon. It is recommended that you also
enable signal logging in the auditing section so that logs are
generated when a process performs an illegal instruction."

If resource limits are configured properly, this feature stops most
of fork bombs, because what grsec checks for is the rate of the _failed_
forks (the actual reason of failure is not important). So when, say,
RLIMIT_NPROC is overstepped, fork() start to fail, and all the forking
processes are forcibly put to sleep...


reason is because we have shell access for users. they are limited to 15
procs through /etc/security/limits.conf. grsec is enabled and on high too.


The limits specified in
/etc/security/limits.conf apply only to the processes started in a PAM
session, (e.g. user shells). Daemons escape this restrictions since
they are started directly by init. To restrict them one has to expilcitly
specify resource limits in the startup scripts. Details depend on your
distribution, init, and the daemon itself.

I use runit ( http://smarden.org/runit) as init. Here is the startup
script of the web server (thttpd, http://www.acme.com/software/thttpd):

Code: Select all
$ cat /etc/sv/thttpd/run
#!/bin/sh

CONFFILE=/etc/thttpd/thttpd.conf

if [ ! -f $CONFFILE ]; then
    exit 1
fi
# Run with nice value
NICE='-n 15'
# Limit number of concurrently running CGIs
NPROC='-p 10'
# Limit the VM available to the daemon to 600Mb
MEM='-m 629145600'

exec chpst $NICE $NPROC $MEM /usr/sbin/thttpd -D -C $CONFFILE


For Sys V init your need to stick ulimit -u NPROC
(and ulimit -u VMSIZE) into /etc/init.d/httpd (or whatever it is on
your system).

the normal bash fork bombs and c fork() don't work, but this cmd is
killing our system:

Code: Select all
python -c 'while 1: __import__("os").fork()'



Interesting. I've just tried it on my system. The thing stopped after
complaining about fork failures (so presumably grsec works as advertised)

Code: Select all
$ ulimit -a
core file size          (blocks, -c) 16384
data seg size           (kbytes, -d) 1048576
max nice                        (-e) 38
file size               (blocks, -f) unlimited
pending signals                 (-i) unlimited
max locked memory       (kbytes, -l) 262144
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 16384
max rt priority                 (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 60
virtual memory          (kbytes, -v) 524288
file locks                      (-x) 1024
$ python -c 'while 1: __import__("os").fork()'
Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable  File "<string>", line 1, in ?

OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
-bash: fork: Resource temporarily unavailable
  File "<string>", line 1, in ?
OSError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in ?


P.S.

@moderator I apologize for being slightly off topic, but I don't like
RTFM as an answer.
Alexei.Sheplyakov
 
Posts: 53
Joined: Sun Feb 19, 2006 11:48 am


Return to grsecurity support

cron