swapoff does not work when romount_protect is enabled

[ThomasKeller]

when I enable romount_protect:

echo 1 > /proc/sys/kernel/grsecurity/romount_protect

I cannot use swapoff

swapoff -a
swapoff: Not superuser.

and in the logs, I have:
grsec: denied write to block device /dev/dm-3 by /sbin/swapon[swapoff:3405] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:2727] uid/euid:0/0 gid/egid:0/0

Is this intended behaviour ?
I don't understand why swapoff needs to write to disk
Is there any way around that