slow policy generating

[KDE]

I'm generating policy with
command: gradm -F -L /etc/grsec/learning.logs -O policy
learn_config: http://pastebin.com/x7jduMdb
learning.logs size: 270 MB
Time already exceeds 1 hour. It will probably take another 2 hours.
According to strace it seems that gradm is reading /etc/grsec/learning.logs during whole generation,
which is probably one of reason of slowness.
Is there any way to make it faster?

[spender]

Is it visibly making progress? You can always sort | uniq the learning log before processing it (may help some small amount). That's not an incredibly large learning log, so it's unusual for it to be taking so long. BTW if you're willing to provide the learning log, I can profile the code to try to resolve any performance bottlenecks.

-Brad

[KDE]

I have already deleted that learning log. Would it possible for gradm to read and parse learning log once?

[KDE]

strace of gradm with 66 MB learning log
read of 8192 byte block was called 1525489 times
total read size 11917 MB, which means learning log was read 183 times

Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
76.27 127.77 127.77 2041062971 0.00 0.00 fulllearn_pass3lex
13.07 149.66 21.89 181 0.12 0.90 fulllearn_pass3parse
6.57 160.66 11.00 34017868 0.00 0.00 match_file_node
0.68 161.80 1.14 181 0.01 0.01 fulllearn_pass3restart
0.61 162.82 1.02 232426 0.00 0.00 do_find_insert_file

[spender]

It has to parse it multiple times because policy generation has to be broken up into multiple parts or else risk running out of memory or consuming the entire 32bit address space. This was so that it could handle very large learning logs and still generate a policy. Based on that profiling, I'll see what I can do to improve it.

BTW, how many roles and subjects were produced for that policy?

Thanks,
-Brad

[KDE]

66 MB learning log:
17 roles
184 subjects

using sort and uniq 66 MB file was reduced to 16 MB one and generation was much faster

I would see some optimisations:
1. conversion of learning log from text file to binary file with tree structure without data duplication
2. using mmap for reading that binary file instead of read

[spender]

184 subjects is unusual. Do you see from the policy why so many subjects were created? Were you performing admin tasks outside of the admin role?

-Brad

[KDE]

I was compiling/installing software using Portage
role root - 79 subjects
role portage - 60 subjects

[spender]

Yea, that will look like a mess, since the learning system will create a new subject for every protected resource accessed. Each of those new subjects causes a rescan of the learning log. I'm not sure I'll be able to improve the speed much while still using flex (it generates the parser code), so I'll write up a simpler parser that uses mmap.

Thanks,
-Brad

[KDE]

Would it be possible to use binary file as cache, which should be much faster than text file?

[spender]

It wouldn't be that much faster, since it consists of mostly strings and is created by the kernel in the order of accesses.

-Brad

[KDE]

I meant gradm could convert log to binary file.
I tried to store all file paths from 270 MB log file in binary files.
These binary files take only 16 MB of disk space.

[spender]

Hi,

The latest gradm in git uses a 16MB buffer for reads from the learning logs. It should significantly reduce syscall overhead, while still keeping lex/yacc around for parsing the logs. If this doesn't help enough, then I'll go ahead and hand-code a parser.

-Brad

[KDE]

It isn't faster, but I changed config file which results in much smaller log file after sort and uniq.

There is problem with installing gradm from git. After "make install" is executed system freezes. Hard reboot is needed.

[spender]

Are you sure it actually freezes? A 'make install' will cause a udev trigger, which among other things, can result in X restarting.

-Brad