It all depends on how many patches there are in the debian queue and whether or not they conflict with grsecurity/pax. And if one conflicts, taking it out may start to make other debian patches start to fail to apply. Its a bad domino effect.
Only real way to find out is to give it a whirl.
An example of this ... at this current moment I grabbed the latest fedora 2.6.25.2 kernel-xen from rawhide and applied pax ... so far there were only a few things that conflicted... the execshield patch, which made four other patches fail when I took it out (all post-execshield related) and I fixed one of them due to compile errors. Then there were 10 minor hunk failures from pax that I had to fix, and it's in the process of building. We'll see if it works out
But if I tried to do this for a RHEL kernel for example, it would be impossible (1600+ patches to go through).
So if you know what you're doing, it isn't too hard. If not - you're going to have to stick with using a vanilla kernel + grsecurity/pax.