When Posturing Meets Reality
Posted: Sat Nov 07, 2015 8:41 pmReaction to the Washington Post article on Linux kernel security seems to have been split: there are those that see the merits of improved security and others that though they may not disagree with adding security, think Linus' views made the most sense. In this post I'd like to get to the root of the issue and hopefully expose the reasons for the predictable reactions. This post won't get into a discussion of those who with no domain-specific knowledge claim the kernel isn't a problem or doesn't need improvement. The fundamental lack of understanding of both offensive strategies and modern software defenses of those critics has been mostly laid bare by others already: http://mjg59.dreamwidth.org/38158.html.
First off, to describe my own views about the article: I called it "very fair" and believe it is. One of my family members said about it: "now I finally understand what it is you do!" -- so from that aspect of making something actually very technical reasonably understandable by a non-technical audience it should be considered a success. Could grsecurity have looked better? Very much so -- the influence of grsecurity and PaX not just on modern OSes but the processors running those OSes was mostly missing. Could I personally have looked worse? Absolutely -- "sometimes brittle" would have been an understatement had some LWN posts been quoted as Linus' LKML mails had. Could Linus have looked better? I think so, focusing on the words in rants leaves less room for the important issues and suggests that somehow his mere words are the reason for upstream Linux not having as advanced of defenses as Apple and Microsoft these days. Could Linus have looked worse? Sure, he got to repeat the same kinds of baseless dismissals devoid of content as we've seen in the past without direct rebuttal. That said, the overall tone of the article lends itself against those claims and fits in with the repeated pattern of the rest of the series where someone points out security problems that are ignored until a breaking point is reached.
"Baseless dismissals devoid of content" is a heavy accusation to throw around, so I'll be specific and deal with each claim of the type present in the article. It is important to note that the group being referred to as "masturbating monkeys" is OpenBSD. In the previous sentence (from http://article.gmane.org/gmane.linux.kernel/706950) it uses the "black and white" phrasing that he repeated in the interview for the article. We then combine this with "the people who care most about this stuff are completely crazy", without progressing past the first page we've established someone in a clear position of authority summarily dismissing the views of those mentioned in the rest of the article. At the same time, we find the crux of Linus' argument (as summarized by the author): "His broader message was this: Security of any system can never be perfect. So it always must be weighed against other priorities -- such as speed, flexibility and ease of use -- in a series of inherently nuanced trade-offs. This is a process, Torvalds suggested, poorly understood by his critics."
It's all very persuasive and the latter argument should sound very reasonable to most everyone. There's a reason for this: no one disagrees that there are often (but not always) performance or usability tradeoffs to be made with security, least of all the critics quoted in the article. Further, none of the people referred to as "black-and-white" in their views are any of the people quoted in the article. Linus' position essentially boils down to a truism and scapegoat. Yes, security will never be perfect -- there will always be logic flaws and other errors, but the answer isn't to point to another completely separate OS that is severely lacking in general usage, performance, and modern features and say (effectively) "if we go down this path of added security, we'll end up like that." It's not necessary to make such a stretch of a comparison because the presence of grsecurity and PaX for the past decade and a half shows Linux can make improvements without crippling the performance and features it's become known for. Security of every system never being perfect should not be an excuse to only offer a small subset of security features to users running the latest versions of Linux with the very latest processors, sometimes nearly a decade after those features have plainly proven their worth. It's a defeatist attitude and an excuse made to justify the past and present security situation.
Had we been "black-and-white" in our views or valued security over any kind of performance considerations, the PaX Team wouldn't have (for instance) initially designed the x86_64 UDEREF feature in a weaker security form compared to its near 0-cost i386 equivalent that used hardware segmentation. Initially on x86_64 due to lack of a sufficient segmentation capability, UDEREF "shifted" the userland address space on entering into the kernel. While userland memory could still be accessed directly by the kernel, it was not possible via dereferencing typical magic values pointing to userland or NULL dereferences. This "shadow" userland was also made entirely non-executable by marking the associated top-level page tables NX (achieving the equivalent of SMEP prior to the existence of the KERNEXEC plugin). As an additional performance consideration, the address space was slightly reduced so that the number of page tables that needed to be copied upon a context switch (or have their present bits toggled on kernel entry/exit) could fit inside a single cache-line. If we had been "black-and-white", the PaX team would not have in 2013 made yet another sweeping change to the x86_64 UDEREF to add PCID support, a newer feature of Intel processors (but still far predating SMAP support). PCID support was used to provide an additional option: users could keep the old UDEREF behavior and benefit from increased performance due to the use of PCID lowering the performance hit of what was now a limited TLB flush, or they could use a new mode that made the security properties of the x86_64 UDEREF closer to its i386 counterpart with a level of performance that was previously impossible. Unlike Apple's no_shared_cr3 option which has unacceptable performance for most users making it only an option for debugging or those with no performance considerations whatsoever, PaX's PCID-enhanced x86_64 UDEREF is a massive improvement. In fact, the upstream Linux kernel to date doesn't make use of PCID for any kind of performance optimizations that other OSes have already caught on to. This is but one of many examples we could give and only scratches the surface of the performance considerations made for this one feature, so we can confidently say the suggestion that we don't understand the tradeoffs involved couldn't be further from the truth.
As the article hinted at, might the cutoff point of what Linus thinks of as "acceptable" with regards to performance be different from ours? Sure, as we do our work in our limited free time, we generally don't seek out or lose much sleep over single-cycle performance improvements like some salaried upstream developers devoted to that task. Yes, we often employ defensive programming strategies to make our changes more resilient against the rapid pace of upstream kernel development. But Linus' cutoff point isn't a universal truth, each user will have their own. Here too lies an important point: since the beginnings of grsecurity and PaX, the user has always had complete control over what individual features they chose to use. The allowance for personal priorities is highlighted by the fact that many years ago I updated the configuration system to allow for different modes: a high-security mode that enabled the majority of features people commonly used, and a performance mode that disabled features that we felt didn't (and couldn't really, in any conceivable form) provide security benefits commensurate with their associated performance hits. This is virtually no different from Linux today carrying a large number of features under the label of "debugging". Attempting to provide proper memory protections for the kernel is one of these "debugging" features of Linux (CONFIG_DEBUG_RODATA,CONFIG_DEBUG_SET_MODULE_RONX). There's code to detect some forms of credential misuse, improper list operations, memory leaks, locking issues, and much more. Nothing stops users from enabling any number of these features, and just like in the case of grsecurity, if they're not used there's no associated performance impact.
Linus says in the article that "most people don't want the Grsecurity system." It may be that even many don't, but it doesn't change the fact that upstream Linux lags behind other modern OSes in security, nor does it avoid the fact that Linux is diminishing its greatest strength here: that it empowers users to take control of their own systems and not have that level of control be prescribed by some other person or corporation. We don't discuss our customers or generally about grsecurity use we're aware of, but given what we know and the enormous influx of mails we've received from companies since our announcement, it's very safe to say that millions of devices and systems clearly want the "grsecurity system". Linus' public statement is both ironic and disrespectful given that not only have most of the security advances present in Linux come from grsecurity, but they're now holding invite-only meetings about what additional features of grsecurity they should adopt.
It's unclear from both the article and several others over the years about security exactly what experience and knowledge Linus has of grsecurity, yet his contentless opinions get repeated by ardent followers as fact. In a recent (currently subscription-only: http://lwn.net/Articles/662907/) article on LWN, describing an invite-only security meeting at the 2015 Kernel Summit, it says:
It's not clear whether KERNEXEC or UDEREF is being mentioned in the vague reference to "protections against executing user-space code"; the article doesn't bother to mention if it's UDEREF, what version of UDEREF, or how Linus would have implemented it differently. It may even be the case that the article has its terminology wrong and what we're seeing in 2015 is yet a repeating from Linus' apparent 2009 misunderstanding of SEGMEXEC from 2001 as what he described more accurately reflected Red Hat's ExecShield (see: https://lwn.net/Articles/315164/, https://lwn.net/Articles/313765/). There's also no mention what options then exist upstream for the vast majority of systems out there that aren't running the very latest Intel Skylake (and a limited number of Broadwell) CPUs. If not doing it "badly" means waiting 9 years for the feature to be implemented in hardware and then only making it available to a tiny subset of current users and leaving the rest in the dust, I don't consider that any form of success.
The article says Linus "is glad he refused to take it", referring to the code. Yet neither the PaX Team nor myself have ever submitted any features of PaX or grsecurity upstream. Not even any of the outside good samaritans had ever attempted to break out and submit KERNEXEC or UDEREF. How can Linus have refused to accept code that has never been offered in the first place? The "deep well" comment appears to be little more than FUD to excuse having missed out on the feature for 9 years and continuing to leave the majority of systems today vulnerable just because of the existence of some new hardware that will take years to saturate a significant portion of the market.
Likewise, he refers to unspecific "slow, horrible code" and then what appears to be an unnamed reference to the KSTACKOVERFLOW code which requires the use of vmalloc to actually detect and prevent stack overflows in the kernel, unlike the existing features of Linux that act as little more than unreliable debugging measures, which are generally not going to trigger in any intentional exploitation attempt. It's easy to criticize over the dubious downside of "adding more use of the high-memory concept" (incorrect terminology to begin with) -- but what alternative have they proposed that can actually prevent exploitation and doesn't require breaking up large pages in the kernel and in the process harming performance even worse than KSTACKOVERFLOW? Does he know of all the experiments and brainstorming that led me to the implementation I created? An academic can point to most modern exploit defenses and proudly proclaim their processor or language doesn't have some particular issue. Yet their processor exists only on paper and the language isn't used. This kind of criticism ignores reality, is unhelpful, and in the end sees its place not in the hands of users but as nothing more than a mental exercise. The alternative of "don't do anything about this problem or provide an option to users to solve this problem" is simply not acceptable.
Some unfamiliar with our work have responded to the article with: "if Linux's security is so bad, it's free software, so why doesn't someone just fork it?". That's exactly what grsecurity is and has been for 14 years. By operating outside of the slow evolution of upstream security and its resistance to change, by focusing on the work itself instead of politics and convincing others of its merits (which by now are nearly universally agreed upon), we've been able to dramatically improve the security of our users and influence that of all OSes in ways that would have otherwise been impossible. Linus' theory of gradual evolution may work for most problems, but the evolution of security itself has quickly outpaced that model. We've pushed the envelope, we understand the threats, we know how the landscape will change in the future, and we'll continue to drive innovation and security in advance of those changes, not in the years after it's needed most. Don't forget: for the past 12 years of our 14 year history (since 2003 with the introduction of PaX's KERNEXEC) we saw how the landscape would shift to kernel exploitation and have been working on kernel self-protection ever since.
Talk is cheap; as I mentioned the last time there was a push to upstream grsecurity features, despite a whole Google SoC behind it and hundreds of mailing list posts, nothing of any consequence came of it: https://lwn.net/Articles/538600/. It's easy to create a mailing list and a wiki pre-populated with the work of others, it's something completely different to have people who actually understand security from both a defensive and offensive perspective come up with original ideas and maintain them for the length of time we have in grsecurity.
So it's clear why many people still believe Linus has a point: he's been arguing for years with a pithy line that no one disagrees with against a boogeyman that doesn't exist.
-Brad
First off, to describe my own views about the article: I called it "very fair" and believe it is. One of my family members said about it: "now I finally understand what it is you do!" -- so from that aspect of making something actually very technical reasonably understandable by a non-technical audience it should be considered a success. Could grsecurity have looked better? Very much so -- the influence of grsecurity and PaX not just on modern OSes but the processors running those OSes was mostly missing. Could I personally have looked worse? Absolutely -- "sometimes brittle" would have been an understatement had some LWN posts been quoted as Linus' LKML mails had. Could Linus have looked better? I think so, focusing on the words in rants leaves less room for the important issues and suggests that somehow his mere words are the reason for upstream Linux not having as advanced of defenses as Apple and Microsoft these days. Could Linus have looked worse? Sure, he got to repeat the same kinds of baseless dismissals devoid of content as we've seen in the past without direct rebuttal. That said, the overall tone of the article lends itself against those claims and fits in with the repeated pattern of the rest of the series where someone points out security problems that are ignored until a breaking point is reached.
"Baseless dismissals devoid of content" is a heavy accusation to throw around, so I'll be specific and deal with each claim of the type present in the article. It is important to note that the group being referred to as "masturbating monkeys" is OpenBSD. In the previous sentence (from http://article.gmane.org/gmane.linux.kernel/706950) it uses the "black and white" phrasing that he repeated in the interview for the article. We then combine this with "the people who care most about this stuff are completely crazy", without progressing past the first page we've established someone in a clear position of authority summarily dismissing the views of those mentioned in the rest of the article. At the same time, we find the crux of Linus' argument (as summarized by the author): "His broader message was this: Security of any system can never be perfect. So it always must be weighed against other priorities -- such as speed, flexibility and ease of use -- in a series of inherently nuanced trade-offs. This is a process, Torvalds suggested, poorly understood by his critics."
It's all very persuasive and the latter argument should sound very reasonable to most everyone. There's a reason for this: no one disagrees that there are often (but not always) performance or usability tradeoffs to be made with security, least of all the critics quoted in the article. Further, none of the people referred to as "black-and-white" in their views are any of the people quoted in the article. Linus' position essentially boils down to a truism and scapegoat. Yes, security will never be perfect -- there will always be logic flaws and other errors, but the answer isn't to point to another completely separate OS that is severely lacking in general usage, performance, and modern features and say (effectively) "if we go down this path of added security, we'll end up like that." It's not necessary to make such a stretch of a comparison because the presence of grsecurity and PaX for the past decade and a half shows Linux can make improvements without crippling the performance and features it's become known for. Security of every system never being perfect should not be an excuse to only offer a small subset of security features to users running the latest versions of Linux with the very latest processors, sometimes nearly a decade after those features have plainly proven their worth. It's a defeatist attitude and an excuse made to justify the past and present security situation.
Had we been "black-and-white" in our views or valued security over any kind of performance considerations, the PaX Team wouldn't have (for instance) initially designed the x86_64 UDEREF feature in a weaker security form compared to its near 0-cost i386 equivalent that used hardware segmentation. Initially on x86_64 due to lack of a sufficient segmentation capability, UDEREF "shifted" the userland address space on entering into the kernel. While userland memory could still be accessed directly by the kernel, it was not possible via dereferencing typical magic values pointing to userland or NULL dereferences. This "shadow" userland was also made entirely non-executable by marking the associated top-level page tables NX (achieving the equivalent of SMEP prior to the existence of the KERNEXEC plugin). As an additional performance consideration, the address space was slightly reduced so that the number of page tables that needed to be copied upon a context switch (or have their present bits toggled on kernel entry/exit) could fit inside a single cache-line. If we had been "black-and-white", the PaX team would not have in 2013 made yet another sweeping change to the x86_64 UDEREF to add PCID support, a newer feature of Intel processors (but still far predating SMAP support). PCID support was used to provide an additional option: users could keep the old UDEREF behavior and benefit from increased performance due to the use of PCID lowering the performance hit of what was now a limited TLB flush, or they could use a new mode that made the security properties of the x86_64 UDEREF closer to its i386 counterpart with a level of performance that was previously impossible. Unlike Apple's no_shared_cr3 option which has unacceptable performance for most users making it only an option for debugging or those with no performance considerations whatsoever, PaX's PCID-enhanced x86_64 UDEREF is a massive improvement. In fact, the upstream Linux kernel to date doesn't make use of PCID for any kind of performance optimizations that other OSes have already caught on to. This is but one of many examples we could give and only scratches the surface of the performance considerations made for this one feature, so we can confidently say the suggestion that we don't understand the tradeoffs involved couldn't be further from the truth.
As the article hinted at, might the cutoff point of what Linus thinks of as "acceptable" with regards to performance be different from ours? Sure, as we do our work in our limited free time, we generally don't seek out or lose much sleep over single-cycle performance improvements like some salaried upstream developers devoted to that task. Yes, we often employ defensive programming strategies to make our changes more resilient against the rapid pace of upstream kernel development. But Linus' cutoff point isn't a universal truth, each user will have their own. Here too lies an important point: since the beginnings of grsecurity and PaX, the user has always had complete control over what individual features they chose to use. The allowance for personal priorities is highlighted by the fact that many years ago I updated the configuration system to allow for different modes: a high-security mode that enabled the majority of features people commonly used, and a performance mode that disabled features that we felt didn't (and couldn't really, in any conceivable form) provide security benefits commensurate with their associated performance hits. This is virtually no different from Linux today carrying a large number of features under the label of "debugging". Attempting to provide proper memory protections for the kernel is one of these "debugging" features of Linux (CONFIG_DEBUG_RODATA,CONFIG_DEBUG_SET_MODULE_RONX). There's code to detect some forms of credential misuse, improper list operations, memory leaks, locking issues, and much more. Nothing stops users from enabling any number of these features, and just like in the case of grsecurity, if they're not used there's no associated performance impact.
Linus says in the article that "most people don't want the Grsecurity system." It may be that even many don't, but it doesn't change the fact that upstream Linux lags behind other modern OSes in security, nor does it avoid the fact that Linux is diminishing its greatest strength here: that it empowers users to take control of their own systems and not have that level of control be prescribed by some other person or corporation. We don't discuss our customers or generally about grsecurity use we're aware of, but given what we know and the enormous influx of mails we've received from companies since our announcement, it's very safe to say that millions of devices and systems clearly want the "grsecurity system". Linus' public statement is both ironic and disrespectful given that not only have most of the security advances present in Linux come from grsecurity, but they're now holding invite-only meetings about what additional features of grsecurity they should adopt.
It's unclear from both the article and several others over the years about security exactly what experience and knowledge Linus has of grsecurity, yet his contentless opinions get repeated by ardent followers as fact. In a recent (currently subscription-only: http://lwn.net/Articles/662907/) article on LWN, describing an invite-only security meeting at the 2015 Kernel Summit, it says:
Linus said that it would indeed be useful for somebody to go through the PaX and grsecurity patches. But that code needs to be looked at carefully. For example, they have implemented protections against executing user-space code, but (according to Linus) they have done so badly. Now hardware offers that protection and we can use it; if we had accepted their code, we would be in a deep well we couldn't dig our way out of. He is glad he refused to take it.
Other parts of those patch sets include "slow, horrible code" that affects fundamental parts of the kernel. He called out in particular the fact that grsecurity adds more use of the high-memory concept, while the kernel developers are instead trying to get rid of it. That said, he allowed that there is probably useful stuff to be found there.
It's not clear whether KERNEXEC or UDEREF is being mentioned in the vague reference to "protections against executing user-space code"; the article doesn't bother to mention if it's UDEREF, what version of UDEREF, or how Linus would have implemented it differently. It may even be the case that the article has its terminology wrong and what we're seeing in 2015 is yet a repeating from Linus' apparent 2009 misunderstanding of SEGMEXEC from 2001 as what he described more accurately reflected Red Hat's ExecShield (see: https://lwn.net/Articles/315164/, https://lwn.net/Articles/313765/). There's also no mention what options then exist upstream for the vast majority of systems out there that aren't running the very latest Intel Skylake (and a limited number of Broadwell) CPUs. If not doing it "badly" means waiting 9 years for the feature to be implemented in hardware and then only making it available to a tiny subset of current users and leaving the rest in the dust, I don't consider that any form of success.
The article says Linus "is glad he refused to take it", referring to the code. Yet neither the PaX Team nor myself have ever submitted any features of PaX or grsecurity upstream. Not even any of the outside good samaritans had ever attempted to break out and submit KERNEXEC or UDEREF. How can Linus have refused to accept code that has never been offered in the first place? The "deep well" comment appears to be little more than FUD to excuse having missed out on the feature for 9 years and continuing to leave the majority of systems today vulnerable just because of the existence of some new hardware that will take years to saturate a significant portion of the market.
Likewise, he refers to unspecific "slow, horrible code" and then what appears to be an unnamed reference to the KSTACKOVERFLOW code which requires the use of vmalloc to actually detect and prevent stack overflows in the kernel, unlike the existing features of Linux that act as little more than unreliable debugging measures, which are generally not going to trigger in any intentional exploitation attempt. It's easy to criticize over the dubious downside of "adding more use of the high-memory concept" (incorrect terminology to begin with) -- but what alternative have they proposed that can actually prevent exploitation and doesn't require breaking up large pages in the kernel and in the process harming performance even worse than KSTACKOVERFLOW? Does he know of all the experiments and brainstorming that led me to the implementation I created? An academic can point to most modern exploit defenses and proudly proclaim their processor or language doesn't have some particular issue. Yet their processor exists only on paper and the language isn't used. This kind of criticism ignores reality, is unhelpful, and in the end sees its place not in the hands of users but as nothing more than a mental exercise. The alternative of "don't do anything about this problem or provide an option to users to solve this problem" is simply not acceptable.
Some unfamiliar with our work have responded to the article with: "if Linux's security is so bad, it's free software, so why doesn't someone just fork it?". That's exactly what grsecurity is and has been for 14 years. By operating outside of the slow evolution of upstream security and its resistance to change, by focusing on the work itself instead of politics and convincing others of its merits (which by now are nearly universally agreed upon), we've been able to dramatically improve the security of our users and influence that of all OSes in ways that would have otherwise been impossible. Linus' theory of gradual evolution may work for most problems, but the evolution of security itself has quickly outpaced that model. We've pushed the envelope, we understand the threats, we know how the landscape will change in the future, and we'll continue to drive innovation and security in advance of those changes, not in the years after it's needed most. Don't forget: for the past 12 years of our 14 year history (since 2003 with the introduction of PaX's KERNEXEC) we saw how the landscape would shift to kernel exploitation and have been working on kernel self-protection ever since.
Talk is cheap; as I mentioned the last time there was a push to upstream grsecurity features, despite a whole Google SoC behind it and hundreds of mailing list posts, nothing of any consequence came of it: https://lwn.net/Articles/538600/. It's easy to create a mailing list and a wiki pre-populated with the work of others, it's something completely different to have people who actually understand security from both a defensive and offensive perspective come up with original ideas and maintain them for the length of time we have in grsecurity.
So it's clear why many people still believe Linus has a point: he's been arguing for years with a pithy line that no one disagrees with against a boogeyman that doesn't exist.
-Brad