Or just x86's virtualization. Nothing different popped in my news feed until now. Has OpenBSD's position changed on the subject of virtualization? And is there a more recent post that explains their position?
Theo's more recent talks (like that one for ruBSD) seem to indicate that OpenBSD's looking to prioritize virtualization in order to stay competitive with other operating systems.
It's also worth noting that OpenBSD has supported virtualization on some other platforms, too; for example, it can be used to manage SPARC logical domains (LDOMs).
Makes sense. I disagree with him in that they can definitely have benefit over regular OS for security. The reason is relative simplicity and ability to easily integrate security tech in versus a whole OS with legacy compatibility issues. This was proven in KVM/370, then VAX Security Kernel, then MILS, then Nizza Security Architecture, and then people started leveraging Xen for similar reasons albeit without the same assurance. The NSA's pentesters repeatedly failed to breach some of these despite years of effort while the OS's... see Snowden leaks and TAO catalog.
Of course, I agree that VM's aren't necessary: one of many approaches one can use and don't get the job done by themselves. They are beneficial, though, as running security-critical components on a 4-12Kloc in kernel mode with mediated, simple interface should have way less impact than running it on 1Mloc+ with POSIX-style interface.
I haven't read it, but I assume the discussion was about sandboxing clients. From that perspective, any additional security would be defeated as soon as a client is able to affect the hypervisor or the host OS. So (according to Theo) if you can't write a secure host and/or client, the VM doesn't improve security.
While I agree with him on a lot of things, this isn't one of them; security is all about layers, and it's not like OpenBSD hasn't steered clear of other forms of sandboxing (like chroots and systrace). This doesn't mean that virtualization should be relied on exclusively or even near-exclusively as a defense (which is what I suspect Theo was more objecting to, along with the point of "well if you can't write a secure operating system, what makes you think you can write a secure hypervisor?"), but rather that it should be used as an additional layer on top of (or rather, underneath) a bunch of others.
It's like bulkheads on modern ships. Yeah, if you get a hole in your hull, you're gonna be in some (literally) deep water, but that bulkhead (so long as it's built right) could mean the difference between limping to the nearest harbor or sinking to the nearest seafloor.
Somehow back when Amazon had to reboot every EC2 server to fix a Xen bug, my "insecure" hypervisor-less server didn't require such action. I think I'll prefer to keep sailing without that particular bulkhead.
The reboots were because Amazon cuts a lot of corners in their Xen setup because instances are supposed to be "disposable". A proper VM cluster would use dedicated storage nodes that export over something like iscsi, which would require just transferring a memory snapshot, or would use the native, slower, disk snapshotting migration. But that's just one of many ways AWS is rather broken from an operations standpoint.
What you call a proper VM cluster has been pretty terrible from my experience as both a developer on an early cloud and a consumer of large government facing clouds.
Typically the network or SAN becomes oversaturated and the vm's shit the bed. AWS on the other hand was considerably more reliable and I'd argue they've made better decisions rather than cut corners.
The SAN becoming over-saturated isn't something that just "happens". Between establishing limits ahead of time and monitoring that shouldn't happen without someone knowing well ahead of time.
Just to be clear, I'm defending Amazon over an accusation that they've cut corners from an install that was up in 2007 or earlier. Now we could have focused on xen guests shitting the bed for no apparent reason or flakey switch port but we decided to focus on storage.
OK so I'm not proclaiming to be an expert but as someone working in the area in 2007, buying something off the shelf like it aint no thing, you're getting something like a netapp with a limit of 512 iscsi inititatiors, or a Sun amber road where your only form of automation is an ssh consol with a big warning stating it's unsupported.
From memory, there was no such thing as setting quotas on the amount of IOPs an iSCSI inititatior can do, in fact, I'm fairly sure IOPs quotas just didn't exist period, as the vendors weren't really up-to-speed with this new selling vm's thing. So basically, we're suggesting that it's a good idea to just buy a SAN to run an indetermined amount of vm's that are going to do an indetermined amount of IOPs.
OK, cool, you're now indebted to storage vendors selling you new shelvs at £80,000 a pop for those extra IOPs you so deperately need. Now to be fair, Amazon could probably afford it, but your VMs would still be a lot more expensive and would probably have still been totally disposible when your switch port decides to blip traffic to the SAN or as previously stated, Xen shits the bed.
None of these things might be a problem today, I don't know, I'm more a consumer than a producer of clouds these days, but I'd suggests these criticisms are bullshit. They come from some obviously smart people, but bullshit none-the-less.
Well true; even the RMS Titanic, with its fifteen bulkheads, was no match against an iceberg. Let's just hope that this new hypervisor for OpenBSD is built with better-quality steel :)
Also, it's worth mentioning that EC2 instances are meant to be ephemeral; Amazon doesn't provide any semblance of a guarantee that your instances won't reboot, and assumes that you intend for all your "machines" to be arbitrarily rebootable. Not saying that any hypervisor implementation right now is particularly good; only that Amazon isn't exactly the best representation here.
How many bugs have OpenBSD team found vs found in Xen? That would be a relevant comparison. From there, an assessment of exploitability of each given OpenBSD's attention to mitigation.
What you said, on other hand, was meaningless given that OpenBSD has had bugs that could lead to a crash. Real question is, "Do Xen or security-focused virtualization schemes (a) reduce number of vulnerabilities with impact of kernel-mode 0-days, and/or (b) prevent, contain, or facilitate easy recovery from OS- and app-level 0-days?" Prior experience in security-focused efforts show yes to both questions. Xen isn't one of them as the existence of the Xenon project shows. However, it's small size and improvements over time make it substantially less risky than an arbitrary OS + software combination esp if above layer is also addressed (eg MirageOS). Even Galois Inc.'s conservative teams are using it in some work.
and why can't you do that with containers or jails? or filesystem snapshots? or instant re-deploy to a known good state from ansible/puppet/salt/chef?
Just because there's a nice mouse-navigable GUI for entry level Windows admins doesn't mean it's a good solution.
if you need snapshots for doing updates it's because your software is fragile, undocumented, and you don't have a deployment procedure. Fix that, and upgrades will be easy and not scary.
Or you could have a consistent repeatable way of doing it across multiple apps and operating systems. I am not averse to your argument that doing it at the app level has benefits, but being able to do it at the VM level in a consistent way is going to be simpler when you expand beyond a handful of applications.
I agree that there is convenience to the execution controls VMs provide. But if you can't repeatably and easily stand up a replacement/duplicates of a system from configuration management and backups, then you don't actually have configuration management or backups.
Sure, to be clear I'm not arguing against those things, I'm just arguing that VM level snapshots and clones are also a tool in the operational arsenal and is in fact a very popular one that works well in practice.
KVM, VMware, and Xen all allow you to do a snapshot of memory as well as of disk. It's one of the tools used to migrate xen instances between nodes, for example.
Yes, people have repeated that line since 2008. Readers certainly saw the timestamp. It was main Q&A and search result, though. Hence asking for modern post.
"This is from 2013"
And that's a start. Appreciate the link. I don't see many of his interviews but that was the first one I saw him admit to being behind. Good they're changing their attitude a bit.
The expected and somewhat disappointing part is when he has no answer to what can be done to raise the bar past a few exploit mitigations. There's something like four decades of work (and worked examples) showing how to increase assurance of security in hardware, software, and systems. Especially in capability, microkernels, static analysis, covert channel analysis, and so on. He could... idk... apply some of that instead of mock and ignore it like most mainstream does. FreeBSD is ahead here with SEBSD and Capsicum work.
One project did port OpenBSD to L4 kernel to isolate it in a protection domain. The idea, as in Nizza Security Architecture, is to be able to split system into legacy, untrusted stuff in VM and trusted, highly-assured components running directly on microkernel. A proven model that would benefit OpenBSD by dramatically reducing attack surface. This is done in embedded space (eg INTEGRITY, PikeOS Hypervisor) for up to 8 ISA's each for those wondering about portability.
Just one of dozens of techs to draw on to increase assurance. Will be interesting to see if they draw on any of this or get left behind [again] by those that do.
Both EROS and Genode managed to do a lot of what I mentioned with less staff & time. Plus, with solid architecture that above components can leverage without doing all the highly assured stuff over and over again: what I call security ROI. Your huge TCB, drivers probably in kernel, and language-related issues are where I predicted most of the trouble would come from. (Did it?) Something like Nizza Security Architecture would benefit you along with interface checks you're already good at and a better language that translates to C with checks automatically inserted.
Far as covert channels, Google Kemmerer's Shared Resource Matrix for a method that worked for amateurs and pro's alike on a budget. Any person handling a subsystem can apply it. Even using it on function call level can tell you a lot. If understaffed, just use it on any component handling secrets for storage and timing channels. Best bang for buck.
Ok, to be more serious for a bit, a goal of the OpenBSD project is to produce a Unix like operating system. To the extent "innovation" is "don't be Unix" it's somewhat counter to project goals.
Well, there we go. That's unfortunate, but understandable. However, it still allows you to build on decades of work in security engineering (incl old secure UNIX's). The easiest route at this point is putting OpenBSD API on top of a microkernel, pulling security-critical functionality out of main system onto microkernel, and bulletproofing your middleware for these. Additionally, writing the code in a way that lets tools such as Astree Analyzer work on as much of it as possible will knock out many bugs. Compiler tools that automatically transform kernel or user-mode code to make it safer might help. Softbound + CETS comes to mind.
Much to draw on or improve while remaining a UNIX. The microkernel + user-mode virtualization approach has already been done in academia and commercial products. So, it could be done here. Will they? Another matter entirely. I doubt it.
Truth be told, though, I voted for the Xen Dom0 to use OpenBSD because 0-days would be its main concern. And we know which team is the best at removing them from a UNIX codebase. ;)
It's fun to do what the sheep or crowds are doing. That's Windows, Mac, BSD, or UNIX at any given time. Throw in C/C++ everywhere, browsers, Flash, HTTP as universal transport, language runtimes too bloated to understand, the Cloud... the more vulnerabilities and maintenance horrors, the more they'll like you and the more fun you'll have with them.
Our security needs to improve dramatically. Some ways are proven to work, some are proven not to. Pushing the second category is extremely fun, you might get famous in Silicon Valley companies, maybe invited to DEFCON, and everyone will make excuses for its problems later. Lots of buzzwords there, I agree: can't even mentally track all of them. Then there's a tinier group pushing methods that work because they're necessary, even if not all fun. Staying with that group on principle if not profit.
Comments
Good to see OpenBSD in process of getting one. I previously thought that OpenBSD, or at least Theo, hated virtualization based on this rant:
http://www.tylerkrpata.com/2007/10/theo-de-raadt-on-x86-virt...
Or just x86's virtualization. Nothing different popped in my news feed until now. Has OpenBSD's position changed on the subject of virtualization? And is there a more recent post that explains their position?
Theo's more recent talks (like that one for ruBSD) seem to indicate that OpenBSD's looking to prioritize virtualization in order to stay competitive with other operating systems.
It's also worth noting that OpenBSD has supported virtualization on some other platforms, too; for example, it can be used to manage SPARC logical domains (LDOMs).
Appreciate it. Makes sense on competitiveness. Didn't know about the LDOM support. Interesting.
Theo is objecting to the view that VMs are a security necessity.
Makes sense. I disagree with him in that they can definitely have benefit over regular OS for security. The reason is relative simplicity and ability to easily integrate security tech in versus a whole OS with legacy compatibility issues. This was proven in KVM/370, then VAX Security Kernel, then MILS, then Nizza Security Architecture, and then people started leveraging Xen for similar reasons albeit without the same assurance. The NSA's pentesters repeatedly failed to breach some of these despite years of effort while the OS's... see Snowden leaks and TAO catalog.
Of course, I agree that VM's aren't necessary: one of many approaches one can use and don't get the job done by themselves. They are beneficial, though, as running security-critical components on a 4-12Kloc in kernel mode with mediated, simple interface should have way less impact than running it on 1Mloc+ with POSIX-style interface.
Aren't they though? With 0-days it makes sense to be able to snapshot non mission-critical systems, bring them offline, until a patch is found.
Snapshotting alone is a security necessity when patching too. Very easy to rollback to a "good" state.
I haven't read it, but I assume the discussion was about sandboxing clients. From that perspective, any additional security would be defeated as soon as a client is able to affect the hypervisor or the host OS. So (according to Theo) if you can't write a secure host and/or client, the VM doesn't improve security.
While I agree with him on a lot of things, this isn't one of them; security is all about layers, and it's not like OpenBSD hasn't steered clear of other forms of sandboxing (like chroots and systrace). This doesn't mean that virtualization should be relied on exclusively or even near-exclusively as a defense (which is what I suspect Theo was more objecting to, along with the point of "well if you can't write a secure operating system, what makes you think you can write a secure hypervisor?"), but rather that it should be used as an additional layer on top of (or rather, underneath) a bunch of others.
It's like bulkheads on modern ships. Yeah, if you get a hole in your hull, you're gonna be in some (literally) deep water, but that bulkhead (so long as it's built right) could mean the difference between limping to the nearest harbor or sinking to the nearest seafloor.
Somehow back when Amazon had to reboot every EC2 server to fix a Xen bug, my "insecure" hypervisor-less server didn't require such action. I think I'll prefer to keep sailing without that particular bulkhead.
The reboots were because Amazon cuts a lot of corners in their Xen setup because instances are supposed to be "disposable". A proper VM cluster would use dedicated storage nodes that export over something like iscsi, which would require just transferring a memory snapshot, or would use the native, slower, disk snapshotting migration. But that's just one of many ways AWS is rather broken from an operations standpoint.
Thing is, the AWS philosophy of considering everything short of data sources "disposable" leads to a lot more robust engineering.
Yes, it makes Ops work more difficult, but as someone smart said regarding software: if something hurts, you're not going it often enough :)
I'm pretty sure "if something hurts, you're not doing it often enough" doesn't apply to, say, arm-breakage or self-immolation :)
What you call a proper VM cluster has been pretty terrible from my experience as both a developer on an early cloud and a consumer of large government facing clouds.
Typically the network or SAN becomes oversaturated and the vm's shit the bed. AWS on the other hand was considerably more reliable and I'd argue they've made better decisions rather than cut corners.
The SAN becoming over-saturated isn't something that just "happens". Between establishing limits ahead of time and monitoring that shouldn't happen without someone knowing well ahead of time.
Just to be clear, I'm defending Amazon over an accusation that they've cut corners from an install that was up in 2007 or earlier. Now we could have focused on xen guests shitting the bed for no apparent reason or flakey switch port but we decided to focus on storage.
OK so I'm not proclaiming to be an expert but as someone working in the area in 2007, buying something off the shelf like it aint no thing, you're getting something like a netapp with a limit of 512 iscsi inititatiors, or a Sun amber road where your only form of automation is an ssh consol with a big warning stating it's unsupported.
From memory, there was no such thing as setting quotas on the amount of IOPs an iSCSI inititatior can do, in fact, I'm fairly sure IOPs quotas just didn't exist period, as the vendors weren't really up-to-speed with this new selling vm's thing. So basically, we're suggesting that it's a good idea to just buy a SAN to run an indetermined amount of vm's that are going to do an indetermined amount of IOPs.
OK, cool, you're now indebted to storage vendors selling you new shelvs at £80,000 a pop for those extra IOPs you so deperately need. Now to be fair, Amazon could probably afford it, but your VMs would still be a lot more expensive and would probably have still been totally disposible when your switch port decides to blip traffic to the SAN or as previously stated, Xen shits the bed.
None of these things might be a problem today, I don't know, I'm more a consumer than a producer of clouds these days, but I'd suggests these criticisms are bullshit. They come from some obviously smart people, but bullshit none-the-less.
Well true; even the RMS Titanic, with its fifteen bulkheads, was no match against an iceberg. Let's just hope that this new hypervisor for OpenBSD is built with better-quality steel :)
Also, it's worth mentioning that EC2 instances are meant to be ephemeral; Amazon doesn't provide any semblance of a guarantee that your instances won't reboot, and assumes that you intend for all your "machines" to be arbitrarily rebootable. Not saying that any hypervisor implementation right now is particularly good; only that Amazon isn't exactly the best representation here.
How many bugs have OpenBSD team found vs found in Xen? That would be a relevant comparison. From there, an assessment of exploitability of each given OpenBSD's attention to mitigation.
What you said, on other hand, was meaningless given that OpenBSD has had bugs that could lead to a crash. Real question is, "Do Xen or security-focused virtualization schemes (a) reduce number of vulnerabilities with impact of kernel-mode 0-days, and/or (b) prevent, contain, or facilitate easy recovery from OS- and app-level 0-days?" Prior experience in security-focused efforts show yes to both questions. Xen isn't one of them as the existence of the Xenon project shows. However, it's small size and improvements over time make it substantially less risky than an arbitrary OS + software combination esp if above layer is also addressed (eg MirageOS). Even Galois Inc.'s conservative teams are using it in some work.
Well, the point is my security would not have been improved, in any way, by running on top of Xen and sharing my server with some rando.
I agree with that. It's why I still recommend BareMetal hosting and physical separation where possible. ;)
Why would you be sharing your server with some rando? You don't have to share your Xen deployments with other people if you don't want to, you know :)
You've never had to reboot your server?
and why can't you do that with containers or jails? or filesystem snapshots? or instant re-deploy to a known good state from ansible/puppet/salt/chef?
Just because there's a nice mouse-navigable GUI for entry level Windows admins doesn't mean it's a good solution.
if you need snapshots for doing updates it's because your software is fragile, undocumented, and you don't have a deployment procedure. Fix that, and upgrades will be easy and not scary.
Or you could have a consistent repeatable way of doing it across multiple apps and operating systems. I am not averse to your argument that doing it at the app level has benefits, but being able to do it at the VM level in a consistent way is going to be simpler when you expand beyond a handful of applications.
I agree that there is convenience to the execution controls VMs provide. But if you can't repeatably and easily stand up a replacement/duplicates of a system from configuration management and backups, then you don't actually have configuration management or backups.
Sure, to be clear I'm not arguing against those things, I'm just arguing that VM level snapshots and clones are also a tool in the operational arsenal and is in fact a very popular one that works well in practice.
It looks like I agree entirely with you, just not the guy that started this whole thread by saying VMs are a security necessity.
Snapshotting is sorta orthogonal to virtualization though, isn't it? Just snapshot at the storage level, no?
KVM, VMware, and Xen all allow you to do a snapshot of memory as well as of disk. It's one of the tools used to migrate xen instances between nodes, for example.
x86 hardware support for virtualization has changed quite a bit since 2007, though. https://en.wikipedia.org/wiki/X86_virtualization#Intel_virtu...
True. Looks to have gotten a lot better, too.
That E-Mail is from 2007...
This is from 2013: https://youtu.be/OXS8ljif9b8?t=395
"That E-Mail is from 2007..."
Yes, people have repeated that line since 2008. Readers certainly saw the timestamp. It was main Q&A and search result, though. Hence asking for modern post.
"This is from 2013"
And that's a start. Appreciate the link. I don't see many of his interviews but that was the first one I saw him admit to being behind. Good they're changing their attitude a bit.
The expected and somewhat disappointing part is when he has no answer to what can be done to raise the bar past a few exploit mitigations. There's something like four decades of work (and worked examples) showing how to increase assurance of security in hardware, software, and systems. Especially in capability, microkernels, static analysis, covert channel analysis, and so on. He could... idk... apply some of that instead of mock and ignore it like most mainstream does. FreeBSD is ahead here with SEBSD and Capsicum work.
One project did port OpenBSD to L4 kernel to isolate it in a protection domain. The idea, as in Nizza Security Architecture, is to be able to split system into legacy, untrusted stuff in VM and trusted, highly-assured components running directly on microkernel. A proven model that would benefit OpenBSD by dramatically reducing attack surface. This is done in embedded space (eg INTEGRITY, PikeOS Hypervisor) for up to 8 ISA's each for those wondering about portability.
Just one of dozens of techs to draw on to increase assurance. Will be interesting to see if they draw on any of this or get left behind [again] by those that do.
The OpenBSD covert channel analysis team is understaffed at the moment.
Both EROS and Genode managed to do a lot of what I mentioned with less staff & time. Plus, with solid architecture that above components can leverage without doing all the highly assured stuff over and over again: what I call security ROI. Your huge TCB, drivers probably in kernel, and language-related issues are where I predicted most of the trouble would come from. (Did it?) Something like Nizza Security Architecture would benefit you along with interface checks you're already good at and a better language that translates to C with checks automatically inserted.
Far as covert channels, Google Kemmerer's Shared Resource Matrix for a method that worked for amateurs and pro's alike on a budget. Any person handling a subsystem can apply it. Even using it on function call level can tell you a lot. If understaffed, just use it on any component handling secrets for storage and timing channels. Best bang for buck.
So go run EROS. Why aren't you already?
Nah, I'm just not running OpenBSD. Gotta stay where the innovation is at, both in productivity and security. ;)
Ok, to be more serious for a bit, a goal of the OpenBSD project is to produce a Unix like operating system. To the extent "innovation" is "don't be Unix" it's somewhat counter to project goals.
Well, there we go. That's unfortunate, but understandable. However, it still allows you to build on decades of work in security engineering (incl old secure UNIX's). The easiest route at this point is putting OpenBSD API on top of a microkernel, pulling security-critical functionality out of main system onto microkernel, and bulletproofing your middleware for these. Additionally, writing the code in a way that lets tools such as Astree Analyzer work on as much of it as possible will knock out many bugs. Compiler tools that automatically transform kernel or user-mode code to make it safer might help. Softbound + CETS comes to mind.
Much to draw on or improve while remaining a UNIX. The microkernel + user-mode virtualization approach has already been done in academia and commercial products. So, it could be done here. Will they? Another matter entirely. I doubt it.
Truth be told, though, I voted for the Xen Dom0 to use OpenBSD because 0-days would be its main concern. And we know which team is the best at removing them from a UNIX codebase. ;)
but its so fun to blast out buzzwords
It's fun to do what the sheep or crowds are doing. That's Windows, Mac, BSD, or UNIX at any given time. Throw in C/C++ everywhere, browsers, Flash, HTTP as universal transport, language runtimes too bloated to understand, the Cloud... the more vulnerabilities and maintenance horrors, the more they'll like you and the more fun you'll have with them.
Our security needs to improve dramatically. Some ways are proven to work, some are proven not to. Pushing the second category is extremely fun, you might get famous in Silicon Valley companies, maybe invited to DEFCON, and everyone will make excuses for its problems later. Lots of buzzwords there, I agree: can't even mentally track all of them. Then there's a tinier group pushing methods that work because they're necessary, even if not all fun. Staying with that group on principle if not profit.