There is a great wired article [1], which outline how Google uses orchestration software to manage linux containers in its day-to-day operations. Mid way through the article there is this great diagram, which shows Omega (Google's orchestration software), and how it deploys containers for images, search, gmail, etc onto the same physical hardware. There is an amazing talk by John Wilkes (Google Cluster Management, Mountain View) about Omega at Google Faculty Summit 2011 [2], I would highly recommend watching it!
By the way, one of the key concepts of containers is, control groups (cgroups) [3, 4], and this was initially added to the kernel back in 2007 by two Google engineers, so they have definitely given back in this area. I know all this because I have spent the last two weeks researching control groups for an upcoming screencast.
I am happy Google released this, and cannot wait to dig though it!
I just spend the last hour or so digging through the code and playing with CLI. It's pretty neat. The "specification" for creating new containers (how to describe limits, namespacing etc) is not very well documented so it takes some trial and error... But this feels like a nice, clean low-level component.
It can be used as a C++ library, too - I'm going to evaluate it as a possible low-level execution backend for docker :)
In response to the other comments mentioning docker - this definitely does not compete with docker. A better comparison would be with the lxc tools or maybe something like libvirt-lxc or systemd's nspawn.
* Building is relatively straightforward on an Ubuntu system. You'll need to install re2 from source, but that's about it.
* No configuration necessary to start playing. lmctfy just straight up mounts cgroups and starts playing in there.
* Containers can be nested which is nice.
* I really couldn't figure out useful values for the container spec. Even the source doesn't seem to have a single reference - it's all dynamically registered by various subsystems in a rather opaque way. I opened a few issues to ask for more details.
* This is a really low-level tool. Other than manipulating cgroups it doesn't seem to do much, which is perfect for my particular use case (docker integration). I couldn't figure out how to set namespaces (including mnt and net namespaces which means all my containers shared the host's filesystem and network interfaces). I don't know if that functionality is already in the code, or has yet to be added.
* Given the fairly small footprint, limited feature set, and clean build experience, this really looks like an interesting option to use as a backend for docker. I like that it carries very little dependencies. Let's see what the verdict is on these missing features.
Most apps running on Google servers are aware that they're running in a shared environment, so they don't need the overhead of virtualized network interfaces. So I doubt that there will be any specific support for network namespaces.
And you can approximate mount namespaces with chroots and bind mounts. (In some ways that's better, since it's a bit easier for a process outside the container to interact with the container's filesystem).
Damn. This means it's much less useful to me (and 99% of applications outside of google). I guess I could combine lmctfy with a namespacing library of my own. But that's more extra work than I was anticipating.
I'm sure they do - but again google is probably holding a proprietary alternative to virtually every piece of software in the world :) Most of it is too tied to "the google way" to be useful to anybody else.
This is more true than the idea that there's a competitive advantage. We would love to open-source more of our stack (and are working towards it), but it's all very tied to the rest of our cluster environment. Piece by piece.
As an ex-googler I agree. Much if it just wouldn't be useful as is to most people as it's an environment that few match in load, resources, and diversity of services. Plus it's all integrated tightly across the board.
Am I the only one who feels like cgroups are extraordinarily complex for the problem they're trying to solve? It seems like a simpler structure could have achieved most of the same goals and not required one or two layers (in the case of docker cgroup->lxc->docker) of abstraction to find widespread use.
In particular, was the ability to migrate a process or have a process in two cgroups really essential to containerization? It seems like without those it'd be a simple matter of nice/setuidgid-style privilege de-escalation commands to get the same kinds of behaviour without adding a whole 'nother resource management to the mix (the named groups).
The cgroups document you link to as [4] has such a weirdly contrived use case example it makes me think they were trying really hard to come up with a way to justify the complexity they baked into the idea.
(Original cgroups developer here, although I've since moved on from Google and don't have time to play an active role anymore.)
It's true that cgroups are a complex system, but they were developed to solve a complex group of problems (packing large numbers of dynamic jobs on servers, with some resources isolated, and some shared between different jobs). I think that pretty much all the features of cgroups come either from real requirements, or from constraints due to the evolution of cgroups from cpusets.
Back when cgroups was being developed, cpusets had fairly recently been accepted into the kernel, and it had a basic process grouping API that was pretty much what cgroups needed. It was much easier politically to get people to accept an evolution of cpusets into cgroups (in a backward-compatible way) than to introduce an entirely new API. With hindsight, this was a mistake, and we should have pushed for a new (binary, non-VFS) API, as having to fit everything into the metaphor of a filesystem (and deal with all the VFS logic) definitely got in the way at times.
If you want to be able to manage/tweak/control the resources allocated to a group after you've created the group, then you need some way of naming that group, whether it be via a filesystem directory or some kind of numerical identifier (like a pid). So I don't think a realistic resource management system can avoid that.
The most common pattern of the need for a process being in multiple cgroups is that of a data-loader/data-server job pair. The data-loader is responsible for periodically loading/maintaining some data set from across the network into (shared) memory, and the data-server is responsible for low-latency serving of queries based on that data. So they both need to be in the same cgroup for memory purposes, since they're sharing the memory occupied by the loaded data. But the CPU requirements of the two are very different - the data-loader is very much a background/batch task, and shouldn't be able to steal CPU from either the data-server or from any other latency-sensitive job on the same machine. So for CPU purposes, they need to be in separate cgroups. That (and other more complex scenarios) is what drives the requirement for multiple independent hierarchies of cgroups.
Since the data-loader and data-server can be stopped/updated/started independently, you need to be able to launch a new process into an existing cgroup. It's true that the need to be able to move a process into a different cgroup would be much reduced if there was an extension to clone() to allow you to create a child directly in a different set of cgroups, but cpusets already provided the movement feature, and extending clone in an intrusive way like that would have raised a lot of resistance, I think.
The good news is that namespaces (the most interesting part of containers) are simpler than cgroups, and the api is stable.
cgroups are indeed a mess. The api is highly unstable and there is an effort underway to sanitize it, with the help of a "facade" userland api. In other words kernel devs are basically saying: "use this userland api while we fix our shit". (I don't claim to understand the intricacies of this problem. All I know is that, as a developer of docker, it is better for my sanity to maintain an indirection between my tool and the kernel - until things settle down, at least).
cgroups are extremely powerful, but they are fairly complex, it took me some hands on experience to wrap my mind around it. Redhat has done a great job on the intigation side. You can watch a demo @ http://www.youtube.com/watch?v=KX5QV4LId_c
yes, cloud foundry has been using warden for PaaS isolation between hosted apps for awhile. it was originally authored by redis and cloud foundry contributor pieter noorduis currently working for vmware [1]. the ongoing work has been continued by the cloud foundry team at pivotal.
warden has a c server core [2] wrapping cgroups and other features currently on the lmctfy roadmap like network and file system isolation [3]. the current file system isolation uses either aufs or overlayfs depending on the distro/linux version you are using [4]. the network uses namespaces and additional features.
warden also has early/experimental support for centos in addition to ubuntu, although some of the capabilities are degraded. for example, disk isolation uses a less efficient, but still workable copy file system approach.
the client orchestration of warden is currently written in ruby, but there was also a branch started to move that to go [5] that has not been hardened and moved into master.
recently cloudfoundry started using bosh-lite [2] leveraging warden to do full dev environments using linux containers instead of separate linux hosts on many virtual machines from an IaaS provider, which has dramatically reduced the resources and time required to create, develop and use the full system.
Comments
There is a great wired article [1], which outline how Google uses orchestration software to manage linux containers in its day-to-day operations. Mid way through the article there is this great diagram, which shows Omega (Google's orchestration software), and how it deploys containers for images, search, gmail, etc onto the same physical hardware. There is an amazing talk by John Wilkes (Google Cluster Management, Mountain View) about Omega at Google Faculty Summit 2011 [2], I would highly recommend watching it!
By the way, one of the key concepts of containers is, control groups (cgroups) [3, 4], and this was initially added to the kernel back in 2007 by two Google engineers, so they have definitely given back in this area. I know all this because I have spent the last two weeks researching control groups for an upcoming screencast.
I am happy Google released this, and cannot wait to dig though it!
[1] http://www.wired.com/wiredenterprise/2013/03/google-borg-twi...
[2] http://www.youtube.com/watch?v=0ZFMlO98Jkc
[3] http://en.wikipedia.org/wiki/Cgroups
[4] https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt
I just spend the last hour or so digging through the code and playing with CLI. It's pretty neat. The "specification" for creating new containers (how to describe limits, namespacing etc) is not very well documented so it takes some trial and error... But this feels like a nice, clean low-level component.
It can be used as a C++ library, too - I'm going to evaluate it as a possible low-level execution backend for docker :)
In response to the other comments mentioning docker - this definitely does not compete with docker. A better comparison would be with the lxc tools or maybe something like libvirt-lxc or systemd's nspawn.
Quick summary of my early explorations:
* Building is relatively straightforward on an Ubuntu system. You'll need to install re2 from source, but that's about it.
* No configuration necessary to start playing. lmctfy just straight up mounts cgroups and starts playing in there.
* Containers can be nested which is nice.
* I really couldn't figure out useful values for the container spec. Even the source doesn't seem to have a single reference - it's all dynamically registered by various subsystems in a rather opaque way. I opened a few issues to ask for more details.
* This is a really low-level tool. Other than manipulating cgroups it doesn't seem to do much, which is perfect for my particular use case (docker integration). I couldn't figure out how to set namespaces (including mnt and net namespaces which means all my containers shared the host's filesystem and network interfaces). I don't know if that functionality is already in the code, or has yet to be added.
* Given the fairly small footprint, limited feature set, and clean build experience, this really looks like an interesting option to use as a backend for docker. I like that it carries very little dependencies. Let's see what the verdict is on these missing features.
Are you referring to the container spec in the proto file? https://github.com/google/lmctfy/blob/master/include/lmctfy.... Which attributes are you having trouble setting a useful value for?
According to the docs all that can be set is cpu and memory limits. So maybe that's the extent of it for now, even though the proto identifies more.
Thank you! I was scanning .h files for a type declaration.. Silly old-school me :)
I only knew because I've worked with the internal analogue of this library recently. Glad I could help.
Most apps running on Google servers are aware that they're running in a shared environment, so they don't need the overhead of virtualized network interfaces. So I doubt that there will be any specific support for network namespaces.
And you can approximate mount namespaces with chroots and bind mounts. (In some ways that's better, since it's a bit easier for a process outside the container to interact with the container's filesystem).
Damn. This means it's much less useful to me (and 99% of applications outside of google). I guess I could combine lmctfy with a namespacing library of my own. But that's more extra work than I was anticipating.
Namespaces will be coming, but we're not there yet. This captures some of what we are already doing internally, but not all of it, yet.
Perhaps they'd be open to a collaboration where you add that functionality and then you use it to make docker (even more!) awesome.
The namespacing part is much simpler, if you have specific use cases.
We (lmctfy team) are in the middle of designing and adding namespace support. It will start trickling in soon.
google is probably holding the piece that competes with docker or openvz, they should have somthing like docker internally
I'm sure they do - but again google is probably holding a proprietary alternative to virtually every piece of software in the world :) Most of it is too tied to "the google way" to be useful to anybody else.
This is more true than the idea that there's a competitive advantage. We would love to open-source more of our stack (and are working towards it), but it's all very tied to the rest of our cluster environment. Piece by piece.
As an ex-googler I agree. Much if it just wouldn't be useful as is to most people as it's an environment that few match in load, resources, and diversity of services. Plus it's all integrated tightly across the board.
Am I the only one who feels like cgroups are extraordinarily complex for the problem they're trying to solve? It seems like a simpler structure could have achieved most of the same goals and not required one or two layers (in the case of docker cgroup->lxc->docker) of abstraction to find widespread use.
In particular, was the ability to migrate a process or have a process in two cgroups really essential to containerization? It seems like without those it'd be a simple matter of nice/setuidgid-style privilege de-escalation commands to get the same kinds of behaviour without adding a whole 'nother resource management to the mix (the named groups).
The cgroups document you link to as [4] has such a weirdly contrived use case example it makes me think they were trying really hard to come up with a way to justify the complexity they baked into the idea.
(Original cgroups developer here, although I've since moved on from Google and don't have time to play an active role anymore.)
It's true that cgroups are a complex system, but they were developed to solve a complex group of problems (packing large numbers of dynamic jobs on servers, with some resources isolated, and some shared between different jobs). I think that pretty much all the features of cgroups come either from real requirements, or from constraints due to the evolution of cgroups from cpusets.
Back when cgroups was being developed, cpusets had fairly recently been accepted into the kernel, and it had a basic process grouping API that was pretty much what cgroups needed. It was much easier politically to get people to accept an evolution of cpusets into cgroups (in a backward-compatible way) than to introduce an entirely new API. With hindsight, this was a mistake, and we should have pushed for a new (binary, non-VFS) API, as having to fit everything into the metaphor of a filesystem (and deal with all the VFS logic) definitely got in the way at times.
If you want to be able to manage/tweak/control the resources allocated to a group after you've created the group, then you need some way of naming that group, whether it be via a filesystem directory or some kind of numerical identifier (like a pid). So I don't think a realistic resource management system can avoid that.
The most common pattern of the need for a process being in multiple cgroups is that of a data-loader/data-server job pair. The data-loader is responsible for periodically loading/maintaining some data set from across the network into (shared) memory, and the data-server is responsible for low-latency serving of queries based on that data. So they both need to be in the same cgroup for memory purposes, since they're sharing the memory occupied by the loaded data. But the CPU requirements of the two are very different - the data-loader is very much a background/batch task, and shouldn't be able to steal CPU from either the data-server or from any other latency-sensitive job on the same machine. So for CPU purposes, they need to be in separate cgroups. That (and other more complex scenarios) is what drives the requirement for multiple independent hierarchies of cgroups.
Since the data-loader and data-server can be stopped/updated/started independently, you need to be able to launch a new process into an existing cgroup. It's true that the need to be able to move a process into a different cgroup would be much reduced if there was an extension to clone() to allow you to create a child directly in a different set of cgroups, but cpusets already provided the movement feature, and extending clone in an intrusive way like that would have raised a lot of resistance, I think.
Cool, thanks for the details.
The good news is that namespaces (the most interesting part of containers) are simpler than cgroups, and the api is stable.
cgroups are indeed a mess. The api is highly unstable and there is an effort underway to sanitize it, with the help of a "facade" userland api. In other words kernel devs are basically saying: "use this userland api while we fix our shit". (I don't claim to understand the intricacies of this problem. All I know is that, as a developer of docker, it is better for my sanity to maintain an indirection between my tool and the kernel - until things settle down, at least).
cgroups are extremely powerful, but they are fairly complex, it took me some hands on experience to wrap my mind around it. Redhat has done a great job on the intigation side. You can watch a demo @ http://www.youtube.com/watch?v=KX5QV4LId_c
There is also warden, but it doesn't get much press.
https://github.com/cloudfoundry/warden/tree/master/warden
yes, cloud foundry has been using warden for PaaS isolation between hosted apps for awhile. it was originally authored by redis and cloud foundry contributor pieter noorduis currently working for vmware [1]. the ongoing work has been continued by the cloud foundry team at pivotal.
warden has a c server core [2] wrapping cgroups and other features currently on the lmctfy roadmap like network and file system isolation [3]. the current file system isolation uses either aufs or overlayfs depending on the distro/linux version you are using [4]. the network uses namespaces and additional features.
warden also has early/experimental support for centos in addition to ubuntu, although some of the capabilities are degraded. for example, disk isolation uses a less efficient, but still workable copy file system approach.
the client orchestration of warden is currently written in ruby, but there was also a branch started to move that to go [5] that has not been hardened and moved into master.
recently cloudfoundry started using bosh-lite [2] leveraging warden to do full dev environments using linux containers instead of separate linux hosts on many virtual machines from an IaaS provider, which has dramatically reduced the resources and time required to create, develop and use the full system.
[1] https://twitter.com/pnoordhuis [2] https://github.com/cloudfoundry/warden/tree/master/warden/sr... [3] https://github.com/cloudfoundry/warden/blob/master/warden/RE... [4] https://github.com/cloudfoundry/warden/blob/master/warden/RE... [5] https://github.com/cloudfoundry/warden/tree/go
Just a follow up to my comment. I have released the "Introduction to Linux Control Groups (cgroups)" screencast which I talked about in my previous comment. View it @ http://sysadmincasts.com/episodes/14-introduction-to-linux-c...
You may want to check out Mesos: http://mesos.apache.org/
My understanding (from reading various articles) is that Mesos is very analogous to Google's original scheduler, Borg.