Am I the only one who feels like cgroups are extraordinarily complex for the problem they're trying to solve? It seems like a simpler structure could have achieved most of the same goals and not required one or two layers (in the case of docker cgroup->lxc->docker) of abstraction to find widespread use.
In particular, was the ability to migrate a process or have a process in two cgroups really essential to containerization? It seems like without those it'd be a simple matter of nice/setuidgid-style privilege de-escalation commands to get the same kinds of behaviour without adding a whole 'nother resource management to the mix (the named groups).
The cgroups document you link to as [4] has such a weirdly contrived use case example it makes me think they were trying really hard to come up with a way to justify the complexity they baked into the idea.
(Original cgroups developer here, although I've since moved on from Google and don't have time to play an active role anymore.)
It's true that cgroups are a complex system, but they were developed to solve a complex group of problems (packing large numbers of dynamic jobs on servers, with some resources isolated, and some shared between different jobs). I think that pretty much all the features of cgroups come either from real requirements, or from constraints due to the evolution of cgroups from cpusets.
Back when cgroups was being developed, cpusets had fairly recently been accepted into the kernel, and it had a basic process grouping API that was pretty much what cgroups needed. It was much easier politically to get people to accept an evolution of cpusets into cgroups (in a backward-compatible way) than to introduce an entirely new API. With hindsight, this was a mistake, and we should have pushed for a new (binary, non-VFS) API, as having to fit everything into the metaphor of a filesystem (and deal with all the VFS logic) definitely got in the way at times.
If you want to be able to manage/tweak/control the resources allocated to a group after you've created the group, then you need some way of naming that group, whether it be via a filesystem directory or some kind of numerical identifier (like a pid). So I don't think a realistic resource management system can avoid that.
The most common pattern of the need for a process being in multiple cgroups is that of a data-loader/data-server job pair. The data-loader is responsible for periodically loading/maintaining some data set from across the network into (shared) memory, and the data-server is responsible for low-latency serving of queries based on that data. So they both need to be in the same cgroup for memory purposes, since they're sharing the memory occupied by the loaded data. But the CPU requirements of the two are very different - the data-loader is very much a background/batch task, and shouldn't be able to steal CPU from either the data-server or from any other latency-sensitive job on the same machine. So for CPU purposes, they need to be in separate cgroups. That (and other more complex scenarios) is what drives the requirement for multiple independent hierarchies of cgroups.
Since the data-loader and data-server can be stopped/updated/started independently, you need to be able to launch a new process into an existing cgroup. It's true that the need to be able to move a process into a different cgroup would be much reduced if there was an extension to clone() to allow you to create a child directly in a different set of cgroups, but cpusets already provided the movement feature, and extending clone in an intrusive way like that would have raised a lot of resistance, I think.
The good news is that namespaces (the most interesting part of containers) are simpler than cgroups, and the api is stable.
cgroups are indeed a mess. The api is highly unstable and there is an effort underway to sanitize it, with the help of a "facade" userland api. In other words kernel devs are basically saying: "use this userland api while we fix our shit". (I don't claim to understand the intricacies of this problem. All I know is that, as a developer of docker, it is better for my sanity to maintain an indirection between my tool and the kernel - until things settle down, at least).
cgroups are extremely powerful, but they are fairly complex, it took me some hands on experience to wrap my mind around it. Redhat has done a great job on the intigation side. You can watch a demo @ http://www.youtube.com/watch?v=KX5QV4LId_c
Comments
Am I the only one who feels like cgroups are extraordinarily complex for the problem they're trying to solve? It seems like a simpler structure could have achieved most of the same goals and not required one or two layers (in the case of docker cgroup->lxc->docker) of abstraction to find widespread use.
In particular, was the ability to migrate a process or have a process in two cgroups really essential to containerization? It seems like without those it'd be a simple matter of nice/setuidgid-style privilege de-escalation commands to get the same kinds of behaviour without adding a whole 'nother resource management to the mix (the named groups).
The cgroups document you link to as [4] has such a weirdly contrived use case example it makes me think they were trying really hard to come up with a way to justify the complexity they baked into the idea.
(Original cgroups developer here, although I've since moved on from Google and don't have time to play an active role anymore.)
It's true that cgroups are a complex system, but they were developed to solve a complex group of problems (packing large numbers of dynamic jobs on servers, with some resources isolated, and some shared between different jobs). I think that pretty much all the features of cgroups come either from real requirements, or from constraints due to the evolution of cgroups from cpusets.
Back when cgroups was being developed, cpusets had fairly recently been accepted into the kernel, and it had a basic process grouping API that was pretty much what cgroups needed. It was much easier politically to get people to accept an evolution of cpusets into cgroups (in a backward-compatible way) than to introduce an entirely new API. With hindsight, this was a mistake, and we should have pushed for a new (binary, non-VFS) API, as having to fit everything into the metaphor of a filesystem (and deal with all the VFS logic) definitely got in the way at times.
If you want to be able to manage/tweak/control the resources allocated to a group after you've created the group, then you need some way of naming that group, whether it be via a filesystem directory or some kind of numerical identifier (like a pid). So I don't think a realistic resource management system can avoid that.
The most common pattern of the need for a process being in multiple cgroups is that of a data-loader/data-server job pair. The data-loader is responsible for periodically loading/maintaining some data set from across the network into (shared) memory, and the data-server is responsible for low-latency serving of queries based on that data. So they both need to be in the same cgroup for memory purposes, since they're sharing the memory occupied by the loaded data. But the CPU requirements of the two are very different - the data-loader is very much a background/batch task, and shouldn't be able to steal CPU from either the data-server or from any other latency-sensitive job on the same machine. So for CPU purposes, they need to be in separate cgroups. That (and other more complex scenarios) is what drives the requirement for multiple independent hierarchies of cgroups.
Since the data-loader and data-server can be stopped/updated/started independently, you need to be able to launch a new process into an existing cgroup. It's true that the need to be able to move a process into a different cgroup would be much reduced if there was an extension to clone() to allow you to create a child directly in a different set of cgroups, but cpusets already provided the movement feature, and extending clone in an intrusive way like that would have raised a lot of resistance, I think.
Cool, thanks for the details.
The good news is that namespaces (the most interesting part of containers) are simpler than cgroups, and the api is stable.
cgroups are indeed a mess. The api is highly unstable and there is an effort underway to sanitize it, with the help of a "facade" userland api. In other words kernel devs are basically saying: "use this userland api while we fix our shit". (I don't claim to understand the intricacies of this problem. All I know is that, as a developer of docker, it is better for my sanity to maintain an indirection between my tool and the kernel - until things settle down, at least).
cgroups are extremely powerful, but they are fairly complex, it took me some hands on experience to wrap my mind around it. Redhat has done a great job on the intigation side. You can watch a demo @ http://www.youtube.com/watch?v=KX5QV4LId_c