Could you perhaps explain what the overall motivations were for not using LXC userspace tools and instead creating an alternative?
Watching the Google Omega talk video linked in the comment above, I am guessing your implementation probably mostly exists to instantiate Google Omega cluster cell-manager specified jobs including intra-google properties of resource shape, constraints and preferences in a local container. I am guessing that part is not released because the current code has too much to do with your internal standards for expressing the above job-related metadata.
We started building a large-scale datacenter automation system in 2003, and by late 2005 it was deployed on most machines but it became apparent that achieving the high density of job packing we wanted was going to be impossible by relying on post-hoc user-space enforcement of resource allocation (killing jobs that used too much memory, nicing jobs that used too much CPU, etc). Sensitive services like websearch were insisting on their own dedicated machines or even entire dedicated clusters, due to the performance penalties of sharing a machine with a careless memory/CPU hog. We clearly needed some kind of kernel support, but back then it didn't really exist - there were several competing proposals for a resource control system like cgroups but none of them made much progress.
One that did get in was cpusets, and on the suggestion of akpm (who had recently joined Google) we started experimenting with using cpusets for very crude CPU and memory control. Assigning dedicated CPUs to a job was pretty easy via cpusets. Memory was trickier - by using a feature originally intended for testing NUMA on non-NUMA systems, we broke memory up into many "fake" NUMA nodes, and dynamically assigned them to jobs on the machine based on their memory demands and importance. This started making it into production in late 2006 (I think), around the same time that we were working on evolving cpusets into cgroups to support new resource controls.
Interesting history, makes sense. (I thought your name was familiar: you are the author of the cgroups.txt kernel documentation! Do you still get to work on this stuff much? What is your take on the apparent popularization of container-based virt? What are the kernel features you would like to see in the area that do not yet exist?)
Was there a reason you guys didn't open source this many years ago?
I left the cluster management group over three years ago, so I've not had much chance to work on / think about containers since then.
This code grew symbiotically with Google's kernel patches (big chunks of which were open-sourced into cgroups) and the user-space stack (which was tightly coupled with Google's cluster requirements). So open-sourcing it wouldn't necessarily have been useful for anyone. It looks like someone's done a lot of work to make this more generically-applicable before releasing it.
What menage said is true. As it came time to clean up what we were using internally it became a decision of adapting the code and semantics we already have to be cleaner, or starting anew with something like LXC (which does more than we need in some ways and less in others).
We have a lot of really well tested and battle-hardened code and behavior, so we chose to keep that.
lmctfy is designed from the ground up as automation which can be used by humans, and never the other way around.
Great to see some of this code finally making it out into the world, although it seems to have changed a bit over the last few years. (I still recognize a few low-level functions.)
Comments
FWIW, we have a lot of these things done internally, but not in a releasable form yet.
Could you perhaps explain what the overall motivations were for not using LXC userspace tools and instead creating an alternative?
Watching the Google Omega talk video linked in the comment above, I am guessing your implementation probably mostly exists to instantiate Google Omega cluster cell-manager specified jobs including intra-google properties of resource shape, constraints and preferences in a local container. I am guessing that part is not released because the current code has too much to do with your internal standards for expressing the above job-related metadata.
LXC didn't really exist in a usable form when Google started work on kernel-based resource isolation.
Hrrm OK... when was that? 2009 or earlier I suppose. I was using it in 2010 with few hiccups.
We started building a large-scale datacenter automation system in 2003, and by late 2005 it was deployed on most machines but it became apparent that achieving the high density of job packing we wanted was going to be impossible by relying on post-hoc user-space enforcement of resource allocation (killing jobs that used too much memory, nicing jobs that used too much CPU, etc). Sensitive services like websearch were insisting on their own dedicated machines or even entire dedicated clusters, due to the performance penalties of sharing a machine with a careless memory/CPU hog. We clearly needed some kind of kernel support, but back then it didn't really exist - there were several competing proposals for a resource control system like cgroups but none of them made much progress.
One that did get in was cpusets, and on the suggestion of akpm (who had recently joined Google) we started experimenting with using cpusets for very crude CPU and memory control. Assigning dedicated CPUs to a job was pretty easy via cpusets. Memory was trickier - by using a feature originally intended for testing NUMA on non-NUMA systems, we broke memory up into many "fake" NUMA nodes, and dynamically assigned them to jobs on the machine based on their memory demands and importance. This started making it into production in late 2006 (I think), around the same time that we were working on evolving cpusets into cgroups to support new resource controls.
Interesting history, makes sense. (I thought your name was familiar: you are the author of the cgroups.txt kernel documentation! Do you still get to work on this stuff much? What is your take on the apparent popularization of container-based virt? What are the kernel features you would like to see in the area that do not yet exist?)
Was there a reason you guys didn't open source this many years ago?
I left the cluster management group over three years ago, so I've not had much chance to work on / think about containers since then.
This code grew symbiotically with Google's kernel patches (big chunks of which were open-sourced into cgroups) and the user-space stack (which was tightly coupled with Google's cluster requirements). So open-sourcing it wouldn't necessarily have been useful for anyone. It looks like someone's done a lot of work to make this more generically-applicable before releasing it.
What menage said is true. As it came time to clean up what we were using internally it became a decision of adapting the code and semantics we already have to be cleaner, or starting anew with something like LXC (which does more than we need in some ways and less in others).
We have a lot of really well tested and battle-hardened code and behavior, so we chose to keep that.
lmctfy is designed from the ground up as automation which can be used by humans, and never the other way around.
p.s. Hello menage! Nice to see you commenting here :)
Great to see some of this code finally making it out into the world, although it seems to have changed a bit over the last few years. (I still recognize a few low-level functions.)
Who came up with the name?
The name was a collaborative effort. We suck at names :)
And yeah, it's been retooled a lot to extract it more cleanly from the surrounding code.