Id strongly recommend adding the ability to implement pipelines as code -- ie: you check in the CI/CD configuration into the repo (or another repo, containing CI/CD configurations). Having to configure the pipelines manually is nasty, and often builds are caused by misconfigured of the pipeline. having this checked it allows easy versioning.
I also like the idea of pipelines as code. I'd love to see it go a step further and become pipelines as testable code. I've yet to see a pipeline of non-trivial complexity that stays easy to reason about. I predict the next wave of CI/CD (or pipeline anything) will make pipeline testing & sanity checking simple as heck. If this already exists, please tell me about it, because I don't see people using it ;)
How does this compare to something like Buildbot (https://buildbot.net/) - I share a lot of your frustrations with Jenkins and this is where we landed.
I don't know, I'm really not trying to make one of those cliched checkbox X vs Y type grids, but when I started building Vespene I was most concerned about being able to make builds more consistent when you had hundreds of microservices developed by a lot of different teams. This is why I did http://docs.vespene.io/variables.html#snippets
I also saw a lot of people trying to do ops style stuff from tools like Jenkins, which is why Ansible has built-in SSH automation so it can hold on to SSH keys and use them in your behalf, and has some really cool built in RBAC so you can decide who can run what and it can be different than who can edit what.
Mostly I just want to build an architecture that can go interesting places - what is here today I think is usuable, but it's just a starting point. My thinking is if you make an architecture that is really pluggable, and the code is easy to read and add to, it can go to some really neat places.
I'd just recommend taking a look at the various chapters of http://docs.vespene.io/ to get a feel for features, and if you like it, spin up a copy using the setup instructions and see what you think.
Snippets of stuff is really missing from a lot of platforms. I use gitlab to store parts of our builds in one location instead of duplicating build code in every repo (and we have lots of them) for stuff like building RPM, where they all use a common model. Making this first-class is a good move.
Hello, Community Advocate from GitLab here. Thanks for sharing your solution with other users, we are happy to hear how you use GitLab to make your work more convenient.
Configuration management is best with operating system packages, so in creating "Ansible", not only did you completely miss the ball, you created a monster. People now hack ad hoc YAML files instead of designing clean OS packages to manipulate the configuration.
It's fine if you don't like it - there are things I don't like about it, but ultimately single packages can't express multi-node configurations very easily. We need something.
While I "grew up" as it were, believing in RPM, we live in a very post-distro, multi-language kind of world, and there's a need for things to glue that together. How many times have I tried to campaign against "wget tarball" as a deployment mechanism, I don't know. It's rough and yes, there's a lack of discipline in ops that needs to improve.
Immutable systems is a VERY interesting way to solve that, but it doesn't work for certain stateful things and you always need something to deploy the undercloud.
I'd encourage you to try to build your own experimental project to try to find different ways to do it, as this is the only real way that technology ever gets ahead.
Funny you should mention RPM's, since that's one of the packaging formats we perform configuration management in.
I have my own configuration management framework which can add and remove configuration from files that each package brings on. It's written in AWK. No knowledge is required to use it - configuration packages deliver just their configuration files' excerpts. In addition to being able to template them with regular shell variables, self-assembly is supported as well.
How is that different than what Puppet, Chef, CFEngine, or any number of other configuration management tools do [and did long before Ansible]? How do you propose to have a package that requires dynamic configuration supplied by the system (something from dhcp, for example). Are you conflating package management and configuration management? I'm genuinely curious.
It's not any different, they all suck. Read my reply above to the author on how it was solved cleanly without needing to hack anything, just deliver a config excerpt via an OS package and it works. It's not open sourced yet, mainly due to lack of spare time to spend on computers and my cronic exhaustion.
The first proof of concept prototype was developed in 2007 based on insights gained with packaging and system engineering at two different companies, one a software one and the other a very large financial institution.
Refined framework (oh how I hate that word now thanks to web developers!) went into production in 2009 on Solaris 10, depending on advanced AT&T SVR4 packaging features. It was then subsequently ported to CentOS / RHEL RPM and went into production around 2011. It's been the corner stone of configuration management for the entire infrastructure, particularly production ever since.
Ugh. Conveying tone on the internet... I mean it sincerely: I do wish you good luck. Please about your project on HN when you release it, I'd love to see it. While I think it's deceptively hard, but config/immutable state are fascinating areas.
I think we owe Jenkins a TON of credit for being a free solution that has a lot of great traction.
I'm used to seeing pages of checkboxes and can't stand how obscure some of the configuration is, but mostly I wanted to write something that could handle configuration differences between hundreds of projects, so that is why there are things like Snippets in Vespene.
Plus, I wanted to make something that was a little better for ops use cases (so people don't also have to go pick up something like Rundeck), so that's why there are things like the SSH integration and http://docs.vespene.io/launch_questions.html
It's more about capabilities and future capabilities than the frustrations IMHO.
Though I do have a bunch of friends who are frustrated with plugin compatibility, plugin hunting (we're doing more of a "batteries included" approach like I ran with ansible - just with much less modules), and stuff like that.
I was also able to add in some stuff like container build isolation really easily, and that's all included stock.
So back around 2003 I was working at Adaptec (an IBM spinoff) and we were trying to put together a build system, I think I ended up front-ending it around DamageControl, which was a Ruby port of CruiseControl, except I had to hand-code a fake "SCM" module to run timed (versus commit-based builds). Our builds took like 8 hours! At that time, I really wanted to help people with build environments but never really did it.
Then, what sort of happened is I kept running into overly complex Jenkins installs. The typical miles of config checkboxes was overwhelming for me, but a larger problem was that organizations would have like 200 microservices projects, and all the build scripts would be slightly different.
The idea was then to take the templating system from Cobbler - which allows lots of variables merged in at various levels, and Jinja2, and allow reuse of build scripts through variables and snippets.
There was some other frustration with Jenkins - the idea that it was a ginormous codebase and still not database based, and you had to hunt for good plugins.
Ansible was pretty successful for getting everybody together to contribute to common modules (it maybe took on too many though) and make sure you didn't have to go on too much of a plugin hunt. So could possibly we start a "batteries included" build system?
Finally, I wanted to try out some opportunity to merge the functionality of a build system with something like Rundeck, so you didn't need two tools. I also found Rundeck complicated to set up, but if you want to run some self-service automation, most people do that within their build system. So I figured I could add in some SSH and Q&A interactivity in there to easily get that going.
My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast, that's too much work to bite off without a lot of help :)
Honestly the main thing is I just wanted to work with a lot of the same open source folks again (and a bunch of new ones), so I'm really looking forward to that!
Maybe it's just me, but I always have trouble figuring out what apps are by reading the documentation. I'd usually understand more if there was an architecture diagram, which also clues me in that I need to do one for Vespene :)
One other reason I didn't build something is I don't have a current need to use something like that, so I'm probably not the one that should be developing it. There's a lot of datacenter use cases I'm ultimately not super familiar with.
I wanted another Science Fiction reference to something I loved.
Vespene gas is very important in build order.
My probable worst name was Cobbler (the PXE server) for those that were into cockney, but it was originally named "Shoe", and the idea was a Cobbler makes boots.
I'm pretty sure it's going to happen, or at least having Docker build files where you can run your own (on the workers, you'd want to install what you wanted anyway). I've had a couple of conversations from folks wanting to do that and it seems like it would be pretty easy, just making the entry point supervisor vs systemd.
Comments
Hey folks, this is my new project. You might possibly remember me from creating Cobbler and Ansible.
I thought I'd share and if you have any questions, let me know!
Id strongly recommend adding the ability to implement pipelines as code -- ie: you check in the CI/CD configuration into the repo (or another repo, containing CI/CD configurations). Having to configure the pipelines manually is nasty, and often builds are caused by misconfigured of the pipeline. having this checked it allows easy versioning.
I also like the idea of pipelines as code. I'd love to see it go a step further and become pipelines as testable code. I've yet to see a pipeline of non-trivial complexity that stays easy to reason about. I predict the next wave of CI/CD (or pipeline anything) will make pipeline testing & sanity checking simple as heck. If this already exists, please tell me about it, because I don't see people using it ;)
See my earlier reply about http://docs.vespene.io/importing.html - though I'm pretty darn open to making this do a lot more in the .vespene file.
How does this compare to something like Buildbot (https://buildbot.net/) - I share a lot of your frustrations with Jenkins and this is where we landed.
(BTW thanks for ansible, it's awesome :) )
I don't know, I'm really not trying to make one of those cliched checkbox X vs Y type grids, but when I started building Vespene I was most concerned about being able to make builds more consistent when you had hundreds of microservices developed by a lot of different teams. This is why I did http://docs.vespene.io/variables.html#snippets
I also saw a lot of people trying to do ops style stuff from tools like Jenkins, which is why Ansible has built-in SSH automation so it can hold on to SSH keys and use them in your behalf, and has some really cool built in RBAC so you can decide who can run what and it can be different than who can edit what.
Mostly I just want to build an architecture that can go interesting places - what is here today I think is usuable, but it's just a starting point. My thinking is if you make an architecture that is really pluggable, and the code is easy to read and add to, it can go to some really neat places.
I'd just recommend taking a look at the various chapters of http://docs.vespene.io/ to get a feel for features, and if you like it, spin up a copy using the setup instructions and see what you think.
Snippets of stuff is really missing from a lot of platforms. I use gitlab to store parts of our builds in one location instead of duplicating build code in every repo (and we have lots of them) for stuff like building RPM, where they all use a common model. Making this first-class is a good move.
Hello, Community Advocate from GitLab here. Thanks for sharing your solution with other users, we are happy to hear how you use GitLab to make your work more convenient.
Configuration management is best with operating system packages, so in creating "Ansible", not only did you completely miss the ball, you created a monster. People now hack ad hoc YAML files instead of designing clean OS packages to manipulate the configuration.
It's fine if you don't like it - there are things I don't like about it, but ultimately single packages can't express multi-node configurations very easily. We need something.
While I "grew up" as it were, believing in RPM, we live in a very post-distro, multi-language kind of world, and there's a need for things to glue that together. How many times have I tried to campaign against "wget tarball" as a deployment mechanism, I don't know. It's rough and yes, there's a lack of discipline in ops that needs to improve.
Immutable systems is a VERY interesting way to solve that, but it doesn't work for certain stateful things and you always need something to deploy the undercloud.
I'd encourage you to try to build your own experimental project to try to find different ways to do it, as this is the only real way that technology ever gets ahead.
Funny you should mention RPM's, since that's one of the packaging formats we perform configuration management in.
I have my own configuration management framework which can add and remove configuration from files that each package brings on. It's written in AWK. No knowledge is required to use it - configuration packages deliver just their configuration files' excerpts. In addition to being able to template them with regular shell variables, self-assembly is supported as well.
How is that different than what Puppet, Chef, CFEngine, or any number of other configuration management tools do [and did long before Ansible]? How do you propose to have a package that requires dynamic configuration supplied by the system (something from dhcp, for example). Are you conflating package management and configuration management? I'm genuinely curious.
It's not any different, they all suck. Read my reply above to the author on how it was solved cleanly without needing to hack anything, just deliver a config excerpt via an OS package and it works. It's not open sourced yet, mainly due to lack of spare time to spend on computers and my cronic exhaustion.
You're not the first person to try this, good luck.
The first proof of concept prototype was developed in 2007 based on insights gained with packaging and system engineering at two different companies, one a software one and the other a very large financial institution.
Refined framework (oh how I hate that word now thanks to web developers!) went into production in 2009 on Solaris 10, depending on advanced AT&T SVR4 packaging features. It was then subsequently ported to CentOS / RHEL RPM and went into production around 2011. It's been the corner stone of configuration management for the entire infrastructure, particularly production ever since.
Ugh. Conveying tone on the internet... I mean it sincerely: I do wish you good luck. Please about your project on HN when you release it, I'd love to see it. While I think it's deceptively hard, but config/immutable state are fascinating areas.
What's frustrating about jenkins?
It almost always turns into a snowflake the second you start using it, at least in my experience.
I think we owe Jenkins a TON of credit for being a free solution that has a lot of great traction.
I'm used to seeing pages of checkboxes and can't stand how obscure some of the configuration is, but mostly I wanted to write something that could handle configuration differences between hundreds of projects, so that is why there are things like Snippets in Vespene.
Plus, I wanted to make something that was a little better for ops use cases (so people don't also have to go pick up something like Rundeck), so that's why there are things like the SSH integration and http://docs.vespene.io/launch_questions.html
It's more about capabilities and future capabilities than the frustrations IMHO.
Though I do have a bunch of friends who are frustrated with plugin compatibility, plugin hunting (we're doing more of a "batteries included" approach like I ran with ansible - just with much less modules), and stuff like that.
I was also able to add in some stuff like container build isolation really easily, and that's all included stock.
What was the inspiration? What problem(s) did you have with other tools that Vespene solves?
So back around 2003 I was working at Adaptec (an IBM spinoff) and we were trying to put together a build system, I think I ended up front-ending it around DamageControl, which was a Ruby port of CruiseControl, except I had to hand-code a fake "SCM" module to run timed (versus commit-based builds). Our builds took like 8 hours! At that time, I really wanted to help people with build environments but never really did it.
Then, what sort of happened is I kept running into overly complex Jenkins installs. The typical miles of config checkboxes was overwhelming for me, but a larger problem was that organizations would have like 200 microservices projects, and all the build scripts would be slightly different.
The idea was then to take the templating system from Cobbler - which allows lots of variables merged in at various levels, and Jinja2, and allow reuse of build scripts through variables and snippets.
There was some other frustration with Jenkins - the idea that it was a ginormous codebase and still not database based, and you had to hunt for good plugins.
Ansible was pretty successful for getting everybody together to contribute to common modules (it maybe took on too many though) and make sure you didn't have to go on too much of a plugin hunt. So could possibly we start a "batteries included" build system?
Finally, I wanted to try out some opportunity to merge the functionality of a build system with something like Rundeck, so you didn't need two tools. I also found Rundeck complicated to set up, but if you want to run some self-service automation, most people do that within their build system. So I figured I could add in some SSH and Q&A interactivity in there to easily get that going.
My other idea was to write a VM/container controller, and while I think there still needs to be a simple one as Kubernetes is getting to be a beast, that's too much work to bite off without a lot of help :)
Honestly the main thing is I just wanted to work with a lot of the same open source folks again (and a bunch of new ones), so I'm really looking forward to that!
I'm curious what you think of https://www.nomadproject.io/
Maybe it's just me, but I always have trouble figuring out what apps are by reading the documentation. I'd usually understand more if there was an architecture diagram, which also clues me in that I need to do one for Vespene :)
One other reason I didn't build something is I don't have a current need to use something like that, so I'm probably not the one that should be developing it. There's a lot of datacenter use cases I'm ultimately not super familiar with.
I would strongly suggest changing the name. "Ves pene" means something pretty much NSFW in Spanish. Besides that, nice job! Keep it up.
As a native speaker I have no problems with the name.
For those who don’t understand it may sound like “(you) see penis”. But to be honest I didn’t even make the connection when I read it the first time.
My first thought on seeing the name is the resource from StarCraft: “We require more vespene gas”
It's got to be a riff on the StarCraft resource, much like Ansible was a riff on Le Guin's faster-than-light communication device.
Indeed!
I wanted another Science Fiction reference to something I loved.
Vespene gas is very important in build order.
My probable worst name was Cobbler (the PXE server) for those that were into cockney, but it was originally named "Shoe", and the idea was a Cobbler makes boots.
Lo siento!
any plans to provide a docker image?
I'm pretty sure it's going to happen, or at least having Docker build files where you can run your own (on the workers, you'd want to install what you wanted anyway). I've had a couple of conversations from folks wanting to do that and it seems like it would be pretty easy, just making the entry point supervisor vs systemd.