When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours.
Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us.
Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero.
I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.
Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s).
Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)
Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).
Yea, quite right, also use runners on ephemeral build targets as that can reduce some of the attack surface for running untrusted code, don't do it all in the box.
We have open tickets about the SLA being broken for Q1 and Q2. They've been open for a while, and we're out of ways to escalate them (despite enterprise).
And GitHub's SLA is not great to begin with: 10% of spend refund at 3 nines (99,9%) and 25% of spend at 2 nines (99%).
In general, don't use Jenkins. It is mediocre piece of software with an even worse ecosystem. It self-implodes every now and then. Big tech companies build a lot of custom stuff to get it to work correctly. Once that is done sure, it works. I am not a fan of Github actions but to host GIT and Jenkins is not really a serious option. Pick another CI platform in that case.
This comment makes a lot of assertions without any backing data.
How is it mediocre? Is it because of the CVEs that have been released in the prior years? I recall GitLab also having quite a bad week of CVEs in February[1].
How is it a bad ecosystem? If this is about plugins in order to do things, I actually like this framework - it lets there be specific owners for portions of the open source development.
Self-implodes? This seems like it would be tracked as a bug. I've encountered an instance where Jenkins wouldn't start due to a crypto issue but that was due to a bug and all I needed to do was install a patch.
I think that using Jenkins can be a thought of a serious option if like anything else, you follow security protocols ie: don't allow public access, maintain RBAC standards, have a maintenance schedule.
I have stolen access to production systems though Jenkins. Cause the scrambled pw is always sent to the Jenkins client. And then by default anybody could decode it from the console… They fixed it in 2.0 I believe.
The problem with dependencies is that different plugins need different versions but there is no solution to that. Even more. You update Jenkins. Plugins break. Data gets corrupted. You try to align the plugins to some dependent version. It fails, cause most of the ecosystem is a hobby-project that never gets updated.
Jenkins has absolutely nothing to do on the public Internet as most of these tools.
There are so many CI/CD services. Most are very cheap. You really need to love yak shaving to pick Jenkins over something from the shelf. No major cloud offers Jenkins Saas. Wonder why…
'I don't like it because it's not secure and I was able to get access to prod via this method.'
'That method has been fixed.'
'Well...I still don't like it.'
Bugs are being found and fixed all the time. As for a the different plugins need different versions. Yes. This also occurs in general coding to where every package or library that you want to use in your code is also going to be versioned. If that package or library gets updated, your code might break.
This goes back to having some things in place. Backups and a maintenance and testing schedule. If you don't test things before pushing them out to prod maybe fix that first.
GitHub is running a complex, planet-scale product. I think w they've crossed the threshold where doing it yourself is more likely to be reliable for some use cases.
We've been running GitLab on GKE for the past three years, no problems outside of initial migration pains.
Also: "you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services."
If your code was in a private repo they supposedly didn't use it for training. I think it was also mentioned they tried to limit it to only repos where they could determine some kind of license they thought would permit such usage.
If you're publishing your IP to GitHub in a public repo and using a widely permissive license, they didn't need an agreement to use that IP to train copilot.
There needs to be a line somewhere that says, "if you use a widely permissive license, we will use your IP to make money for ourselves" or something like that, then, right?
Do they need to have such a statement? What makes them need to make such a statement? You've given them the permission with the copyright license you've published your code with.
If you're not just being facetious then that someone who can help is an IP lawyer, and I congratulate you on your new boat. I'm not being fake with that.
Buf if you published your code with an MIT or GPL kind of license chances are they don't owe you anything nor do they have to say that it's using your code. IANAL, but nothing in those common licenses leads me to believe they owe you even that statement. If I'm wrong feel free to point to it.
You're not really engaging in this discussion much more than giving extremely vague hypotheticals and then making demands that seem like quite a leap. Where in, say the GPL, would they have to make a statement like you claim they need to? What are the actual terms in your public license and code that you have that they would have infringed in some manner? The repo would have to be public and you just said it would be free, so please do link it. Otherwise, what are we even talking about here? Just making noise on the internet I guess?
Its disparaging to ask for the public codebase that you claim Microsoft is using inappropriately or for you to explain why you think they owe you something?
And how is it disparaging to suggest that if you really think Microsoft somehow harmed you that you go and do the thing that would actually resolve the issue?
I try and learn from my mistakes. Can you describe exactly how I am being disparaging by these comments?
Also, following up, will you share the codebase in question? You said its free, and its clearly something publicly hosted in GitHub otherwise its entirely out of scope from this conversation.
When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours.
Comments
When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours.
Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us.
Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero.
I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.
PSA:
Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s).
Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)
Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).
Yea, quite right, also use runners on ephemeral build targets as that can reduce some of the attack surface for running untrusted code, don't do it all in the box.
We have open tickets about the SLA being broken for Q1 and Q2. They've been open for a while, and we're out of ways to escalate them (despite enterprise).
And GitHub's SLA is not great to begin with: 10% of spend refund at 3 nines (99,9%) and 25% of spend at 2 nines (99%).
In general, don't use Jenkins. It is mediocre piece of software with an even worse ecosystem. It self-implodes every now and then. Big tech companies build a lot of custom stuff to get it to work correctly. Once that is done sure, it works. I am not a fan of Github actions but to host GIT and Jenkins is not really a serious option. Pick another CI platform in that case.
This comment makes a lot of assertions without any backing data.
How is it mediocre? Is it because of the CVEs that have been released in the prior years? I recall GitLab also having quite a bad week of CVEs in February[1].
How is it a bad ecosystem? If this is about plugins in order to do things, I actually like this framework - it lets there be specific owners for portions of the open source development.
Self-implodes? This seems like it would be tracked as a bug. I've encountered an instance where Jenkins wouldn't start due to a crypto issue but that was due to a bug and all I needed to do was install a patch.
I think that using Jenkins can be a thought of a serious option if like anything else, you follow security protocols ie: don't allow public access, maintain RBAC standards, have a maintenance schedule.
[1]https://about.gitlab.com/releases/2022/02/25/critical-securi...
I have stolen access to production systems though Jenkins. Cause the scrambled pw is always sent to the Jenkins client. And then by default anybody could decode it from the console… They fixed it in 2.0 I believe.
The problem with dependencies is that different plugins need different versions but there is no solution to that. Even more. You update Jenkins. Plugins break. Data gets corrupted. You try to align the plugins to some dependent version. It fails, cause most of the ecosystem is a hobby-project that never gets updated.
Jenkins has absolutely nothing to do on the public Internet as most of these tools.
There are so many CI/CD services. Most are very cheap. You really need to love yak shaving to pick Jenkins over something from the shelf. No major cloud offers Jenkins Saas. Wonder why…
This is an odd response in my perspective.
'I don't like it because it's not secure and I was able to get access to prod via this method.'
'That method has been fixed.'
'Well...I still don't like it.'
Bugs are being found and fixed all the time. As for a the different plugins need different versions. Yes. This also occurs in general coding to where every package or library that you want to use in your code is also going to be versioned. If that package or library gets updated, your code might break.
This goes back to having some things in place. Backups and a maintenance and testing schedule. If you don't test things before pushing them out to prod maybe fix that first.
I have always been happy using GitLabs CI/CD tooling [1]. Also, the integration with the source code this way is like Github with Github Actions.
[1] https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html
GitHub is running a complex, planet-scale product. I think w they've crossed the threshold where doing it yourself is more likely to be reliable for some use cases.
We've been running GitLab on GKE for the past three years, no problems outside of initial migration pains.
Also: "you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services."
https://www.microsoft.com/en-ca/servicesagreement/upcoming.a...
Am I missing something, or is GitHub distinctly not listed in the Covered Services section of that services agreement?
Nope, looks like I missed that. My mistake.
So what part of the github terms gives them permission to sell my IP back to me through copilot?
https://docs.github.com/en/site-policy/github-terms/github-t...
GitHub has a separate policy...
Thank you, I missed that part.
As I asked elsewhere, where is the part of these agreements that permits them to profit off my IP through copilot?
If your code was in a private repo they supposedly didn't use it for training. I think it was also mentioned they tried to limit it to only repos where they could determine some kind of license they thought would permit such usage.
If you're publishing your IP to GitHub in a public repo and using a widely permissive license, they didn't need an agreement to use that IP to train copilot.
There needs to be a line somewhere that says, "if you use a widely permissive license, we will use your IP to make money for ourselves" or something like that, then, right?
That's what I want. To find that line.
Do they need to have such a statement? What makes them need to make such a statement? You've given them the permission with the copyright license you've published your code with.
Yes it's required, at no point did any license I assign my code give them permission to sell my IP as a part of copilot.
Which licenses do you use and where in the licenses does it state it couldn't be used as training data?
My license says something along the lines of "This is totally free, except for use by Microsoft and Github in anything they develop or sell."
So, where does it say they can use my IP?
Cool. Call an IP lawyer, sue Microsoft, and enjoy retirement on your new boat.
Or I guess you didn't really have any IP of value that was infringed on whatever license you claim to have made.
Dang, I was hoping you or someone could help. Anyways thanks for participating in the discussion none the less.
If you're not just being facetious then that someone who can help is an IP lawyer, and I congratulate you on your new boat. I'm not being fake with that.
Buf if you published your code with an MIT or GPL kind of license chances are they don't owe you anything nor do they have to say that it's using your code. IANAL, but nothing in those common licenses leads me to believe they owe you even that statement. If I'm wrong feel free to point to it.
You're not really engaging in this discussion much more than giving extremely vague hypotheticals and then making demands that seem like quite a leap. Where in, say the GPL, would they have to make a statement like you claim they need to? What are the actual terms in your public license and code that you have that they would have infringed in some manner? The repo would have to be public and you just said it would be free, so please do link it. Otherwise, what are we even talking about here? Just making noise on the internet I guess?
That's now the second time you've disparaged me. I think you're wrong and now I also think you're a jerk.
Its disparaging to ask for the public codebase that you claim Microsoft is using inappropriately or for you to explain why you think they owe you something?
And how is it disparaging to suggest that if you really think Microsoft somehow harmed you that you go and do the thing that would actually resolve the issue?
I try and learn from my mistakes. Can you describe exactly how I am being disparaging by these comments?
Also, following up, will you share the codebase in question? You said its free, and its clearly something publicly hosted in GitHub otherwise its entirely out of scope from this conversation.
Oh you sweet summer child