But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage)
When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaranteed` for critical process.
The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.
IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj.
That's not what you're saying though. You're also putting requests/limits for CPU and memory in the same class, which they're not.
Lack of CPU will never cause a pod to be killed. Evicted, yes. Killed, no.
Lack of memory can cause a pod to be OOM killed. That's an entirely different failure mode.
Eviction should always be expected by your pods. They should be able to handle eviction gracefully. On the other hand, there is nothing you can do to handle an OOM kill gracefully.
K8s works best (economically) when it can bin pack things. The only way it can bin pack things safely is by having user provided limits--its not smart enough to right size your app (out of the box at least). Not setting them means you're going to end up paying more or have resource contention/outages.
It's better to set the limits higher than you need than to not set them at all. Ideally this is easily done since you're profiling/load testing your app and you understand the appropriate sizing for it, right?
The problem is that the Kubernetes CPU scheduler is... not great.
There are many instances where it will throttle CPU of a pod way before it even reaches the requested amount, let alone the limits. This is only true if you set limits. If you don't, it will always have the requested amount available, plus whatever can be spared that's not in use by other processes. It will throttle at most down to the requested amount, and in most cases not at all.
Users come to expect the performance that was never promised or even properly requested. Once you efficiently load the system they complain because you overdelivered and they got used to it.
Comments
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
I can't comment about about other statement.
But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage)
When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaranteed` for critical process.
1. oom_score_adj - https://kubernetes.io/docs/concepts/scheduling-eviction/node...
2. Guaranteed - https://kubernetes.io/docs/tasks/configure-pod-container/qua...
3. QoS - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.
This doens't match what I've read.
IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj.
OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...
OOM means "Out Of Memory". The post is about CPU.
I am aware of it, are you aware that Kubernetes assigned Burstable Qos if you don't defined CPU requests and limits?
What I'm trying to say that while try to avoid one pitfall you fall right into an other - in short there are good reasons to use cpu limits.
That's not what you're saying though. You're also putting requests/limits for CPU and memory in the same class, which they're not.
Lack of CPU will never cause a pod to be killed. Evicted, yes. Killed, no.
Lack of memory can cause a pod to be OOM killed. That's an entirely different failure mode.
Eviction should always be expected by your pods. They should be able to handle eviction gracefully. On the other hand, there is nothing you can do to handle an OOM kill gracefully.
K8s works best (economically) when it can bin pack things. The only way it can bin pack things safely is by having user provided limits--its not smart enough to right size your app (out of the box at least). Not setting them means you're going to end up paying more or have resource contention/outages.
It's better to set the limits higher than you need than to not set them at all. Ideally this is easily done since you're profiling/load testing your app and you understand the appropriate sizing for it, right?
The problem is that the Kubernetes CPU scheduler is... not great.
There are many instances where it will throttle CPU of a pod way before it even reaches the requested amount, let alone the limits. This is only true if you set limits. If you don't, it will always have the requested amount available, plus whatever can be spared that's not in use by other processes. It will throttle at most down to the requested amount, and in most cases not at all.
Unless your app can use all the cpu you assign it
Ha! Always end with a joke, I love it.
Users come to expect the performance that was never promised or even properly requested. Once you efficiently load the system they complain because you overdelivered and they got used to it.
We've been fixing that over time and now no one expects a computer to be as fast as it accidentally was in 2010.
Why, by 2030, computers ought to be even slower!
I'm sorry, in what way are modern servers not fast enough to serve a few hundred CRUD requests per second?