470385 (3) [Avatar] Offline
#1
Hi, Marko
First of all your book is the best I've read on kubernetes and it's not finished yet smilie So great job and thank you.
Second, I do have a question regarding kubernetes 1.6. I want faster pod eviction after a node failure in the cluster.
Where are the following options set in k8s 1.6:
--node-monitor-grace-period (kube-controller-manager)
--pod-eviction-timeout (kube-controller-manager)
--node-status-update-frequency (kubelet on the nodes)
I knew where they were set in k8s 1.5 but in 1.6 I just can't seem to get around it.Is it possible to change these settings in 1.6 at all? If yes could you put that in the book or answer to this feed? I'd be eternally grateful.
If no, please tell me so I can stop busting my head off my lap top smilie
And keep up the GREAT work!
Cheers,
Denis.
Marko Lukša (67) [Avatar] Offline
#2
Looking at the source code, all three options are still there. In the latest MEAP update (v10), you'll find a new chapter about Advanced scheduling. Section 16.1.4 talks about the
node.alpha.kubernetes.io/notReady
and
node.alpha.kubernetes.io/unreachable
tolerations, which allow you to configure how soon after a node failure the pod should be rescheduled. This defaults to 5 minutes (or whatever you set with
--pod-eviction-timeout
.

Hope this helps. If not, just tell me more about the actual problem you're seeing, and I'll try to help.

Thanks for the kind words. I'm glad the time I've put into the book wasn't wasted.
470385 (3) [Avatar] Offline
#3
The problem I have now is I have a deployment with 2 replicas. In the deployment I have this:
tolerations:
- effect: "NoExecute"
key: "node.alpha.kubernetes.io/notReady"
operator: "Exists"
tolerationSeconds: 3
- effect: "NoExecute"
key: "node.alpha.kubernetes.io/unreachable"
operator: "Exists"
tolerationSeconds: 3
(I tried it both with and without quotation marks). I do "k get po <name> -o yaml" and it says the tolSec: 3 for the pod.
But I shut down a node. After 10-20 sec k8s sees node as NotReady. 5m after that the pod is still in Running state. After 5m one unknown status of the pod and it gets rescheduled immediately. I tried the tolerations with deployment, replicasets, and with pod yaml files. No luck.
Is it because it's still beta feature or is my cluster not right somewhere?
Thanks.
Marko Lukša (67) [Avatar] Offline
#4
Ah, yes, sorry. Because this is an alpha feature, it's not enabled by default. You add the following cmd-line option to the kube-controller-manager:

--feature-gates=TaintBasedEvictions=true


If you're running Control Plane components as Pods (such as in the cluster from appendix B), you add the option to /etc/kubernetes/manifests/kube-controller-manager.yaml.

Thanks for reporting this. I'll add a callout to chapter 16.