Difference between node_status_capacity_memory and node_status_allocateable_memory

[kubernetes]

Difference between node_status_capacity_memory and node_status_allocateable_memory

[ahakanbaba]

Would this code help to understand the difference between capacity and allocatable ?

 // Set Allocatable.
 node.Status.Allocatable = make(api.ResourceList)
 for k, v := range node.Status.Capacity {
    value := *(v.Copy())
    if kl.reservation.System != nil {
       value.Sub(kl.reservation.System[k])
    }
    if kl.reservation.Kubernetes != nil {
       value.Sub(kl.reservation.Kubernetes[k])
    }
    if value.Sign() < 0 {
       // Negative Allocatable resources don't make sense.
       value.Set(0)
    }
    node.Status.Allocatable[k] = value
 }

This is from here https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go#L459

Reservation is defined here
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go#L974

 // reservation specifies resources which are reserved for non-pod usage, including kubernetes and
 // non-kubernetes system processes.
 reservation kubetypes.Reservation

[brancz]

Interesting, in the code we seem to use the correct values and the values from the status struct should reflect the state you are looking for. I believe these numbers come from cAdvisor through the apiserver. I believe we have seen that behaviour from cAdvisor before, right @fabxc?

[brancz]

Looking further into it there are quite a few more constants that describe memory resources. We should figure out which ones are applicable for the purpose you described. At least from our current naming I would expect the same numbers/behavior as you do @rvrignaud .

[brancz]

Looking at my local dev cluster it looks like the resources we take should be correct, but I get the same numbers for capacity/allocatable as well directly from the apiserver.

{
  "capacity": {
    "alpha.kubernetes.io/nvidia-gpu": "0",
    "cpu": "1",
    "memory": "1021160Ki",
    "pods": "110"
  },
  "allocatable": {
    "alpha.kubernetes.io/nvidia-gpu": "0",
    "cpu": "1",
    "memory": "1021160Ki",
    "pods": "110"
  },
  ...
}

[fabxc]

I recall seeing this issue before and it being an issue in cAdvisor.

Found something similar at least: google/cadvisor#1404

[therc]

Yes, by default capacity and allocatable are the same, but a good administrator will reserve some resources from the node for system processes (systemd, journald, sshd and in many cases kubelet/kube-proxy).

[andyxning]

@rvrignaud @brancz Kubelet now supports two command line arguments: --kube-reserved and --system-reserved. By setting any of both, node allocatable and capacity should be different. Just as what has been mentioned in comment 264933854 by @fabxc.

For now, the value for node allocatable and capacity are correct.

Maybe we can close this. @rvrignaud

[rvrignaud]

Closing it