Autoscaling in a Managed Kubernetes cluster
Autoscaling is unavailable:
- for node groups with GPU without drivers;
- node groups on dedicated servers.
In a Managed Kubernetes cluster, you can use Cluster Autoscaler for node group autoscaling. It helps use cluster resources efficiently: depending on the cluster load, the number of nodes in a group will automatically decrease or increase. Cluster Autoscaler is installed automatically when creating a cluster; to get started, simply enable it.
When using Cluster Autoscaler, consider the recommendations.
For pod autoscaling in Managed Kubernetes, Metrics Server is used.
Recommendations
For optimal autoscaling performance, we recommend:
- do not use more than one autoscaling tool at the same time;
- make sure that the project has quotas for vCPU, RAM, GPU, and disk size to create the maximum number of nodes in the group;
- specify resource requests in pod manifests. For more information, see the Resource Management for Pods and Containers guide in Kubernetes documentation;
- configure PodDisruptionBudget for pods that cannot tolerate disruptions. This will help avoid downtime during migration between nodes;
- do not change node resources manually via the control panel. Cluster Autoscaler will not take these changes into account;
- make sure that the nodes in the group have the same configuration and labels.
Autoscaling using Cluster Autoscaler
Cluster Autoscaler does not need to be installed in the cluster — it is installed automatically when creating a cluster. To use Cluster Autoscaler in a cluster, enable node group autoscaling. After autoscaling is enabled, default settings are used, but you can configure Cluster Autoscaler for each node group.
Operating principle
Cluster Autoscaler works with existing node groups and pre-selected configurations. If a node group is in the ACTIVE status, Cluster Autoscaler checks every 10 seconds whether there are pods in the PENDING status and analyzes the load — requests from pods for vCPU, RAM, and GPU. Depending on the check results, nodes are added or deleted. The node group enters the PENDING_SCALE_UP or PENDING_SCALE_DOWN status during this time. The cluster status during autoscaling is ACTIVE. For more information about cluster statuses, see the View cluster status guide.
The minimum and maximum number of nodes in a group can be set when enabling autoscaling — Cluster Autoscaler will change the number of nodes only within these limits. If at least two worker nodes remain in other node groups of the cluster, you can configure autoscaling to zero nodes.
Adding a node
If there are pods in the PENDING status and the cluster does not have enough free resources to place them, the required number of nodes is added to the cluster. In a cluster with Kubernetes version 1.28 or higher, Cluster Autoscaler will work across multiple groups simultaneously and distribute nodes evenly.
For example, you have two node groups with autoscaling enabled. The cluster load has increased and requires four nodes to be added. Two new nodes will be created in each node group simultaneously.
In a cluster with Kubernetes version 1.27 or lower, nodes are added one at a time per check cycle.
Deleting a node
If there are no pods in the PENDING status, Cluster Autoscaler checks the amount of resources requested by pods.
If the requested amount of resources for pods on a node is less than 50% of its resources, Cluster Autoscaler marks the node as unneeded. If the number of resource requests on the node does not increase within 10 minutes, Cluster Autoscaler will check if the pods can be migrated to other nodes.
Cluster Autoscaler will not move pods and, consequently, will not delete a node if one of the following conditions is met:
- pods use PodDisruptionBudget;
- In kube-system pods there is no PodDisruptionBudget;
- pods are created without a controller — for example, Deployment, ReplicaSet, StatefulSet;
- pods use local storage;
- there are no resources on other nodes for the pod requests;
- there is a mismatch of nodeSelector, affinity and anti-affinity rules, or other parameters.
You can allow such pods to be moved — to do this, add the following annotation:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
If there are no restrictions, the pods will be migrated, and the underloaded nodes will be removed. Nodes are removed one by one per check cycle.
Autoscaling to zero nodes
In a node group, you can configure autoscaling to zero nodes—if the load is low, all nodes in the group are removed. The node group card with all settings is not removed. When the load increases, nodes can be added to this node group again.
Autoscaling to zero nodes works only if there are at least two working nodes remaining in other node groups in the cluster. Working nodes must remain in the cluster to host system components required for cluster operation.
For example, autoscaling to zero nodes will not work if the cluster has:
- two node groups with one working node in each;
- one node group with two working nodes.
When there are no nodes in a group, you do not pay for unused resources.
Enable autoscaling using Cluster Autoscaler
If you set the minimum number of nodes in a group greater than the current number, it will not increase to the lower limit immediately. The node group will only scale after pods in the PENDING status appear. The same applies to the upper limit of nodes in a group: if the current number of nodes exceeds the upper limit, deletion will start only after checking the pods.
You can enable autoscaling using Cluster Autoscaler in the control panel, via the Managed Kubernetes API, or via Terraform.
- In the control panel, in the top menu, click Products and select Managed Kubernetes.
- In the Clusters section, open the cluster page → Cluster topology tab.
- From the menu of the node group, select Change number of nodes.
- In the Number of nodes field, open the With autoscaling tab.
- Set the minimum and maximum number of nodes in the group; the number of nodes will only change within this range. For fault-tolerant operation of system components, we recommend using at least two working nodes in the cluster. Nodes can be located in different groups.
- Click Save.
Configure Cluster Autoscaler
You can configure Cluster Autoscaler separately for each node group.
Parameters, their descriptions, and default values can be found in the Cluster Autoscaler parameters table. If you do not specify a parameter in the manifest, the default value will be used.
Example manifest:
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-autoscaler-nodegroup-options
namespace: kube-system
data:
config.yaml: |
150da0a9-6ea6-4148-892b-965282e195b0:
scaleDownUtilizationThreshold: 0.55
scaleDownUnneededTime: 7m
zeroOrMaxNodeScaling: true
e3dc24ca-df9d-429c-bcd5-be85f8d28710:
scaleDownGpuUtilizationThreshold: 0.25
ignoreDaemonSetsUtilization: true
Here, 150da0a9-6ea6-4148-892b-965282e195b0 and e3dc24ca-df9d-429c-bcd5-be85f8d28710 are unique identifiers (UUIDs) of node groups in the cluster. You can view them in the control panel: in the top menu, click Products ⟶ Managed Kubernetes ⟶ Kubernetes section ⟶ cluster page ⟶ copy the UUID above the node group card, next to the pool segment.