Autoscaling in a Managed Kubernetes cluster
Autoscaling is unavailable:
- for node groups with GPU without drivers;
- node groups on dedicated servers.
In a Managed Kubernetes cluster, you can use Cluster Autoscaler to autoscale node groups. It helps to optimize cluster resource usage—the number of nodes in a group will automatically decrease or increase depending on the cluster load. Cluster Autoscaler is installed automatically when a cluster is created; to start using it, you just need to enable it.
When using Cluster Autoscaler, keep the recommendations in mind.
For pod autoscaling in Managed Kubernetes, Metrics Server is used.
Recommendations
For optimal autoscaling performance, we recommend:
- do not use more than one autoscaling tool at the same time;
- make sure the project has quotas for vCPU, RAM, GPU, and disk capacity to create the maximum number of nodes in the group;
- specify resource requests in the manifests for pods. For more details, see the Resource Management for Pods and Containers instruction in the Kubernetes documentation;
- configure a PodDisruptionBudget for pods that cannot be stopped. This will help avoid downtime during migration between nodes;
- do not manually modify node resources through the control panel. Cluster Autoscaler will not account for these changes;
- check that nodes in the group have the same configuration and labels.
Autoscaling using Cluster Autoscaler
Cluster Autoscaler does not need to be installed in the cluster—it is installed automatically when a cluster is created. To use Cluster Autoscaler in a cluster, enable autoscaling for the node group. Once autoscaling is enabled, default settings are used, but you can configure Cluster Autoscaler for each node group.
Operating principle
Cluster Autoscaler works with existing node groups and preselected configurations. If a node group is in the ACTIVE status, Cluster Autoscaler checks every 10 seconds whether there are pods in the PENDING status and analyzes the load—requests from pods for vCPU, RAM, and GPU. Depending on the check results, nodes are added or deleted. During this time, the node group switches to PENDING_SCALE_UP or PENDING_SCALE_DOWN. The cluster status during autoscaling is ACTIVE. For more details on cluster statuses, see the View cluster status instruction.
The minimum and maximum number of nodes in a group can be set when enabling autoscaling —Cluster Autoscaler will scale the number of nodes only within these limits. If at least two working nodes remain in other node groups of the cluster, you can configure autoscaling to zero nodes.
Adding a node
If there are pods in the PENDING status and the cluster lacks free resources to host them, the required number of nodes is added to the cluster. In a cluster with Kubernetes version 1.28 and higher, Cluster Autoscaler will work across multiple groups simultaneously and distribute nodes evenly.
For example, you have two node groups with autoscaling enabled. The load on the cluster has increased and requires adding four nodes. Two new nodes will be created in each node group simultaneously.
In a cluster with Kubernetes version 1.27 or lower, nodes are added one at a time per check cycle.
Deleting a node
If there are no pods in the PENDING status, Cluster Autoscaler checks the amount of resources requested by the pods.
If the requested resource amount for pods on a single node is less than 50% of its total resources, Cluster Autoscaler flags the node as unnecessary. If the resource request volume does not increase on the node within 10 minutes, Cluster Autoscaler checks whether the pods can be migrated to other nodes.
Cluster Autoscaler will not move pods and, consequently, will not delete a node if one of the following conditions is met:
- pods use PodDisruptionBudget;
- In kube-system pods there is no PodDisruptionBudget;
- pods are created without a controller — for example, Deployment, ReplicaSet, StatefulSet;
- pods use local storage;
- there are no resources on other nodes for the pod requests;
- there is a mismatch of nodeSelector, affinity and anti-affinity rules, or other parameters.
You can allow such pods to be moved — to do this, add the following annotation:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
If there are no restrictions, the pods will be migrated, and the underutilized nodes will be removed. Nodes are removed one by one per check cycle.
Autoscaling to zero nodes
You can configure autoscaling to zero nodes for a node group—all nodes in the group are removed during low load. The node group card with all settings is not removed. When the load increases, nodes can be added to this node group again.
Autoscaling to zero nodes only works if at least two working nodes remain in other cluster node groups. The cluster must have working nodes for hosting system components required for the cluster to function.
For example, autoscaling to zero nodes will not work if the cluster has:
- two node groups with one working node in each;
- one node group with two working nodes.
When there are no nodes in a group, you do not pay for unused resources.
Enable autoscaling using Cluster Autoscaler
If you set the minimum number of nodes in a group higher than the current number, it will not increase to the lower limit immediately. The node group will only scale after pods appear in the PENDING status. Similarly, for the upper limit of nodes—if the current number of nodes is higher than the upper limit, removal will begin only after checking the pods.
You can enable autoscaling using Cluster Autoscaler in the control panel, via the Managed Kubernetes API or via Terraform.
- In the control panel, on the top menu, click Products and select Managed Kubernetes.
- Open the cluster page → Cluster Resources tab.
- In the menu of the node group, select Change number of nodes.
- In the Number of nodes field, open the With autoscaling tab.
- Set the minimum and maximum number of nodes in a group; the value will change only within this range. For fault-tolerant operation of system components, we recommend using at least two working nodes in the cluster. Nodes can be in different groups.
- Click Save.
Configure Cluster Autoscaler
You can configure Cluster Autoscaler separately for each node group.
Parameters, their descriptions, and default values can be found in the Cluster Autoscaler parameters table. If you do not specify a parameter in the manifest, the default value will be used.
Example manifest:
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-autoscaler-nodegroup-options
namespace: kube-system
data:
config.yaml: |
150da0a9-6ea6-4148-892b-965282e195b0:
scaleDownUtilizationThreshold: 0.55
scaleDownUnneededTime: 7m
zeroOrMaxNodeScaling: true
e3dc24ca-df9d-429c-bcd5-be85f8d28710:
scaleDownGpuUtilizationThreshold: 0.25
ignoreDaemonSetsUtilization: true
Here, 150da0a9-6ea6-4148-892b-965282e195b0 and e3dc24ca-df9d-429c-bcd5-be85f8d28710 are unique identifiers (UUIDs) of node groups in the cluster. You can view them in the control panel: in the top menu, click Products ⟶ Managed Kubernetes ⟶ the Kubernetes section ⟶ cluster page ⟶ copy the UUID above the node group card, next to the pool segment.