Connect file storage to a Managed Kubernetes cluster in a different pool

If you plan to use file storage for backing up data, we recommend creating the storage and the Managed Kubernetes cluster in pools from different availability zones or regions to increase fault tolerance. If the file storage and the cluster are in different pools, you must configure private network connectivity at the L3 level via a global router.

1. Create a global router.

2. Connect a network and subnet for a Managed Kubernetes cluster to the global router.

3. Connect a network and subnet for file storage to the global router.

4. Assign an IP address to a Managed Kubernetes cluster node.

5. Add a route on the Managed Kubernetes cluster node.

6. Add a route in the file storage subnet.

7. Create file storage.

8. Mount the file storage to the Managed Kubernetes cluster.

See an example of connecting file storage to a Managed Kubernetes cluster in another pool.

If you need to increase disk space using file storage, we recommend creating the storage in the same pool as the Managed Kubernetes cluster. See more details in the guide Connect file storage to a Managed Kubernetes cluster in one pool.

Example of connecting file storage to a Managed Kubernetes cluster

For example, you need to connect file storage in pool ru-2 to a Managed Kubernetes cluster in pool ru-8.

1. Create a global router.

2. Connect two private networks to the global router — 192.168.0.0/29 with gateway 192.168.0.1 for pool ru-8 and 172.16.0.0/29 with gateway 172.16.0.1 for pool ru-2.

3. Assign an address from the 192.168.0.0/29 subnet to the Managed Kubernetes cluster node, for example 192.168.0.2.

4. Add a route on the Managed Kubernetes cluster node in pool ru-8 — to the 172.16.0.0/29 subnet via gateway 192.168.0.1.

5. Create file storage in the 172.16.0.0/29 subnet.

6. Mount the file storage to the Managed Kubernetes cluster.

1. Create a global router

1. In the Control panel, in the top menu, click Products and select Global Router.
2. Click Create router. A limit of five global routers is set for each account.
3. Enter the router name.
4. Click Create.
5. If the router was created with the status ERROR or is stuck in one of the statuses, create a ticket.

2. Connect a network and subnet for the Managed Kubernetes cluster to the router

You need to connect a network and subnet to the global router for the project and pool where the Managed Kubernetes cluster is created.

You can connect a new network to the router or an existing network if it is not already connected to any of the account's global routers.

Connect a new network

1. In the control panel, on the top menu, click Products and select Global Router.

2. Open the router page → Networks tab.

3. Click Create network.

4. Enter a network name. It will only be used in the control panel.

5. Select the Cloud Platform service.

6. Select the location where the Managed Kubernetes cluster was created.

7. Select the project where the Managed Kubernetes cluster was created.

8. Enter a subnet name. It will only be used in the control panel.

9. Enter the CIDR — the IP address and subnet mask. The subnet must meet the following conditions:

   • belong to the RFC 1918 private address range: 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16;
   • have a size of at least /29, as three addresses will be occupied by Servercore network equipment;
   • do not overlap with other subnets added to this router — there must be no identical IP addresses in the subnets of one router;
   • if a Managed Kubernetes cluster on cloud servers is included in the global router network, the subnet must not overlap with the 10.10.0.0/16, 10.96.0.0/12, 10.250.0.0/16 and 10.251.0.0/24. If a cluster on dedicated servers is included in the network — with the 10.10.0.0/16, 10.222.0.0/16, 10.250.0.0/16, 10.251.0.0/24 and 172.250.0.0/14. These subnets are used for internal Managed Kubernetes addressing; using them may lead to network conflicts in the global router.

10. Enter the gateway IP or leave the first address from the subnet that is assigned by default. Do not assign this address to your devices so as not to disrupt network operation.

11. Enter the service IPs or leave the last addresses from the subnet that are assigned by default. Do not assign these addresses to your devices so as not to disrupt network operation.

12. Click Create network.

13. Optional: check the network topology on the global router. In the control panel, on the top menu, click Products and select Global Router. Open the page of the required router and click Network map.

Connect an existing network

1. Check that the network has not been added to any of the account's global routers yet. In the control panel, on the top menu, click Products → Cloud Servers → Network → Private networks tab → make sure the network card does not have a Global Router tag.

2. Ensure that the subnet meets the following conditions:

   • belong to the RFC 1918 private address range: 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16;
   • have a size of at least /29, as three addresses will be occupied by Servercore network equipment;
   • do not overlap with other subnets added to this router — there must be no identical IP addresses in the subnets of one router;
   • if a Managed Kubernetes cluster on cloud servers is included in the global router network, the subnet must not overlap with the 10.10.0.0/16, 10.96.0.0/12, 10.250.0.0/16 and 10.251.0.0/24. If a cluster on dedicated servers is included in the network — with the 10.10.0.0/16, 10.222.0.0/16, 10.250.0.0/16, 10.251.0.0/24 and 172.250.0.0/14. These subnets are used for internal Managed Kubernetes addressing; using them may lead to network conflicts in the global router.

3. In the control panel, on the top menu, click Products and select Cloud Servers.

4. Go to Network → Private networks.

5. In the network menu, select Connect to Global Router.

6. Select a global router.

7. For each of the network subnets, enter the IP address that will be assigned to the router, or leave the first free address from the subnet that is assigned by default. Do not assign this address to your devices so as not to disrupt network operation. The last two free subnet addresses will be reserved as service addresses.

8. Click Connect. Do not close the window until you see a message stating that the network is connected. Then, in the control panel:

   • the network will appear in the Servercore Global Router section on the page of the router you connected it to. You can check this in the control panel: on the top menu, click Products → Global Router → router page;
   • a Global Router tag will appear on the network. You can check this in the control panel: on the top menu, click Products → Cloud Servers → cloud server page → Network section → network card.

3. Connect a network and subnet for the file storage to the router

You need to connect a network and subnet to the global router for the project and pool where the file storage will be created later.

You can connect a new network to the router or an existing network if it is not already connected to any of the account's global routers.

Connect a new network

1. In the control panel, on the top menu, click Products and select Global Router.

2. Open the router page → Networks.

3. Click Create network.

4. Enter a network name. It will only be used in the control panel.

5. Select the Cloud Platform service.

6. Select the location where the file storage will be created.

7. Select the project in which the file storage will be created.

8. Enter a subnet name. It will only be used in the control panel.

9. Enter the CIDR — the IP address and subnet mask. The subnet must meet the following conditions:

   • belong to the RFC 1918 private address range: 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16;
   • have a size of at least /29, as three addresses will be occupied by Servercore network equipment;
   • do not overlap with other subnets added to this router — there must be no identical IP addresses in the subnets of one router;
   • if a Managed Kubernetes cluster on cloud servers is included in the global router network, the subnet must not overlap with the 10.10.0.0/16, 10.96.0.0/12, 10.250.0.0/16 and 10.251.0.0/24. If a cluster on dedicated servers is included in the network — with the 10.10.0.0/16, 10.222.0.0/16, 10.250.0.0/16, 10.251.0.0/24 and 172.250.0.0/14. These subnets are used for internal Managed Kubernetes addressing; using them may lead to network conflicts in the global router.

10. Enter the gateway IP or leave the first address from the subnet that is assigned by default. Do not assign this address to your devices so as not to disrupt network operation.

11. Enter the service IPs or leave the last addresses from the subnet that are assigned by default. Do not assign these addresses to your devices so as not to disrupt network operation.

12. Click Create network.

13. Optional: check the network topology on the global router. In the control panel, on the top menu, click Products and select Global Router. Open the router page and click Network map.

Connect an existing network

1. Check that the network has not been added to any of the account's global routers yet. In the control panel, on the top menu, click Products → Cloud Servers → Network → Private networks tab → make sure the network card does not have a Global Router tag.

2. Ensure that the subnet meets the following conditions:

   • belongs to the RFC 1918 private address range: 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16;
   • is at least /29, as three addresses will be occupied by Servercore network equipment;
   • does not overlap with other subnets added to this router: the IP addresses of each subnet on the router must not overlap with the IP addresses of other subnets on the router;
   • if a Managed Kubernetes cluster on cloud servers is included in the global router network, the subnet must not overlap with the 10.10.0.0/16, 10.96.0.0/12, 10.250.0.0/16 and 10.251.0.0/24. If a cluster on dedicated servers is included in the network — with the 10.10.0.0/16, 10.222.0.0/16, 10.250.0.0/16, 10.251.0.0/24 and 172.250.0.0/14. These subnets are used for internal Managed Kubernetes addressing; using them may lead to network conflicts in the global router.

3. In the control panel, on the top menu, click Products and select Cloud Servers.

4. Go to Network → Private networks.

5. In the network menu, select Connect to Global Router.

6. Select a global router.

7. For each of the subnets, enter the IP gateway or leave the first free address from the subnet that is assigned by default. Do not assign this address to your devices so as not to disrupt network operation. The last two free subnet addresses will be reserved as service addresses.

8. Click Connect. Do not close the window until you see a message stating that the network is connected. Then, in the control panel:

   • the network will appear in the Servercore Global Router section on the page of the router you connected it to. You can check this in the control panel: on the top menu, click Products → Global Router → router page;
   • a Global Router tag will appear on the network. You can check this in the control panel: on the top menu, click Products → Cloud Servers → cloud server page → Network section → network card.

4. Assign an IP address to a Managed Kubernetes cluster node

Configure a local port on the Managed Kubernetes cluster node that is included in the global router network. Assign an IP address to the port from the subnet that you connected to the global router in step 2.

1. Add the Managed Kubernetes cluster node to the subnet that you connected to the global router in step 2. If you do not have a Managed Kubernetes cluster yet, create one. When creating the cluster, select the subnet you connected to the global router in step 2.

2. Apply changes depending on the Apply changes parameter in the Port settings block. You can check the parameter value in the Control panel: in the top menu Products → Cloud Servers → cloud server page → Ports:

   • Upon server reboot — reboot the node programmatically or manually make changes to the network configuration file on the node;
   • Manually in the network configuration file on the server — manually make changes to the network configuration file on the node.

5. Add a route on the Managed Kubernetes cluster node

You need to add a static route to the file storage on each cluster node. To do this, in the Managed Kubernetes cluster subnet that you connected to the global router in step 2, add a static route to the file storage subnet that you connected to the global router in step 3.

Use the Configure static routes in a subnet section of the Static Routes guide.

6. Add a route in the file storage subnet

In the file storage subnet that you connected to the global router in step 3, you need to add a static route to the Managed Kubernetes cluster subnet that you connected to the global router in step 2.

Use the Configure static routes in a subnet section of the Static Routes guide.

7. Create file storage

Control panel

1. In the Control panel, on the top menu, click Products and select File Storage.

2. Click Create Storage.

3. Enter a storage name or leave the name that is automatically created.

4. Select a location where the storage will be created.

   If you need to increase the disk space of a cloud server or Managed Kubernetes cluster using file storage, select the location where the server or cluster is located. To increase the disk space of dedicated servers, file storage can be created in any location.

   If you plan to use storage for backups, we recommend choosing a location different from the main infrastructure location to improve fault tolerance.

5. Fill in the blocks:

   • Subnet
   • Settings
   • Access rules

6. Check the price of the file storage.

7. Click Create.

Subnet

1. Select a private subnet for the file storage that you connected to the global router in step 3. You will not be able to change the subnet after the storage is created.

2. Enter a private IP address for the storage or leave the first available address from the subnet, which is assigned by default. You will not be able to change the IP address after the storage is created.

Settings

1. Select a file storage type:

   • HDD Basic,
   • SSD Universal,
   • SSD Fast.

   Once created, the storage type cannot be changed.

2. Specify the storage size: from 50 GB to 50 TB. After creation, you can increase file storage, but you cannot decrease it.

3. Select a protocol:

   • NFSv4 — for connecting storage to servers running Linux and other Unix systems;
   • CIFS SMBv3 — for connecting storage to Windows servers.

   Once created, the protocol cannot be changed.

Access rules

NFSv4

1. Configure access rules for the file storage:

   • accessible by everyone — the storage will be accessible from any IP address in the private subnet where it is created;
   • access restricted — the storage will be accessible only to specific IP addresses or private subnets. If you create file storage without rules, access will be restricted for all IP addresses.

2. If you selected the Access restricted option, click Add rule.

3. Enter the IP address or CIDR of the private subnet and select the access level.

   After the storage is created, you can configure new access rules.

CIFS SMBv3

1. Configure access rules for the file storage:

   • accessible by everyone — the storage will be accessible from any IP address in the private subnet where it is created;
   • access restricted — the storage will be accessible only to specific IP addresses or private subnets. If you create file storage without rules, access will be restricted for all IP addresses.

2. If you selected the Access restricted option, click Add rule.

3. Enter the IP address or CIDR of the private subnet.

   After the storage is created, you can configure new access rules.

OpenStack CLI

1. Open the OpenStack CLI.

2. Create a file storage network:

   openstack share network create \
     --name <share_network> \
     --neutron-net-id <net_uuid> \
     --neutron-subnet-id <subnet_uuid>

   Specify:

   • <share_network> — the file storage network name;
   • <net_uuid> — the ID of the private or public network to which the file storage will be connected. You can view the list of networks using the openstack network list; command;
   • <subnet_uuid> — the subnet ID. You can view the list of subnets using the openstack subnet list. command.

3. Create the file storage:

   openstack share create <share_protocol> <share_size>\
     --name <share_name> \
     --share-network <share_network_id> \
     --share-type <share_type_name> 

   Specify:

   • <share_protocol> — the file storage connection protocol, NFSv4 or CIFS SMBv3;
   • <share_size> — the file storage size in GB;
   • <share_name> — the file storage name;
   • <share_network_id> — the ID of the file storage network you created in step 2. You can view the list of file storage networks using the openstack share network list; command;
   • <share_type_name> — file storage type. You can view the list of types using the openstack share type list. command.

Terraform

Follow the Create file storage instructions in the Terraform documentation.

8. Mount the file storage to the Managed Kubernetes cluster

The mounting process depends on the file storage protocol: NFSv4 or CIFS SMBv3.

NFSv4

1. Create a PersistentVolume.

2. Create a PersistentVolumeClaim.

3. Add file storage to the container.

1. Create a PersistentVolume

1. Connect to the Managed Kubernetes cluster.

2. Create a yaml file with a manifest for the PersistentVolume object:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv_name
   spec:
     storageClassName: storageclass_name
     capacity:
       storage: <storage_size>
     accessModes:
       - ReadWriteMany
     nfs:
       path: /shares/share-<mountpoint_uuid>
       server: <filestorage_ip_address>

   Specify:

   • <storage_size> — PersistentVolume size in GB (file storage size), for example 100 Gi. The limit is from 50 GB to 50 TB;
   • <mountpoint_uuid> — mount point ID. You can view it in the control panel: from the top menu, click Products → File Storage → storage page → block Connection → tab GNU/Linux;
   • <filestorage_ip_address> — file storage IP address. You can view it in the control panel: from the top menu, click Products → File Storage → storage page → tab Settings → field IP.

3. Apply the manifest:

   kubectl apply -f <persistent_volume.yaml>

   Specify <persistent_volume.yaml> — the name of the yaml file with the manifest to create the PersistentVolume.

4. Make sure the PersistentVolume object is created:

   kubectl get pv

2. Create a PersistentVolumeClaim

1. Create a yaml file with a manifest for the PersistentVolumeClaim object:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc_name
   spec:
     storageClassName: storageclass_name
     accessModes:
       - ReadWriteMany
     resources:
       requests:
         storage: <storage_size>

   Specify <storage_size> — PersistentVolume (file storage) size in GB, for example 100 Gi. The limit is from 50 GB to 50 TB.

2. Apply the manifest:

   kubectl apply -f <persistent_volume_claim.yaml>

   Specify <persistent_volume_claim.yaml> — the name of the yaml file with the manifest to create the PersistentVolumeClaim.

3. Make sure the PersistentVolumeClaim object is created:

   kubectl get pvc

3. Add storage to the container

1. Create a yaml file with a manifest for the Deployment object:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: filestorage_deployment_name
     labels:
       project: filestorage_deployment_name
   spec:
     replicas: 2
     selector:
       matchLabels:
         project: filestorage_project_name
     template:
       metadata:
         labels:
           project: filestorage_project_name
       spec:
         volumes:
           - name: volume_name
             persistentVolumeClaim:
               claimName: pvc_name
         containers:
           - name: container-nginx
             image: nginx:stable-alpine
             ports:
               - containerPort: 80
                 name: "http-server"
             volumeMounts:
               - name: volume_name
                 mountPath: <mount_path>

   Specify <mount_path> — the path to the folder inside the container where the file storage will be mounted.

2. Apply the manifest:

   kubectl apply -f <deployment.yaml>

   Specify <deployment.yaml> — the name of the yaml file with the manifest to create the Deployment.

CIFS SMBv3

1. Install the CSI driver for Samba.

2. Create a secret to store the username and password.

3. Create a StorageClass.

4. Create a PersistentVolumeClaim.

5. Add file storage to the container.

1. Install the CSI driver for Samba

1. Connect to the Managed Kubernetes cluster.

2. Install the Helm package manager.

3. Download the CSI driver from GitHub Kubernetes CSI.

4. Install the latest version of the driver:

   helm repo add csi-driver-smb https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/charts
   helm install csi-driver-smb csi-driver-smb/csi-driver-smb --namespace kube-system --version v1.4.0

5. Check that the pods are installed and running:

   kubectl --namespace=kube-system get pods --selector="app=csi-smb-controller"

2. Create a secret

File storage does not support access rights differentiation. Access via the CIFS SMBv3 protocol is granted under the guest user.

Create a secret to store the username and password (the default is guest/guest):

kubectl create secret generic smbcreds --from-literal username=guest --from-literal password=guest

3. Create a StorageClass

1. Create a yaml file with a manifest for the StorageClass object:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: storageclass_name
   provisioner: smb.csi.k8s.io
   parameters:
     source: "//<filestorage_ip_address>/share-<mountpoint_uuid>"
     csi.storage.k8s.io/provisioner-secret-name: "smbcreds"
     csi.storage.k8s.io/provisioner-secret-namespace: "default"
     csi.storage.k8s.io/node-stage-secret-name: "smbcreds"
     csi.storage.k8s.io/node-stage-secret-namespace: "default"
   reclaimPolicy: Delete
   volumeBindingMode: Immediate
   mountOptions:
     - dir_mode=0777
     - file_mode=0777

   Specify:

   • <mountpoint_uuid> — mount point ID. You can view it in the control panel: from the top menu, click Products → File Storage → storage page → block Connection → tab GNU/Linux;
   • <filestorage_ip_address> — file storage IP address. You can view it in the control panel: from the top menu, click Products → File Storage → storage page → tab Settings → field IP.

2. Apply the manifest:

   kubectl apply -f <storage_class.yaml>

   Specify <storage_class.yaml> — the name of the yaml file with the manifest to create the StorageClass.

3. Make sure the StorageClass object is created:

   kubectl get storageclass

4. Create a PersistentVolumeClaim

1. Create a yaml file with a manifest for the PersistentVolumeClaim object:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc_name
     annotations:
       volume.beta.kubernetes.io/storage-class: smb
   spec:
     accessModes: ["ReadWriteMany"]
     resources:
       requests:
         storage: <storage_size>

   Specify <storage_size> — PersistentVolume (file storage) size in GB, for example 100 Gi. The limit is from 50 GB to 50 TB.

2. Apply the manifest:

   kubectl apply -f <persistent_volume_claim.yaml>

   Specify <persistent_volume_claim.yaml> — the name of the yaml file with the manifest to create the PersistentVolumeClaim.

3. Make sure the PersistentVolumeClaim object is created:

   kubectl get pvc

5. Add storage to the container

1. Create a yaml file with a manifest for the Deployment object:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: filestorage_deployment_name
     labels:
       project: filestorage_deployment_name
   spec:
     replicas: 2
     selector:
       matchLabels:
         project: filestorage_project_name
     template:
       metadata:
         labels:
           project: filestorage_project_name
       spec:
         volumes:
           - name: volume_name
             persistentVolumeClaim:
               claimName: pvc_name
         containers:
           - name: container-nginx
             image: nginx:stable-alpine
             ports:
               - containerPort: 80
                 name: "http-server"
             volumeMounts:
               - name: volume_name
                 mountPath: <mount_path>

   Specify <mount_path> — the path to the folder inside the container where the file storage will be mounted.

2. Apply the manifest:

   kubectl apply -f <deployment.yaml>

   Specify <deployment.yaml> — the name of the yaml file with the manifest to create the Deployment.