A three-node cluster with Apache Kafka

Apache Kafka is a distributed software message broker for real-time data stream processing from The Apache Software Foundation. The broker allows you to collect, store, and process large volumes of data from various sources and transfer them to data processing and analysis systems. Apache Kafka uses a three-node distributed cluster architecture for fault tolerance.

You can create a three-node cluster with a pre-installed Apache Kafka application. In Russia, the cluster nodes run on a configured SelectOS 1 64-bit operating system. In other countries, they run on Ubuntu 24.04.

Before creating a cloud server with an application, review the software license agreements included in the image.

Minimum resource requirements

	Test environment	Production environment
Number of vCPUs	4	8
RAM	8 GB	32 GB
Boot volume	16 GB	from 16 GB *

* The boot disk size for a production environment depends on your needs.

Create a three-node cluster with Apache Kafka

In Servercore, to create a cluster, you must first create a temporary cloud server with Apache Kafka. Creating a temporary server initiates the creation of a three-node cluster. Each node is a cloud server with the configuration selected when creating the temporary cloud server.

The name of each server in the cluster will follow the format cs-<server_name>-<cluster_number>, where <server_name> is the server name, and <cluster_number> is the server number in the cluster from 1 to 3.

The three created cloud servers are in the same subnet as the temporary cloud server.

Next, a Kafka cluster will be configured on three cloud servers. Once the cluster is configured, the temporary cloud server is deleted. If the temporary cloud server was not deleted, an issue occurred while configuring the Kafka cluster. To identify the problem, view the logs.

The cluster cost will be calculated as the cost of three cloud servers. Learn more about the Cloud Platform billing model in the article Billing model and prices.

To configure Apache Kafka, when creating a server, specify the application parameters used to connect to the OpenStack API.

1. Create a service user.

2. Create a public floating IP address.

3. Create a cloud server with Apache Kafka.

1. Create a service user

Create a service user, select the member or reader role in the permission, and the Projects scope. Users can be created by the Account owner or users with the iam.admin.

2. Create a public floating IP address

Create a public floating IP address so the cloud server with Zabbix is accessible from the internet.

Use the Create public floating IP address subsection of the instructions Public floating IP addresses.

3. Create a cloud server with Apache Kafka

1. In the Control panel, from the top menu, click Products and select Cloud Servers.

2. Click Create server.

3. Fill in the blocks:

   • Name and placement;
   • Source;
   • Configuration;
   • Disks;
   • Internet;
   • Private network;
   • Security;
   • Access;
   • Additional settings.

4. Check the cloud server price. Note that after configuration, the price for a three-node cluster will be calculated using the formula: cloud server price * 3.

5. Click Create.

Name and placement

1. Enter the server name. It will be set as the hostname in the operating system.

2. Select the location where the server will be created. The list of available server configurations and resource costs depends on the location. You cannot change the location after the server is created.

Source

1. Open the tab Applications.

2. Select Cloud Kafka.

3. Optional: if you need a different current or archived application version, in the Version field, select the desired version.

4. Fill in the application parameters:

   4.1. Enter the service user name. You can find it in the Control panel: in the top menu, click IAM and go to the Service Users section (the section is only available to the Account owner and a user with the iam.admin role).

   4.2. Enter the service user password. You can view it when creating a user or change it to a new one.

   4.3. Enter the project ID. You can copy it in the Control panel: in the top menu, click Products → Cloud Servers → open the project menu → in the project row, click .

   4.4. Specify the pool where the cloud server will be created, for example, ru-3.

   4.5. Specify the pool segment where the cloud server will be created, for example, ru-3b.

   4.6. Enter the account number in the control panel. You can view it in the Control panel in the upper-right corner.

Configuration

Select a configuration with at least 2 vCPUs and at least 8 GB RAM if the server is needed for development and testing. If the server is needed for a moderate workload in a production environment, select a configuration with at least 8 vCPUs and at least 32 GB RAM. The boot volume size for the production environment depends on your needs.

For all product lines except Shared and Dedicated, two types of server configurations are available:

• fixed configurations — product line configurations with different technical specifications and fixed resource ratios;
• custom configurations — configurations in which any resource ratio can be specified.

Configurations use different processors depending on the product line and pool segment. You can customize the selected configuration. After the server is created, you can change the configuration.

Fixed configuration

1. Open the tab with the line.

2. Click Fixed.

3. Optional: you can customize the configuration if you are creating a server in a segment of the multi-zone pool ru-6 or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select the CPU manufacturer. Selecting a manufacturer is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be assigned to the cloud server vCPUs, uncheck the Dedicated cores checkbox. Learn more in the instructions Dedicated cores.

   3.4. Optional: if you want to disable Hyper-Threading for a server with dedicated cores, uncheck Hyper-Threading (SMT).

   3.5. Optional: if you are creating a server with dedicated cores and want to place a multi-processor server on a single NUMA node, select the Mandatory placement on a single NUMA node checkbox. A server with 4 vCPUs or more can be placed on a single NUMA node. If the cloud server resources cannot be placed on a single node, it will not be created. Learn more in the Placement on a single NUMA node subsection of the instructions Dedicated cores.

4. Select a configuration.

5. If both local and network volumes are available in the selected configuration, select the volume to be used as the boot volume:

   • local disk — select the Local NVMe SSD checkbox. A server with a local disk can only be created from images and applications;
   • network drive — do not select the checkbox Local NVMe SSD.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves part of the RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the command sudo dmesg | grep Memory.

Custom configuration

1. Open the tab with the line.

2. Click Custom.

3. Optional: you can customize the configuration if you are creating a server in a segment of the multi-zone pool ru-6 or pool segments ru-3b, ru-7a, and ru-7b:

   3.1. Expand the block with the configuration settings description.

   3.2. Optional: select the CPU manufacturer. Selecting a manufacturer is not available in all pools.

   3.3. Optional: if you do not want physical processor cores to be assigned to the cloud server vCPUs, uncheck the Dedicated cores checkbox. Learn more in the instructions Dedicated cores.

   3.4. Optional: if you want to disable Hyper-Threading for a cloud server with dedicated cores, uncheck Hyper-Threading (SMT).

   3.5. Optional: if you are creating a cloud server with dedicated cores and want to place a multi-processor server on a single NUMA node, select the Mandatory placement on a single NUMA node checkbox. A cloud server with 4 vCPUs or more can be placed on a single NUMA node. If the cloud server resources cannot be placed on a single node, it will not be created. Learn more in the Placement on a single NUMA node subsection of the instructions Dedicated cores.

4. Specify the number of vCPUs.

5. Specify the RAM size.

6. If you need to add GPUs to the server, click Add GPU, select the GPU type, and specify the number of GPUs. The selected product line will change to the GPU line. If you selected an image without GPU optimization in the Source section and the Auto-select GPU image toggle is turned on, the selected image will automatically be replaced with a GPU-optimized image.

7. If both local and network volumes are available in the selected configuration, select the volume to be used as the boot volume:

   • local disk — select the Local NVMe SSD checkbox. A server with a local disk can only be created from images and applications;
   • network drive — do not select the checkbox Local NVMe SSD.

   The amount of RAM allocated to the server may be less than specified in the configuration — the operating system kernel reserves part of the RAM depending on the kernel version and distribution. You can check the allocated amount on the server using the command sudo dmesg | grep Memory.

Volumes

1. If you did not check the Local NVMe SSD checkbox when setting up the configuration, the first specified network drive will be used as the server boot disk. To configure it:

   1.1. Select the network boot drive type.

   1.2. Specify the network boot drive size in GB or TB. Take into account the network drive limits on the maximum size.

   1.3. If you selected the Universal v2 or SSD Fast v2 drive type, specify the total number of read and write operations in IOPS. After creating the drive, you can change the number of IOPS by increasing or decreasing it. The number of IOPS changes is unlimited.

2. Optional: add an additional server network drive:

   2.1. Click Add.

   2.2. Select the network drive type.

   2.3. Specify the network drive size in GB or TB. Take into account the network drive limits on the maximum size.

   2.4. If you selected the Universal v2 or SSD Fast v2 drive type, specify the total number of read and write operations in IOPS. After creating the drive, you can change the number of IOPS by increasing or decreasing it. The number of IOPS changes is unlimited.

   After the server is created, you can attach new additional drives.

Internet

Configure public access to the server.

The cloud server will be added to a private subnet connected to a cloud router with 1:1 NAT and internet access. Internet access to and from the server will go through the cloud router. The server will be accessible from the internet via a public floating IP address.

1. In the Internet connection field, select the access type Public floating IP address.

2. Select the public floating IP address you created in Step 2.

Private network

A cloud server can be added to an existing or new private subnet.

Existing private subnet

1. In the Subnet field, select a private subnet.

2. Optional: in the IP address field, change the default IP address.

3. In the Router field, select an existing router or create a new one.

   If the router is not connected to the internet, it will be automatically connected after the server is created.

New private subnet

1. In the Subnet field, select New private subnet.

2. In the Network field, select an existing network or create a new one.

3. Optional: change the subnet CIDR.

4. Optional: in the IP address field, change the default IP address.

5. In the Router field, select an existing router or create a new one.

   If the router is not connected to the internet, it will be automatically connected to the internet after the server is created.

Security

Select security groups to filter traffic on the server ports. Without security groups, traffic will be denied. If the block is not present, traffic filtering (port security) is disabled in the server network. With traffic filtering disabled, all traffic will be allowed.

Access

1. Place an SSH key for the project on the server for secure connection:

   1.1. If the SSH key for the project has not been added to the Cloud Platform, click Add SSH key, enter a key name, paste the public key in OpenSSH format, and click Add.

   1.2. If the SSH key for the project has been added to the Cloud Platform, in the SSH key field, select an existing key. The SSH key is only available in the pool where it was placed.

2. Optional: in the Password for "root":

   2.1. Copy the password for the root user — a user with unlimited permissions for all actions in the system.

   2.2. Save the password in a secure place and do not transfer it in plain text.

Additional settings

1. Optional: if you plan to create multiple servers and want to increase infrastructure fault tolerance, add the server to a placement group:

   1.1. To create a new group, in the Placement group field, click Create.

   1.2. Select New group and enter the group name.

   1.3. Select the placement policy across different hosts:

   • preferred — soft-anti-affinity. The system will try to place servers on different hosts. If there is no suitable host when creating the server, it will be created on the same host;
   • mandatory — anti-affinity. Servers in the group must be located on different hosts. If there is no suitable host when creating the server, the server will not be created.

   1.4. If the group has been created, in the Placement group field, select the placement group.

2. Optional: to add additional information or filter servers in the list, add server tags. Operating system and configuration tags are added automatically. To add a new tag, in the Tags field, enter the tag.

3. To add a script that will run using the cloud-init agent at the first operating system startup, in the Automation section, in the field User data:

   • open the Text tab and paste the script as text;
   • or open the File tab and upload the script file.

View logs

If the temporary cloud server was not deleted, there was an issue during Apache Kafka cluster configuration. For example, the service user does not have sufficient permissions, an incorrect password was entered, or the user data settings contain errors.

1. Connect to the server you created earlier.

2. To view the error log, open the cloud-init-output.log file in the text editor vi:

   vi /var/log/cloud-init-output.log

3. Depending on the error log, grant access to the service user or change the user data settings.

4. Recreate the server.

5. If you cannot solve the problem on your own, create a ticket and attach the error logs.