Skip to main content

Cluster of three nodes with Apache Kafka

Apache Kafka is a distributed software message broker for processing real-time data streams from The Apache Software Foundation. The broker allows you to collect, store, and process large amounts of data from a variety of sources and feed it into data processing and analysis systems.Apache Kafka uses a distributed cluster architecture with three nodes for resilience to failures.

It is possible to create a cluster of three nodes with a ready-made Apache Kafka application.In Russia, the cluster nodes work with the configured SelectOS 1.1 operating system .In other countries — Ubuntu 24.04.

In Servercore, to create a cluster, you must first create a temporary cloud server with Apache Kafka.Creating a temporary server initiates the creation of a cluster of three nodes.Each node is a cloud server with the configuration that was selected when the temporary cloud server was created.

The name of each server in the cluster will be in the format cs-<server_name>-<cluster_number>, where <server_name> is the server name and <cluster_number> is the number of the server in the cluster from 1 to 3.

The three cloud servers created are on the same subnet as the temporary cloud server.

Next, a Kafka cluster will be configured on the three cloud servers.If the cluster is configured, the temporary cloud server is removed.If the temporary cloud server is not removed, there was a problem when configuring the Kafka cluster.To identify the problem, look at the logs.

The cost of the cluster will be calculated as the cost of three cloud servers.Read more about the cloud platform payment model in the article Payment Model and Pricing.

Create a cluster of three nodes with Apache Kafka

To configure Apache Kafka at creation, you must specify user data — user configuration parameters that are used to connect to the OpenStack API.

  1. Create a service user.

  2. Create a cloud server with Apache Kafka.

1. Create a service user

Create a service user In the permission, select the role member or reader and access area Projects.Create users can Account Owner or users with the role iam_admin.

2. Create a cloud server with Apache Kafka

  1. In the Dashboard, on the top menu, click Products and select Cloud Servers.

  2. Click Create Server.

  3. Fill in the blocks:

  4. Check the price of the cloud server. Note that once configured, the price for a three-node cluster will be calculated using the formula: cloud server price * 3.

  5. Click Create.

Name and location

  1. Enter the server name. This will be set as the host name in the operating system.

  2. Select the region and pool segment in which the server will be created.The list of available server configurations and resource costs depends on the pool segment.Once the server is created, you cannot change the pool segment.

Source

  1. Open the Applications tab.

  2. Select Cloud Kafka.

  3. Optional: if you need another current or archived version of the application, select the desired version in the Version field.

Configuration

Choose a configuration of 2 vCPUs and RAM of 8 GB if the server is needed for development and testing.If the server is needed for moderate workloads in a productive environment, choose a configuration of 8 vCPUs and RAM of 32 GB.Two types of configurations are available for the server:

The configurations use different processors depending on the line and pool segment.After the server is created, you can change the configuration.

  1. Click Fixed.

  2. Open the tab with the desired ruler.

  3. Select a configuration.

  4. If both local and network disks are available in the selected configuration, select the disk to be used as the boot disk:

    • local disk — check the box Local SSD NVMe disk. A server with a local disk can only be created from images and applications;
    • network drive — do not check the checkbox.

    The amount of RAM allocated to the server may be less than the amount specified in the configuration — the operating system kernel reserves some RAM depending on the kernel version and distribution. You can check the allocated amount on the server with sudo dmesg | grep Memory.

Disks

  1. If you did not check the Local SSD NVMe disk checkbox when setting up the configuration. the first specified network drive will be used as the server boot disk.To configure it:

    1.1 Select the type of network boot disk.

    1.2 Specify the size of the network boot disk in GB or TB. Observe the maximum size limits for network disks.

    1.3 If you selected the Universal v2 disk type, specify the total number of read and write operations in IOPS. After creating a disk, you can change the number of IOPS — decrease or increase.The number of IOPS changes is unlimited.

  2. Optional: add an additional server network disk:

    2.1 Click Add.

    2.2 Select the type of network drive.

    2.3 Specify the size of the network disk in GB or TB. Observe the maximum size limits of the network disks.

    2.4 If you selected the Universal v2 disk type, specify the total number of read and write operations in IOPS. After creating a disk, you can change the number of IOPS — decrease or increase. There is no limit to the number of IOPS changes.

    After the server is created, you can connect new additional disks.

Network

The server can be added to a new private subnet or to an existing one.

Private — A subnet without access from the Internet or with a single static public IP address.The public IP address connects to a server that will be accessible from the Internet.

  1. Click Private Subnet.

  2. Expand the block with private subnet settings.

  3. In the Subnet field, select an existing subnet that is connected to the cloud router with access to the external network.

  4. In the Private IP field, specify the private IP address of the server.

Safety

Select security groups to filter traffic on server ports. Without security groups, traffic will not be allowed. If there is no block, port security is disabled on the server network . With traffic filtering disabled, all traffic will be allowed.

Access

  1. Place an SSH key for the project on the server for secure connection:

    1.1 If the SSH key for the project is not added to the cloud platform, click Add SSH Key, enter the key name, insert the public key in OpenSSH format, and click Add.

    1.2. If an SSH key for the project is added to the cloud platform. select the existing key in the SSH key field.SSH key is available only in the pool in which it is hosted.

  2. Optional: in the Password field for "root":

    2.1 Copy the password of the root user — a user with unlimited rights to all actions on the system.

    2.2 Save the password in a safe place and do not transmit it in public.

Additional settings

  1. If you plan to create multiple servers and want to improve the fault tolerance of your infrastructure, add the server to a placement group:

    1.1 To create a new group, in the Placement Group field, click New.

    1.2. Select New Group and enter a group name.

    1.3 Select a policy for hosting on different hosts:

    • preferably soft-anti-affinity. The system will try to place servers on different hosts. If there is no suitable host when creating a server, it will be created on the same host;
    • anti-affinity is mandatory. Servers in a group must be located on different hosts. If there is no suitable host when creating a server, the server will not be created.

    1.4 If a group has been created, select the placement group in the Placement Group field.

  2. To add additional information or filter servers in the list, add server tags. Operating system and configuration tags are automatically added. To add a new tag, enter a tag in the Tags field.

  3. To add a script that will be executed by the cloud-init agent when the operating system first starts up, in the Automation block in the User data field:

    • open the Text tab and paste the script with text;
    • or open the File tab and upload the file with the script.
    #cloud-config

    write_files:
    - path: "/opt/gomplate/values/user-values.yaml"
      permissions: "0644"
      content: |
        ServiceUsername: <username>
        ServicePassword: <password>
        ProjectId: <project_id>
        CloudRegion: <pool>
        CloudZone: <pool_segment>
        AccountNumber: <account_id>

    Specify:

    • <username> — service user name. You can look it up in the control panel: in the top menu, click Account and go to the section Users → tab Service Users (the section is available only to the Account Owner and the user with the role of iam_admin);
    • <password> — service user password. You can view it when creating a user or change it to a new one. change it to a new one.
    • <project_id> — Project ID.Can be copied in control panel: in the top menu click Products → Cloud Servers → open the projects menu → in the project line, click ;
    • <pool> — pool in which the cloud server will be created, e.g. ru-3;
    • <pool_segment> — pool segment in which the cloud server will be created, e.g. ru-3b;
    • <account_id> — account number in the control panel.Can be viewed in the control panel in the upper right corner.

View logs

If the temporary cloud server is not removed, there is a problem when configuring the Apache Kafka cluster, for example, the service user does not have sufficient rights, the password is incorrect, or the settings in user data are incorrect.

  1. Connect to the server that you you created earlier.

  2. To see the error log, open the cloud-init-output.log file in a vi text editor:

    vi /var/log/cloud-init-output.log

  3. Depending on the error log, grant service user access or change the user data settings.

  4. Re-create the server.

  5. If you cannot fix the problem yourself, create a ticket and attach logs with errors.