Skip to main content

Cloud server with ELK

ELK (Elasticsearch, Logstash, Kibana) is a set of open source tools for analyzing and visualizing logs and other data.

In Servercore, a cloud server with ELK contains:

  • Elasticsearch is an open source distributed data management and search system based on the Apache Lucene search engine. The system provides powerful capabilities for indexing, storing, searching and analyzing large amounts of information in real time;
  • Kibana is a web interface for monitoring, analyzing Elasticsearch logs and presenting data on dashboards;
  • Filebeat is an agent for collecting logs and passing them to Elasticsearch;
  • Metricbeat is an agent for collecting metrics and passing them to Elasticsearch.

It is possible to create a cloud server with a ready-made ELK application.In Russia, the server works with a customized operating system SelectOS 1.1. In other countries — Ubuntu 22.04.

Create a cloud server with ELK

For ELK to work, the cloud server must be accessible from the Internet.For this purpose, when creating the server, create a private subnet and connect a public IP address.To configure ELK, also when creating the server, specify user data — user configuration parameters of the operating system.

After creating a server with ELK, a free TLS certificate from Let's Encrypt® is automatically issued for the domain you specify.To issue the certificate, you must add an A-record for the domain and specify the public IP address of the server in the record value.The domain can be added to Servercore's DNS hosting (actual).

  1. Create a public IP address.

  2. Add an A-record for the domain.

  3. Create a cloud server with ELK.

1. Create a public IP address

Create a public IP address so that the cloud server with ELK can be accessed from the internet.

Use the Create a Public IP Address subsection of the Public IP Addresses instruction.

2. Add an A-record for the domain

Add a resource record to access the ELK by domain.

Specify:

3. Create a cloud server with ELK

  1. In the Dashboard, on the top menu, click Products and select Cloud Servers.

  2. Click Create Server.

  3. Fill in the blocks:

  4. Check the price of the cloud server.

  5. Click Create.

Name and location

  1. Enter the server name. This will be set as the host name in the operating system.

  2. Select the region and pool segment in which the server will be created.The list of available server configurations and resource costs depends on the pool segment.Once the server is created, you cannot change the pool segment.

Source

  1. Open the Applications tab.

  2. Select Cloud ELK.

  3. Optional: if you need another current or archived version of the application, select the desired version in the Version field.

Configuration

Select a configuration from 4 vCPU, RAM от 10 ГБ и размером загрузочного диска от 30 ГБ.Two types of configurations are available for the server:

The configurations use different processors depending on the line and pool segment.After the server is created, you can change the configuration.

  1. Click Fixed.

  2. Open the tab with the desired ruler.

  3. Select a configuration.

  4. If both local and network disks are available in the selected configuration, select the disk to be used as the boot disk:

    • local disk — check the box Local SSD NVMe disk. A server with a local disk can only be created from images and applications;
    • network drive — do not check the checkbox.

    The amount of RAM allocated to the server may be less than the amount specified in the configuration — the operating system kernel reserves some RAM depending on the kernel version and distribution. You can check the allocated amount on the server with sudo dmesg | grep Memory.

Disks

  1. If you did not check the Local SSD NVMe disk checkbox when setting up the configuration. the first specified network drive will be used as the server boot disk.To configure it:

    1.1 Select the type of network boot disk.

    1.2 Specify the size of the network boot disk in GB or TB. Observe the maximum size limits of network disks.

    1.3 If you selected the Universal v2 disk type, specify the total number of read and write operations in IOPS. After creating a disk, you can change the number of IOPS — decrease or increase.The number of IOPS changes is unlimited.

  2. Опционально: добавьте дополнительный network disk server :

    2.1 Click Add.

    2.2 Select the type of network drive.

    2.3. Укажите размер сетевого диска в ГБ или ТБ. consider the maximum size limits of network drives.

    2.4 If you selected the Universal v2 disk type, specify the total number of read and write operations in IOPS. After creating a disk, you can change the number of IOPS — decrease or increase. There is no limit to the number of IOPS changes.

    After the server is created, you can connect new additional disks.

Network

The server can be added to a new private subnet or to an existing one.

Private — A subnet without access from the Internet or with a single static public IP address.The public IP address connects to a server that will be accessible from the Internet.

  1. Click Private Subnet.

  2. In the Public IP address for Internet access field, select the public IP address you created earlier or create a new one.

  3. Expand the block with private subnet settings.

  4. In the Subnet field, select an existing subnet.

  5. In the Private IP field, specify the private IP address of the server. The public IP address will be automatically connected to the private address.

Safety

Select security groups to filter traffic on server ports. Without security groups, traffic will not be allowed. If there is no block, port security is disabled on the server network . With traffic filtering disabled, all traffic will be allowed.

Access

  1. Place an SSH key for the project on the server for secure connection:

    1.1 If the SSH key for the project is not added to the cloud platform, click Add SSH Key, enter the key name, insert the public key in OpenSSH format, and click Add.

    1.2. If an SSH key for the project is added to the cloud platform. select the existing key in the SSH key field.SSH key is available only in the pool in which it is hosted.

  2. Optional: in the Password field for "root":

    2.1 Copy the password of the root user — a user with unlimited rights to all actions on the system.

    2.2 Save the password in a safe place and do not transmit it in public.

Additional settings

  1. If you plan to create multiple servers and want to improve the fault tolerance of your infrastructure, add the server to a placement group:

    1.1 To create a new group, in the Placement Group field, click New.

    1.2. Select New Group and enter a group name.

    1.3 Select a policy for hosting on different hosts:

    • preferably soft-anti-affinity. The system will try to place servers on different hosts. If there is no suitable host when creating a server, it will be created on the same host;
    • anti-affinity is mandatory. Servers in a group must be located on different hosts. If there is no suitable host when creating a server, the server will not be created.

    1.4 If a group has been created, select the placement group in the Placement Group field.

  2. To add additional information or filter servers in the list, add server tags. Operating system and configuration tags are automatically added. To add a new tag, enter a tag in the Tags field.

  3. To add a script that will be executed by the cloud-init agent when the operating system first starts up, in the Automation block in the User data field:

    • open the Text tab and paste the script with text;
    • or open the File tab and upload the file with the script.
#cloud-config

write_files:
- path: "/opt/gomplate/values/user-values.yaml"
  permissions: "0644"
  content: |   
    username: "<kibana_username>"
    password: "<kibana_password>"
    domain: "<example.com>"
    email: "<root@example.com>"

Specify:

  • <kibana_username> — the administrator name for logging in to Kibana. You cannot use the name elastic — it is reserved by the system;
  • <kibana_password> — password of the Kibana administrator;
  • <example.com> — the domain to access the ELK that you added in step 2;
  • <root@example.com> — Kibana administrator email to create an account and receive Let's Encrypt® notifications.

Install Filebeat on monitored servers

Filebeat is installed on servers from which you want to collect data for monitoring. Filebeat collects log messages from log files and forwards them to a cloud server with Elasticsearch for indexing.

  1. Create a Kibana user with the necessary rights and roles in the ELK web interface. See Create a user documentation of Elastic for more details.

  2. Install Filebeat. See Install Filebeat documentation Elastic for more information.

  3. Download a public CA certificate from a cloud server to connect Filebeat over HTTPS protocol:

    cd /etc/filebeat && wget https://<example.com>/ca.crt

    Specify <example.com> is the domain to access Elasticsearch that you added in step 2..

  4. Open the configuration file /etc/filebeat/filebeat/filebeat.yml in a text editor.

  5. Add connection parameters to Elasticsearch in the configuration file:

    outputs:
      default:
        type: elasticsearch
        hosts: [https://<example.com>:9200]
        username: "<username>"
        password: "<password>"
        ssl.enabled: true
        ssl.certificate_authorities: "/etc/filebeat/ca.crt"

    Specify:

    • <example.com> — the domain to access the ELK that you added in step 2;
    • <username> — the username of the Kibana user you created in step 1;
    • <password> — password of the Kibana user you created in step 1.

  6. Restart the agent:

    systemctl restart filebeat

Install Elastic Agent on the monitored servers

Elastic Agent is installed on servers from which you want to collect metrics, logs, and security events. The agent collects the information and forwards it to the cloud server with Elasticsearch.

  1. Create a Kibana user with the necessary rights and roles in the ELK web interface. See Create a user documentation of Elastic for more details.

  2. Install Elastic Agent. See Install Elastic Agents in the Elastic documentation for more information.

  3. Download the public CA certificate from the cloud server to connect Elastic Agent over HTTPS protocol:

    cd /etc/elastic-agent && wget https://<example.com>/ca.crt

    Specify <example.com> is the domain to access Elasticsearch that you added in step 2..

  4. Open the configuration file /etc/elastic-agent/elastic-agent.yml in a text editor.

  5. Add connection parameters to Elasticsearch in the configuration file:

    outputs:
      default:
        type: elasticsearch
        hosts: [https://<example.com>:9200]
        username: "<username>"
        password: "<password>"
        ssl.enabled: true
        ssl.certificate_authorities: "/etc/elastic-agent/ca.crt"

    Specify:

    • <example.com> — the domain to access the ELK that you added in step 2;
    • <username> — the username of the Kibana user you created in step 1;
    • <password> — password of the Kibana user you created in step 1.

  6. Restart the agent:

    systemctl restart elastic-agent