Monitoring a Kafka cluster

In Kafka Managed Databases, you can track the status of the cluster.

To evaluate the overall status of the cluster, view its status.

For a more detailed analysis, you can:

• view the status of cluster nodes — as charts in the control panel;
• export cluster node metrics in Prometheus format.

When analyzing charts, note that the time in the Control Panel corresponds to the time on your device and does not depend on the region where the cluster is located.

note

For example, you have created a cluster in Tashkent, in the uz-1 pool. Tashkent is in the UTC+5 time zone. On the device from which you accessed the control panel, the UTC+3 time zone is set. The time on the metrics charts will be displayed in UTC+3.

View cluster status

1. In the control panel, on the top menu click Products and select Managed Databases.

2. Open the Active tab.

3. View the status in the cluster row.

   ACTIVE	Cluster is available
   CREATING	Cluster is being created
   UPDATING	Cluster is being updated
   RESIZING	Cluster is scaling
   ERROR	An error occurred, create a ticket
   DISK FULL	The disk is full, the cluster is working in read-only mode. To make the cluster work in read-write mode, free up disk space or scale the cluster and choose a configuration with a larger disk size
   DEGRADED	Some cluster nodes are unavailable
   DELETING	Cluster is being deleted

View cluster node status

1. In the control panel, on the top menu click Products and select Managed Databases.
2. Open the Active tab.
3. Open the cluster page → Monitoring tab.
4. In the Cluster monitoring block, view the available cluster node metrics.

Cluster node metrics in the control panel

Memory	Memory used excluding operating system cache and buffers, in percent or gigabytes
vCPU	Percentage of cluster node core utilization
CPU iowait	Percentage of time the processor spent waiting for I/O operations
Disk	Occupied disk space in percentages or gigabytes. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more information about reserving disk space, see the instructions Using disk space in a Kafka cluster
Load Average	Average system load over a period of time.Shows how many processes are being handled by the cluster cores.The metric is presented as three values — for one minute, five minutes, and 15 minutes.These values should not exceed the number of cores on the node
OOM	number of processes that finished with an Out of Memory error due to lack of RAM
Disk load	Read and write speed in KB/s or number of read/write operations per second
Network load	Number of bits or packets sent and received via the network interface

Export metrics in Prometheus format

Historical information for clusters is not available — metrics are requested only in real time.A list of all metrics supported in Managed Databases and their descriptions can be viewed in the table Metrics in Prometheus format.

1. Get a token.
2. Get metrics in Prometheus format.

1. Get a token

The token provides access to metrics for all clusters in a project in a single pool.

1. In the control panel, on the top menu click Products and select Managed Databases.

2. Open the Active tab.

3. Open the cluster page → Monitoring tab.

4. In the Prometheus tokens block, click Create token. The token will be generated automatically.

5. Copy the token.To do this, in the token line, click .

2. Get metrics in Prometheus format

Configuration file

1. Add to the Prometheus configuration file:

   scrape_configs:
     - job_name: get-metrics-from-dbaas
       scrape_interval: 1m
       static_configs:
         - targets:
           - '<domain>'
       scheme: https
       authorization:
         type: Bearer
         credentials: <token>

   Specify:

   • <domain> is the Managed Databases API domain. This is the part of the URL used to access the API without https:// and /v1, for example, uz-1.dbaas.api.servercore.com. The URL depends on the region and pool, and can be viewed in the list of URLs;
   • <token> — the token you copied when getting the token in step 5.

2. Open the page in your browser where Prometheus-format metrics will be available:

   http://<ip_address>:9090/targets

   Specify <ip_address> — the IP address where Prometheus is installed.

3. Set up monitoring and alerts for database clusters yourself.

CLI

1. Open the CLI.

2. To get metrics, send a request:

   curl -L "https://<domain>/metrics" -H "Authorization: Bearer <token>"

   Specify:

   • <domain> is the Managed Databases API domain. This is the part of the URL used to access the API without https:// and /v1, for example, uz-1.dbaas.api.servercore.com. The URL depends on the region and pool, and can be viewed in the list of URLs;
   • <token> — the token you copied when getting the token in step 5.

   The available metrics in Prometheus format will appear in the response.

3. Set up monitoring and alerts for database clusters yourself.

Metrics in Prometheus format

metrics in Prometheus format are provided for all clusters.A specific cluster can be found by the database cluster ID in the label ds_id.

dbaas_memory_percent	Memory used excluding operating system cache and buffers (RAM), in percent
dbaas_memory_bytes	Memory used excluding operating system cache and buffers (RAM), in bytes
dbaas_oom_count	number of processes that finished with an Out of Memory error due to lack of RAM
dbaas_cpu	vCPU usage on database cluster nodes, in percent
dbaas_cpu_iowait	I/O wait time, in percent
dbaas_disk_percent	Occupied disk space in percentages. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more information about reserving disk space, see the instructions Using disk space in a Kafka cluster
dbaas_disk_bytes	Occupied disk space in bytes. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more information about reserving disk space, see the instructions Using disk space in a Kafka cluster
dbaas_disk_read_iops	Number of read operations per second
dbaas_disk_write_iops	Number of write operations per second
dbaas_disk_read_bytes	Disk read speed, in bytes per second
dbaas_disk_write_bytes	Disk write speed, in bytes per second
dbaas_node_load1	Average system load over one minute. Shows how many processes are being handled by cluster cores
dbaas_node_load5	Average system load over five minutes. Shows how many processes are being handled by cluster cores
dbaas_node_load15	Average system load over 15 minutes. Shows how many processes are being handled by cluster cores
dbaas_network_receive_bytes	Number of bytes received via the network interface
dbaas_network_transmit_bytes	Number of bytes sent via the network interface
dbaas_network_receive_packets	Number of packets received via the network interface per second
dbaas_network_transmit_packets	Number of packets sent via the network interface per second
dbaas_role	Node role: 0 — role unknown; ; 1 — master; ; 2 — replica