Monitoring a MySQL semi-sync cluster, nodes, and databases

In Managed Databases MySQL semi-sync you can monitor the cluster state.

To estimate the general cluster state, view its status.

For a more detailed analysis, some metrics can be viewed as charts in the control panel:

• cluster node metrics;
• database metrics.

A full set of available metrics can be exported in Prometheus format.

The time in the control panel corresponds to the time set on your device and does not depend on the region where the cluster is hosted.

note

For example, you have created a cluster in Tashkent, in the uz-1 pool. Tashkent is in the UTC+5 time zone. On the device from which you logged into the control panel, the UTC+3 time zone is set. The time on the metrics charts will be displayed in UTC+3.

View cluster status

1. In the control panel, on the top menu, click Products and select Managed Databases.

2. Open the Active tab.

3. Check the status in the cluster row.

   ACTIVE	Cluster is available
   CREATING	Cluster is being created
   UPDATING	Changes are being applied to the cluster
   RESIZING	Cluster is scaling
   ERROR	An error occurred, create a ticket
   DISK FULL	Disk is full; the cluster is operating in read-only mode. To make the cluster work in read/write mode, scale the cluster and select a configuration with a larger disk size
   DEGRADED	Some cluster nodes are unavailable
   DELETING	Cluster is being deleted

View cluster node status

1. In the control panel, on the top menu, click Products and select Managed Databases.
2. Open the Active tab.
3. Open the cluster page → Monitoring tab.
4. In the Cluster monitoring block, click Cluster nodes.
5. Select the nodes whose metrics you want to view.
6. View available cluster node metrics.

Cluster node metrics in the control panel

Memory	Consumed memory excluding operating system cache and buffers, in percent or gigabytes
vCPU	Percentage of cluster node cores load
CPU iowait	Percentage of time the CPU spent waiting for I/O
Disk	Occupied disk space in percent or gigabytes. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more details on disk space reservation, see the Using disk space in cluster MySQL semi-sync instructions
Load Average	System load average over a period of time. Shows the number of processes being processed by cluster cores. The indicator is presented as three values — for one minute, five minutes, and 15 minutes. These values must not exceed the number of cores on the node
OOM	Number of processes that finished with an Out of Memory error due to insufficient RAM
Disk load	Data read/write speed in KB/s or number of read/write operations per second
Network load	Number of bits or packets sent and received via the network interface

View database status

1. In the control panel, on the top menu, click Products and select Managed Databases.
2. Open the Active tab.
3. Open the cluster page → Monitoring tab.
4. In the Cluster monitoring block, click Cluster.
5. Select the nodes whose metrics you want to view.
6. View available database metrics.

Database metrics in the control panel

Binlog file size	Total size of binlog files in bytes
Cache hit	Percentage of requests hitting the cache
Temporary file size	Total size of temporary files in bytes
Slow queries	Number of slow queries per second. Slow queries are defined by the long_query_time parameter. You can change the value of the long_query_time parameter
Queries	Number of queries of a specific type per second: DELETE; INSERT; SELECT; UPDATE
Threads	Number of threads (threads) in MySQL: Cached — number of threads in the cache; ; Connected — number of open connections; ; Running — number of running threads
Database size	Total size of the selected database in bytes

Export metrics in Prometheus format

Historical information for clusters is not available — metrics are requested only in real time. A list of all metrics supported in Managed Databases and their descriptions can be viewed in the Metrics in Prometheus format table.

1. Get a token.
2. Get metrics in Prometheus format.

1. Get a token

The token provides access to metrics for all clusters of a project in one location.

1. In the control panel, on the top menu, click Products and select Managed Databases.

2. Open the Active tab.

3. Open the cluster page → Monitoring tab.

4. In the Prometheus tokens block, click Create token. The token will be generated automatically.

5. Copy the token. To do this, click in the token line.

2. Get metrics in Prometheus format

Configuration file

1. Add the scrape_configs: block to the Prometheus configuration file:

   scrape_configs:
     - job_name: get-metrics-from-dbaas
       scrape_interval: 1m
       static_configs:
         - targets:
           - '<domain>'
       scheme: https
       authorization:
         type: Bearer
         credentials: <token>

   Specify:

   • <domain> — Managed Databases API domain. This is the part of the URL for accessing the API, excluding https:// and /v1 uz-1.dbaas.api.servercore.com. The URL depends on the region and pool; you can view it in the list of URLs;
   • <token> — the token you copied when getting a token in step 5.

2. Open the page in your browser where the metrics in Prometheus format will be available:

   http://<ip_address>:9090/targets

   Specify <ip_address> — the IP address where Prometheus is installed.

3. Set up your own monitoring and alerts for database clusters.

CLI

1. Open the CLI.

2. To get the metrics, send a request:

   curl -L "https://<domain>/metrics" -H "Authorization: Bearer <token>"

   Specify:

   • <domain> — Managed Databases API domain. This is the part of the URL for accessing the API, excluding https:// and /v1 uz-1.dbaas.api.servercore.com. The URL depends on the region and pool; you can view it in the list of URLs;
   • <token> — the token you copied when getting a token in step 5.

   The available metrics in Prometheus format will appear in the response.

3. Set up your own monitoring and alerts for database clusters.

Metrics in Prometheus format

Metrics in Prometheus format are provided for all clusters. A specific cluster can be found by its database cluster identifier in the ds_id label.

Infrastructure level metrics

dbaas_memory_percent	Occupied memory excluding operating system cache and buffers (RAM), in percent
dbaas_memory_bytes	Occupied memory excluding operating system cache and buffers (RAM), in bytes
dbaas_oom_count	Number of processes that finished with an Out of Memory error due to insufficient RAM
dbaas_cpu	vCPU usage on database cluster nodes, in percent
dbaas_cpu_iowait	I/O wait time, in percent
dbaas_disk_percent	Occupied disk space in percent. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more details on disk space reservation, see the Using disk space in cluster MySQL semi-sync instructions
dbaas_disk_bytes	Occupied disk space in bytes. It takes into account the part of disk space reserved for service needs and unavailable for database placement. For more details on disk space reservation, see the Using disk space in cluster MySQL semi-sync instructions
dbaas_disk_read_iops	Number of read operations per second
dbaas_disk_write_iops	Number of write operations per second
dbaas_disk_read_bytes	Disk read speed, in bytes per second
dbaas_disk_write_bytes	Disk write speed, in bytes per second
dbaas_node_load1	System load average over one minute. Shows the number of processes being processed by cluster cores
dbaas_node_load5	System load average over five minutes. Shows the number of processes being processed by cluster cores
dbaas_node_load15	System load average over 15 minutes. Shows the number of processes being processed by cluster cores
dbaas_network_receive_bytes	Number of bytes received via the network interface
dbaas_network_transmit_bytes	Number of bytes sent via the network interface
dbaas_network_receive_packets	Number of packets received via the network interface per second
dbaas_network_transmit_packets	Number of packets sent via the network interface per second
dbaas_role	Node role: 0 — role unknown; 1 — master; 2 — replica

Application level metrics

dbaas_threads_cached	Number of threads (threads) in the cache
dbaas_threads_connected	Number of open connections to the cluster
dbaas_threads_running	Number of running threads (threads)
dbaas_commands_total_select	Number of SELECT queries in MySQL per second
dbaas_commands_total_insert	Number of INSERT queries in MySQL per second
dbaas_commands_total_update	Number of UPDATE queries in MySQL per second
dbaas_commands_total_delete	Number of DELETE queries in MySQL per second
dbaas_slow_queries	Number of slow queries per second. Slow queries are defined by the long_query_time parameter
dbaas_innodb_buffer_pool_hit_ratio	Percentage of requests hitting the cache
dbaas_mysql_tmp_size_bytes	MySQL temporary file size, in bytes
dbaas_mysql_total_binlogs_size_bytes	Total binlog file size, in bytes
dbaas_databases_size_bytes	Total database size in bytes. You can use the datname label, which is the database name. The metric is missing if there is no data in the database