Using disk space in a Kafka cluster

In Cloud Databases, a portion of disk space is reserved for system needs. The total amount of reserved space depends on the version of Kafka.

	Kafka 3.5	Kafka 4.1
File system reservation	4% of disk capacity	4% of disk capacity
Reservation for the operating system, service components and logs	8 GB	15 GB

The reserved portion of disk space is not available for hosting databases. Keep this in mind when choosing a configuration line.

You can track disk usage using disk full notifications and metrics. For more information about metrics, see the Kafka cluster and node monitoring instructions.

When the cluster disk is 95% full or more, the cluster will switch to the DISK_FULL status and become read-only. This is necessary to prevent complete locking or corruption of the cluster due to lack of free space. To make the cluster available for read and write, free up disk space or scale the cluster and select a configuration with a larger disk size than in the previous configuration.

Disk occupancy notifications

Disk usage notifications are sent to the email address of the Account Owner and control panel users who are subscribed to the notification category Products and services. Notifications are sent when disk usage reaches 80% and 95%.

Free up disk space

To free up disk space, you can:

• delete the topic;
• configure the data cleanup policy.

Delete a topic

1. In the control panel, in the top menu, click Products and select Cloud Databases.
2. Open the Active tab.
3. Open the cluster page → Topics tab.
4. In the topic menu, select Delete.
5. Enter the topic name to confirm deletion.
6. Click Delete.

Configure a data cleanup policy

warning

After changing the log.retention.minutes and log.retention.hours parameters in the DBMS settings, the cluster will be restarted and will become unavailable during the restart.

To speed up disk cleanup, change the data retention policy in the cluster settings. Kafka will automatically delete data based on the new rules.

1. In the control panel, in the top menu, click Products and select Cloud Databases.
2. Open the Active tab.
3. Open the cluster page → Settings tab.
4. In the DBMS settings block, click Edit.
5. Specify new values for the parameters for configuring the data cleanup policy.
6. Click Save.

Parameters for configuring a data cleanup policy

log.segment.bytes	The maximum size of a single segment in a topic partition in bytes. When the current segment is full, Kafka will create a new segment. Default value is 1 073 741 824
log.retention.bytes	The maximum size of a topic partition in bytes. When the partition is full, Kafka will start deleting old segments. Default value is -1 (maximum size is not limited)
log.retention.ms	Segment retention time in milliseconds. If not set, the log.retention.minutes parameter is used. If set to -1, the retention time is not limited, even if the log.retention.minutes or log.retention.hours
log.retention.minutes	Segment retention time in minutes. If not set, the log.retention.hours parameter is used
log.retention.hours	Segment retention time in hours. Used if the log.retention.ms and log.retention.minutes parameters are not set. Default value is 168