Using disk space in a Kafka cluster

In Managed Databases, some disk space is reserved for service needs.The total amount of reserved space depends on the version of Kafka.

	Kafka 3.5	Kafka 4.1
File system reservation	4% of disk capacity	4% of disk capacity
Reservation for the operating system, service components, and logs	8 GB	15 GB

The reserved portion of the disk space is not available to host databases.Take this into account when selecting a configuration lineup.

You can monitor disk occupancy using disk occupancy notifications and metrics. For more information about metrics, see the Kafka cluster and node monitoring.

When the cluster disk is 95% full or more, the cluster will enter DISK_FULL status and will be read-only.This is necessary to prevent total cluster lockup or damage due to lack of free space.To make the cluster read-write, free up disk space or scale the cluster and select a configuration with a larger disk size than the previous one.

Disk occupancy notifications

Disk occupancy notifications are sent to the email address of the Account Owner and users who are subscribed to the notification category “Services & Products”.Notifications are sent when the disk is 80% and 95% full.

Free up disk space

To free up disk space, you can:

• delete a topic;
• configure a data cleanup policy.

Delete a topic

1. In the Dashboard, on the top menu, click Products and select Managed Databases.
2. Open the Active tab.
3. Open the cluster page → Topics tab.
4. In the menu of the topic, select Delete.
5. Enter the topic name to confirm deletion.
6. Click Delete.

Configure a data cleanup policy

warning

After changing the log.retention.minutes and log.retention.hours parameters in the DBMS settings, the cluster will be restarted and will be unavailable during the restart.

To speed up disk cleanup, change the data retention policy in the cluster settings.Kafka will automatically delete data based on the new rules.

1. In the Dashboard, on the top menu, click Products and select Managed Databases.
2. Open the Active tab.
3. Open the cluster page → Settings tab.
4. In the DBMS settings block, click Edit.
5. Specify the new values for the parameters for configuring a data cleanup policy.
6. Click Save.

Parameters for configuring a data cleanup policy

log.segment.bytes	The maximum size of a single segment in a topic partition in bytes. When the current segment is full, Kafka will create a new one. The default value is 1 073 741 824
log.retention.bytes	The maximum size of a topic partition in bytes. When the partition is full, Kafka will start deleting old segments. The default value is -1 (maximum size is unlimited)
log.retention.ms	The segment retention time in milliseconds. If not set, the log.retention.minutes parameter is used. If -1 is specified, the retention time is unlimited, even if the log.retention.minutes or log.retention.hours parameters are set
log.retention.minutes	The segment retention time in minutes. If not set, the log.retention.hours parameter is used
log.retention.hours	The segment retention time in hours. Used if the log.retention.ms and log.retention.minutes parameters are not specified. The default value is 168