Skip to main content

Fluent Bit

Fluent Bit is an agent that allows you to configure the collection, processing, and exporting of logs, as well as import logs from third-party storage. Fluent Bit is extensible with output and input plugins. You can view the list of plugins on the Fluent Bit website (Outputs, Inputs).

With Fluent Bit, you can manage logs in the Logs service: receive logs or add logs from your own storage.

Before you begin, set up Fluent Bit.

Configure Fluent Bit

  1. Add a service user.
  2. Issue an S3 key to the user.
  3. Configure the agent to receive or add events.

1. Add a service user

Add a service user with the Projects access scope and the role:

Users can be added by the Account Owner or users with the iam.admin role.

2. Issue an S3 key to the user

Only the Account Owner or a user with the iam.admin role can issue S3 keys to other users.

You need to create a separate key for each project. You can issue multiple keys for a single project.

  1. In the control panel, on the top menu, click IAM.

  2. Go to the section.

  3. Open the user page → the Access tab.

  4. In the S3 keys block, click Add key.

  5. Enter the key name.

  6. Select the project that the key will apply to.

  7. Click Generate. Two values will be generated:

    • Access key — Access Key ID, the key identifier;
    • Secret key — Secret Access Key, the secret key.
  8. Click Copy and save the key — it cannot be viewed after the window is closed.

3. Configure the agent

Using the built-in fluent-bit-cloudwatch-input-plugin, you can configure exporting events from the Logs service to your infrastructure, for example, to Elasticsearch, Loki, or Kafka.

  1. Download the config/fluent-bit.yaml configuration file.

  2. Open the config/fluent-bit.yaml configuration file in a text editor.

  3. In the configuration file parameters, specify:

    • region — pool, for example, kz-1 ;
    • endpoint — URL for accessing the Logs API. For a list of URLs, see the Logs section of the URL list;
    • log_group_name — log group name, for example, s/lbaas/Chromie-lb;
    • log_stream_name — event stream name, for example, http-21380357;
    • sqlite_path — path to the SQLite database that stores information about the latest received events. A single database can be used for multiple groups and streams.

    The list of events from the GetLogEvents Amazon CloudWatch API method is split into separate records using a Lua filter. Each received event is converted into a separate independent Fluent Bit record. This allows you to process events individually, tag them, and send them to different output plugins. The configuration file uses the stdout output plugin, but you can use any other, for example, Elasticsearch or Kafka.

  4. Optional: to configure data collection from multiple groups and streams within a single configuration file, in the pipeline block, add the corresponding inputs blocks.

  5. Run Fluent Bit:

    docker run \
    --name fluent-bit-cloudwatch \
    --rm \
    -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
    -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
    -e AWS_ACCESS_KEY_ID=<access_key> \
    -e AWS_SECRET_ACCESS_KEY=<secret_key> \
    ghcr.io/servercore/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    • <access_key> — value of the Access key field from the S3 key that you issued to the user;
    • <secret_key> — value of the Secret key field from the S3 key that you issued to the user.

    You can add <access_key> and <secret_key> to the container using any method:

    Example of output in stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...