Skip to main content

Fluent Bit

Fluent Bit is an agent that allows you to configure the collection, processing, and exporting of logs, as well as import logs from third-party storage. Fluent Bit is extensible with output and input plugins. You can view the list of plugins on the Fluent Bit website (Outputs, Inputs).

With Fluent Bit, you can manage logs in the Logs service: receive logs or add logs from your own storage.

Before you begin, set up Fluent Bit.

Configure Fluent Bit

  1. Add a service user.
  2. Issue an S3 key to the user.
  3. Configure the agent to receive or add events.

1. Add a service user

Add a service user with permissions in the Projects scope and the role:

Users can be added by the Account Owner or users with the iam.admin role.

2. Issue an S3 key to the user

Control panel users can issue S3 keys to themselves, but we recommend creating service users and issuing S3 keys to them.

Only the Account Owner or a user with the iam.admin role can issue S3 keys to other users. A service user cannot obtain an S3 key independently because they do not have access to the control panel — the Account Owner or iam.admin must issue a key to them.

You must create a separate key for each project. You can issue multiple keys for a single project.

  1. In the control panel, in the top menu, click IAM.

  2. Go to the section with the desired user type:

    • Control panel users — for users with access to the control panel;
    • Service users — for users with programmatic access without access to the control panel.

  3. Open the user page → Access tab.

  4. In the S3 keys block, click Add key.

  5. Enter a key name.

  6. Select the project for which the key will work.

  7. Click Generate. Two values will be generated:

    • Access key — Access Key ID, a key identifier;
    • Secret key — Secret Access Key, a secret key.

  8. Click Copy and save the key — it cannot be viewed after closing the window.

3. Configure the agent

Using the built-in fluent-bit-cloudwatch-input-plugin, you can configure exporting events from the Logs service to your infrastructure, for example, to Elasticsearch, Loki, or Kafka.

  1. Download the config/fluent-bit.yaml configuration file.

  2. Open the config/fluent-bit.yaml configuration file in a text editor.

  3. In the configuration file parameters, specify:

    • region — pool, for example, kz-1 ;
    • endpoint — URL for accessing the Logs API. For a list of URLs, see the Logs section of the URL list;
    • log_group_name — log group name, for example, s/lbaas/Chromie-lb;
    • log_stream_name — event stream name, for example, http-21380357;
    • sqlite_path — path to the SQLite database that stores information about the latest received events. A single database can be used for multiple groups and streams.

    The list of events from the GetLogEvents Amazon CloudWatch API method is split into separate records using a Lua filter. Each received event is converted into a separate independent Fluent Bit record. This allows you to process events individually, tag them, and send them to different output plugins. The configuration file uses the stdout output plugin, but you can use any other, for example, Elasticsearch or Kafka.

  4. Optional: to configure data collection from multiple groups and streams within a single configuration file, in the pipeline block, add the corresponding inputs blocks.

  5. Run Fluent Bit:

    docker run \
      --name fluent-bit-cloudwatch \
      --rm \
      -v ${PWD}/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro \
      -v ${PWD}/sqlite:/var/lib/fluent-bit/cloudwatch/sqlite:rw \
      -e AWS_ACCESS_KEY_ID=<access_key> \
      -e AWS_SECRET_ACCESS_KEY=<secret_key> \
      ghcr.io/servercore/fluent-bit-cloudwatch-input-plugin:latest

    Specify:

    • <access_key> — value of the Access key field from the S3 key that you issued to the user;
    • <secret_key> — value of the Secret key field from the S3 key that you issued to the user.

    You can add <access_key> and <secret_key> to the container using any method:

    Example of output in stdout in MessagePack format:

    ...
    [1029] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750606362344, "message"=>"{"level":"System", "msg":"InnoDB initialization has started."}", "timestamp"=>1750606335369}]
    [1031] cloudwatch-input.0: [[1751577177.000000000, {}], {"ingestion_time"=>1750608768923, "message"=>"{"level":"System", "msg":"InnoDB initialization has ended."}", "timestamp"=>1750608747624}]
    ...