Skip to main content

Configure Apache Kafka® metrics sent to Datadog

When creating a Datadog service integration, you can customize which metrics are sent to the Datadog endpoint using the Aiven CLI.

Prerequisites​

note

Datadog integration is not available for new Startup-2 plans in Aiven for Apache Kafka. Existing customers using Startup-2 with Datadog integration can continue to create new Startup-2 plans with Datadog integration and use their existing services without upgrading to a higher plan.

Aiven recommends using a Business-4 plan or higher for Aiven for Apache Kafka services with Datadog integration to avoid resource pressure on Startup-2 plans.

If you are an existing customer and cannot create a Startup-2 plan with Datadog integration in a new project, contact Aiven Support.

Supported metrics​

The following metrics are currently supported for each topic and partition in Apache Kafka®:

  • kafka.log.log_size
  • kafka.log.log_start_offset
  • kafka.log.log_end_offset

All metrics are tagged with topic and partition, enabling independent monitoring of each topic and partition.

Variables​

Replace the following placeholders in the code samples:

VariableDescription
SERVICE_NAMEAiven for Apache Kafka® service name
INTEGRATION_IDID of the integration between Aiven for Apache Kafka® service and Datadog

To find the INTEGRATION_ID parameter, run:

avn service integration-list SERVICE_NAME

Customize metrics for Datadog​

Before customizing metrics, configure and enable a Datadog endpoint in your Aiven for Apache Kafka service. For setup instructions, see Send metrics to Datadog.

Format any listed parameters as a comma-separated list: ['value0', 'value1', 'value2', ...].

To customize the metrics sent to Datadog, use the service integration-update command with the kafka_custom_metrics parameter. Specify a comma-separated list of custom metrics, such as kafka.log.log_size, kafka.log.log_start_offset, and kafka.log.log_end_offset.

For example, to send the kafka.log.log_size and kafka.log.log_end_offset metrics, run:

avn service integration-update                                                \
    -c 'kafka_custom_metrics=["kafka.log.log_size","kafka.log.log_end_offset"]' \
    INTEGRATION_ID

After updating settings, view the collected metrics in your Datadog explorer.

Customize consumer metrics for Datadog​

Apache Kafka Consumer Integration collects metrics for message offsets. To customize the metrics sent from this Datadog integration to Datadog, use the service integration-update command with the following parameters:

  • include_topics: A comma-separated list of topics to include.

    note

    By default, all topics are included.

  • exclude_topics: A comma-separated list of topics to exclude.

    warning

    To use exclude_topics, you must specify at least one include_consumer_groups value. Otherwise, exclude_topics does not take effect.

  • include_consumer_groups: A comma-separated list of consumer groups to include.

  • exclude_consumer_groups: A comma-separated list of consumer groups to exclude.

For example, to include topics topic1 and topic2, and exclude topic3, run:

avn service integration-update                                                  \
    -c 'kafka_custom_metrics=["kafka.log.log_size","kafka.log.log_end_offset"]' \
    -c 'include_topics=["topic1","topic2"]'                                     \
    INTEGRATION_ID

After updating settings, view the collected metrics in your Datadog explorer.

Related pages