Skip to content

write_kafka plugin
Plugin name write_kafka
Type write
Callbacks config, write
Status supported
Since 5.5
Copyright 2014 Pierre-Yves Ritschard
License MIT License
Configuration collectd.conf(5)

The Write Kafka plugin sends metrics to Apache Kafka, a distributed message bus.

The data sent can be formatted in three different ways:

  • PUTVAL commands, one line per metric.
  • collectd's standard JSON format.
  • Graphite's ASCII format.

Synopsis

 LoadPlugin "write_kafka"
 <Plugin write_kafka>
   Property "metadata.broker.list" "localhost:9092"
   <Topic "mytopic">
     Format JSON
   </Topic>
 </Plugin>

This will register a write callback with the name write_kafka/mytopic.

Dependencies

  • librdkafka
  • pthread
  • zlib

See also

Configuration

Configuration options · from collectd.conf(5)

See this section in the full configuration reference

The write_kafka plugin will send values to a Kafka topic, a distributed queue. Synopsis:

<Plugin "write_kafka">
  Property "metadata.broker.list" "broker1:9092,broker2:9092"
  <Topic "collectd">
    Format JSON
  </Topic>
</Plugin>

The following options are understood by the write_kafka plugin:

<Topic Name>

The plugin's configuration consists of one or more Topic blocks. Each block is given a unique Name and specifies one kafka producer. Inside the Topic block, the following per-topic options are understood:

Property String String
Configure the named property for the current topic. Properties are forwarded to the kafka producer library librdkafka.
Key String
Use the specified string as a partitioning key for the topic. Kafka breaks topic into partitions and guarantees that for a given topology, the same consumer will be used for a specific key. The special (case insensitive) string Random can be used to specify that an arbitrary partition should be used.
Format Command|JSON|Graphite

Selects the format in which messages are sent to the broker. If set to Command (the default), values are sent as PUTVAL commands which are identical to the syntax used by the Exec and UnixSock plugins.

If set to JSON, the values are encoded in the JavaScript Object Notation, an easy and straight forward exchange format.

If set to Graphite, values are encoded in the Graphite format, which is <metricE<gt <valueE<gt> <timestampE<gt>\n>.

StoreRates true|false

Determines whether or not COUNTER, DERIVE and ABSOLUTE data sources are converted to a rate (i.e. a GAUGE value). If set to false (the default), no conversion is performed. Otherwise the conversion is performed using the internal value cache.

Please note that currently this option is only used if the Format option has been set to JSON.

GraphitePrefix (Format=Graphite only)

A prefix can be added in the metric name when outputting in the Graphite format.

When GraphiteUseTags is false, prefix is added before the Host name. Metric name will be <prefixE<gt<hostE<gt><postfixE<gt><pluginE<gt><typeE<gt><nameE<gt>>

When GraphiteUseTags is true, prefix is added in front of series name.

GraphitePostfix (Format=Graphite only)

A postfix can be added in the metric name when outputting in the Graphite format.

When GraphiteUseTags is false, postfix is added after the Host name. Metric name will be <prefixE<gt<hostE<gt><postfixE<gt><pluginE<gt><typeE<gt><nameE<gt>>

When GraphiteUseTags is true, prefix value appended to the end of series name (before the first ; that separates the name from the tags).

GraphiteEscapeChar (Format=Graphite only)
Specify a character to replace dots (.) in the host part of the metric name. In Graphite metric name, dots are used as separators between different metric parts (host, plugin, type). Default is _ (Underscore).
GraphiteSeparateInstances false|true

If set to true, the plugin instance and type instance will be in their own path component, for example host.cpu.0.cpu.idle. If set to false (the default), the plugin and plugin instance (and likewise the type and type instance) are put into one component, for example host.cpu-0.cpu-idle.

Option value is not used when GraphiteUseTags is true.

GraphiteAlwaysAppendDS true|false
If set to true, append the name of the Data Source (DS) to the "metric" identifier. If set to false (the default), this is only done when there is more than one DS.
GraphitePreserveSeparator false|true

If set to false (the default) the . (dot) character is replaced with GraphiteEscapeChar. Otherwise, if set to true, the . (dot) character is preserved, i.e. passed through.

Option value is not used when GraphiteUseTags is true.

GraphiteUseTags false|true

If set to true Graphite metric names will be generated as tagged series.

Default value: false.

StoreRates true|false

If set to true (the default), convert counter values to rates. If set to false counter values are stored as is, i.e. as an increasing integer number.

This will be reflected in the ds_type tag: If StoreRates is enabled, converted values will have "rate" appended to the data source type, e.g. ds_type:derive:rate.

Property String String
Configure the kafka producer through properties, you almost always will want to set metadata.broker.list to your Kafka broker list.