aggregation plugin¶


Plugin name	`aggregation`
Type	read / write
Callbacks	config, read, write
Status	supported
Since	5.2
Copyright	2012 Florian octo Forster
License	MIT License
Configuration	collectd.conf(5)

The Aggregate plugin allows to aggregate multiple values into a single value using one or several consolidation functions, e.g. summation and average. This has a broad range of applications, e.g. calculating the average CPU utilization over all cores of each host. The full specification for the aggregation plugin can be found in the manpage for collect.conf(5): 1.

Naming schema¶

New names are automatically generated for the aggregated values. They all have the plugin field set to aggregation. The exact naming follows these rules:

Host: The host is copied from the original value-lists unless the Host option is left unspecified in which case it will be set to global.
Plugin: The plugin is set to aggregation.
Plugin instance: The plugin instance contains plugin and plugin instance of the original value list and the aggregation function used. The three parts are separated by dashes. If Plugin or PluginInstance (or both) is unspecified, that part will be omitted.
Type: The type cannot be left unspecified and is copied from the original value lists.
Type instance: The type instance is copied from the original value lists unless the TypeInstance option is left unspecified in which case it will be left empty.
GroupBy: The field to group by. It has the following options "Host", "Plugin", "PluginInstance", "TypeInstance". You can repeat this clause to include multiple fields.

Synopsis¶

- → See: Plugin-Aggregation-Config

 <Plugin "aggregation">
   <Aggregation>
     Plugin "example"
     Type "gauge"

     GroupBy "Host"
     GroupBy "TypeInstance"

     CalculateNum false
     CalculateSum false
     CalculateAverage true
     CalculateMinimum false
     CalculateMaximum false
     CalculateStddev false
   </Aggregation>
 </Plugin>

This registers a writer by the name

aggregation

Dependencies¶

None

Configuration examples¶

This page contains some example configurations for the Aggregation plugin. This page is meant as a cookbook, so if you have a configuration for an aspect not handled here or a daemon not present, please feel free to add anything that's useful for you.

Aggregated CPU statistics per host¶

The following configuration assumes your cpu plugin is configured this way:

 <Plugin cpu>
   ValuesPercentage false
   ReportByCpu true
 </Plugin cpu>

The following configuration aggregates the CPU statistics of all CPUs into one set using the sum and average consolidation functions.

 <Plugin "aggregation">
   <Aggregation>
     Plugin "cpu"
     Type "cpu"

     GroupBy "Host"
     GroupBy "TypeInstance"

     CalculateSum true
     CalculateAverage true
   </Aggregation>
 </Plugin>

This will result in a per-CPU-state set of values for each host. The values will be names according to this schema:

 host.example.com/aggregation-cpu-sum/cpu-idle
 host.example.com/aggregation-cpu-sum/cpu-user
 host.example.com/aggregation-cpu-sum/cpu-system
 …
 host.example.com/aggregation-cpu-average/cpu-idle
 host.example.com/aggregation-cpu-average/cpu-user
 host.example.com/aggregation-cpu-average/cpu-system
 …

If you don't want to keep the per-CPU statistics around, you can use the filter mechanism to get rid of them:

 LoadPlugin "match_regex" # we want to use this for our Matching
 <Chain "PostCache">
   <Rule> # Send "cpu" values to the aggregation plugin.
     <Match regex>
       Plugin "^cpu$"
       PluginInstance "^[0-9]+$"
     </Match>
     <Target write>
       Plugin "aggregation"
     </Target>
     Target stop
   </Rule>
   <Target write> # Write everything else via "rrdtool".
     Plugin "rrdtool"
   </Target>
 </Chain>

Aggregated CPU frequency per host¶

This example plots the average CPU frequency for a given host. It changes the GroupBy to "PluginInstance" to group all fields.

  <Plugin "aggregation">
    <Aggregation>
      Plugin "cpufreq"
      Type "cpufreq"
      GroupBy "Host"
      GroupBy "PluginInstance"
      CalculateAverage true
    </Aggregation>
  </Plugin>

Configuration¶

Configuration options · from collectd.conf(5)

See this section in the full configuration reference

The Aggregation plugin makes it possible to aggregate several values into one using aggregation functions such as sum, average, min and max. This can be put to a wide variety of uses, e.g. average and total CPU statistics for your entire fleet.

The grouping is powerful but, as with many powerful tools, may be a bit difficult to wrap your head around. The grouping will therefore be demonstrated using an example: The average and sum of the CPU usage across all CPUs of each host is to be calculated.

To select all the affected values for our example, set Plugin cpu and Type cpu. The other values are left unspecified, meaning "all values". The Host, Plugin, PluginInstance, Type and TypeInstance options work as if they were specified in the WHERE clause of an SELECT SQL statement.

Plugin "cpu"
Type "cpu"

Although the Host, PluginInstance (CPU number, i.e. 0, 1, 2, ...) and TypeInstance (idle, user, system, ...) fields are left unspecified in the example, the intention is to have a new value for each host / type instance pair. This is achieved by "grouping" the values using the GroupBy option. It can be specified multiple times to group by more than one field.

GroupBy "Host"
GroupBy "TypeInstance"

We do neither specify nor group by plugin instance (the CPU number), so all metrics that differ in the CPU number only will be aggregated. Each aggregation needs at least one such field, otherwise no aggregation would take place.

The full example configuration looks like this:

<Plugin "aggregation">
  <Aggregation>
    Plugin "cpu"
    Type "cpu"

GroupBy "Host"
GroupBy "TypeInstance"

    CalculateSum true
    CalculateAverage true
  </Aggregation>
</Plugin>

There are a couple of limitations you should be aware of:

The Type cannot be left unspecified, because it is not reasonable to add apples to oranges. Also, the internal lookup structure won't work if you try to group by type.
There must be at least one unspecified, ungrouped field. Otherwise nothing will be aggregated.

As you can see in the example above, each aggregation has its own Aggregation block. You can have multiple aggregation blocks and aggregation blocks may match the same values, i.e. one value list can update multiple aggregations. The following options are valid inside Aggregation blocks:

Host Host

Plugin Plugin

PluginInstance PluginInstance

Type Type

TypeInstance TypeInstance

Selects the value lists to be added to this aggregation. Type must be a valid data set name, see types.db(5) for details.

If the string starts with and ends with a slash (/), the string is interpreted as a regular expression. The regex flavor used are POSIX extended regular expressions as described in regex(7). Example usage:

Host "/^db[0-9]\\.example\\.com$/"

GroupBy Host|Plugin|PluginInstance|TypeInstance

Group valued by the specified field. The GroupBy option may be repeated to group by multiple fields.

SetHost Host

SetPlugin Plugin

SetPluginInstance PluginInstance

SetTypeInstance TypeInstance

Sets the appropriate part of the identifier to the provided string.

The PluginInstance should include the placeholder %{aggregation} which will be replaced with the aggregation function, e.g. "average". Not including the placeholder will result in duplication warnings and/or messed up values if more than one aggregation function are enabled.

The following example calculates the average usage of all "even" CPUs:

<Plugin "aggregation">
  <Aggregation>
    Plugin "cpu"
    PluginInstance "/[0,2,4,6,8]$/"
    Type "cpu"

SetPlugin "cpu"
SetPluginInstance "even-%{aggregation}"

GroupBy "Host"
GroupBy "TypeInstance"

    CalculateAverage true
  </Aggregation>
</Plugin>

This will create the files:

foo.example.com/cpu-even-average/cpu-idle
foo.example.com/cpu-even-average/cpu-system
foo.example.com/cpu-even-average/cpu-user
...

CalculateNum true|false

CalculateSum true|false

CalculateAverage true|false

CalculateMinimum true|false

CalculateMaximum true|false

CalculateStddev true|false

Boolean options for enabling calculation of the number of value lists, their sum, average, minimum, maximum andE<nbsp>/ or standard deviation. All options are disabled by default.