Skip to content

logparser plugin

Plugin name logparser
Type read
Callbacks init, config, read, shutdown
Status supported
Since 5.11
Copyright 2017-2018 Intel Corporation, Krzysztof Matczak , Marcin Mozejko ,
License MIT License
Configuration collectd.conf(5)

The logparser plugin is used to parse different kinds of logs. Setting proper options you can choose strings to collect. Plugin searches the log file for messages which contain several matches (two or more). When all mandatory matches are found then it sends proper notification containing all fetched values.

This utility simplifies filtering and parsing specific messages from given log file. It tracks log file using utils_tail_match.h API.

Main feature is automatic messages assembly based on set of user provided regular expressions containing distinguished expressions for recognizing start and the end of the message. That makes assembling single message from multiple log lines quite easy. Utility supports multiple parsing jobs and also user defined message part validation flags (mandatory or not).

Synopsis

 <Plugin logparser>
   <Logfile "/var/log/syslog">
     FirstFullRead false
     <Message "pcie_errors">
       DefaultType "pcie_error"
       DefaultSeverity "warning"
       <Match "aer error">
         Regex "AER:.*error received"
         SubmatchIdx -1
       </Match>
       <Match "incident time">
         Regex "(... .. ..:..:..) .* pcieport.*AER"
         SubmatchIdx 1
         IsMandatory false
       </Match>
       <Match "root port">
         Regex "pcieport (.*): AER:"
         SubmatchIdx 1
         IsMandatory true
       </Match>
       <Match "device">
         PluginInstance true
         Regex " ([0-9a-fA-F:\\.]*): PCIe Bus Error"
         SubmatchIdx 1
         IsMandatory false
       </Match>
       <Match "severity_mandatory">
         Regex "severity="
         SubMatchIdx -1
       </Match>
       <Match "nonfatal">
         Regex "severity=.*\\([nN]on-[fF]atal"
         TypeInstance "non_fatal"
         IsMandatory false
       </Match>
       <Match "fatal">
         Regex "severity=.*\\([fF]atal"
         Severity "failure"
         TypeInstance "fatal"
         IsMandatory false
       </Match>
       <Match "corrected">
         Regex "severity=Corrected"
         TypeInstance "correctable"
         IsMandatory false
       </Match>
       <Match "error type">
         Regex "type=(.*),"
         SubmatchIdx 1
         IsMandatory false
       </Match>
      <Match "id">
         Regex ", id=(.*)"
         SubmatchIdx 1
       </Match>
     </Message>
   </Logfile>
 </Plugin>

Example graph

None yet. Add one now!

Dependencies

Caveats

See also

Configuration

Configuration options · from collectd.conf(5)

See this section in the full configuration reference

The logparser plugin is used to parse different kinds of logs. Setting proper options you can choose strings to collect. Plugin searches the log file for messages which contain several matches (two or more). When all mandatory matches are found then it sends proper notification containing all fetched values.

Synopsis:

<Plugin logparser>
  <Logfile "/var/log/syslog">
    FirstFullRead false
    <Message "pcie_errors">
      DefaultType "pcie_error"
      DefaultSeverity "warning"
      <Match "aer error">
        Regex "AER:.*error received"
        SubmatchIdx -1
      </Match>
      <Match "incident time">
        Regex "(... .. ..:..:..) .* pcieport.*AER"
        SubmatchIdx 1
        IsMandatory false
      </Match>
      <Match "root port">
        Regex "pcieport (.*): AER:"
        SubmatchIdx 1
        IsMandatory true
      </Match>
      <Match "device">
        PluginInstance true
        Regex " ([0-9a-fA-F:\\.]*): PCIe Bus Error"
        SubmatchIdx 1
        IsMandatory false
      </Match>
      <Match "severity_mandatory">
        Regex "severity="
        SubMatchIdx -1
      </Match>
      <Match "nonfatal">
        Regex "severity=.*\\([nN]on-[fF]atal"
        TypeInstance "non_fatal"
        IsMandatory false
      </Match>
      <Match "fatal">
        Regex "severity=.*\\([fF]atal"
        Severity "failure"
        TypeInstance "fatal"
        IsMandatory false
      </Match>
      <Match "corrected">
        Regex "severity=Corrected"
        TypeInstance "correctable"
        IsMandatory false
      </Match>
      <Match "error type">
        Regex "type=(.*),"
        SubmatchIdx 1
        IsMandatory false
      </Match>
     <Match "id">
        Regex ", id=(.*)"
        SubmatchIdx 1
      </Match>
    </Message>
  </Logfile>
</Plugin>

Options:

Logfile File
The Logfile block defines file to search. It may contain one or more Message blocks which are defined below.
FirstFullRead true|false
Set to true if the file has to be parsed from the beginning on the first read. If false only subsequent writes to log file will be parsed.
Message Name
Message block contains matches to search the log file for. Each Message block builds a notification message using matched elements if its mandatory Match blocks are matched.
DefaultPluginInstance String
Sets the default value for the plugin instance of the notification.
DefaultType String
Sets the default value for the type of the notification.
DefaultTypeInstance String
Sets the default value for the type instance of the notification.
DefaultSeverity String
Sets the default severity. Must be set to "OK", "WARNING" or "FAILURE". Default value is "OK".
Match Name
Multiple Match blocks define regular expression patterns for extracting or excluding specific string patterns from parsing. First and last Match items in the same Message set boundaries of multiline message and are mandatory. If these matches are not found then the whole message is discarded.
Regex Regex
Regular expression with pattern matching string. It may contain subexpressions, so next option SubmatchIdx specifies which subexpression should be stored.
SubmatchIdx Integer
Index of subexpression to be used for notification. Multiple subexpressions are allowed. Index value 0 takes whole regular expression match as a result. Index value -1 does not add result to message item. Can be omitted, default value is 0.
Excluderegex Regex
Regular expression for excluding lines containing specific matching strings. This is processed before checking Regex pattern. It is optional and can be omitted.
IsMandatory true|false
Flag indicating if Match item is mandatory for message validation. If set to true, whole message is discarded if it's missing. For false its presence is optional. Default value is set to true.
PluginInstance true|String
If set to true, it sets plugin instance to string returned by regex. It can be overridden by user string.
Type true|String
Sets notification type using rules like PluginInstance.
TypeInstance true|String
Sets notification type instance using rules like above.
Severity String
Sets notification severity to one of the options: "OK", "WARNING", "FAILURE".