logparser plugin¶
| Plugin name | logparser |
| Type | read |
| Callbacks | init, config, read, shutdown |
| Status | supported |
| Since | 5.11 |
| Copyright | 2017-2018 Intel Corporation, Krzysztof Matczak , Marcin Mozejko , |
| License | MIT License |
| Configuration | collectd.conf(5) |
The logparser plugin is used to parse different kinds of logs. Setting proper options you can choose strings to collect. Plugin searches the log file for messages which contain several matches (two or more). When all mandatory matches are found then it sends proper notification containing all fetched values.
This utility simplifies filtering and parsing specific messages from given log file. It tracks log file using utils_tail_match.h API.
Main feature is automatic messages assembly based on set of user provided regular expressions containing distinguished expressions for recognizing start and the end of the message. That makes assembling single message from multiple log lines quite easy. Utility supports multiple parsing jobs and also user defined message part validation flags (mandatory or not).
Synopsis¶
<Plugin logparser>
<Logfile "/var/log/syslog">
FirstFullRead false
<Message "pcie_errors">
DefaultType "pcie_error"
DefaultSeverity "warning"
<Match "aer error">
Regex "AER:.*error received"
SubmatchIdx -1
</Match>
<Match "incident time">
Regex "(... .. ..:..:..) .* pcieport.*AER"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "root port">
Regex "pcieport (.*): AER:"
SubmatchIdx 1
IsMandatory true
</Match>
<Match "device">
PluginInstance true
Regex " ([0-9a-fA-F:\\.]*): PCIe Bus Error"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "severity_mandatory">
Regex "severity="
SubMatchIdx -1
</Match>
<Match "nonfatal">
Regex "severity=.*\\([nN]on-[fF]atal"
TypeInstance "non_fatal"
IsMandatory false
</Match>
<Match "fatal">
Regex "severity=.*\\([fF]atal"
Severity "failure"
TypeInstance "fatal"
IsMandatory false
</Match>
<Match "corrected">
Regex "severity=Corrected"
TypeInstance "correctable"
IsMandatory false
</Match>
<Match "error type">
Regex "type=(.*),"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "id">
Regex ", id=(.*)"
SubmatchIdx 1
</Match>
</Message>
</Logfile>
</Plugin>
Example graph¶
None yet. Add one now!
Dependencies¶
Caveats¶
See also¶
- Plugin-logparser/config
- Plugin-logparser/tests
- Logparser high level design document
- PR and discussion on github
Configuration¶
Configuration options · from collectd.conf(5)
See this section in the full configuration reference
The logparser plugin is used to parse different kinds of logs. Setting proper options you can choose strings to collect. Plugin searches the log file for messages which contain several matches (two or more). When all mandatory matches are found then it sends proper notification containing all fetched values.
Synopsis:
<Plugin logparser>
<Logfile "/var/log/syslog">
FirstFullRead false
<Message "pcie_errors">
DefaultType "pcie_error"
DefaultSeverity "warning"
<Match "aer error">
Regex "AER:.*error received"
SubmatchIdx -1
</Match>
<Match "incident time">
Regex "(... .. ..:..:..) .* pcieport.*AER"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "root port">
Regex "pcieport (.*): AER:"
SubmatchIdx 1
IsMandatory true
</Match>
<Match "device">
PluginInstance true
Regex " ([0-9a-fA-F:\\.]*): PCIe Bus Error"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "severity_mandatory">
Regex "severity="
SubMatchIdx -1
</Match>
<Match "nonfatal">
Regex "severity=.*\\([nN]on-[fF]atal"
TypeInstance "non_fatal"
IsMandatory false
</Match>
<Match "fatal">
Regex "severity=.*\\([fF]atal"
Severity "failure"
TypeInstance "fatal"
IsMandatory false
</Match>
<Match "corrected">
Regex "severity=Corrected"
TypeInstance "correctable"
IsMandatory false
</Match>
<Match "error type">
Regex "type=(.*),"
SubmatchIdx 1
IsMandatory false
</Match>
<Match "id">
Regex ", id=(.*)"
SubmatchIdx 1
</Match>
</Message>
</Logfile>
</Plugin>
Options:
- Logfile File
- The Logfile block defines file to search. It may contain one or more Message blocks which are defined below.
- FirstFullRead true|false
- Set to true if the file has to be parsed from the beginning on the first read. If false only subsequent writes to log file will be parsed.
- Message Name
- Message block contains matches to search the log file for. Each Message block builds a notification message using matched elements if its mandatory Match blocks are matched.
- DefaultPluginInstance String
- Sets the default value for the plugin instance of the notification.
- DefaultType String
- Sets the default value for the type of the notification.
- DefaultTypeInstance String
- Sets the default value for the type instance of the notification.
- DefaultSeverity String
- Sets the default severity. Must be set to "OK", "WARNING" or "FAILURE". Default value is "OK".
- Match Name
- Multiple Match blocks define regular expression patterns for extracting or excluding specific string patterns from parsing. First and last Match items in the same Message set boundaries of multiline message and are mandatory. If these matches are not found then the whole message is discarded.
- Regex Regex
- Regular expression with pattern matching string. It may contain subexpressions, so next option SubmatchIdx specifies which subexpression should be stored.
- SubmatchIdx Integer
- Index of subexpression to be used for notification. Multiple subexpressions are allowed. Index value 0 takes whole regular expression match as a result. Index value -1 does not add result to message item. Can be omitted, default value is 0.
- Excluderegex Regex
- Regular expression for excluding lines containing specific matching strings. This is processed before checking Regex pattern. It is optional and can be omitted.
- IsMandatory true|false
- Flag indicating if Match item is mandatory for message validation. If set to true, whole message is discarded if it's missing. For false its presence is optional. Default value is set to true.
- PluginInstance true|String
- If set to true, it sets plugin instance to string returned by regex. It can be overridden by user string.
- Type true|String
- Sets notification type using rules like PluginInstance.
- TypeInstance true|String
- Sets notification type instance using rules like above.
- Severity String
- Sets notification severity to one of the options: "OK", "WARNING", "FAILURE".