Skip to content

curl_xml plugin

Plugin name curl_xml
Type read
Callbacks config, read
Status supported
Since 4.10
Copyright 2009,2010 Amit Gupta
License GPLv2
Configuration collectd.conf(5)

The cURL-XML plugin reads files using libcurl and parses it as Extensible Markup Language (XML). The parsed tree is then traversed according to the user's configuration, using the XML Path Language (XPath).

This plugin is a generic plugin, i.e. it cannot work without configuration, because there is no reasonable default behavior. Please read the Plugin curl_xml section of the collectd.conf(5) manual page for an in-depth description of the plugin's configuration.

Synopsis

- → See: Plugin-cURL-XML/Config

 <Plugin "curl_xml">
   <URL "<nowiki>http://localhost/stats.xml</nowiki>">
     Host "my_host"
     Instance "some_instance"
     User "collectd"
     Password "thaiNg0I"
     VerifyPeer true
     VerifyHost true
     CACert "/path/to/ca.crt"

     <xpath "table[@id=\"magic_level\"]/tr">
       Type "magic_level"
       #InstancePrefix "prefix-"
       InstanceFrom "td[1]"
       ValuesFrom "td[2]/span[@class=\"level\"]"
     </xpath>
   </URL>
 </Plugin>

Example graphs

None yet. Add one now!

Dependencies

See also

Configuration examples

This page contains some example configurations for the cURL-XML plugin. This page is meant as a cookbook, so if you have a configuration for an aspect not handled here or a daemon not present, please feel free to add anything that's useful for you.

Sun Java System Web Server 7.0

Sun Java System Web Server 7.0 is a web server which delivers a single, secure infrastructure for all web technologies and applications. The web server exposes the monitoring statistics via a URL which can be easily consumed by the cURL-XML plugin.

The example config collects the rate of incoming and outgoing traffic, rate of 3xx,4xx and 5xx errors and the rate of the web server requests for all the virtual servers. The config also collects the web server Java statistics like Java heap usage, current Java threads and total number of Java garbage collections for all the web server processes (if more than one).

 <Plugin curl_xml>
   <URL "<nowiki>http://localhost:80/stats-xml</nowiki>">
       Instance "sunwebserver-inst1"
       <xpath "/stats/server/virtual-server">
           InstanceFrom "@id"
           ValuesFrom "request-bucket/@countBytesReceived"
                      "request-bucket/@countBytesTransmitted"
           Type "sjsws_bytes"
       </xpath>
       <xpath "/stats/server/virtual-server">
           InstanceFrom "@id"
           ValuesFrom "request-bucket/@count3xx" "request-bucket/@count4xx"
                      "request-bucket/@count5xx"
           Type "sjsws_errors"
       </xpath>
       <xpath "/stats/server/virtual-server">
           InstanceFrom "@id"
           ValuesFrom "request-bucket/@countRequests"
           Type "sjsws_requests"
       </xpath>
       <xpath "/stats/server/process">
           InstanceFrom "@pid"
           ValuesFrom "jvm/@countTotalClassesLoaded" "jvm/@countTotalClassesUnloaded"
           Type "sjsws_jvm_classes"
       </xpath>
       <xpath "/stats/server/process">
           InstanceFrom "@pid"
           ValuesFrom "jvm/@countThreads" "jvm/@countTotalThreadsStarted"
           Type "sjsws_jvm_threads"
       </xpath>
       <xpath "/stats/server/process">
           InstanceFrom "@pid"
           ValuesFrom "jvm/@sizeHeap"
           Type "sjsws_jvm_heap"
       </xpath>
       <xpath "/stats/server/process">
           InstanceFrom "@pid"
           ValuesFrom "jvm/@countGarbageCollections"
           Type "sjsws_jvm_gc"
       </xpath>
   </URL>
 </Plugin>

Twitter statistics

The following configuration uses the REST API for Twitter to read the number of status updates ("Tweets") and followers. The configuration uses the screen name "foobar" which you will have to replace by the screen name you're actually interested in.

This example demonstrates how data can be extracted from text nodes using the text() syntax.

 <Plugin "curl_xml">
   <nowiki><URL "http://api.twitter.com/1/users/show.xml?screen_name=foobar"></nowiki>
     Instance "twitter"

     <XPath "/user">
       Type "records"
       InstancePrefix "tweets"
       #InstanceFrom ""
       ValuesFrom "statuses_count/text()"
     </XPath>

     <XPath "/user">
       Type "users"
       InstancePrefix "followers"
       #InstanceFrom ""
       ValuesFrom "followers_count/text()"
     </XPath>
   </URL>
 </Plugin>

Because Twitter does some rate-limiting on the use that this API, you should not use a 10 second interval when querying this data! Currently (July 2010) the limit is 150 requests per hour, so querying the information every 30 seconds should be fine. For more information, see http://dev.twitter.com/pages/rate-limiting.

Fritz!Box Internet Router

Some models of Fritz!Box routers (e.g. 3270) allow retrieving some status information with a SOAP call. The following example shows how to call a SOAP function and parse the XML response using namespaces in order to retrieve the number of bytes sent and received by the router.

 <Plugin "curl_xml">
    <URL <nowiki>"http://192.168.0.1:49000/upnp/control/WANCommonIFC1"></nowiki>
    Instance "router_stats"
    Header <nowiki>"SoapAction: \"urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1#GetAd
donInfos\""</nowiki>
    Header "Content-Type: text/xml"
    Post <nowiki>"<?xml version=\"1.0\"?><s:Envelope xmlns:s=\"http://schemas.xmlsoap.org/soap/
envelope/\" s:encodingStyle=\"http://schemas.xmlsoap.org/soap/encoding/\"><s:Body><u:G
etAddonInfos xmlns:u=\"urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1\" /></s
:Body></s:Envelope>"</nowiki>

    Namespace "s" <nowiki>"http://schemas.xmlsoap.org/soap/envelope/"</nowiki>
    Namespace "u" "urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1"

    <XPath "/s:Envelope/s:Body/u:GetAddonInfosResponse">
      Type "ipt_bytes"
      InstancePrefix "tx"
      ValuesFrom "NewTotalBytesSent"
    </XPath>
    <XPath "/s:Envelope/s:Body/u:GetAddonInfosResponse">
      Type "ipt_bytes"
      InstancePrefix "rx"
      ValuesFrom "NewTotalBytesReceived"
    </XPath>
  </URL>
 </Plugin>

See also

Configuration

Configuration options · from collectd.conf(5)

See this section in the full configuration reference

The curl_xml plugin uses libcurl (http://curl.haxx.se/) and libxml2 (http://xmlsoft.org/) to retrieve XML data via cURL.

<Plugin "curl_xml">
  <URL "http://localhost/stats.xml">
    AddressFamily "any"
    Host "my_host"
    #Plugin "curl_xml"
    Instance "some_instance"
    User "collectd"
    Password "thaiNg0I"
    VerifyPeer true
    VerifyHost true
    CACert "/path/to/ca.crt"
    Header "X-Custom-Header: foobar"
    Post "foo=bar"

    <XPath "table[@id=\"magic_level\"]/tr">
      Type "magic_level"
      #InstancePrefix "prefix-"
      InstanceFrom "td[1]"
      #PluginInstanceFrom "td[1]"
      ValuesFrom "td[2]/span[@class=\"level\"]"
    </XPath>
  </URL>
</Plugin>

In the Plugin block, there may be one or more URL blocks, each defining a URL to be fetched using libcurl. Within each URL block there are options which specify the connection parameters, for example authentication information, and one or more XPath blocks.

Each XPath block specifies how to get one type of information. The string argument must be a valid XPath expression which returns a list of "base elements". One value is dispatched for each "base element". The type instance and values are looked up using further XPath expressions that should be relative to the base element.

Within the URL block the following options are accepted:

AddressFamily Type
IP version to resolve URL to. Useful in cases when hostname in URL resolves to both IPv4 and IPv6 addresses, and you are interested in using one of them specifically. Use ipv4 to enforce IPv4, ipv6 to enforce IPv6, or any to keep the default behavior of resolving addresses to all IP versions your system allows. If libcurl is compiled without IPv6 support, using ipv6 will result in a warning and fallback to any. If Type cannot be parsed, a warning will be printed and the whole URL block will be ignored.
Host Name
Use Name as the host name when submitting values. Defaults to the global host name setting.
Plugin Plugin
Use Plugin as the plugin name when submitting values. Defaults to 'curl_xml'.
Instance Instance
Use Instance as the plugin instance when submitting values. May be overridden by PluginInstanceFrom option inside XPath blocks. Defaults to an empty string (no plugin instance).
Interval Interval
Sets the interval (in seconds) in which the values will be collected from this URL. By default the global Interval setting will be used.
Namespace Prefix URL

If an XPath expression references namespaces, they must be specified with this option. Prefix is the "namespace prefix" used in the XML document. URL is the "namespace name", an URI reference uniquely identifying the namespace. The option can be repeated to register multiple namespaces.

Examples:

Namespace "s" "http://schemas.xmlsoap.org/soap/envelope/"
Namespace "m" "http://www.w3.org/1998/Math/MathML"
User User
Password Password
Digest true|false
VerifyPeer true|false
VerifyHost true|false
CACert CA Cert File
Header Header
Post Body
Timeout Milliseconds
These options behave exactly equivalent to the appropriate options of the cURL plugin. Please see there for a detailed description.
<StatisticsE<gt>
One Statistics block can be used to specify cURL statistics to be collected for each request to the remote URL. See the section "cURL Statistics" above for details.
<XPath XPath-expression>

Within each URL block, there must be one or more XPath blocks. Each XPath block specifies how to get one type of information. The string argument must be a valid XPath expression which returns a list of "base elements". One value is dispatched for each "base element".

Within the XPath block the following options are accepted:

Type Type
Specifies the Type used for submitting patches. This determines the number of values that are required / expected and whether the strings are parsed as signed or unsigned integer or as double values. See types.db(5) for details. This option is required.
InstancePrefix InstancePrefix
Prefix the type instance with InstancePrefix. The values are simply concatenated together without any separator. This option is optional.
InstanceFrom InstanceFrom
Specifies a XPath expression to use for determining the type instance. The XPath expression must return exactly one element. The element's value is then used as type instance, possibly prefixed with InstancePrefix (see above).
PluginInstanceFrom PluginInstanceFrom
Specifies a XPath expression to use for determining the plugin instance. The XPath expression must return exactly one element. The element's value is then used as plugin instance.

If the "base XPath expression" (the argument to the XPath block) returns exactly one argument, then InstanceFrom and PluginInstanceFrom may be omitted. Otherwise, at least one of InstanceFrom or PluginInstanceFrom is required.

ValuesFrom ValuesFrom [ValuesFrom ...]
Specifies one or more XPath expression to use for reading the values. The number of XPath expressions must match the number of data sources in the type specified with Type (see above). Each XPath expression must return exactly one element. The element's value is then parsed as a number and used as value for the appropriate value in the value list dispatched to the daemon. This option is required.