Influxdb and SNMP perfomance Data: Separate Interface-Names and Interface-Metrics ("_in_Octets")

snmp
influxdb

(sam) #1

Hello,

I set up Icinga2 with Influxdb and Grafana and SNMP checks for network devices. I got everything running and the performance data is parsed into Influxdb and can be querried by Grafana.

Till now I tried the check-interfaces[1] snmp plugin and the manubulon check_snmp_int.pl[2] plugin. My problem is, that with both plugins the performance data is stored in Influxdb in a way I don’t know how to make much use of. Icinga2 parses the following to Influx

check_snmp_int.pl:

Port-channel1_in_octet 24,200,351,098.00
Port-channel1_out_octet 420,872,142.00
Port-channel1_in_error 0.00
Port-channel1_in_discard 0.00
Port-channel1_out_error 0.00
Port-channel1_out_error 0.00

check-interfaces:

"TenGigabitEthernet1-1-2::check_snmp::inOctets=15720371925c",
"TenGigabitEthernet1-1-2::check_snmp::outOctets=337359874c",
"TenGigabitEthernet1-1-2::check_snmp::inDiscards=0c",
"TenGigabitEthernet1-1-2::check_snmp::outDiscards=0c",
"TenGigabitEthernet1-1-2::check_snmp::inErrors=0c",
"TenGigabitEthernet1-1-2::check_snmp::outErrors=0c",
"TenGigabitEthernet1-1-2::check_snmp::inUcast=1029508c",
"TenGigabitEthernet1-1-2::check_snmp::outUcast=813564c",
"TenGigabitEthernet1-1-2::check_snmp::speed=10000000000",

which results (using [3] for /etc/icinga2/features-enabled/influxdb.conf) in the measurements “check_interfaces” and “check_snmp_int” in influxdb with the colum “service” having the value “check-interfaces” respectively “snmp-interfaces” (which is not of much use, due to the non-surplus information in regards to the measurement, which provides this information) the “value” column with the correct values of the checks and the “metrics” column with e. g. “Port-channel1_in_discard”, “Port-channel1_in_error”, “Port-channel1_out_octet”, “Port-channel1_in_octet”…

Now I would like to creat a Grafana Dashboard with variables and would like to select Host and Interface in the dashboards and then plot “in_octets”, “out-octets”, “in_discards” and so of the host and the respective interface. But due to the conglomeration of Interface-Name and Interface-Metric (“in_octet”…) beeing in one column I fail to seperate them.

Is it maybe somehow posssible to perform string manipulation in “/etc/icinga2/features-enabled/influxdb.conf” in the “service_template” section and parse the interface-name part in e. g. the service column and the Interface-Metric (without the name) part in the “metrics” column? I studied [4] and [5] but failed to find anything.

Or does anyone know Influxdb queries to sperate the stuff in a good way? Also any other suggestions are much apprechiated!

Thanks a lot in advance!

Sam

[1] github . com/NETWAYS/check_interfaces
[2] github . com/dnsmichi/manubulon-snmp
[3] www.icinga . com/2016/08/31/icinga-2-meets-influxdb/
[4] www.icinga . com/docs/icinga2/latest/doc/17-language-reference/
[5] www.icinga . com/docs/icinga2/latest/doc/18-library-reference/


(Michael Friedrich) #2

Performance data keys are provided by the plugin, and cannot be manipulated by Icinga 2 as such. You can either update/fork the plugin, or find a way to combine them in InfluxDB.

Do you have an example how this data looks like to be selected in your Grafana graphs? I’m not sure if I understood the requirement of removing the prefix entirely.


(sam) #3

thanks for your answer @dnsmichi!

Do you have an example how this data looks like to be selected in your Grafana graphs? I’m not
sure if I understood the requirement of removing the prefix entirely.

I have a couple of switches, I’d like to have two drop down menus (via Grafana Dashboard Variables) on the dashboard, one for the switch and one for the interface to be displayed. the metrics shown shoud be octets, errors and discards; all in and out.

I’m unable to find a good way to achieve this with grafana with the given data structure. The best I could get is the following (example for in_octets, has to be added for each query seperately)

DashboardVariaable “$inOctets”:

SHOW TAG VALUES FROM "check_interfaces"  WITH KEY = "metric" WHERE ("hostname" =~ /^$hostname$/ AND "metric" =~ /.*inOctets$/)

Graph Query:
SELECT 8 * derivative(mean("value"),1s) AS "value" FROM "check_interfaces" WHERE ("hostname" =~ /^$hostname$/ AND "metric" =~ /^$inOctets$/ AND "metric" =~ /.*inOctets$/) AND $timeFilter GROUP BY time($interval) fill(none)

With this I have to select every Interface for each query.

Performance data keys are provided by the plugin, and cannot be manipulated by Icinga 2 as such

Ok, is there a way to parse the perfomance data more “differentiated” to icinga? The only Documentation I found is [1], which I read as, everything after “|” is performance data and a simple key:value store and therefore only two columns in influxdb (plus the hostname). I’d need (so I think) the hostname and three more columns with interface, metric (in_octets…), value.

anyway, I’ve literally no idea about perl (package names in cpan are case sensitive, who would have guessed that :wink: ) but started with a spaghetti on wall approach to make use of the “influxdb:lineprotocol” perl module and parse the perofmance output of check_snmp_int.pl directly to influxdb and got a proof of work running.

But how do other people monitor switch-ports/-performance-data with icinga2? Not at all? Poll every single interface instead of al of them in bulk?

[1] https://nagios-plugins.org/doc/guidelines.html#PLUGOUTPUT


(Carsten Köbke) #4

Why do you use a templated variable in your query when you want to get all “inOctets” from a host/service at once?
Just remove the templated variable from your query

SELECT 8 * derivative(mean("value"),1s) AS "value" FROM "check_interfaces" WHERE ("hostname" =~ /^$hostname$/ AND "metric" =~ /.*inOctets$/) AND $timeFilter GROUP BY time($interval) fill(none)

I would add the servicename in the where clause if there is more then one service that uses “check_interfaces”

SELECT 8 * derivative(mean("value"),1s) AS "value" FROM "check_interfaces" WHERE ("hostname" =~ /^$hostname$/ AND "service" =~ /^$service$/ "metric" =~ /.*inOctets$/) AND $timeFilter GROUP BY time($interval) fill(none)

(sam) #5

thank you for your suggestion @Carsten. But I need the utilisation for each port non-aggregated and not in one graph (several hundred ports in one graph are not very usefull :slight_smile: ).

I managed to achieve what I need with extending the manubulon interfaces plugin to directly write influxdb and this waay seperating intferace-ame from the metric (octets, discards…). the following dashboard is what I intended to achieve:

here I used variables in combination with “row options” -> “Repeat for” -> “service” and can select all inetrfaces or just the ones which are of interest.