Filter specific nested dictionary keys in a function


(watermelon) #1

Hi all,

I’m having some trouble with using functions in assign where expressions. I followed the docs pretty closely on this – however, the problem is not due to the functions I don’t think.

My main goal is to make it so certain hosts (they are switches) do not send email notifications for certain interfaces (since each interface is configured as a service) to me during work hours.

Here’s what I have so far:
switches.conf

object Host "<mySwitch>" {
  import "generic-host"
  address = "<IP>"

  vars.int["Port 0/1"] = {
    int = "gigabitethernet1/0/1"
    port_type = "uplink"
  }
  vars.int["Port 0/2"] = {
    int = "gigabitethernet1/0/2"
    port_type = "workstation"
  }

...

  vars.int["Port 0/8"] = {
    int = "gigabitethernet1/0/8"
    port_type = "workstation"
  }
}

functions.conf

globals.check_int_port_type = function(host, type) {
  if (typeof(host.vars.int) != Dictionary) {
    return false
  }

  for (key => val in host.vars.int) {
    if (typeof(val) == Dictionary && val.port_type == type) {
      return true
    }
  }

  return false
}

services.conf

/* use this service definition for any port type other than a workstation */
apply Service for (int => config in host.vars.int) {
  import "generic-service"
  check_command = "check_iftraffic"
  vars += config

  vars.notification["mail1"] = {
    groups = [ "test" ]
  }

  assign where !check_int_port_type(host, "workstation")
}

/* use this service definition for workstations */
apply Service "Workstation: " for (int => config in host.vars.int) {
  import "generic-service"
  check_command = "check_iftraffic"
  vars += config

  vars.notification["mail2"] = {
    groups = [ "test" ]
  }

  assign where check_int_port_type(host, "workstation")
}

Then, I thought I would create a Notification object to apply only during certain hours based on the type of notification (mail1, mail2). I haven’t written this part out yet because it seems I haven’t gotten the function in functions.conf to work.

As you can see, in services.conf, I’m using the following assign where expressions:

assign where check_int_port_type(host, "workstation")

  • where I want to assign the service for a host that has a port_type of “workstation”, indicating that the port is connected to a patch panel which is connected to a workstation (and I don’t want notifications during work hours for)

assign where !check_int_port_type(host, "workstation")

  • where I want to assign the service for a host that has any port_type except “workstation” (where I do want notifications during work hours for)
  • this actually does not work as I thought it would with the ! and I had to pass a different string into the function (I put “uplink” and got rid of the !. Maybe @dnsmichi could clarify this? Should it work the way I have it written above?

The problem is that Icinga creates two sets of identical services: currently I have 48 services shown without the "Workstation: " prefix and 48 services shown with the "Workstation: " prefix for my mySwitch host.

This is most definitely because the apply Service is too broad; as soon as it sees one port_type with value “workstation”, it assigns the Service and then iterates through each and every dictionary key defined as int regardless of their port_type because of this part in the apply service definition:
for (int => config in host.vars.int)

Does anybody know a way to work around this, other than creating individual service objects for each port that corresponds to a workstation that I have on my switches? This is the only way that I can think of how to fix this.

Thanks in advance!!

Sorry if this seems like a “wall of text”.


(Carsten Köbke) #2

Tried it with 2 Dictionaries? One that hold all Ports with Workstation and the other without it.


(Michael Friedrich) #3

I’d say, you’ll use the debug console and connect to your running Icinga 2 instance where the global registered function is available. Then you can just call it, and test various negations or different return types.

icinga2 console --connect 'https://root:icinga@localhost:5665/'
=> var my_host = get_host("oneofyourchoice")
=> check_int_port_type(my_host, "workstation")

I don’t have the time atm to rebuild your configuration and logic here.


(watermelon) #4

@Carsten
hmm, I’m not sure if this is correct solution because I have realized a different issue that I’ll discuss below.

@dnsmichi
I used the debug console and tried a few negations/return types and now I realize that the example from the docs is not quite exactly what I want. The function from the example in the docs will return true as soon as it sees one occurrence of “workstation” for instance, which is why I think the service is being applied for every single interface like I mentioned initially.

Icinga 2 (version: r2.8.2-1)
<1> => var my_host = get_host("<myHost>")
null
<2> => check_int_port_type(my_host, "workstation")
true
<3> => check_int_port_type(my_host, "uplink")
true
<4> => check_int_port_type(my_host, "test")
false
<5> => !check_int_port_type(my_host, "test")
true

I think ideally I want to maybe return an array of the interfaces of port_type “workstation”, and maybe from there have a service object apply for the objects in the array – although I’m not sure how to go about this and don’t even know if services can apply like this. What do you guys think?


(Michael Friedrich) #5

Hmmm ok, now I’ve understood it a little better. I would try to avoid two locations which check for a specific key in host.vars.

Let’s try it. You’re an awesome community member helping with so many answers, you’ll deserve it :slight_smile:

Filter function for assign where

  • The function should only take a dictionary as parameter, not the entire host object itself. This leaves away hardcoding host.vars.int all over.
  • Give the parameters and variables telling names. For a function, you’ll type them just once.
  • The typeof() checks are already awesome, maybe add some log() calls for better troubleshooting

Standalone config below.

object Host "<mySwitch>" {
  check_command = "dummy"

  vars.int["Port 0/1"] = {
    int = "gigabitethernet1/0/1"
    port_type = "uplink"
  }
  vars.int["Port 0/2"] = {
    int = "gigabitethernet1/0/2"
    port_type = "workstation"
  }
}

globals.check_int_port_type = function(interfaces, port_type) {
  if (typeof(interfaces) != Dictionary) {
    log(LogWarning, "check_int_port_type", "interfaces parameter is not a Dictionary, but " + typeof(interfaces).to_string() + " (value: '" + Json.encode(interfaces) + "')")
    return false
  }

  for (ifname => ifvalues in interfaces) {
    log(LogInformation, "check_int_port_type", "Evaluating interface " + ifname + " with values: '" + Json.encode(ifvalues) + "'")
    if (typeof(ifvalues) == Dictionary && ifvalues.port_type == port_type) {
      log(LogInformation, "check_int_port_type", "Matched port_type: " + port_type)
      return true
    }
  }

  return false
}

Doesn’t satisfy yet, but looks better.

  • If I would need to negate a function boolean return value, why not just use ignore where. We’re already in a for loop anyways.
/* use this service definition for any port type other than a workstation */
apply Service for (int => config in host.vars.int) {
  check_command = "dummy"
  vars += config

  ignore where check_int_port_type(host.vars.int, "workstation") // <--- this for the exclusion, and the changed function parameter
}

/* use this service definition for workstations */
apply Service "Workstation: " for (int => config in host.vars.int) {
  check_command = "dummy"
  vars += config

  assign where check_int_port_type(host.vars.int, "workstation") // <--- and this, just using the different function argument here
}

Doesn’t work yet though, as the function always returns true, since we loop in there again over all interfaces, and one always is the workstation key. Though we’ve learned that from programming the DSL now, with fancy logs.

information/ApiListener: My API identity: imagine
information/check_int_port_type: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/check_int_port_type: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/check_int_port_type: Matched port_type: workstation
information/check_int_port_type: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/check_int_port_type: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/check_int_port_type: Matched port_type: workstation
information/check_int_port_type: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/check_int_port_type: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/check_int_port_type: Matched port_type: workstation
information/check_int_port_type: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/check_int_port_type: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/check_int_port_type: Matched port_type: workstation

Doesn’t work here

Pre-filter data for apply for rules

Ok, new idea with the function pre-filtering all data. This is a little brainfuck since you need to supply two conditions:

  • exclude a specific port_type (best to be matched with a filter) which means cutting it off the dictionary
  • include a specific port_type (and only this, or go by a wildcard pattern match), only returning this as a result

In order to avoid two different functions, I’ve developed a function which takes the port_type_filter and controls exclude or include with a boolean parameter.

This is the function signature.

globals.get_host_interfaces = function(interfaces, port_type_filter, exclude_enabled) {

This can then be used inside the service apply rules, shortcut and simple.

/* use this service definition for any port type other than a workstation */
apply Service for (int => config in get_host_interfaces(host.vars.int, "*workstation*", true)) {
  check_command = "dummy"
  vars += config
}

/* use this service definition for workstations */
apply Service "Workstation: " for (int => config in get_host_interfaces(host.vars.int, "*workstation*", false)) {
  check_command = "dummy"
  vars += config
}

Filter specific nested dictionary keys in a function

I’m using lots of logging for development here, this keeps it simple to know if conditions are matched, and at which place a problem might occur.

I’m also a fan of re-using any DSL feature which makes programming easier. Especially for complex value types like a Dictionary, a JSON encoded string is similar to Perl’s Data::Dumper or PHP’s var_dump().

First, define the result storage.

  var res = {}

Then check for specific parameter values, if they do meet our requirements. If not, return an empty result. This makes apply for think that no objects matched. Log something for better error handling.

  if (typeof(interfaces) != Dictionary) {
    log(LogWarning, "get_host_interfaces", "interfaces parameter is not a Dictionary, but " + typeof(interfaces).to_string() + " (value: '" + Json.encode(interfaces) + "')")
    return res // empty
  }

Loop through the interfaces we’ve gotten as interfaces parameter. Log some debug information to see which values are currently processed.

  for (ifname => ifvalues in interfaces) {
    log(LogInformation, "get_host_interfaces", "Evaluating interface " + ifname + " with values: '" + Json.encode(ifvalues) + "'")

If the nested dictionary is not of the type Dictionary, avoid to process any further and just return. The user must fix the configuration then.

    if (typeof(ifvalues) != Dictionary) {
      log(LogInformation, "get_host_interfaces", "Wrong type for ifvalues for ifname + " + ifname)
      return {}
    }

Add some debug logging to see whether excluding or including port types is enabled (you can leave that away, it just helps with the logging chain).

    log(LogInformation, "get_host_interfaces", "Port_type_filter: " + port_type_filter + " ifvalues.port_type: " + ifvalues.port_type)
    if (exclude_enabled) {
      log(LogInformation, "get_host_interfaces", "exclude enabled")
    } else {
      log(LogInformation, "get_host_interfaces", "exclude disabled")
    }

If the first condition is matched, which means that exclude the given filter if matching on the port_type attribute, then we want to store the ifvalues from the nested dictionary just again in our result storage res with the same key ifvalues. Add some logging to see what’s stored.

    if (exclude_enabled == true && !match(port_type_filter, ifvalues.port_type)) {
       log(LogInformation, "get_host_interfaces", "Matched port_type: " + ifvalues.port_type)

       res[ifname] = ifvalues

       log(LogInformation, "get_host_interfaces", "Adding to result: " + Json.encode(res))

If the second condition is matched, which means that include the given filter if matching in the port_type attribute, then we want to store the values in a similar manner just like above. Again, specific scoped logging so that I know which condition is currently met.

    } else if (exclude_enabled == false && match(port_type_filter, ifvalues.port_type)) {
       log(LogInformation, "get_host_interfaces", "Matched port_type: " + ifvalues.port_type)

       res[ifname] = ifvalues

       log(LogInformation, "get_host_interfaces", "Adding to result: " + Json.encode(res))

Add an else condition, just to log if something doesn’t match at all. This helps to refine the filters. The reason I am going for a wildcard pattern match here is that you may want to re-use this function for other apply rules, where the port_type isn’t necessarily only “workstation”. Or do whatever with it :smiley:

    } else {
       log(LogInformation, "get_host_interfaces", "Nothing matched")
    }

Finally, log the result in res again, to see where we are at, then return the value. This is what the apply for rule actually gets for processing in the for loop. If your brain didn’t explode yet, continue implementing it.

  }

  log(LogInformation, "get_host_interfaces", "Result: " + Json.encode(res))

  return res
}

Tests

Now let’s see what happens with that code.

information/get_host_interfaces: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/get_host_interfaces: Port_type_filter: *workstation* ifvalues.port_type: uplink
information/get_host_interfaces: exclude enabled
information/get_host_interfaces: Matched port_type: uplink
information/get_host_interfaces: Adding to result: {"Port 0/1":{"int":"gigabitethernet1/0/1","port_type":"uplink"}}
information/get_host_interfaces: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/get_host_interfaces: Port_type_filter: *workstation* ifvalues.port_type: workstation
information/get_host_interfaces: exclude enabled
information/get_host_interfaces: Nothing matched
information/get_host_interfaces: Result: {"Port 0/1":{"int":"gigabitethernet1/0/1","port_type":"uplink"}}
information/get_host_interfaces: Evaluating interface Port 0/1 with values: '{"int":"gigabitethernet1/0/1","port_type":"uplink"}'
information/get_host_interfaces: Port_type_filter: *workstation* ifvalues.port_type: uplink
information/get_host_interfaces: exclude disabled
information/get_host_interfaces: Nothing matched
information/get_host_interfaces: Evaluating interface Port 0/2 with values: '{"int":"gigabitethernet1/0/2","port_type":"workstation"}'
information/get_host_interfaces: Port_type_filter: *workstation* ifvalues.port_type: workstation
information/get_host_interfaces: exclude disabled
information/get_host_interfaces: Matched port_type: workstation
information/get_host_interfaces: Adding to result: {"Port 0/2":{"int":"gigabitethernet1/0/2","port_type":"workstation"}}
information/get_host_interfaces: Result: {"Port 0/2":{"int":"gigabitethernet1/0/2","port_type":"workstation"}}

That’s good, we have two apply for rule calls here, and both of them loop over the host object interfaces with two of them, makes it four evaluation rounds.

We can also see that 2 conditions matched, 2 did not. That’s what I’ve expected.

Now let’s verify this with object list:

imagine /etc/icinga2/tests # icinga2 object list --type Service 
Object '<mySwitch>!Port 0/1' of type 'Service':
  % declared in '/etc/icinga2/tests/watermelon.conf', lines 65:1-65:94
  * __name = "<mySwitch>!Port 0/1"
  * action_url = ""
  * check_command = "dummy"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 66:3-66:25
  * check_interval = 300
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "Port 0/1"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "<mySwitch>"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 65:1-65:94
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 3
  * name = "Port 0/1"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 65:1-65:94
  * notes = ""
  * notes_url = ""
  * package = "_etc"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 65:1-65:94
  * retry_interval = 60
  * source_location
    * first_column = 1
    * first_line = 65
    * last_column = 94
    * last_line = 65
    * path = "/etc/icinga2/tests/watermelon.conf"
  * templates = [ "Port 0/1" ]
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 65:1-65:94
  * type = "Service"
  * vars
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 67:3-67:16
    * int = "gigabitethernet1/0/1"
    * port_type = "uplink"
  * volatile = false
  * zone = ""

No name prefix, as obviously port_type = "uplink". Good.

Object '<mySwitch>!Workstation: Port 0/2' of type 'Service':
  % declared in '/etc/icinga2/tests/watermelon.conf', lines 71:1-71:111
  * __name = "<mySwitch>!Workstation: Port 0/2"
  * action_url = ""
  * check_command = "dummy"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 72:3-72:25
  * check_interval = 300
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "Workstation: Port 0/2"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "<mySwitch>"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 71:1-71:111
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 3
  * name = "Workstation: Port 0/2"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 71:1-71:111
  * notes = ""
  * notes_url = ""
  * package = "_etc"
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 71:1-71:111
  * retry_interval = 60
  * source_location
    * first_column = 1
    * first_line = 71
    * last_column = 111
    * last_line = 71
    * path = "/etc/icinga2/tests/watermelon.conf"
  * templates = [ "Workstation: Port 0/2" ]
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 71:1-71:111
  * type = "Service"
  * vars
    % = modified in '/etc/icinga2/tests/watermelon.conf', lines 73:3-73:16
    * int = "gigabitethernet1/0/2"
    * port_type = "workstation"
  * volatile = false
  * zone = ""

Name prefix "Workstation: " as obviously port_type = "workstation".

Awesome, it works :heart:

Working configuration

Attached is the full configuration, works standalone. Just include it in your icinga2.conf and drop all other config object includes.

mkdir -p /etc/icinga2/tests

vim /etc/icinga2/tests

include "tests/watermelon.conf"

watermelon.conf (2.4 KB)


(watermelon) #6

@dnsmichi

This is INCREDIBLE. Thank you so much for putting the time and effort into not only answering my question, but also explaining every step of the process. I never thought it would get that complicated! I had to reread this post more than a few times over, but now I think I understand what you’ve done here.

Your function works perfectly with my set up, all I had to do was drop it in like you said.

Thanks again!!! The #monitoringlove is strong!


#7

WOW!!! Just want to say THANK YOU for this great post of yours, dnsmichi.

It helped me very much, to simplify my host-configuration files by using this function in combination with the apply-rules. Can’t tell you, how useful this was for me!

Whoever struggles to filter apply rules by entries in a dictionary - this is the solution!


(Michael Friedrich) #8

(Rafael Voss) #9

@dnsmichi’s post shows me that there is still so much ways so solve things in icinga2 that I’am not aware of :slight_smile:

At the result I’am now missing Dictionaries in the Director again :smirk:


(Michael Friedrich) #10

Thanks. I am planning to show you more in a blog post series, also what’s coming with 2.10 and namespaces. Still, I got too many things on my plate for now (IcingaDB, 2.9/2.10, GitLab training, etc.) so that may take some time.