Can icinga monitor re-send a notification if the status hasn't changed but the plugin output has

Hi,

We have a monitor called device-events which looks for a set of eventid in logs tied to a device. We categorize the eventid as “set-eventid” and “clear-eventid”

For e.g. lets pick devicename as localhost-a and has 4 eventid where event id [1,2] are set event id and [3,4] are clear event id and the monitor will send a warning alert if it saw any of the set-eventid in last 10 mins and reset it as OK if we see any of clear-eventid. the plugin is set to run once every 5 mins. When monitor check runs it looks for all event id reported in last 10 mins range from elasticsearch index

We have a scenario where let’s say at 9AM the device has logged event id 1 and the monitor check runs at 9:01 AM

the icinga monitor output looks something like
“localhost-a has reported event id 1.” and we set the status as WARNING and send a notification for this alert

Now during the next poll we see event id 2 reported at 9:04 AM the icinga monitor output looks like
"localhost-a has reported event id 1.localhost-a has reported event id 2. " . But the status is still WARNING since it’s a set-eventid but there was a new event id 2 present in icinga service output which never gets reported. Is there a capability in icinga where we can know if the output of monitor has changed even though the status hasn’t changed and still send out a notification.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.2-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-9-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 022328c363ac```

Icinga can’t deal with it. I’ve asked a similar question and in another thread the recommendation points to Elasticsearch.

Thanks @rsx . Did you find a better alternative to kick on notification for these scenarios. since our monitors are already alerting based on events from elasticsearch what would be a viable option. I was thinking

  1. write the no of events in plugin output with state in local file and for next poll compare the new no of events in plugin output and if it’s > than previous logged count then force a state change that would send out a notification
  2. build out other monitor that queries this 1st icinga monitor plugin output using icinga api and have them deal with individual events

Both options seems like re-engineering stuffs and was wondering if someone has a better approach for this.

Or may be is there a way to reset the monitor status to OK from warning right after notification is being sent out within same plugin. That way once a notification is sent after event is detected we reset it OK and then if there are new events in output during next poll we don’t miss them

Unfortunately, I still have no satisfying solution. I’ve written a check plugin for vSphere alarms with a similar idea like yours reset. But the other way round means when the plugins detects a new alarm it exits with exitcode 0 and output “reset”. With then next run it sends the notification. For detection of changes I send the current output to the plugin with:

vars.icinga_last_output = {{get_service(host.name, “vsphere_alarms”).last_check_result.output}}

I was trying to see if there is a way to update/modify the state of icinga monitor itself in following sequence

  1. once we detect event id 1 and monitor is warning -> send out notification for us. As soon as notification is sent update the monitor state to be OK. so that in next poll if we get event id 2 in output the monitor status will be reset from OK to warning and we will get a notification for 2nd event.

But seems like i cannot update the state of monitor, I can disable checks/ notifications via api calls but not the state of monitor(not sure if it’s supposed to work that way) but I found this in their api doc

User-specified filters are run in a sandbox environment which ensures that filters cannot modify Icinga’s state, for example object attributes or global variables.

curl -k -u <username>:<password> -H 'Accept: application/json' -X POST 'https://<master_ip>:5665/v1/objects/services/<hostname>!<service_name>' -d '{ "attrs": { "state": 1.0 }, "pretty": true }'
it does return 200 code but monitor state is not WARNING and the subsequent get api call doesn’t seem to have made the update.