Icinga2 - monitoring of sending notifications

This forum was archived to /woltlab and is now in read-only mode.
  • Hi Everybody,

    I have 3 different self written notification scripts, which do Email, SMS and creating a Ticket in our Ticket System.

    I have the possiblities in the scripts to do some error checking and see if the notification did actually work.

    But it seems that Icinga2 just dont care about what happens with notifications, there is no simple way to check the status code of the notification itself.

    Is there some configuration to archive this?

    The only way I see currently is monitoring the Icinga logs, writing my own logs from the notification scripts and do a compare between, so that every log entry for a sending notification in Icinga log needs to have the same in the logs of my scripts.

    How do you handle this?

    Thanks in advance.

  • What do you expect icinga should do if the notification script returns non-zero ?

    How would you like that to be processed ?

  • Well, I'm not sure yet.

    Something like a Notification_Check object where you can define who gets notified, which command and in which time ranges and escalations a notification should be send, maybe try to resend the notification, or allow a notification to have a fallback notification command or an event handler if something fails, it would also be good to actually see the check as a object in the icingaweb2 frontend.

  • Hi Wolfgang, thank you for your reply.

    What do you mean answer parts of my questions?

    What I can see there are the notification objects that are created during the apply rules.

    What I am asking for is, if there are best practise ways to check if the notification scripts that are used actually worked and really sent a notification or not just excited with some error.

    The background is: I've written a script that parses icinga notification information to an online SMS gateway provider (via REST API) and the api sends back some status codes where I can determine if it worked or if there are some issues. When there are issues I need to either retry the command or try another way of notification.

    As I said, I have ideas to implement this on my own, but I'm wondering if this is not already implemented somehow as this is a major issue when it comes to stable notification for problems and wake people up in the midnight.

    Thanks in advance!

  • "icinga2 object --list ..." shows the definition of an object contained in a running instance. It should display some of the information you specified. As these things (mostly?) don't change during runtime I see no reason why it should be displayed in the frontend.

    I'd guess that the return code of notification commands is stored in the database.

  • What do you mean mostly dont change during runtime? The notification objects wont change, but thats not what I want to monitor.

    I checked the following tables in the DB:

    - icinga_contactnotificationmethods

    - icinga_contact_notificationcommands

    - icinga_notifications

    - icinga_systemcommands

    But I cant find something in the DB that points to the return code.

    Why do I want so see this as an Monitoring Object in Icinga? Because I want to know if my notification scripts are working as expected or if there are any problems.

    I cant periodically check if the notifications work because every notification costs money (for the sms provider) so I need to check the notification return codes and notifiy someone via another notification method if the first notification somehow fails.

    So it seems that there is no simple way to archive this via config, I will think about my own implementation.

  • The exit code is not stored anywhere; there were ideas to store such similar to last_check_result but for last_notification_result with the user as primary key. Ticket is somewhere, won't happen anytime soon.

    If you want to manually trace the (notification) command execution and their exit code, enable the log with level notice at least.