Notification not escalating

  • Hi!


    I'm building a quite big icinga2 setup and faced a problem I wasn't able to solve within two days.


    We have a custom notification command that is used for very important services. The configs are like this:


    The notification command definition:


    The notification service:

    Code
    1. apply Notification "emergency-call" to Service {
    2. import "emergency-calltemplate"
    3. users = [ "emergency-call" ]
    4. assign where service.vars.type == "emergency"
    5. }


    The notification template:

    Code
    1. template Notification "emergency-calltemplate" {
    2. command = "emergency-call"
    3. interval = 1m
    4. times.begin = 0s
    5. period = "24x7"
    6. states = [ Critical ]
    7. types = [ Problem ]
    8. }


    The user:


    Code
    1. object User "emergency-call" {
    2. display_name = "emergency-call"
    3. period = "24x7"
    4. states = [ Critical ]
    5. types = [ Problem ]
    6. }



    And finally the service which it's applied to:





    As far as I understand, when I get a CRITICAL for this alert, notifications should be sent... but they they don't.

    It CRITICAL for about 1 hour, but still no reaction:





    From the debug log (when I force a notification in the web gui):




    As far as I understand, it doesn't execute the notification command, looks like a bug.



    I'll appreciated for any help with this issue.


    Best regards, Petr

  • There is a current bug about Notifications not being sent. I'm not sure if this applies here, can you post the log around the time the Service goes critical? Also the output of `icinga2 object list --name icinga01.somecompanydomain.com!EMERGENCY test2!emergency-call` just to make sure the Notification object is applied with all the correct values.

  • Thanks for the reply!


    I found out why it didn't worked


    I checked the object and got this:



    As you can see the "command_endpoint" is empty.


    After that I changed the config this way:



    Notification started working.




    I have two hosts in the "master" zone, both with notification feature enabled. Shouldn't it use them as command_endoint by default without hardcoding it?


    Best regards, Petr

  • Nice :thumbsup:


    When you use command_endpoint the host is different from the check source. This can lead to confusing dependencies and associations (esp. in icingaweb).

    According to your object the command_endpoint is set to "" in /etc/icinga2/zones.d/global-templates/templates/generic/service.conf, how does that config look?

  • Here is the config. Actually I thought it should had used the definitions defined here:

  • Yes of course. Two masters are in one zone, all the configs are shared through a global zone, scripts are identical with same permission etc.

  • Ok, I'm kind of lost now. Everything looks correct, and it being a bug in icinga2 wouldn't make much sense as it works on one but not the other... All the other notifications work correctly? Maybe it would help recreating the config for this Notification step by step, making sure it works along the way.


    Sorry, but I'm running out of ideas :(

  • Actually that's why I'm here - I know that the config is OK. But it doesn't work without forcing the beast to use a hardcoded command_endpoint. The notification function is only enabled within the master zone, when the notification is triggered it should just pick up a host from a zone as a command_endpoint, but it doesn't. It looks like a bug to me. I'm running various setups on icinga2 for 3 years now and this issue is very confusing to me. I hoped that you may point me to an error I made in the configuration, but it seems that the configs are OK. So it looks like a bug.... Isn't it? Email sending works just fine nevertheless.

    If you need any info on this: logs, configs - just tell me. I have a big production over there and I'm very interested in helping developers to find and fix bugs.

  • i was able to reproduce the issue with the attached config. Its cropped down to very few elements and is anonymized for security reasons since this discussion and file share is public.


    Once again, how to reproduce:
    1) Launch the config with minor changes (ip, hostname) on a host

    2) configure second master on a different host

    3) ensure services are started


    Launch this:




    Info about the servers I have:

    1) All CentOS7 (fully updated)

    2) The package:


    Files

    • icinga2.zip

      (28.69 kB, downloaded 6 times, last: )

    The post was edited 1 time, last by noret ().