Notifications (sometimes) not sent

This forum was archived to /woltlab and is now in read-only mode.
  • Hi all,


    I am not sure if this is an Icinga2 issue or an´Icinga Director issue, but as I used the Director to configure the notifications I'll put it here.


    I have the following notification apply rule:

    And normally the notifications are correctly sent after the 10 minute delay.

    For example for this service:



    But then I have this service for example, for which no notification is sent, even after 4 hours:

    Screenshot from today.


    I'm not sure if the intervals of the check may be a problem?!


    Maybe someone can enlighten me :D


    Thx and greetings

  • some additional info:


    tried sending a custom notification, but no notification was created:

    Code
    1. [2017-12-14 10:48:58 +0100] information/ExternalCommandListener: Executing external command: [1513244938] SEND_CUSTOM_SVC_NOTIFICATION;ST-XXX-RZ;xxx_Funktion;0;pba;
    2. [2017-12-14 10:48:58 +0100] information/Checkable: Checking for configured notifications for object 'ST-XXX-RZ!xxx_Funktion'

    Then forced sending the custom notification and two notifications were created:

    Code
    1. [2017-12-14 10:51:19 +0100] information/Notification: Sending 'Custom' notification 'ST-XXX-RZ!xxx_Funktion!Autoticketing-Services' for user 'Autoticket-Intern'
    2. [2017-12-14 10:51:19 +0100] information/Notification: Completed sending 'Custom' notification 'ST-XXX-RZ!xxx_Funktion!Autoticketing-Services' for checkable 'ST-XXX-RZ!xxx_Funktion' and user 'Autoticket-Intern'.
    3. [2017-12-14 10:51:19 +0100] information/Notification: Sending reminder 'Problem' notification 'ST-XXX-RZ!xxx_Funktion!Autoticketing-Services' for user 'Autoticket-Intern'
    4. [2017-12-14 10:51:19 +0100] information/Notification: Completed sending 'Problem' notification 'ST-XXX-RZ!xxx_Funktion!Autoticketing-Services' for checkable 'ST-XXX-RZ!xxx_Funktion' and user 'Autoticket-Intern'.


    Also found something very similar in another setup:

    Service with a 5min check interval, 1min retry interval and 5 check attempts.


    The service history looks like this, no notifications since Nov 10.




    On Nov 10 I configured some scheduled downtimes under /etc/icinga2/conf.d/downtimes.conf

    Also just noticed thatservice icinga2 status looks like this:

    The post was edited 1 time, last by log1c ().

  • Some more info from the api (/v1/objects/notifications)

    Code
    1. last_notification:1513245079.8763101101 (my forced notif at 14.12.2017 10:51:19)
    2. last_problem_notification:1513245079.8763101101
    3. next_notification:1513333829.0140430927 (15.12.2017 11:30:29, but why?)
    4. no_more_notifications:true


    I don't get why the notifications don't work for the service

  • Added a new test notification rule, forced a re-check and the notification got sent ?(:/



    Output from icinga2 object list --type Notification --name *Autoticketing-xxx-Services*



    edit:

    found this in the debug log after updating the notification rule (added a new service which also got notified right away after a re-check):

    Code
    1. [2017-12-18 10:21:14 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_services SET action_url = '', active_checks_enabled = '1', check_command_args = '', check_command_object_id = 31, check_interval = '60',
    2. config_hash = '994555da2ea2f8e348378fd8e97dc78320a941bb497e291f8e82d4433413138c', config_type = '1', display_name = 'xxx_Funktion', event_handler_enabled = '1', flap_detection_enabled = '0',
    3. freshness_checks_enabled = '1', freshness_threshold = '3600', high_flap_threshold = '30', host_object_id = 2821, icon_image = '', icon_image_alt = '', instance_id = 1, is_volatile = '0',
    4. low_flap_threshold = '25', max_check_attempts = '3', notes = 'bla', notes_url = '', notification_interval = '0', notifications_enabled = '1', notify_on_critical = '0', notify_on_downtime = '0',
    5. notify_on_flapping = '0', notify_on_recovery = '0', notify_on_unknown = '0', notify_on_warning = '0', passive_checks_enabled = '0', process_performance_data = '1', retry_interval = '10',
    6. service_object_id = 6878, stalk_on_critical = '0', stalk_on_ok = '0', stalk_on_unknown = '0', stalk_on_warning = '0' WHERE service_object_id = 6878

    for my everything looks good in there EXCEPT notify_on_critical = '0'


    The post was edited 1 time, last by log1c ().

  • Right. But I didn't force a notification, only a re-check of the service(s).

    After updating the notification rule with a new service this new service got notified after the next check.

    I updated my last post with more logs

  • That log is from the IDO feature, I was more looking for details on the Notification feature itself, how states/types filters are evaluated etc.

  • ah, ok.

    here is what I got after adding a new notification rule for me to the service. This new rule has the same config (in director) as the Autoticketing rule:

    Untiln now there is nothing in the whole log concerning my service and the Autoticket notification rule (searched the log for ST-xxx-RZ!xxx_Funktion!Autoticketing-Services (Hostname!Service-Name!Notifcation-name)

  • Maybe that notification object does not exist as config object, tried "icinga2 object list --type Notification --name *Autoticketing-Services*" already?

  • yes. It does exist

    Output from icinga2 object list --type Notification --name *Autoticketing-xxx-Services*


  • little update:

    excluded the service (and others) from my "big" notification rule and created a new one with a one hour delay.

    As the checks were already critical for several days/hours the notification worked instantly.


    Will monitor if it works normally now, after the checks switched to "OK" again, or if this was just a "one timer".


    cheers and merry christmas (in advance) :)