Delaying notification for a service only


(Lucas Possamai) #1

Hi there.

Is it possible to delay notifications for a service only?

According to the documentation, it only shows how to delay all notifications.

Cheers.


(Carsten Köbke) #2

Hello,

just add a custom variable to your service with the delay like vars.notification_delay and add this to your service notification rule

if(service.vars.notification_delay) {
   times.begin = service.vars.notification_delay
}

Regards,
Carsten


(Lucas Possamai) #3

That solved my problem. Thanks for the help! :slight_smile:

UPDATE: I’m trying to do that with PagerDuty. pagerduty-icinga2.conf doesn’t allow me to do that:

[2019-01-11 08:58:39 +1300] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable ‘service’
Location: in /etc/icinga2/zones.d/master/pagerduty-icinga2.conf: 85:6-85:12
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(83): // interval = 0 // disable re-notification
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(84): // times.begin = 10m // delay notification window
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(85): if(service.vars.notification_delay) {
^^^^^^^
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(86): times.begin = service.vars.notification_delay
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(87): }

Do you know why? (Sorry I didn’t mention this earlier).

UPDATE: I may have found a solution. Just trying to test it now.

pagerduty-icinga2.conf:

  if(vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }

services.conf:

// Ping Check
apply Service "Ping" {
  import "generic-service"
  check_command = "ping4"
  vars.ping_crta = "350"
  vars.ping_wrta = "150"
  vars.ping_wpl = "10"
  vars.ping_cpl = "20"
  vars.notification_delay = "10m"

(Lucas Possamai) #4

Apparently it is not working; it should have waited 10 minutes before sending the notification:

[2019-01-11 19:12:03 +1300] information/Checkable: Checking for configured notifications for object 'db1.datacentre.example.com.au!ping4'

[2019-01-11 19:12:03 +1300] information/Notification: Sending 'Problem' notification 'db1.datacentre.example.com.au!ping4!pagerduty-service' for user 'pagerduty'

[2019-01-11 19:12:03 +1300] information/Notification: Completed sending 'Problem' notification 'db1.datacentre.example.com.au!ping4!pagerduty-service' for checkable 'db1.datacentre.example.com.au!ping4' and user 'pagerduty'.

Note that if I do

if(service.vars.notification_delay) {
times.begin = service.vars.notification_delay
}

instead of

if(vars.notification_delay) {
times.begin = service.vars.notification_delay
}

It throws me an error: **critical** /config: Error: Error while evaluating expression: Tried to access undefined script variable 'service'


(Carsten Köbke) #5

Can you post the complete notification rule please.


(Lucas Possamai) #6

Sure:

apply Notification "pagerduty-service" to Service {
  command = "notify-service-by-pagerduty"
  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

// This part is the one I added.
  if(service.vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }
  
  assign where service.vars.enable_pagerduty == true
}

apply Notification "pagerduty-host" to Host {
  command = "notify-host-by-pagerduty"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

// This part is the one I added.
  if (service.vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }

  assign where host.vars.enable_pagerduty == true
}

(Carsten Köbke) #7

This is your problem

pply Notification "pagerduty-host" to Host {
  command = "notify-host-by-pagerduty"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

// This part is the one I added.
  if (service.vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }

you cant use a service var in a host notification, here you have to use host vars and put the variable also to the host object directly or via template

apply Notification "pagerduty-host" to Host {
  command = "notify-host-by-pagerduty"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

// This part is the one I added.
  if (host.vars.notification_delay) {
    times.begin = host.vars.notification_delay
  }

(Lucas Possamai) #8

yeah… that makes sense. That was my fault. However, I still got the problem:

New notification rules (pagerduty-icinga2.conf):

apply Notification "pagerduty-service" to Service {
  command = "notify-service-by-pagerduty"
  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

  if(service.vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }
  
  assign where service.vars.enable_pagerduty == true
}

apply Notification "pagerduty-host" to Host {
  command = "notify-host-by-pagerduty"
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

  if (host.vars.notification_delay) {
    times.begin = host.vars.notification_delay
  }

  assign where host.vars.enable_pagerduty == true
}

Error:

[2019-01-12 00:33:28 +1300] critical/config: Error: Validation failed for object 'db1.datacentre.example.com.au!Ping!pagerduty-service' of type 'Notification'; Attribute 'times' -> 'begin': Invalid type.
Location: in /etc/icinga2/zones.d/master/pagerduty-icinga2.conf: 68:5-68:49
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(66): 
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(67):   if(service.vars.notification_delay) {
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(68):     times.begin = service.vars.notification_delay
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(69):   }
/etc/icinga2/zones.d/master/pagerduty-icinga2.conf(70):

(Carsten Köbke) #9

Invald type, how did you set the variable in services?

it should be like vars.notification_delay = 10m without qoutes around 10m


(Lucas Possamai) #10

ohhhh that was the problem!

It is working now. Thanks for your help Carsten!


(Carsten Köbke) #11

your welcome :slight_smile:


(Lucas Possamai) #12

Hi @Carsten… sorry to get back to you. But I think this is not working. Is there a way I could test this? maybe via the API or something?

Cheers!


(Carsten Köbke) #13

You can do icinga2 object list --type notification --name *<NAMEOFNOTIFICATIONRULE>


(Lucas Possamai) #14

hmm… thanks for that!

This is the output:

Object 'webserver.datacentre.example.com!ping4!pagerduty-service' of type 'Notification':
  % declared in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49
  * __name = "webserver.datacentre.example.com!ping4!pagerduty-service"
  * command = "notify-service-by-pagerduty"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 62:3-62:41
  * command_endpoint = ""
  * host_name = "webserver.datacentre.example.com"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49
  * interval = 1800
  * name = "pagerduty-service"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49
  * period = "24x7"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 66:3-66:17
  * service_name = "ping4"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49
  * source_location
    * first_column = 1
    * first_line = 61
    * last_column = 49
    * last_line = 61
    * path = "/etc/icinga2/zones.d/master/pagerduty-icinga2.conf"
  * states = [ "OK", "Warning", "Critical" ]
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 63:3-63:36
  * templates = [ "pagerduty-service" ]
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery" ]
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 65:3-65:48
  * user_groups = null
  * users = [ "pagerduty" ]
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 67:3-67:25
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/master/pagerduty-icinga2.conf', lines 61:1-61:49

Shouldn’t I be seeing a time.begin value in there? Thanks!

notification rule:

apply Notification "pagerduty-service" to Service {
  command = "notify-service-by-pagerduty"
  states = [ OK, Warning, Critical ]
  //states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery ]
  period = "24x7"
  users = [ "pagerduty" ]

  if(service.vars.notification_delay) {
    times.begin = service.vars.notification_delay
  }
  
  assign where service.vars.enable_pagerduty == true
  ignore where "uat" in host.groups
}

service.conf:

apply Service "Ping" {
  import "generic-service"
  check_command = "ping4"
  vars.ping_crta = "350"
  vars.ping_wrta = "150"
  vars.ping_wpl = "20"
  vars.ping_cpl = "30"
  vars.notification_delay = 10m
  assign where host.address // check is executed on the master node
}

#15

This is a notification for service “ping4”:

Here you assign the notification pagerduty-service to services which have vars.enable_pagerduty = true

This is service “Ping” not “ping4” and it also doesn’t have vars.enable_pagerduty set.

Thus it looks as if your service “ping4” has vars.enable_pagerduty set but not vars.notification_delay while your service “Ping” has vars.enable_pagerduty not set but vars.notification_delay.


(Lucas Possamai) #16

Thanks @gvde. You are right.

I found the problem. There are two services.conf files.

1. /etc/icinga2/conf.d/services.conf
2. /etc/icinga2/zones/master/services.conf

the “ping4” service was the one that I didn’t change, that is why it wasn’t working. Now, when querying Icinga I get:

  * times
    * begin = 600

So, it should be all good now. Thanks for all the help!


#17

If you are using zones I wouldn’t use local config as well. I would move all configuration from /etc/icinga2/conf.d/ to the master zone or global-templates depending of the purpose of each object and remove the include_recursive “conf.d” from the main configuration file /etc/icinga2/icinga2.conf.

I would use local configuration objects only as last resort if you are also using zones. It defies the purpose of the distributed configuration with zones and makes the configuration of all masters, satellites and clients much more complicated if you have to consider the local configuration on all nodes…