Kentix service doesn't trigger Notifications

notifications
icingaweb2

#1

Hello there.

I’m currently looking into expanding our existing monitoring setup.
I already did setup notifications via Signal Messenger, and while configuring which services should trigger a Signal notification, I noticed that our existing Kentix sensor checks are not sending notifications for some reason.

I compared the configuration of the Kentix checks and other services which send out notifications, but I can’t find a reason it would block notifications.
I have to say, that I’m not quite sure if the system is configured nicely, since some things are setup directly in the server config files, while some are configured via the web interface.

Here are the configs for Signal notifications:

/etc/icinga2/conf.d/notifications.conf:

apply Notification "signal-icingaadmin" to Host {
  import "signal-host-notification"
  users = ["--censored--"]

  assign where "signal-notification" in host.groups

  if (host.vars.notification_interval) {
    interval = host.vars.notification_interval
  }
}

apply Notification "signal-icingaadmin" to Service {
  import "signal-service-notification"
  users = ["--censored--"]

  assign where "signal-notification" in host.groups

  if (host.vars.notification_interval) {
    interval = host.vars.notification_interval
  }
}

/etc/icinga2/conf.d/notifications.conf:

template Notification "signal-host-notification" {
  command = "signal-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  period = "24x7"
}

template Notification "signal-service-notification" {
  command = "signal-service-notification"

  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  period = "24x7"
}

/etc/icinga2/conf.d/commands.conf:

object NotificationCommand "signal-host-notification" {
  import "plugin-notification-command"
  command = [ SysconfDir + "/icinga2/scripts/signal-host-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    HOSTSTATE = "$host.state$"
    LONGDATETIME = "$icinga.long_date_time$"
    HOSTOUTPUT = "$host.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"
    SIGNALNUMBER = "--censored--"
    USERPHONE = "$user.pager$"
  }
  timeout = 10m
}

object NotificationCommand "signal-service-notification" {
  import "plugin-notification-command"
  command = [ SysconfDir + "/icinga2/scripts/signal-service-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"
    SERVICEDESC = "$service.name$"
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    SERVICESTATE = "$service.state$"
    LONGDATETIME = "$icinga.long_date_time$"
    SERVICEOUTPUT = "$service.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"
    SERVICEDISPLAYNAME = "$service.display_name$"
    SIGNALNUMBER = "--censored--"
    USERPHONE = "$user.pager$"
  }
  timeout = 10m
}

/etc/icinga2/scripts/signal-host-notification.sh:

#!/bin/sh
template=`cat <<TEMPLATE
$NOTIFICATIONTYPE - $HOSTDISPLAYNAME is $HOSTSTATE

Notification Type: $NOTIFICATIONTYPE

Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $HOSTSTATE

Date/Time: $LONGDATETIME

Additional Info: $HOSTOUTPUT

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`

/usr/bin/printf "%b" "$template" | /usr/local/bin/signal-cli -u $SIGNALNUMBER send $USERPHONE

/etc/icinga2/scripts/signal-service-notification.sh:

#!/bin/sh
template=`cat <<TEMPLATE
$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE

Notification Type: $NOTIFICATIONTYPE

Service: $SERVICEDESC
Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $SERVICESTATE

Date/Time: $LONGDATETIME

Additional Info: $SERVICEOUTPUT

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`

/usr/bin/printf "%b" "$template" | /usr/local/bin/signal-cli -u $SIGNALNUMBER send $USERPHONE

Anyone of you is seeing an error here? With this config I am getting notifications about other hosts and services, but not about Kentix alarms. The alarm is shown in Icingaweb and Nagstamon.

Another side question I have, why do you assign the service notification to hosts instead of services? There is a Service scope as well, why don’t use that? Or is that just a personal preference thing?

Greetings,
Raphael


#2

Is there any more information I need to provide for this issue?

Greetings,
Raphael


(Michael Friedrich) #3

Verify that the kentix objects get notification objects generated from your apply rules, e.g. with icinga2 object list.


#4

Ah, I didn’t know that command yet, thanks.

I have that object in the list.
Here it is for the vibration alarm, which I am able to trigger easily:

Object 'Kentix.--censored--!Vibration alarm' of type 'Service':
  % declared in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/master/services.conf', lines 94:1-94:32
  * __name = "Kentix.--censored--!Vibration alarm"
  * action_url = ""
  * check_command = "dummy"
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 55:5-55:27
  * check_interval = 120
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 2:5-2:23
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "Vibration alarm"
  * enable_active_checks = false
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 56:5-56:32
  * enable_event_handler = true
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 58:5-58:31
  * enable_flapping = false
  * enable_notifications = true
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 4:5-4:31
  * enable_passive_checks = true
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 57:5-57:32
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 30
  * groups = [ "signal-notification" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 60:5-60:38
  * host_name = "Kentix.--censored--"
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/master/services.conf', lines 95:5-95:41
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 3
  * name = "Vibration alarm"
  * notes = ""
  * notes_url = ""
  * package = "director"
  * retry_interval = 30
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 3:5-3:24
  * templates = [ "Vibration alarm", "Kentix Sensor Check", "default-service" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/master/services.conf', lines 94:1-94:32
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 52:1-52:38
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 1:0-1:33
  * type = "Service"
  * vars
  * dummy_state = "1"
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 61:5-61:26
  * dummy_text = "Manual reset."
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 62:5-62:37
  * volatile = true
    % = modified in '/var/lib/icinga2/api/packages/director/icinga2-1539093315-0/zones.d/director-global/service_templates.conf', lines 59:5-59:19
  * zone = "master"

Greetings,
Raphael


(Michael Friedrich) #5

Is there a Notification object generated too? If not, the assign where expression filters don’t match and I would look into this region.


#6

Oh whoopsie, I was somehow assuming that I copied the Notification Object, here it is:

Object 'Kentix.--censored--!Vibration alarm!signal-icingaadmin' of type 'Notification':
    % declared in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50
  * __name = "Kentix.--censored--!Vibration alarm!signal-icingaadmin"
  * command = "signal-service-notification"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 67:3-67:41
  * command_endpoint = ""
  * host_name = "Kentix.--censored--"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50
  * interval = 1800
  * name = "signal-icingaadmin"
  * package = "_etc"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50
  * period = "24x7"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 74:3-74:17
  * service_name = "Vibration alarm"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50
  * states = [ "OK", "Warning", "Critical", "Unknown" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 69:3-69:45
  * templates = [ "signal-icingaadmin", "signal-service-notification" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 66:1-66:51
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd", "DowntimeStart", "DowntimeEnd", "DowntimeRemoved" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 70:3-72:57
  * user_groups = null
  * users = [ "--censored--" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 97:3-97:25
  * vars = null
  * zone = "master"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 93:1-93:50

So it indeed exists, not sure what I am doing wrong.


(Michael Friedrich) #7

I would remove the condition in the notification apply rule which inherits this value, and test again. It shouldn’t make a difference, but who knows.


#8

I tried this now and it didn’t make any difference as well.

Greetings,
Raphael


(Michael Friedrich) #9

Is there anything in the debug log when this notification should occur? Specifically, filters which prevent this from sending.


#10

This is the only thing popping up in the debug log regarding the Kentix Sensor when I trigger it:

[2018-10-23 09:38:19 +0200] information/ExternalCommandListener: Executing external command: [1540280296] PROCESS_SERVICE_CHECK_RESULT;Kentix.--censored--;Vibration alarm;1;Vibration alarm,

Greetings,
Raphael


(Michael Friedrich) #11

Does this really trigger a HARD state change with a notification involved? To me this sounds more like a SOFT state, but without further log lines, hard to guess.


#12

I think I finally resolved the mystery. I don’t have a solution yet, but the command for the service is a dummy check which is always returning that it’s working. I guess the Kentix itself is somehow triggering something, which then doesn’t trigger a notification. I’ll look further into that and hopefully find something :slight_smile:

Greetings,
Raphael