# Struggeling with Host Notifications

notifications
icinga2
icingaweb2

(Benjamin Schlattmann) #1

Hi All

We have change our Monitoring System from Nagios to Icinga2 for our Linux Appliances here.

Everything looks great and I like they way how Icinga has implemented the template structure and custom variables for each object type. But somehow. Everyhing works fine but i cannot get Host notifications to be sent out. All works fine for our service checks but somehow it doesnt seem to apply any notification for host alerts (manually created or real hard states).

I can only get it to work when i check the “force” option which made me think there must be something wrong with an ‘enable_notifications’ flag but everything looks fine to me in the configuration.

-------------------------------------------------

Here are the logs I get from sending a custom notification without “force flag”:

-------------------------------------------------

[2018-12-06 13:21:28 +0100] information/HttpServerConnection: Request: POST /v1/actions/send-custom-notification (from [127.0.0.1]:36862, user: icingaweb2)
[2018-12-06 13:21:28 +0100] information/Checkable: Checking for configured notifications for object ‘servername’

-------------------------------------------------

Here are the logs I get from sending a custom notification WITH the “force flag”:

-------------------------------------------------

[2018-12-06 13:21:39 +0100] information/ApiListener: New client connection from [127.0.0.1]:36874 (no client certificate)
[2018-12-06 13:21:39 +0100] information/HttpServerConnection: Request: POST /v1/actions/send-custom-notification (from [127.0.0.1]:36874, user: icingaweb2)
[2018-12-06 13:21:39 +0100] information/Checkable: Checking for configured notifications for object ‘sg007449.tauri.ch’
[2018-12-06 13:21:39 +0100] information/Notification: Sending ‘Custom’ notification ‘servername!benitest’ for user ‘test’

Let me put together our config here:

VERSION

-------------------------------------

icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.2-1)

Copyright © 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

System information:
Platform: Red Hat Enterprise Linux Server
Platform version: 7.5 (Maipo)
Kernel: Linux
Kernel version: 3.10.0-862.14.4.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: unknown

Host Template in use

-------------------------------------

template Host “messaging_host_template” {
check_command = “hostalive4”
max_check_attempts = “3”
check_interval = 10s
retry_interval = 10s
check_timeout = 10s
enable_notifications = true
enable_active_checks = true
enable_passive_checks = true
enable_event_handler = true
enable_flapping = false
enable_perfdata = true
volatile = false
}

Host i’m testing with

-------------------------------------

object Host “” {
import “messaging_host_template”

display_name = "<secret>"
address = "<secret>"
vars.Clustername = "<secret>"
vars.Customer = "Shared"
vars.Environment = "Production"
vars.Hardewaretype = "Appliance"
vars.Productname = "Sentrion"
vars.Role = "Agent"

}

User to test with

-------------------------------------

apply Notification “benitest” to Host {
import “host_notifications”

assign where host.address
states = [ Down, Up ]
users = [ "test" ]

}

Notification Template

-------------------------------------

template Notification “host_notifications” {
command = “mail-host-notification”
interval = 30m
states = [ Down, Up ]
types = [ DowntimeEnd, DowntimeStart ]
}

And the director basics file

-------------------------------------

const DirectorStageDir = dirname(dirname(current_filename))

globals.directorWarnedOnceForThresholds = false;
globals.directorWarnOnceForThresholds = function() {
if (globals.directorWarnedOnceForThresholds == false) {
globals.directorWarnedOnceForThresholds = true
log(LogWarning, “config”, “Director: flapping_threshold_high/low is not supported in this Icinga 2 version!”)
}
}

const DirectorOverrideTemplate = “host var overrides (Director)”
if (! globals.contains(DirectorOverrideTemplate)) {
const DirectorOverrideVars = “_override_servicevars”

globals.directorWarnedOnceForServiceWithoutHost = false;
globals.directorWarnOnceForServiceWithoutHost = function() {
if (globals.directorWarnedOnceForServiceWithoutHost == false) {
globals.directorWarnedOnceForServiceWithoutHost = true
log(
LogWarning,
“config”,
“Director: Custom Variable Overrides will not work in this Icinga 2 version. See Director issue #1579
)
}
}

template Service DirectorOverrideTemplate {
/**
* Seems that host is missing when used in a service object, works fine for
* apply rules
*/
if (! host) {
var host = get_host(host_name)
}
if (! host) {
globals.directorWarnOnceForServiceWithoutHost()
}

if (vars) {
  vars += host.vars[DirectorOverrideVars][name]
} else {
  vars = host.vars[DirectorOverrideVars][name]
}

}
}

I was now searching the issue for hours in all those config files and didnt find anything. I was also playing arround with different host check-commands, different notification configs, different user configs but didnt get it to work…

Would appreaciate it a lot if you can give me a hint what I’m missing here…

Best Regards
Beni


(Michael Friedrich) #2

Custom notifications are of a specific type, and do not necessarily fit the Problem or Recovery scenario. Use them with care, and better have a real test case with a dummy host going down into a HARD state triggering the notifications on its own.

How does the corresponding Notification and Users object look like when extract with object list? More hints btw in the Troubleshooting docs.


(Benjamin Schlattmann) #3

Hi Michael

Thank you so much for your quick Reply. I didnt make the link between “Transition Types” and “States” in my mind as I thought Host alerts would always trigger an “UP” or “DOWN” state. So my Notification profiles were missing the “Problem” and “Recovery” Transition Types for host alerts which prevented it from sending any notifications.

I’ve also added the “Custom” transition type to my notification template and made sure to configure all these options in the template only as I dont want to specfy these settings on every notification apply rule again.

All works fine like a charm now. Thank you so much for your support here.

Best Regards and a great Weekend
Beni