Notfication setup guide in director

(Fredde) #1

Hi,

Clean install with director, no config under /etc/icinga2 that can create conflicts. However, I try to configure notifications director configuration fail.

How do you configure notifications in director, step by step.

Thank you.

(Michael Friedrich) #2

I would look into the Icinga 2 docs for the required objects and how notifications work actually. Then start to investigate on the forms inside the Director. A NotificationCommand, User, UserGroup and a Notification object or apply rule. Maybe a Timeperiod for filtering away such stuff. Or specific filters for states and types.

(Fredde) #3

Ok, I have reproduced as mush as possible from configuration provided by default:

User groups,User template, User/Contacts & the Notification Templates. The trickey part is how to create the same Apply rules in director. When I try to get as near as possible (create, deploy and then look in preview) director deploy breaks! If I enable the notifications.conf (in /etc/icinga2/conf.d/ the output from icinga2 daemon -C is:

warning/ApplyRule: Apply rule ‘mail-icingaadmin’ (in /etc/icinga2/conf.d/notifications.conf: 11:1-11:45) for type ‘Notification’ does not match anywhere!
warning/ApplyRule: Apply rule ‘mail-icingaadmin’ (in /etc/icinga2/conf.d/notifications.conf: 23:1-23:48) for type ‘Notification’ does not match anywhere!

Code from notifications.conf:

apply Notification “mail-icingaadmin” to Host {
import "mail-host-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users

//interval = 2h

//vars.notification_logtosyslog = true

assign where host.vars.notification.mail
}

apply Notification “mail-icingaadmin” to Service {
import "mail-service-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users

//interval = 2h

//vars.notification_logtosyslog = true

assign where host.vars.notification.mail
}

How should I configure this in director, do I have to adjust the default mail notification commands imported in director? Create vars?

The docs on the website is just confusing, refering to use icinga2 with local configfiles insted of focus on guides to show how to use director.

(Michael Friedrich) #4

It doesn’t help much to nitpick on docs which were written for the core, while the Director is just an addon.

You really need to have understood the objects and their relationship, as well as the apply rule logic from Icinga 2, to fully step through the capabilities of the Director.

Still, it requires your own motivation to try things out. There isn’t a so-called “step by step” guide, especially not for a GUI.

So, https://www.icinga.com/demo/ has the Director installed. This is pretty straght forward, especially if you know about the object types already.

I’ve looked for creating my first user.

Then I would need a notification command from the commands tab.

Later I could create a Time Period object if needed.

Then I’ll stash everything commonly used into a notification template, and once that’s created, I’ll look into notifications applied to my hosts and services.

The more you are investing to try things out by yourself, the more others are willing to help in case you’re stuck. If you don’t want to do that, I’d suggest to get a hands-on workshop or training.

(Fredde) #5

Thank you for taking time to help. Im sorry but its really frustrating that most of the documentation is focus on core, not the future - director. Director is the main big factor for my customer to chose migrate from old Nagios to Icinga2/Icingaweb2/Director, without Director - no Icinga2. Its really nice.

As we started with a clean install, based on director, I had to clean up all demo configuration, to get rid of any risk of conflict problems. At the moment I have quite much configuration done, but no notification.

As I noticed demo config works with host and service notification, I tried to use it to reproduce working notification in director. I have successfully manage to:

  • Created icingaadmins group, generic-user template and icingaadmin user.
  • Created TimePeriods never,9to5 and 24x7.
  • Verified that director kickstart wizard has imported mail-host-notification & mail-service-notification command correct.
  • Created mail-host-notification & mail-service-notification Notification Templates.

Finally, I try to create matching Notification Apply Rules, resulting error:

icinga2 daemon -C

warning/ApplyRule: Apply rule ‘mail-icingaadmin’ (in /var/lib/icinga2/api/packages/director/master.example.com-1518736406-0/zones.d/master.example.com/notification_apply.conf: 1:0-1:44) for type ‘Notification’ does not match anywhere!
warning/ApplyRule: Apply rule ‘mail-icingaadmin’ (in /var/lib/icinga2/api/packages/director/master.example.com-1518736406-0/zones.d/master.example.com/notification_apply.conf: 9:1-9:48) for type ‘Notification’ does not match anywhere!

demo config looks like this:

apply Notification “mail-icingaadmin” to Host {
import "mail-host-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users

//interval = 2h

//vars.notification_logtosyslog = true

assign where host.vars.notification.mail
}

apply Notification “mail-icingaadmin” to Service {
import "mail-service-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users

//interval = 2h

//vars.notification_logtosyslog = true

assign where host.vars.notification.mail
}

I think this var is the problem, host.vars.notification.mail, cant find it in my config. So… to apply to my generic-host template, how should I configure the apply rule for hosts (same problem with services)

Thank you for input.

(Michael Friedrich) #6

I don’t think that one should force push users into the Director module. One should be able to use it, accompanied with other integrations and tools for automation, .e.g the Puppet modules where team members invest resources into.

The main problem is expectation - if you install the Director, everything must be self-explanatory, no additional knowledge required. The Director follows a different approach, you really need knowledge about the objects, and you need to know about attributes and specific things possible.

You will also learn during your evaluation that the Director doesn’t support all language features the Icinga 2 DSL provides. It abstracts certain things into web forms, apis, and sync rules but cannot provide anything capable within a static configuration file with free-form code.

One of these is the nested dictionary example you’ll see inside the example docs. As said, get familiar with the documentation on notifications in Icinga 2, understand the requirements for the specific objects, and then model them into the Director forms on your own.

I don’t see why such a thing is “frustrating” as you call it. It may be because you’re under pressure for a customer, but that really isn’t something you should put in here if project plans don’t work out. Instead, focus on the things mentioned and try them out.

(Fredde) #7

I have read the docs, they asume that I configure the notify in core, not director. As I tried to explain, all pices is in place in director, commands,users, groups, notification templates, the only part thats not working is the apply rule. I guess in very basic setup one should configure host notification to apply to generic-host template?

I loged in to your demo site, its broken, try to configure a user group under notification and you get permission error (No permission for director/userGroups). Is it possible to fix this and I can try to reproduce my config there so you can look whats missing, ok?

(Michael Friedrich) #8

You can also use the Vagrant boxes, if you need a different demo. Or you go by your own installation, Director really is a breeze to setup.

As said before, the apply rule assign where expression uses nested dictionaries. Those are 1:1 copied from the example config with static files. Your host objects likely don’t have such attributes specified, so I’d suggest to find your own pattern for applying a notification object to a host/service.

https://www.icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#apply-rules-prerequisites

(Fredde) #9

Now I have configured your demo with same config as core, I had to skip user group and time periods due to permission problems on your server. The check commands are not imported so I had to create them mauual. Please, check the output on last deploy and you will se my error.

(Fredde) #10

In short, I have to figure out how to create the host.vars.notification.mail attribute to the generic-host template right?, to apply notifications on all hosts?

(Michael Friedrich) #11

I’m not looking at the online demo, that’s not my way of doing free support. If you need hands-on support, feel free to contact an Icinga partner. The demo was just an example where one could try possible things. I’d expect to share and collect all details in here, to save the story for others later.

My point is - find a suitable and easy rule to assign notifications to hosts and services. That is the same as with services applied to hosts. Example: assign where "linux-servers" in host.groups.

(Fredde) #12

That makes sense, but still no go. When I try to create the apply rule using an hostgroup i get:

information/cli: Icinga application loader (version: r2.8.1-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: icinga2-master01.iisfredde.com
critical/config: Error: Validation failed for object ‘wiki.iisfredde.com!mail-icingaadmin’ of type ‘Notification’; Attribute ‘command’: Object ‘mail-host-notification’ of type ‘NotificationCommand’ does not exist.
Location: in [stage]/zones.d/AWS Irland Prod/notification_templates.conf: 2:5-2:38
[stage]/zones.d/AWS Irland Prod/notification_templates.conf(1): template Notification “mail-host-notification” {
[stage]/zones.d/AWS Irland Prod/notification_templates.conf(2): command = “mail-host-notification”
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[stage]/zones.d/AWS Irland Prod/notification_templates.conf(3): period = “24x7”
[stage]/zones.d/AWS Irland Prod/notification_templates.conf(4): states = [ Down, Up ]

critical/config: 1 error

Found this:

I use r2.8.1-1 (newer than 2.7.x) and have cluster/satellite setup. Does that demand adjustments in mail scrit or other types of configuration for ex the check commands?

Thank you for helping out.

(Michael Friedrich) #13

The error says, that it doesn’t know about the command itself. Where is that defined? You’ve probably just imported it as external command object inside the Director.

(Fredde) #14

Thats true, imported. Still other imported commands works fine, shouldent imported notification commans work? I just found this in core commands.cfg:

/*

  • If you prefer to use the notification scripts with environment
  • variables instead of command line parameters, you can use
  • the following commands. They have been updated from < 2.7
  • to support the new notification scripts and should help
  • with an upgrade.
  • Remove the comment blocks and comment the notification commands above.
    */

/*

object NotificationCommand “mail-host-notification” {
command = [ SysconfDir + “/icinga2/scripts/mail-host-notification.sh” ]

env = {
NOTIFICATIONTYPE = "$notification.type$"
HOSTDISPLAYNAME = "$host.display_name$"
HOSTNAME = "$host.name$"
HOSTADDRESS = "$address$"
HOSTSTATE = "$host.state$"
LONGDATETIME = "$icinga.long_date_time$"
HOSTOUTPUT = "$host.output$"
NOTIFICATIONAUTHORNAME = "$notification.author$"
NOTIFICATIONCOMMENT = "$notification.comment$"
HOSTDISPLAYNAME = "$host.display_name$"
USEREMAIL = “$user.email$”
}
}

object NotificationCommand “mail-service-notification” {
command = [ SysconfDir + “/icinga2/scripts/mail-service-notification.sh” ]

env = {
NOTIFICATIONTYPE = "$notification.type$"
SERVICENAME = "$service.name$"
HOSTNAME = "$host.name$"
HOSTDISPLAYNAME = "$host.display_name$"
HOSTADDRESS = "$address$"
SERVICESTATE = "$service.state$"
LONGDATETIME = "$icinga.long_date_time$"
SERVICEOUTPUT = "$service.output$"
NOTIFICATIONAUTHORNAME = "$notification.author$"
NOTIFICATIONCOMMENT = "$notification.comment$"
HOSTDISPLAYNAME = "$host.display_name$"
SERVICEDISPLAYNAME = "$service.display_name$"
USEREMAIL = “$user.email$”
}
}

*/

Should I have uncomment this part insted before the import in director?

(Michael Friedrich) #15

No, the 2.7+ commands are the ones which better fir the Director’s needs for parameters and custom attributes.

To me, it seems that the command isn’t available on the host itself. Highly likely that you’ve imported it from conf.d, and then commented it out. You need to add that commands.conf file to your global configuration zone, e.g. global-templates as shown in the Icinga 2 docs. This is a static config file (and now comes the important part you don’t like to hear) - which is part of Icinga 2 itself and distributed throughout the cluster. You need to understand at least how global config zones are configured.

(Fredde) #16

Ok, so I should use the not adjust the commands.cfg, just import to director and then put the content in global-templates to get the config dist to work properly for notifications? (I cant find this part in docs)

As far as zones.conf I figured out erlier that zone and endpoint must be configured on all masters and satellites to get it to work.

(Michael Friedrich) #17

It is advised to start with a fresh setup, one may of course import some required commands if they are not fully managed by the Director. More or less ITL, but sometimes notification commands might do the trick too.

Global zones used for config sync are documented here, https://www.icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#global-zone-for-config-sync and as such, since you have imported them already, putting the commands there should be sufficient.

(Fredde) #18

Qiick reply:

The /etc/icinga2/zones./d should be empty if using director? Yes/no?

(Fredde) #19

Hi I tried out the Vagrant boxes in the weekend to try to clearify things, nice job. I cant find any config examples in standalone with notifications in director and the distributed not using director.

(Michael Friedrich) #20

Director comes without configuration in these boxes, that’s correct. It is a demo afterall.

Did you try to setup a static configuration zone for syncing the commands.conf file to your satellites meanwhile?