So after reading docs and other forums got icinga HA working on masters. Overview of setup
cat /etc/icinga2/zones.conf
object Endpoint "ossipemasteradc0101a.softlayer.local" {
host = "172.16.0.aa"
}
object Endpoint "ossipesatadc0101c.softlayer.local" {
host = "172.16.0.bb"
}
object Zone "local-dev-vergil-adc01-master" {
endpoints = ["ossipemasteradc0101a.softlayer.local","ossipesatadc0101c.softlayer.local"]
}
object Endpoint "ossipesatadc0101a.softlayer.local" {
host = "172.16.0.cc"
}
object Endpoint "ossipesatadc0101b.softlayer.local" {
host = "172.16.0.dd"
}
object Zone "local-dev-vergil-adc01" {
endpoints = ["ossipesatadc0101a.softlayer.local","ossipesatadc0101b.softlayer.local"]
parent = "local-dev-vergil-adc01-master"
}
# The icinga2 web2 director module uses this to manage the whole configuration
object Zone "director-global" {
global = true
}
object Zone "global-templates" {
global = true
}
172.16.0.aa is the true primary master. 172.16.0.bb is the secondary master setup as satellite as per docs and 172.16.0.cc,172.16.0.dd are the satellite nodes. Communication across the board works and when icinga process on primary master 172.16.0.aa is shutdown I can see the checks moving to icingaweb2 on 172.16.0.bb which is what we want. I would like to know what happens when we deploy new monitors while true icinga master 172.16.0.aa is down but 172.16.0.bb is up and running and doing all master zone object checks.
For instance I create a new host via API command
curl -k -u <user>:<pass> -H 'Accept: application/json' -X PUT 'https://172.16.0.bb:5665/v1/objects/hosts/example.localdomain' \
-d '{ "templates": [ "generic-host" ], "attrs": { "address": "192.168.1.1", "check_command": "hostalive", "vars.os" : "Linux" }, "pretty": true }'
and see this host object when I load https://172.16.0.bb/icingaweb2on the UI but when I start back icinga process on true master 172.16.0.aa I don’t see this host object example.localdomain on /var/lib/icinga2 or anywhere on node and also doesn’t show up in https://172.16.0.aa/icingaweb2 . So it looks like icinga objects deployed on master zone when 172.16.0.aa process is down doesn’t syncup when icinga on 172.16.0.aa is online. Is this the expected behavior or are there some workaround for this
Misc Information:
root@ossipemasteradc0101a:~# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.2-1)
Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Application information:
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64
Build information:
Compiler: GNU 6.3.0
Build host: 022328c363ac
root@ossipemasteradc0101a:~# icinga2 feature list
Disabled features: debuglog elasticsearch gelf graphite influxdb opentsdb perfdata statusdata syslog
Enabled features: api checker command compatlog ido-mysql livestatus mainlog notification
the same icinga features are enabled in secondary aster 172.16.0.bb too
I tried changing accept_api = true on api.conf on true/primary master 172.16.0.aa to see if it can sync the new host object but didn’t help. Please let me know if you need further config details