Deploying new monitors when icinga master HA is working and primary master is down

So after reading docs and other forums got icinga HA working on masters. Overview of setup

cat /etc/icinga2/zones.conf
object Endpoint "ossipemasteradc0101a.softlayer.local" {
  host = "172.16.0.aa"
}

object Endpoint "ossipesatadc0101c.softlayer.local" {
  host = "172.16.0.bb"
}

object Zone "local-dev-vergil-adc01-master" {
  endpoints = ["ossipemasteradc0101a.softlayer.local","ossipesatadc0101c.softlayer.local"]
}

object Endpoint "ossipesatadc0101a.softlayer.local" {
  host = "172.16.0.cc"
}
object Endpoint "ossipesatadc0101b.softlayer.local" {
  host = "172.16.0.dd"
}
object Zone "local-dev-vergil-adc01" {
  endpoints = ["ossipesatadc0101a.softlayer.local","ossipesatadc0101b.softlayer.local"]
  parent = "local-dev-vergil-adc01-master"
}
# The icinga2 web2 director module uses this to manage the whole configuration
object Zone "director-global" {
      global = true
}

object Zone "global-templates" {
  global = true
}

172.16.0.aa is the true primary master. 172.16.0.bb is the secondary master setup as satellite as per docs and 172.16.0.cc,172.16.0.dd are the satellite nodes. Communication across the board works and when icinga process on primary master 172.16.0.aa is shutdown I can see the checks moving to icingaweb2 on 172.16.0.bb which is what we want. I would like to know what happens when we deploy new monitors while true icinga master 172.16.0.aa is down but 172.16.0.bb is up and running and doing all master zone object checks.

For instance I create a new host via API command

curl -k -u <user>:<pass> -H 'Accept: application/json' -X PUT 'https://172.16.0.bb:5665/v1/objects/hosts/example.localdomain' \
-d '{ "templates": [ "generic-host" ], "attrs": { "address": "192.168.1.1", "check_command": "hostalive", "vars.os" : "Linux" }, "pretty": true }'

and see this host object when I load https://172.16.0.bb/icingaweb2on the UI but when I start back icinga process on true master 172.16.0.aa I don’t see this host object example.localdomain on /var/lib/icinga2 or anywhere on node and also doesn’t show up in https://172.16.0.aa/icingaweb2 . So it looks like icinga objects deployed on master zone when 172.16.0.aa process is down doesn’t syncup when icinga on 172.16.0.aa is online. Is this the expected behavior or are there some workaround for this

Misc Information:

root@ossipemasteradc0101a:~# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.8.2-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 022328c363ac
root@ossipemasteradc0101a:~# icinga2 feature list
Disabled features: debuglog elasticsearch gelf graphite influxdb opentsdb perfdata statusdata syslog
Enabled features: api checker command compatlog ido-mysql livestatus mainlog notification

the same icinga features are enabled in secondary aster 172.16.0.bb too
I tried changing accept_api = true on api.conf on true/primary master 172.16.0.aa to see if it can sync the new host object but didn’t help. Please let me know if you need further config details

The primary master should get the runtime created objects synced back once the connection is established to the secondary master. Maybe the primary master is configured to not accept any configuration (you can see such warnings in the logs), or something else prevents the config object sync. Logs on both sides would be helpful.

Further, you’re using 2.8.2 which is known to have bugs. I’d suggest to move to 2.9.1+.

Hmm, the logs on primary master 172.16.0.aa has this

[2018-09-12 11:02:46 -0500] warning/ApiListener: Ignoring config update. 'api' does not accept config.
[2018-09-12 11:02:46 -0500] warning/ApiListener: Ignoring config update from 'ossipesatadc0101c.softlayer.local' for object 'example.localdomain' of type 'Host'. 'api' does not accept config.

setting accept_config = true in /etc/icinga2/feature-enabled/api.conf doesn’t have any further effect and throws the same error. technically the secondary master 172.16.0.bb is set as satellite to primary master 172.16.0.aa and both the masters are in same zone. not sure why it wouldn’t sync the monitors created via api calls on master “b” 172.16.0.bb to master “a” once “a” is brought online. Are we sure if this is a 2.8.1 limitation/bug? Thanks!!

You’ll need to restart the instance after modifying the api.conf file. Did you do that?

Yes, I did restart and it still shows cannot accept api config after setting accept_config = true in true/primary master. Any config changes we make on icinga nodes we always restart the instance to pick up the changes