Post icinga upgrade - Last zone sync stage validation failed

Hi All,

Recently upgraded icinga setup to 2.11.3-1 and post upgrade i am getting few errors.
“Last zone sync stage validation failed at 2020-04-28 03:44:26 +0000”

Setup details:
Its a distributed setup with 1 Master and 2 Satellites. All 3 instances are running on 2.11.3-1

Master:
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.3-1)

System information:
Platform: Red Hat Enterprise Linux Server
Platform version: 7.7 (Maipo)
Kernel: Linux
Kernel version: 3.10.0-1062.9.1.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: runner-LTrJQZ9N-project-322-concurrent-0

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

Satellite:
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.3-1)

System information:
Platform: Red Hat Enterprise Linux Server
Platform version: 7.7 (Maipo)
Kernel: Linux
Kernel version: 3.10.0-1062.9.1.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: runner-LTrJQZ9N-project-322-concurrent-0

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

Post upgrade i can see few issues like

  1. Satellite is showing message
    Icinga 2 has been running for 8 minutes and 43 seconds. Version: 2.11.3-1; Last zone sync stage validation failed at 2020-04-28 03:44:26 +0000

  2. icinga2.log says
    [2020-04-28 03:54:37 +0000] critical/config: Error: An object with type ‘Service’ and name ‘uw2010250160098.corp.test.com!Haproxy_process’ already exists (in /var/lib/icinga2/api/packages/_api/b01bcd25-136e-4fad-a882-d7e8291200bf/conf.d/services/uw2010250160098.corp.test.com!Haproxy_process.conf: 1:0-1:31), new declaration: in /var/lib/icinga2/api/packages/_api/b01bcd25-136e-4fad-a882-d7e8291200bf/conf.d/services/uw2010250160098.corp.test.com!Haproxy_process.conf: 1:0-1:31

Error mentioned in point 2 is coming for many checks and multiple servers.

Was going through some of the threads and it was mentioned to delete /var/lib/icinga2/api/zones-stage , zones icinga2.state

Tried deleting those files as well but could recover. icinga2 daemon -C on satellite looks clean with some warning messages

Master global template looks like this
ls -ltr /etc/icinga2/zones.d/global-templates
total 48
-rw-------. 1 icinga icinga 179 Apr 27 18:02 api-users.conf
-rw-------. 1 icinga icinga 803 Apr 27 18:08 users.conf
-rw-------. 1 icinga icinga 906 Apr 27 18:08 timeperiods.conf
-rw-------. 1 icinga icinga 5541 Apr 27 18:08 services.conf
-rw-------. 1 icinga icinga 801 Apr 27 18:08 satellite.conf
-rw-------. 1 icinga icinga 2451 Apr 27 18:08 notifications.conf
-rw-------. 1 icinga icinga 638 Apr 27 18:08 groups.conf
-rw-------. 1 icinga icinga 547 Apr 27 18:08 downtimes.conf
-rw-------. 1 icinga icinga 2597 Apr 27 18:08 commands.conf
-rw-------. 1 icinga icinga 35 Apr 27 18:08 app.conf
-rw-------. 1 icinga icinga 2498 Apr 27 18:31 templates.conf

Sounds like an issued I had recently. The solution is described here.

BTW: Please format your message as described here.

Hi Roland,

Thanks for your response. I did tried deleting zones and zones-stage directories while searching for the solution for this issue, but it didnt helped much at that moment.

And the strange thing is, this issue appeared almost after 7 days of upgrade.

Deleted the complete path /var/lib/icinga2/api after restarting services on master. Which resolved the issue for now.

Thanks
Rahul

packages should have been also deleted. This is what you’ve got by deleting the complete path.