Check PENDING on satellite-node

satellite
check
pending

#1

Hi there,

i’ve got a master-satellite setup with one master-node and two satellites. The satellite-nodes are located in external datacenters and should check all the hosts which are located in this datacenter.
All hosts are currently checked with “hostalive4”, while the check_endpoint is my satellite-node - it’s working fine.
Because of a apply rule, all hosts are additionally checked with “ping4”. I can see this “ping4”-service in the “service”-tab of each server. On all hosts in a datacenter, this “ping4” remains on “PENDING” forever. For other services, the same problem occurs.

I already checked the www for a solution and noticed, that maybe my zone-configuration is incorrect. I deployed my whole infrastructure with the director.

First question
Please see following output:

Object 'monitoring-001.satellite‘ of type ‘Endpoint’:
% declared in ‘/var/lib/icinga2/api/packages/director/monitoring-001.master-1543394575-0/zones.d/zone-master/endpoints.conf’, lines 6:1-6:45
* __name = “monitoring-001.satellite.de
* host = “monitoring-001.satellite.de
% = modified in ‘/var/lib/icinga2/api/packages/director/monitoring-001.master-1543394575-0/zones.d/zone-master/endpoints.conf’, lines 7:5-7:40
* log_duration = 86400
* name = “monitoring-001.satellite.de
* package = “director”
* port = “5665”
% = modified in ‘/var/lib/icinga2/api/packages/director/monitoring-001.master-1543394575-0/zones.d/zone-master/endpoints.conf’, lines 8:5-8:17
* source_location
* first_column = 1
* first_line = 6
* last_column = 45
* last_line = 6
* path = “/var/lib/icinga2/api/packages/director/monitoring-001.master-1543394575-0/zones.d/zone-master/endpoints.conf”
* templates = [ "monitoring-001.satellite.de“ ]
% = modified in ‘/var/lib/icinga2/api/packages/director/monitoring-001.master-1543394575-0/zones.d/zone-master/endpoints.conf’, lines 6:1-6:45
* type = “Endpoint”
* zone = “zone-master”

i dont understand, why the last line shows “zone-master” and not (like configured in my director) “zone-satellite”. Is this correct?

Secound question
Should the /etc/icinga2/zones.conf contain all my satellite-nodes? I can find my satellites in config-files in /var/lib/icinga2/api… but not in the /etc/zones.conf. Is this correct?

zones.conf of master:

/*
* Generated by Icinga 2 node setup commands
* on 2018-02-21 11:55:37 +0100
*/
object Endpoint "monitoring.mydomain.de“ {
host = "monitoring.mydomain.de
}
object Zone “zone-master” {
endpoints = [ “monitoring.mydomain.de” ]
}
object Zone “global-templates” {
global = true
}
object Zone “director-global” {
global = true
}

zones.conf of satellite:

/*
* Generated by Icinga 2 node setup commands
* on 2018-05-30 16:37:11 +0200
*/
object Endpoint “monitoring.mydomain.de” {
host = “monitoring.mydomain.de
port = “5665”
}
object Zone “master” {
endpoints = [ “monitoring.mydomain.de” ]
}
object Endpoint “monitoring-001.satellite.de” {
host = “monitoring-001.satellite.de
port = 5665
}
object Zone "zone-satellite“ {
endpoints = [ “monitoring-001.satellite.de” ]
parent = “master”
}
object Zone “director-global” {
global = true
}

I am running r2.8.4-1 on all nodes.


(Brian LaVallee) #2

Yes, each Satellite should be in /etc/zones.conf for each Master.

object Endpoint "master1.fqdn.tld" {}
object Endpoint "master2.fqdn.tld" {
        host = "master2.fqdn.tld"
        port = "5665"
}

object Zone "master" {
	endpoints = [ "master1.fqdn.tld", "master2.fqdn.tld" ]
}

object Zone "satellites" {
        endpoints = [ "satellite1.fqdn.tld", "satellite2.fqdn.tld" ]
        parent = "master"
}

object Endpoint "satellite1.fqdn.tld" {}
object Endpoint "satelitte2.fqdn.tld" {}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

Assuming you placed the conf files in zones.d/zone-master this would be the expected behavior, on the configuration Master.

You need to specify the zone -or- place the conf files in the proper directory.


#3

I edited my zones.conf and added my satellite-nodes. A config-checks show following error:

root@monitoring-001  /etc/icinga2/zones.d/master  icinga2 daemon -C
[2018-11-29 08:41:58 +0100] warning/icinga-app: Sysconfig file ‘/etc/sysconfig/icinga2’ cannot be read. Using default values.
[2018-11-29 08:41:58 +0100] warning/icinga-app: Sysconfig file ‘/etc/sysconfig/icinga2’ cannot be read. Using default values.
information/cli: Icinga application loader (version: r2.8.4-1)
information/cli: Loading configuration file(s).
critical/config: Error: Object ‘monitoring-002.satellite.de’ of type ‘Endpoint’ re-defined: in /var/lib/icinga2/api/packages/director/monitoring-001.master-1543410792-0/zones.d/zone-master/endpoints.conf: 1:0-1:43; previous definition: in /etc/icinga2/zones.conf: 34:1-34:44
Location: in /var/lib/icinga2/api/packages/director/monitoring-001.master-1543410792-0/zones.d/zone-master/endpoints.conf: 1:0-1:43
/var/lib/icinga2/api/packages/director/monitoring-001.master-1543410792-0/zones.d/zone-master/endpoints.conf(1): object Endpoint “monitoring-002.satellite.de” {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/packages/director/monitoring-001.master-1543410792-0/zones.d/zone-master/endpoints.conf(2): host = “monitoring-002.satellite.de
/var/lib/icinga2/api/packages/director/monitoring-001.master-1543410792-0/zones.d/zone-master/endpoints.conf(3): port = “5665”

Therefore i assume, that the config-files in /var/lib/icinga2/api/packages/director[…] will be used to build the “zones.conf” icinga is working with.
Is there a possibility to show the “final compiled” zones.conf-file? I mean, the config-files in /var/lib/icinga2/api/packages/director[…] are obviously used.

I checked the config-files in my director-config-folder. All satellites are specified as “endpoint” with corrent zone and parent. For me it looks like my zone-configuration is fine, but i’m still having the same error like i described in my first post. Any ideas?


(Aflatto) #4

One of the issues I noticed when deploying checks to Satellites is that the name of the zone in Director that you defined , and the name created in the contants.conf on the satellite do not match,and that can cause sync issues and pending status.

Check that the ZoneName in the constants.conf matches the name of the zone defined in Director.


#5

Thanks for your reply.
You were right, the const “ZoneName” was set to a wrong value. As i noticed, the field “TicketSalt” was empty too. So i edited both parameters.
Before:

const NodeName = “monitoring-001.satellite.de
/* Our local zone name. /
const ZoneName = “monitoring-001.satellite.de
/
Secret key for remote node tickets */
const TicketSalt = “”

After:

const NodeName = “monitoring-001.satellite.de
/* Our local zone name. /
const ZoneName = “zone-satellite”
/
Secret key for remote node tickets */
const TicketSalt = “1cc160fdd0bba2430f1bfa7222f4f221”

I created the “TicketSalt” value with

md5sum /etc/icinga2/pki/monitoring-001.satellite.de.key

If i add new checks like “check_http” or “check_snmp” to my satellite-hosts, they will be executed correctly by the icinga satellite and i get an “OK” from icinga2-UI.

The last issue is the “ping4” - it’s still hanging in “PENDING”-state. In a tcpdump i can see how the Ping is sent and replied by the remote host (maybe this could be the hostalive4-check too).
One dirty but acceptable solution would be, to delete the “PING4”-check from all satellite-hosts, so the PING4-check would not be executed there anymore (no problem, because hostalive4 is beeing executed as well).

How can i check where the “PING4” in all of my hosts is coming from? I checked for an apply-rule, but i did not find anything.


(Aflatto) #6

the ping check is in /etc/icinga2/conf.d files

this is part of the default icinga installation ( ping anything with a host address)


(Michael Friedrich) #7

You cannot sync Endpoint/Zone objects from the master to the satellite, where these objects need to exist statically before any cluster communication. Remove this configuration from the Director itself, but only import such as an external reference object using the kickstart wizard.