Late check results when running more than one master node

This forum was archived to /woltlab and is now in read-only mode.
  • Hi,


    I run Icinga2 version r2.6.0-1 in a cluster mode for ± 120 zones, 700 hosts and 4000 services, the top zone has two masters.


    It appears to work fine when I only run one master node, as soon as I start a second master, most of my checks become overdue.
    It doesn't matter which master is running, as long as only one of them is running it works fine, the configuration are synchronised and the checks are being executed without being late.


    When I turn on the two masters, I start seeing my checks being late.
    I have noticed that when I run a check from Icingaweb2 "Check now" button, I can see it executed straight away in the client debuglog.
    The CheckResult is sent back to one of the masters but Icingaweb2 keeps showing a negative "Next check" counter and never get the result updated.


    I have checked the MySQL connection and I think everything is ok on this side, it just works well with one master or the other alone.


    See below an extract of the zone configuration which is the same on every node (master or client):

    Code
    1. object Zone "global-templates"  { global = true }object Zone "global-commands"   { global = true }
    2. object Endpoint "mgmt01.mydomain" { host = "10.X.X.131" }object Endpoint "mgmt02.mydomain" { host = "10.X.X.132" }
    3. object Zone "datacenter" { endpoints = [    "mgmt01.mydomain",    "mgmt02.mydomain",] }
    4. object Endpoint "vmig.kit46.mydomain" {  }object Zone "vmig.kit46.mydomain" { endpoints = [ "vmig.kit46.mydomain" ] ; parent = "datacenter" }


    I am sorry for the poor debug output I can provide but I really don't know where to start with this issue :s
    If you have any idea or question, please let me know.


    Thanks.

  • I have just upgraded to 2.6.2.1 but I can't see any difference, the results are still late with two masters only.

  • webx

    Have you solve this? I struggle on this problem. All is working fine, but if i run with two masters some checks gets overdue. If i stop the second master, the overdues are checked within some minutes.


    I disabled the checker feature on seconder master, but this seems not to solve the problem.

    Code
    1. icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1
  • I think I found my problem.


    dnsmichi I Like the documentation but my personal fail was the configuration of the second master. I followed:

    https://www.icinga.com/docs/ic…ility-master-with-clients


    Quote


    Set up icinga2-master2.localdomain as client (we will modify the generated configuration)


    But there is no explicit declaration how the icinga2-master2 is configured. The node wizard creats following config:


    Correct should be

    With the "wrong" configuration i see this entries in log

    Quote

    [2017-09-14 13:48:43 +0200] debug/ApiListener: Not connecting to Zone 'icinag2-client.localdomain' because it's not in the same zone, a parent or a child zone.

    I firstly thought this is some other problem and my zone is correctly know by the declaration of the parent. Maybe a hint in docu, can help other people.

  • Aha. Yes i read this part, so i wouldn't quote it here and on which point in docu is shown how the config of icinga2-master2.localdomain has to be modified? I also searched in browser after that string part?

  • When you read https://www.icinga.com/docs/ic…ility-master-with-clients , just a few lines below it tells you:


    Code
    1. The zone hierarchy could look like this. It involves putting the two master nodes icinga2-master1.localdomain and icinga2-master2.localdomain into the master zone.

    this is followed by a example zones.conf


  • Not sure what else we could add to the documentation to make it more clear that you have to modify/create the configuration yourself.