newly added host and service stuck on pending state

This forum was archived to /woltlab and is now in read-only mode.
  • Hello all,


    I have checked many forumthreads and worked my way through many web pages, but did nog manage to fix my problem.

    This is driving me crazy! :(


    Problem

    I have added a first node* via Director. It appears, but is stuck at the state PENDING.

    * (i am not counting the default 'localhost' node)


    Symptoms :

    - Added host via Director, host (and services appear in overview), but remain on status PENDING.

    - 'Check now', does not result in any data being presented about the data.

    - The default host (localhost) has information about 'next check, and last check' the new node does not have this.

    - Option 'Check source' on the host page is empy on the newly added node, while the default node shows 'Check Source icinga.qdelft.nl is reachable ' where 'icinga.qdelft.nl' is a zone.

    - Check state, remain on 'soft state' for hours.

    - Debug logging, does not show that the Icinga2 core is even trying to execute on of the checks..

    - All items are installed on a single machine, no HA-cluster or stuff like that.



    Details on the installation :

    • CentOS Linux release 7.3.1611 (Core)
    • icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1)
    • Icinga2 director (version unknown, not more then 2 days old).
    • Icingaweb2 (version unknown, not more then 2 days old)
    • Graphite V0.0.0.5 (operational).
    • mariadb-server-5.5.52-1.el7.x86_64



    Disabled features: compatlog gelf influxdb opentsdb

    Enabled features: api checker command debuglog graphite ido-mysql livestatus mainlog notification perfdata statusdata syslog

  • Output of debug.log, after a clean of the log, fresh start of the icinga2 service, on debug level, and a forced check of the host (10.33.10.44)

  • Apparently there is something wrong with the way director adds the new hosts. It seems that a timeperiod check is missing.

    Ditched Director (for now, will investigate on test box), and added configs manually.


    Icinga2 is now showing hosts ( and checking services) as expected.

  • Possible Solution to PPS (perpetual pending state)

    In director, try adding one time range per day. Creating separate entries for each day seems to work when multi-day entries don't.

    The ranges page will look something like this:

    Day(s) Timeperiods
    monday 00:00-24:00
    tuesday 00:00-24:00
    and so on...


    Not like this: monday - friday


    My experiential background of this issue: (YMMV, RTFM, DKTM, IMOIMABOFH)

    Might by, monday, tuesday, 2016-01-28 - have a look at the documentation for more examples

    That statement as well as the field label "Day(s)" seem to indicate that an entry of "monday,tuesday" would work, but it fails to validate. Most formats listed in the docs and above, when created in director, fail to validate. However, this format: "sunday - saturday" PASSES VALIDATION but DOES NOT WORK. Objects referencing a timeperiod created in director with that specific format for the Day(s) field will silently fail in the following ways:

    • Scheduled jobs don't run (no error)
    • Hosts and services enter PPS (no error)

    There may be other formats that pass validation but fail to work, this is the one I encountered.


    Kudos to the OP for figuring out that timeperiods created in director could be causing perpetual pending states. Searching combinations of pending / director / icinga, etc. pointed to zone configuration problems, api credential issues, and other rabbit holes which took considerable time to verify as not relevant.


    Cilantro is really good in salad.