troubleshooting "critical/Application: Found error in config: reloading aborted" on client startup

This forum was archived to /woltlab and is now in read-only mode.
  • Running Icinga2 r2.7.1-1 on master and client. Upgraded Icinga2 on master to r2.7.1-1 recently.


    The client is a new instance for a new customer/network and has always thrown this error.


    critical/Application: Found error in config: reloading aborted


    But the connection between the client and master looks fine. I'd ignore the above error but all my checks for this client's monitored hosts are stuck pending.


    If I run icinga2 daemon -C -x notice on the client or the server I see no errors.


    The only thing I wonder about is the below output from the master but I'm guessing this is a result of me running "icinga2 daemon -C -x notice" while the Icinga2 is already running, and already has compiled these objects.

    notice/config: Ignoring non local config include for zone 'global-templates': We already have an authoritative copy included.

    notice/config: Ignoring non local config include for zone 'cccap': We already have an authoritative copy included.

    notice/config: Ignoring non local config include for zone 'hssk': We already have an authoritative copy included.

    notice/config: Ignoring non local config include for zone 'hoffman': We already have an authoritative copy included.

    notice/ConfigCompiler: Registered authoritative config directories for zone 'gimpie': /etc/icinga2/zones.d/gimpie

    notice/config: Ignoring non local config include for zone 'gimpie': We already have an authoritative copy included.

    notice/config: Ignoring non local config include for zone 'kmberx': We already have an authoritative copy included.

    notice/ConfigCompiler: Registered authoritative config directories for zone 'mcow': /etc/icinga2/zones.d/mcow

    notice/config: Ignoring non local config include for zone 'mcow': We already have an authoritative copy included.

    notice/config: Ignoring non local config include for zone 'mondo': We already have an authoritative copy included.





  • I just did a restart of the master and I'm seeing this message. In 2.7 does the master want


    accept_config = true


    in api.conf ?


    [2017-11-01 08:47:27 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

    [2017-11-01 08:47:42 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

    [2017-11-01 08:47:42 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

    [2017-11-01 08:48:14 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

    [2017-11-01 08:48:20 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

  • Did a stop ; rm -rf /var/lib/icinga2/api ; start on the client and saw this.


    notice/Process: PID 21768 ('/usr/lib64/icinga2/sbin/icinga2' '--no-stack-rlimit' 'daemon' '-x' 'notice' '--reload-internal' '21710') terminated with exit code 1

    [2017-11-01 17:25:04 +0000] critical/Application: Found error in config: reloading aborted

  • Here is what I see when I click the "Inspect" link on Icingaweb2.


    {

    attrs: {

    __name: "MUNISUPER-MELBOURNE-AUS-RPSD",

    acknowledgement: 0,

    acknowledgement_expiry: 0,

    action_url: "",

    active: true,

    address: "10.15.0.21",

    address6: "",

    check_attempt: 1,

    check_command: "hostalive",

    check_interval: 60,

    check_period: "",

    check_timeout: null,

    command_endpoint: "",

    display_name: "Munisuper Melbourne AU RPSD 0",

    downtime_depth: 0,

    enable_active_checks: true,

    enable_event_handler: true,

    enable_flapping: false,

    enable_notifications: true,

    enable_passive_checks: true,

    enable_perfdata: true,

    event_command: "",

    flapping: false,

    flapping_last_change: 0,

    flapping_negative: 0,

    flapping_positive: 0,

    flapping_threshold: 30,

    force_next_check: false,

    force_next_notification: false,

    groups: [

    "polywog-RPSD",

    "mutton"

    ],

    ha_mode: 0,

    icon_image: "",

    icon_image_alt: "",

    last_check: -1,

    last_check_result: null,

    last_hard_state: 1,

    last_hard_state_change: 1509466995.145961,

    last_reachable: true,

    last_state: 1,

    last_state_change: 1509466995.145961,

    last_state_down: 0,

    last_state_type: 0,

    last_state_unreachable: 0,

    last_state_up: 0,

    max_check_attempts: 3,

    name: "MUNISUPER-MELBOURNE-AUS-RPSD",

    next_check: 1509563608,

    notes: "",

    notes_url: "",

    original_attributes: null,

    package: "_etc",

    paused: false,

    retry_interval: 30,

    severity: 24,

    source_location: {

    first_column: 1,

    first_line: 31,

    last_column: 41,

    last_line: 31,

    path: "/etc/icinga2/zones.d/mutton/mutton-infra.conf"

    },

    state: 1,

    state_type: 0,

    templates: [

    "MUNISUPER-MELBOURNE-AUS-RPSD",

    "generic-host",

    "mutton"

    ],

    ],

    type: "Host",

    vars: {

    adminpass: "notThepassWd",

    adminuser: "admin-not",

    company: "mutton",

    devtype: "rpsd",

    icmp_cpl: 55,

    icmp_crta: 700,

    icmp_wpl: 45,

    icmp_wrta: 600,

    region: "APAC",

    snmp_community: "mutton-chop"

    },

    version: 0,

    volatile: false,

    zone: "mutton"

    },

    joins: {

    check_command: {

    __name: "hostalive",

    active: true,

    arguments: {

    -H: {

    description: "host to ping",

    value: "$ping_address$"

    },

    -c: {

    description: "critical threshold pair",

    value: "$ping_crta$,$ping_cpl$%"

    },

    -p: {

    description: "number of ICMP ECHO packets to send (Default: 5)",

    value: "$ping_packets$"

    },

    -t: {

    description: "Seconds before connection times out (default: 10)",

    value: "$ping_timeout$"

    },

    -w: {

    description: "warning threshold pair",

    value: "$ping_wrta$,$ping_wpl$%

    }

    },

    command: [

    "/usr/lib64/nagios/plugins/check_ping"

    ],

    env: null,

    execute: {

    arguments: [

    "checkable",

    "cr",

    "resolvedMacros",

    "useResolvedMacros"

    ],

    deprecated: false,

    name: "Internal#PluginCheck",

    side_effect_free: false,

    type: "Function"

    },

    ha_mode: 0,

    name: "hostalive",

    original_attributes: null,

    package: "_etc",

    paused: false,

    source_location: {

    first_column: 1,

    first_line: 99,

    last_column: 31,

    last_line: 99,

    path: "/usr/share/icinga2/include/command-plugins.conf"

    },

    templates: [

    "hostalive",

    "plugin-check-command",

    "ping",

    "plugin-check-command",

    "ping-common",

    "ipv4-or-ipv6",

    "hostalive-common"

    ],

    ],

    timeout: 60,

    type: "CheckCommand",

    vars: {

    check_address: {

    arguments: [],

    deprecated: false,

    name: "",

    side_effect_free: false,

    type: "Function"

    },

    check_ipv4: false,

    check_ipv6: false,

    ping_address: "$check_address$",

    ping_cpl: 100,

    ping_crta: 5000,

    ping_wpl: 80,

    ping_wrta: 3000

    },

    version: 0,

    zone: ""

  • How can I tell which schema version my MySQL is running?


    grep schema /var/log/icinga2/debug.log


    [2017-11-01 16:23:52 -0700] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.2')

  • How can I tell which schema version my MySQL is running?


    grep schema /var/log/icinga2/debug.log


    [2017-11-01 16:23:52 -0700] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.2')

    There is an table like icginga2_version in your IDO DB.

    I can take a look tomorrow (better later) and will post it here

  • There is an table like icginga2_version in your IDO DB.

    I can take a look tomorrow (better later) and will post it here



    ist "icinga2_dbversion"

    Code
    1. select * from icinga_dbversion;


    My installation has also the 1.14.2 Version installed. so your good and up2date there

  • Here is an object for a host that is stuck in pending state.


    {"results":[{"attrs":{"__name":"MUNISUPER-MELBOURNE-AUS-DMA","acknowledgement":0.0,"acknowledgement_expiry":0.0,"action_url":"","active":true,"address":"10.15.0.25","address6":"","check_attempt":1.0,"check_command":"hostalive","check_interval":60.0,"check_period":"","check_timeout":null,"command_endpoint":"","display_name":"Munisuper Melbourne AU DMA 0","downtime_depth":0.0,"enable_active_checks":true,"enable_event_handler":true,"enable_flapping":false,"enable_notifications":true,"enable_passive_checks":true,"enable_perfdata":true,"event_command":"","flapping":false,"flapping_last_change":0.0,"flapping_negative":0.0,"flapping_positive":0.0,"flapping_threshold":30.0,"force_next_check":false,"force_next_notification":false,"groups":["polywog-RPSD","munisuper"],"ha_mode":0.0,"icon_image":"","icon_image_alt":"","last_check":-1.0,"last_check_result":null,"last_hard_state":1.0,"last_hard_state_change":1509639953.0182840824,"last_reachable":true,"last_state":1.0,"last_state_change":1509639953.0182840824,"last_state_down":0.0,"last_state_type":0.0,"last_state_unreachable":0.0,"last_state_up":0.0,"max_check_attempts":3.0,"name":"MUNISUPER-MELBOURNE-AUS-DMA","next_check":1509640015.830258131,"notes":"","notes_url":"","original_attributes":null,"package":"_etc","paused":false,"retry_interval":30.0,"severity":24.0,"source_location":{"first_column":0.0,"first_line":1.0,"last_column":39.0,"last_line":1.0,"path":"/etc/icinga2/zones.d/munisuper/munisuper-infra.conf"},"state":1.0,"state_type":0.0,"templates":["MUNISUPER-MELBOURNE-AUS-DMA","generic-host","munisuper"],"type":"Host","vars":{"adminpass":"admin","adminuser":"sportsguitar","company":"munisuper","devtype":"dma","icmp_cpl":55.0,"icmp_crta":700.0,"icmp_wpl":45.0,"icmp_wrta":600.0,"region":"APAC","snmp_community":"mutton"},"version":0.0,"volatile":false,"zone":"munisuper"},"joins":{},"meta":{},"name":"MUNISUPER-MELBOURNE-AUS-DMA","type":"Host"}]}

  • I think this issue is only related to your Zone and not the Client Host configuration.


    Did you change any settings manuelly after running the node wizard ?

    If the message "critical/Application: Found error in config: reloading aborted" appreas, should there also a specific message what is wrong.

    Could you please check again ?

  • This thread is going ping pong between problems. We did not yet see your zones.conf file from involved endpoints, neither do we know how the tree in zones.d/ looks like on all involved nodes.

  • Sorry for the lack of focus. Been pulling my hair out over this mess I made.


    Master zones.conf file looks much like this:


    And the master zone "core" is not, and never has been, in the zones.conf file. It appears only in constants.conf. I've never understood why this must be.


    Code
    1. const NodeName = "den-snmp-prod.pollywog.net"
    2. const ZoneName = "core"
    3. const TicketSalt = "c373ea93ah81o05d178a56"

    The client's constants.conf looks like this:


    Code
    1. const NodeName = "den-munisuper-snmp.pollywog.net"
    2. const ZoneName = "munisuper"
    3. const PluginDir = "/usr/lib64/nagios/plugins"
    4. const ManubulonPluginDir = "/usr/lib64/nagios/plugins"
    5. const PluginContribDir = "/usr/lib64/nagios/plugins"
    6. const TicketSalt = "ecf324234dfaerw4t9g23763"


    The client's zones.conf file looks like this:


  • Can you comment out the recursive include of conf.d directory


    Code
    1. /**
    2. * Although in theory you could define all your objects in this file
    3. * the preferred way is to create separate directories and files in the conf.d
    4. * directory. Each of these files must have the file extension ".conf".
    5. */
    6. //include_recursive "conf.d"

    Clean API directory


    Code
    1. rm -rf /var/lib/icinga2/api/repository/*
    2. rm -rf /var/lib/icinga2/api/packages/*
    3. rm -rf /var/lib/icinga2/api/zones/*

    and finaly post output of


    Code
    1. icinga2 daemon -C
  • Yeah, I already had all the files in conf.d moved aside and there was never anything in zones.d.


    So I downgraded to v2.6.3 on the client and at first that seemed to help but the monitored hosts are still stuck in pending mode.

  • Code
    1. [2017-11-03 14:06:51 -0700] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [127.0.0.1]:56784, user: web2)
    2. [2017-11-03 14:06:59 -0700] information/ApiListener: New client connection from [127.0.0.1]:56914 (no client certificate)
    3. [2017-11-03 14:06:59 -0700] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [127.0.0.1]:56914, user: web2)
    4. [2017-11-03 14:07:09 -0700] information/ApiListener: New client connection from [127.0.0.1]:57030 (no client certificate)
    5. [2017-11-03 14:07:09 -0700] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [127.0.0.1]:57030, user: web2)
    6. [2017-11-03 14:07:12 -0700] information/ApiListener: New client connection from [127.0.0.1]:57054 (no client certificate)


    Do these messages in the master's log tell us anything useful?

  • No, those messages are coming from 127.0.0.1, are of the HTTP protocol, and refer for the web2 ApiUser which is your Icinga Web 2 action when clicking on "recheck now".


    I would rather look for that specific client, which messages are sent, and how the connection goes. I would believe that it is not connected, or anything similar. Do you have the cluster-zone health check applied to this client? Furthermore you can look into the REST API on /v1/status to see the connected endpoints in the ApiListener struct.