check command works on master, 'No such file or directory' on agent

  • I really hope that I'm missing something basic here, but I'm running into trouble with a custom check command with Icinga 2.6.2. When I apply a custom check command called 'check_dropped_packets' on my icinga master, it executes successfully. When I apply it to an agent, the IcingaWeb2 GUI says "Check command 'check_dropped_packets' does not exist." and the icinga2 log says

    1. warning/PluginCheckTask: Check command for object '[hostname]' (PID: 19256, arguments: '/usr/lib64/nagios/plugins/check_dropped_packets') terminated with exit code 128, output: execvpe(/usr/lib64/nagios/plugins/check_dropped_packets) failed: No such file or directory

    I am certain that the command exists on both master and agent and has the same permissions as the builtin check commands, which work, and selinux is disabled:

    1. # ls -al /usr/lib64/nagios/plugins/check_dropped_packets
    2. -rwxr-xr-x 1 root root 1304 Feb 16 15:31 /usr/lib64/nagios/plugins/check_dropped_packets

    And I can verify that the icinga user can execute the command:

    # sudo -u icinga /usr/lib64/nagios/plugins/check_dropped_packets     DROPPED PACKETS OK 0 dropped packets| eth0=0;;; lo=0;;;
    and I believe I have defined the command correctly, and it exists on both my icinga master and the agent I'm testing it on:

    and yet still, it says it does not exist. Does anyone have any ideas what I'm doing wrong? I would greatly appreciate any help, this is driving me crazy.

  • Also here is the check_dropped_packets script in case that is relevant:

  • You really did everything already i would have advised.

    Great !

    So what is left is not much:

    At the satellite, switch on the debug log: icinga2 feature enable debuglog && service icinga2 reload

    Wait until the service check has been executed and then see what icinga2 has run:

    grep -i 'running command' /var/log/icinga2/debuglog | grep -i dropped and post that here.

    disable the debuglog: icinga2 feature disable debuglog && service icinga2 reload

  • sru: I just tried that. Unfortunately grepping debug.log did not produce any results. It did show results for the built-in checks so at least I know the debug log is working.

  • A coworker just figured it out... in /etc/icinga2/icinga2.conf on the agent I was missing the line 'include_recursive "conf.d"' but it was present on the master. Thanks!

  • I am not sure that this is what you are after.

    Setting the include on an agent results in config objects defined in the agents conf.d/ being (re-)evaluated.

    But normally you either like to push config objects from the master down to the agent or you push commands to be executed down to the agent.

    I personally feel you should remove that include line on the agent and create the missing objects (most likely the check_command) at the masters zones.d/[agentzone]/objects.conf file or in thezones.d/global-templates/objects.conf file so that it will be replicated down to the agent.

    Your solution may work fine in the moment, but if you create an object at the master later on that already exists in the agents conf.d directory,

    your agent will mis-validate the configuration and stop working.