Check_disk returning a critical error on directory that doesn't exist

This forum was archived to /woltlab and is now in read-only mode.
  • Hello,


    Please we need your help with check_disk plugin with icinga2.


    System Details (Ubuntu 16.04):

    $ /usr/lib/nagios/plugins/check_disk -V

    check_disk v2.1.2 (monitoring-plugins 2.1.2)


    $ icinga2 -V

    icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)



    Error (from email output):

    Notification Type: PROBLEM

    Service: disk

    Additional Info: DISK CRITICAL - /var/lib/docker/aufs/mnt/2746bf457464d10db12bfdff0f9ac604bd7213ae7f9c51fee8b8ef0ae64548ec is not accessible: Permission denied


    The issue is appearing in more than a server and the weird part is that i have no docker installed on any of them, even the directory ../docker/aufs/mnt/ doesn't exist!


    Thank you in advance

  • That sounds like that the check_disk is executed somewhere else, not on the remote client. Please verify the "check_source" attribute of these check results, either via REST API or Icinga Web 2.

  • Connect to that box and check the mounts. Post your findings here.


    Side note - you're using the bottom up approach with repository.d and node update-config. This one has been deprecated in 2.6, I'd suggest you plan your migration (details on the release announcement: https://www.icinga.com/2016/12…inga-web-2-v2-4-released/).

  • $ df -h

    Filesystem Size Used Avail Use% Mounted on

    udev 1.7G 0 1.7G 0% /dev

    tmpfs 344M 36M 309M 11% /run

    /dev/sda1 29G 9.1G 20G 33% /

    tmpfs 1.7G 4.0K 1.7G 1% /dev/shm

    tmpfs 5.0M 0 5.0M 0% /run/lock

    tmpfs 1.7G 0 1.7G 0% /sys/fs/cgroup

    none 64K 0 64K 0% /etc/network/interfaces.dynamic.d

    /dev/sdb1 7.0G 23M 7.0G 1% /mnt

    tmpfs 344M 0 344M 0% /run/user/1001

    tmpfs 344M 0 344M 0% /run/user/1000


    Thanks for the note, I will be checking about the new release.

  • Tried the below on remote host, but didn't work:

    $ vi /etc/icinga2/conf.d/hosts.conf

    vars.disks["disk"] = {

    disk_partitions_excluded = [ "/var/lib/docker" ]

    }


    And tried to create "../docker/aufs/mnt/2746bf457464d10db12bfdff0f9ac604bd7213ae7f9c51fee8b8ef0ae64548ec" directories with 777 permissions (nagios user) and didn't work as well.

  • $ mount

    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

    udev on /dev type devtmpfs (rw,nosuid,relatime,size=1741628k,nr_inodes=435407,mode=755)

    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

    tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=352180k,mode=755)

    /dev/sda1 on / type ext4 (rw,relatime,discard,data=ordered)

    securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)

    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)

    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)

    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)

    pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)

    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)

    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)

    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)

    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)

    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)

    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)

    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)

    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)

    mqueue on /dev/mqueue type mqueue (rw,relatime)

    debugfs on /sys/kernel/debug type debugfs (rw,relatime)

    fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

    none on /etc/network/interfaces.dynamic.d type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64k)

    /dev/sdb1 on /mnt type fuseblk (rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096)

    tmpfs on /run/lxcfs/controllers type tmpfs (rw,relatime,size=100k,mode=700)

    hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,relatime,hugetlb)

    blkio on /run/lxcfs/controllers/blkio type cgroup (rw,relatime,blkio)

    memory on /run/lxcfs/controllers/memory type cgroup (rw,relatime,memory)

    cpu,cpuacct on /run/lxcfs/controllers/cpu,cpuacct type cgroup (rw,relatime,cpu,cpuacct)

    pids on /run/lxcfs/controllers/pids type cgroup (rw,relatime,pids)

    freezer on /run/lxcfs/controllers/freezer type cgroup (rw,relatime,freezer)

    devices on /run/lxcfs/controllers/devices type cgroup (rw,relatime,devices)

    cpuset on /run/lxcfs/controllers/cpuset type cgroup (rw,relatime,cpuset)

    perf_event on /run/lxcfs/controllers/perf_event type cgroup (rw,relatime,perf_event)

    net_cls,net_prio on /run/lxcfs/controllers/net_cls,net_prio type cgroup (rw,relatime,net_cls,net_prio)

    name=systemd on /run/lxcfs/controllers/name=systemd type cgroup (rw,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)

    lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

    tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=352180k,mode=700,uid=1001,gid=1001)

    binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

    tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

    tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=352180k,mode=700,uid=1000,gid=1000)

  • You can't modify a runtime created docker mount, that's probably one image created on disk.


    check_disk allows you to exclude certain patterns or specific types. Test-drive the plugin options on the command line too.


    disk_ignore_eregi_path & disk_ignore_eregi_path

    https://docs.icinga.com/icinga…plugin-check-command-disk

  • $ vi /etc/icinga2/conf.d/hosts.conf

    vars.disks["disk"] = {

    disk_all = true

    disk_ignore_eregi_path = "/var/lib/docker"

    }


    I will be adding the above mentioned and test, is the syntax correct?

  • It is stable now and am not receiving any alerts. But each time i reload icinga2 service on the monitored host, I receive a critical alert once then it disappear/stops after 3-4 minutes. what do you think?

  • The below email alert:


    ***** Icinga *****

    Notification Type: PROBLEM

    Service: disk
    Host: ElevatorExpressServiceLive
    Address:
    State: CRITICAL

    Date/Time: 2017-06-23 11:22:46 +0000

    Additional Info: DISK CRITICAL - /var/lib/docker/aufs/mnt/5bfde9753c6f61b2b56a6da69464ba4676501497902256d83006370fe97f0b07 is not accessible: Permission denied

  • So I would assume your regex is not entirely correct. Did you manually verify it is working on the shell?

    I am having the same problem. Can you share the command to verify manually on the shell?


    I don't understand how to translate the hosts.conf parameters from here: https://www.icinga.com/docs/ic…ga-template-library/#disk


    to what is found here: https://www.monitoring-plugins.org/doc/man/check_disk.html


    On the command line, I get an error stating I need to define threshold values, but I have not defined threshold values in hosts.conf.


    /usr/lib/nagios/plugins/check_disk -A -v --ignore-eregi-path=/var/lib/docker

    DISK UNKNOWN: Must set a threshold value before using -r/-R


    If I kill docker on the client, the previous configuration without the ignore regex works.

  • On the command line, I get an error stating I need to define threshold values, but I have not defined threshold values in hosts.conf.

    As described in the documentation you mentioned there are thresholds for disk_wfree and disk_cfree which are used when you omit them in your configuration. So when you execute the plugin on the command line you have to specify them explicitly. Executing ./check_disk -h will show some help which leads to something like

    Code
    1. ./check_disk -w 20% -c 10% -A -v --ignore-eregi-path=/var/lib/docker