Backend icinga is not running

This forum was archived to /woltlab and is now in read-only mode.
  • Hi,


    I've checked multiple threads with the same issue, but couldn't find a solution. Here's what I have set up:


    • icinga2 2.6.3-1~ppa1~trusty1
    • icingaweb2 2.4.1-1~ppa1404+1
    Code: /etc/icingaweb2/resources.ini
    1. [icinga_ido]
    2. type = "db"
    3. db = "mysql"
    4. host = "localhost"
    5. dbname = "icinga2"
    6. username = "icinga2"
    7. password = "password"
    8. charset = "utf8"

    Code: ps aux|grep icinga
    1. nagios 4302 0.0 0.1 496712 4984 pts/0 S 10:44 0:05 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon -d -e /var/log/icinga2/icinga2.err
    2. nagios 4309 4.9 2.1 1134264 70356 ? Ssl 10:44 6:52 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon -d -e /var/log/icinga2/icinga2.err


    I tried to do apt-get install --reinstall following packages: icingaweb2 icinga2-ido-mysql but that didn't solve the issue. I still see "Backend icinga is not running" in Icingaweb2 interface. I have another master running with completely same setup and it works fine. Maybe I missed something? Any idea where to dig for an issue?

  • Anything in the icinga2 logs? and did you create the database & import the icinga2 sql file?


    Edit: Forgot to ask if you enable the ido-mysql feature? (icinga2 feature list)

    The post was edited 1 time, last by Mikesch ().

  • At least no errors in icinga log.


    I didn't create the database, afaik it's created by package when you install icinga2. You mean database for icingaweb2 or for icinga2? If for icingaweb2, I think I don't need it because I use external authentication (pwauth).

  • AFAIK it is not necessary to set a charset for the IDO database, only for the Director database (if you use the Director). Please remove charset = "utf8" in your resources.ini and test it again.

  • Note to myself "ask more specific" :)


    Please post output of:

    Code
    1. cat /etc/os-release
    Quote

    icinga2 feature list

    Code
    1. icinga2 --version
    Code
    1. systemctl stop icinga2 && rm -f /var/log/icinga2/icinga2.log && systemctl start icinga2 && sleep 10 && tail -n 1000 /var/log/icinga2/icinga2.log
    Code
    1. mysql -h localhost -u icinga2 --password=password -e "show databases; use icinga ; show tables ; select * from icinga_dbversion"
  • AFAIK it is not necessary to set a charset for the IDO database, only for the Director database (if you use the Director). Please remove charset = "utf8" in your resources.ini and test it again.

    Nope, that doesn't help.

  • Mikesch

    Code
    1. NAME="Ubuntu"
    2. VERSION="14.04.5 LTS, Trusty Tahr"
    3. ID=ubuntu
    4. ID_LIKE=debian
    5. PRETTY_NAME="Ubuntu 14.04.5 LTS"
    6. VERSION_ID="14.04"
    7. HOME_URL="http://www.ubuntu.com/"
    8. SUPPORT_URL="http://help.ubuntu.com/"
    9. BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    Code
    1. Disabled features: debuglog gelf graphite influxdb opentsdb perfdata syslog
    2. Enabled features: api-users api checker command compatlog ido-mysql livestatus mainlog notification statusdata

    Since I'm getting error about message more than 10,000 chars, here's the log output: https://pastebin.com/gjAyJMVt

  • hmm to slow for the log. Please run this


    Code
    1. fgrep -i ido /var/log/icinga2/icinga2.log

    Rest looks good so far.

    Another thing, can you validate the connection in icingaweb2 also please.





    I log is ok, then it might be selinux and/or firewall. but lets see what the log says about ido

  • Mikesch sorry for late reply, I was on holiday. Here's the output from today's log:


    Code
    1. [2017-07-24 11:05:38 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
    2. [2017-07-24 11:05:38 +0200] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.2')
    3. [2017-07-24 11:05:39 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
    4. [2017-07-24 11:05:39 +0200] critical/IdoMysqlConnection: Error "MySQL server has gone away" when executing query "DELETE FROM icinga_comments WHERE instance_id = 1 AND session_token <> 1500470117"
    5. [2017-07-24 11:05:39 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!

    ido connection also validates successfully:




    Selinux can't be the issue, as it's Ubuntu Server 14.04 and since we don't use it, it has default settings. It could the firewall, but I doubt that, however, let's find that out. I just checked, I can connect to ports 5665.5665 from main master and vice versa.


    Code
    1. root@ops-monmaster1:~# nc -vz "tools-monmaster.xxx.net" 5665
    2. Connection to tools-monmaster.xxx.net 5665 port [tcp/*] succeeded!
    3. root@ops-monmaster1:~# nc -vz "tools-monmaster.xxx.net" 5666
    4. Connection to tools-monmaster.xxx.net 5666 port [tcp/nrpe] succeeded!
    5. root@tools-monmaster1:~# nc -vz "ops-monmaster.xxx.net" 5665
    6. Connection to ops-monmaster.xxx.net 5665 port [tcp/*] succeeded!
    7. root@tools-monmaster1:~# nc -vz "ops-monmaster.xxx.net" 5666
    8. Connection to ops-monmaster.xxx.net 5666 port [tcp/nrpe] succeeded!


    Afaik there shouldn't be any other open ports between masters.

  • Welcome back vaisov


    you have an problem with your ido connection


    Code
    1. [2017-07-24 11:05:39 +0200] critical/IdoMysqlConnection: Error "MySQL server has gone away" when executing query "DELETE FROM icinga_comments WHERE instance_id = 1 AND session_token <> 1500470117"
    2. [2017-07-24 11:05:39 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!

    Check why the server looses the connection to you database. Maybe not enough rights for the mysql user or still firewall

  • Mikesch connect from localhost to database with ido credentials works fine:


    It looks like that error you pointed out was a one time error. As I can't find it in recent logs. But I still see the same issue in IcingaWeb2 UI.

  • He is able to validate the IDO connection but the backend is not running ?

    What about additional (misconfigured) backends ?

  • He is able to validate the IDO connection but the backend is not running ?

    What about additional (misconfigured) backends ?

    Backend is running, and IDO connection works. It's just IcingaWeb2 reports that backend is not running.

    Where usually can you configure additional backends? If you mean IcingaWeb2, there's only one backend.

  • I bet its the MySQL user

    How should the validation run with non-working credentials ?

    But hey, you might be right that the user does not have full permissions to the database, i did not thought about that.

  • MySQL user has following grants:

    Code
    1. | GRANT USAGE ON *.* TO 'icinga2'@'localhost' IDENTIFIED BY PASSWORD '****' |
    2. | GRANT ALL PRIVILEGES ON `icinga2`.* TO 'icinga2'@'localhost' |

    So I don't think it's a mysql permissions issue.


    Just tried to connect with icinga2 user and get program status from db, it works:

  • Nope, I didn't. Should I? Is it official Icinga support? I'll have to clarify the access with our security team first.