Icinga2-ido-mysql fails on master

(Marco) #1

This is a weird one.

We have a master - slave setup. With two icinga2 Servers, both on Ubuntu 18.04.
Our slave is a hard copy from the master, this means same packages and everything (in theory).

Now heres the problem, our master cannot write in the MySQL DB on our DB server but the slave is able to. So now when our slave is turned off, no notifications are seen in icingaweb2.

This error message gets displayed in the logs:

[2018-11-21 13:38:55 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2018-11-21 13:38:55 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2018-11-21 13:38:55 +0100] critical/IdoMysqlConnection: Error "Malformed packet" when executing query
 "SELECT version FROM icinga_dbversion WHERE name='idoutils'"
        (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2018-11-21 13:38:55 +0100] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!

We see no blocks on our firewall or elsewhere, in fact we can connect to the db server and get the specific value mentioned in the log.

master                                                                                                                                                 slave
mysql> SELECT version FROM icinga_dbversion WHERE name='idoutils';                                    │mysql> SELECT version FROM icinga_dbversion WHERE name='idoutils';
+---------+                                                                                           │+---------+
| version |                                                                                           │| version |
+---------+                                                                                           │+---------+
| 1.14.3  |                                                                                           │| 1.14.3  |
+---------+                                                                                           │+---------+
1 row in set (0.00 sec)                                                                               │1 row in set (0.01 sec)

The ido module has been reinstalled and mysql-client is on version 5.7.

(Roland) #2

Weird? Yes, your setup seems to be weird. :wink: Your slave aka satellite shall never write to the master DB but to its own DB. Notification are then send from slave to master an the master will store it to its DB (which is then seen on Icingaweb2 on the master).

(Marco) #3

Well maybe master - slave is the wrong explanation. master - master maybe, because the “slave” could be a standalone server, it receives the config from the master and when the master is up, it communicates via api which checks it has to make. but if master is down it will work as standalone.

And we do not have any mysql servers installed on the servers themselves. They write everything onto a third server, which does work, as proven by the “satelite”

Im sorry if it isnt clearly explained. :sweat_smile:

(Marco) #4

master1													                                          master2
object Endpoint "master1" {   		                                                              │object Endpoint "master1" {
        host = ""                                                                       │        host = ""
        port = "5665"                                                                                 │        port = "5665"
}                                                                                                     │}
object Endpoint "master2" {		                                                              │object Endpoint "master2" {
        host = ""                                                                       │        host = ""
        port = "5665"                                                                                 │        port = "5665"
}                                                                                                     │}
object Zone "WatchManCluster" {                                                                       │object Zone "WatchManCluster" {
        endpoints = [ "master1", "master2" ]           				                      │        endpoints = [ "master1", "master2" ]
#       parent = "global-templates"                                                                   │#        parent = "global-templates"


master1													                                            master2
/**                                                                                                   │/**
 * The API listener is used for distributed monitoring setups.                                        │ * The API listener is used for distributed monitoring setups.
 */                                                                                                   │ */
object ApiListener "api" {                                                                            │object ApiListener "api" {
  accept_config = true                                                                                │  accept_config = true
  ticket_salt = TicketSalt                                                                            │  accept_commands = true
}                                                                                                     │}

My zones.conf and api.conf.

(Carsten Köbke) #5

Did you try to login from the masters to the database using the username/password configured in /etc/icinga2/features-available/ido-mysql.conf ?

(Marco) #6

Yes, and it seems successfull, we could even ask for the value.

(Carsten Köbke) #7

This works from “both” masters? Also can you post your ido-mysql.conf from both masters please

(Marco) #8

Yes, it works for both masters.

object IdoMysqlConnection "ido-mysql" {
  user = "username",
  password = "password",
  port = 3308,
  host = "x.x.x.x",
  database = "icinga_ido"

The configuration is identical to both servers.

(Carsten Köbke) #9

Looks good to me.
Is a firewall between the the one master and the database server?
Also you said you have mysql client/libs version 5.7 on the master servers. What version is the database server?

(Marco) #10

There are in fact 3 firewalls. (iptables outgoing icinga, cisco firewalls, iptables incoming db server)
but as tested we can look at the db, just icinga2 can not use it. so i suppose firewalls are not the problem ?

mka@db-server:/$ sudo mysql -V
mysql  Ver 14.14 Distrib 5.7.19, for Linux (x86_64) using  EditLine wrapper

(Michael Friedrich) #11

Is there a packet filter in place which optimizes traffic?

(Carsten Köbke) #12

Network is never the problem :wink: Its just a possible failure source.
The error does not say you can not connect, it says its malformed, so i think there is something blocking or modify packets of the connection to the database.

I see @dnsmichi was faster :slight_smile:

(Michael Friedrich) #13

Debugged such problems in the past for many useless hours, it happens too often not to ask that in the very first place. Anything which sounds like a packet problem or connection drops is needs a move away from the application layer down to network and packets.

My next suggestion would have been - tcpdump the entire connection handling til the error happens to gain more insights. Especially a comparison between the sender’s pcap and the receiver’s pcap would then highlight such “optimizations”.

(Brian LaVallee) #14

I like to do a litmus test when possible.

  1. Stop iptables
  2. Disable selinux
  3. permit any any on the firewall

If the problem persists, you’ve eliminated a few suspects. Don’t forget to re-enable things.

Sounds like you’re using a clone. I would compare the interface settings on both Master nodes. I’m thinking the HWADDR may be the same, which could cause packet issues. You might not see this kind of issue with the CLI, because of error correction.

(Marco) #15

@dnsmichi i will try to record the traffic and analyse it. if i find something i’ll tell you.

@poing i checked the HWADDR, and they are not the same. And traffic gets through the firewalls, if we just tried it in normal client server mode.

mysql> SELECT version FROM icinga_dbversion WHERE name='idoutils';
| version | 
| 1.14.3  | 
1 row in set (0.00 sec)  

do you think the firewall blocks it when icinga tries to connect?

Thanks for your help!

(Brian LaVallee) #16

Since you’re probably looking for a network issue, you could try ping. Or setup iperf3 between the two nodes to check the throughput.

ping -c 100 mysql_host.fqdn.tld

Any lost packets, any high latency?

(Marco) #17

we found the error.

On master1 our icinga2 repo was set to debian instead of ubuntu, so it had to work, since ubuntu is based of debian. but maybe the mysql request are handled different, i dont know. :man_shrugging: Anyway, when we corrected this error and set the repo to ubuntu it worked out.

Thanks a lot for your help.