we recently created a wrapper around the shutdown command, which sets a short downtime for the affected system using the REST API.
This seemed to work in general. Unfortunately after restarting Icinga2, we see a database corruption:
[2017-11-09 18:42:53 +0100] critical/IdoMysqlCoKilledon: Error "Duplicate entry '1-1272-2017-11-06 08:34:11-4372' for key 'instance_id'" when executing query "UPDATE icinga_scheduleddowntime
When I select the corresponding entries:
Quote| 77004 | 1 | 1 | 9706 | 2017-11-07 07:11:55 | hpc-shutdown on vca-mdc-37-21.fe.hhi.de | Automated downtime for host shutdown. | 6497 | 0 | 1 | 0 | 2017-11-07 07:11:48 | 2017-11-10 07:11:48 | 1 | 2017-11-07 07:11:48 | 0 | 1 | 2017-11-07 07:11:58 | vca-mdc-37-21.fe.hhi.de!CPU Temperature!hpc-monitoring.fe.hhi.de-1510035115-2444 | 1510240220 | 199
as one example of the affected rows, which indicates the shutdown wrapper as the problem source, but in total:
Quote11851 rows in set, 1 warning (0.07 sec)
This seems like quite a lot, especially since the script is rather new.
But it leads to two questions:
a) How can I get rid of these duplicate entries? Running
Allows Icinga2 to start and run. But as soon as I restart Icinga, the entries are back in the database.
b) Is there a bug in the REST API? I think, even if we are misunderstanding the usage, we should not be able to corrupt the database.
Question a) is the most important for me here, since I'd like to get back to a stable system again as soon as possible.
We can provide code examples, if required.
Thanks and best regards,